Adaptive use of video models for holistic video understanding

Abstract

Systems and techniques for performing overall video understanding are provided. For example, a process may include: obtaining a first video; and determining a first machine learning model from a set of machine learning models using a machine learning model decision engine, for processing at least a portion of the first video. The first machine learning model may be determined based on one or more features of at least that portion of the first video. The process may include: processing at least that portion of the first video using the first machine learning model.

Description

Technical Field

[0001] This disclosure generally relates to models for understanding video content. In some examples, aspects of this disclosure relate to systems and techniques for selecting machine learning models based on characteristics of the video content. Background Technology

[0002] Many machine learning models can be used to analyze and categorize information contained in video data (or sequences of image frames). To gain a holistic understanding of the information contained in video data, it is necessary to understand the video data across various categories, such as actions, attributes, events, objects, scenes, etc. Machine learning models trained to detect actions (e.g., 3D convolutional neural networks, optical flow neural networks, etc.) may be computationally too intensive when applied to video data containing very little action. On the other hand, computationally efficient machine learning models, well-suited for detecting objects in static video scenes, may not be sufficient for detecting actions in videos containing significant motion. In some cases, it is desirable to have a machine learning model that can perform a holistic understanding of a video across multiple categories in the most efficient way possible. Summary of the Invention

[0003] In some examples, systems and techniques for adaptively selecting among video processing models for overall video understanding are described. According to at least one illustrative example, a method for processing video data is provided. The method includes: obtaining a first video; determining a first machine learning model from a set of machine learning models using a machine learning model decision engine, the first machine learning model being determined based on one or more features of at least a portion of the first video; and processing at least that portion of the first video using the first machine learning model.

[0004] In another example, an apparatus for processing video data is provided, the apparatus comprising: a memory configured to store at least one video or a portion of a video; and one or more processors (e.g., implemented in circuit form) coupled to the memory. The one or more processors are configured and capable of: acquiring a first video; determining a first machine learning model from a set of machine learning models using a machine learning model decision engine, the first machine learning model being determined based on one or more features of at least a portion of the first video; and processing at least that portion of the first video using the first machine learning model.

[0005] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: obtain a first video; determine a first machine learning model from a set of machine learning models using a machine learning model decision engine, the first machine learning model being determined based on one or more features of at least a portion of the first video; and process at least that portion of the first video using the first machine learning model.

[0006] In another example, an apparatus for processing video data is provided. The apparatus includes: components for acquiring a first video; components for determining a first machine learning model from a set of machine learning models using a machine learning model decision engine, the first machine learning model being determined based on one or more features of at least a portion of the first video; and components for processing at least that portion of the first video using the first machine learning model.

[0007] In some aspects, the methods, apparatus, and computer-readable media described above also include: determining the classification of at least that portion of the first video based on processing at least that portion of the first video using the first machine learning model.

[0008] In some aspects, the methods, apparatus, and computer-readable media described above further include: extracting first or more features from at least that portion of the first video using a first machine learning model; processing the first or more features using a first supplementary model; and determining a first classification of at least that portion of the first video based on processing the first or more features using the first supplementary model. In some cases, the first supplementary model includes a neural network. In some cases, the first supplementary model includes a one-dimensional convolutional neural network. In this case, the first or more features include: a first feature vector based on a first frame of at least that portion of the first video and a second feature vector based on a second frame of at least that portion of the first video. In this case, the first supplementary model generates temporal information of at least that portion of the first video from at least the first feature vector and the second feature vector. In some cases, the first supplementary model includes a multilayer perceptron. In some cases, the first supplementary model includes a graph convolutional network.

[0009] In some aspects, the methods, apparatus, and computer-readable media described above further include: determining a second machine learning model from the set of machine learning models using the machine learning model decision engine, the second machine learning model being determined based on one or more features of at least another portion of the first video; and processing the at least another portion of the first video using the second machine learning model. In some aspects, the second machine learning model includes a neural network.

[0010] In some aspects, the methods, apparatus, and computer-readable media described above further include: extracting a second or more features from the at least other portion of the first video using the second machine learning model; processing the second or more features using a second supplementary model; and determining a second classification of the at least other portion of the first video based on processing the second or more features using the second supplementary model.

[0011] In some aspects, the first machine learning model includes a neural network. In some aspects, the first machine learning model includes a two-dimensional convolutional neural network. In some cases, the two-dimensional convolutional neural network extracts first one or more features from at least that portion of the first video in two spatial dimensions.

[0012] In some respects, the first machine learning model includes a three-dimensional (3D) CNN. In some cases, the 3D CNN extracts first one or more features from at least that portion of the first video in two spatial dimensions and one temporal dimension.

[0013] In some aspects, the methods, apparatus, and computer-readable media described above further include: obtaining a second video; determining a second machine learning model from a set of machine learning models using a machine learning model decision engine, the second machine learning model being determined based on one or more features of at least a portion of the second video; and processing at least that portion of the second video using the second machine learning model.

[0014] In some aspects, the machine learning model decision engine shares a common neural network with the first machine learning model. In some aspects, the methods, apparatus, and computer-readable media described above further include: extracting one or more first features from at least that portion of the first video using the first machine learning model, in parallel with determining the first machine learning model from the set of machine learning models for processing at least that portion of the first video.

[0015] In some respects, at least that portion of the first video has one or more characteristics including spatial and temporal characteristics.

[0016] In some respects, at least one or more characteristics of the first video include audio characteristics.

[0017] In some aspects, one or more of the above-described devices may include or be a subset of the following: mobile devices (e.g., mobile phones or so-called "smartphones" or other mobile devices), wearable devices, extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), vehicles (e.g., computing devices of vehicles), personal computers, laptop computers, server computers, or other devices. In some aspects, a device includes one or more cameras for capturing one or more images. In some aspects, the device further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors that can be used to determine the position and / or posture of the device, the state of the device, and / or for other purposes.

[0018] This invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. The subject matter should be understood by referring to appropriate portions of the entire specification, any or all of the drawings, and each claim.

[0019] The foregoing, along with other features and embodiments, will become more apparent when the following description, claims, and drawings are taken into account. Attached Figure Description

[0020] The illustrative embodiments of this application are described in detail below with reference to the accompanying drawings:

[0021] Figure 1 This is a block diagram illustrating the overall video understanding system based on some examples;

[0022] Figure 2A - Figure 2D This is a diagram illustrating example operations of a one-dimensional convolutional filter based on some examples;

[0023] Figure 3 This is a block diagram illustrating an example of a comprehensive video understanding system based on several examples;

[0024] Figure 4 This is a block diagram illustrating another example of a comprehensive video understanding system based on some examples;

[0025] Figure 5 This is a block diagram illustrating another example of a comprehensive video understanding system based on some examples;

[0026] Figure 6 This is a block diagram illustrating another example of a comprehensive video understanding system based on some examples;

[0027] Figure 7This is a block diagram illustrating another example of a comprehensive video understanding system based on some examples;

[0028] Figure 8 It is a graph illustrating the overall performance of the system based on some examples and videos.

[0029] Figure 9 This is a flowchart illustrating an example of a process for processing video data, based on some examples.

[0030] Figure 10 This is a block diagram illustrating examples of deep learning networks based on some examples;

[0031] Figure 11 This is a block diagram illustrating an example of a convolutional neural network based on some examples; and

[0032] Figure 12 This is a diagram illustrating an example of a computing system used to implement some of the aspects described herein. Detailed Implementation

[0033] Certain aspects and embodiments of this disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some can be applied in combination, as will be apparent to those skilled in the art. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of embodiments of this application. However, it will be apparent that various embodiments can be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0034] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of the exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0035] Machine learning models can be used to process and / or analyze video data. Video understanding using machine deep learning systems is challenging in many situations. For example, video data comprises a large number of interconnected categories of information, such as objects, actions, events, attributes, concepts, and scenes. These categories can be highly interconnected, making it difficult for machine learning models to perform certain functions. For instance, a machine learning model may not be able to perform action classification without recognizing objects, as actions often involve objects. For example, a machine learning model might be used to distinguish between playing the piano and playing the guitar. The motion of playing varies depending on the type of object (e.g., distinguishing between strumming a guitar and striking the keys), indicating that actions and objects are interconnected. For any given video, one or more categories associated with the video may be more important for understanding the video content than other categories. In one example, in a video depicting a static scene, recognizing objects may be more important than detecting actions. In another example, in a video depicting a sports event with a motion camera following the action, recognizing actions and recognizing objects as they move may be equally important.

[0036] Machine learning models can possess prior knowledge of the important categories present in the video dataset they will be processing. Based on this prior knowledge, machine learning models can be specifically designed to efficiently process that dataset. For example, when detecting motion in various types of videos with dynamic content (e.g., videos of sporting events, concerts, etc.), such prior knowledge can be used to optimize a three-dimensional (3D) convolutional neural network (CNN) analyzing the video in two spatial and temporal dimensions to reduce computational costs. However, processing videos depicting static scenes with 3D CNNs can introduce significant computational costs (e.g., numerous floating-point operations (FLOPs)) that may be unnecessary given the content. This additional computational cost may stem from the 3D convolutional network performing certain functions (e.g., convolutions) simultaneously across multiple frames, functions that might not be needed when objects in the scene are not moving between frames.

[0037] Two-dimensional (2D) CNNs can identify objects in a video based on a single video frame, rather than performing the function simultaneously across multiple frames, and therefore can more efficiently identify objects in one or more video frames. However, because 2D CNNs do not operate on multiple frames captured at different times, they will not return any temporal information indicating how objects within a scene move over time. For example, 2D CNNs themselves may not be well-suited for detecting actions (e.g., those occurring across multiple video frames).

[0038] Various methods can be used to overcome the limitation of 2D CNNs, which only generate spatial information from a single frame. In one example, one-dimensional (1D) convolutions can be used as a complementary model to 2D CNNs. 1D convolutions can be used to generate temporal information from features generated by 2D CNNs across consecutive frames. In another example, non-local neural networks with forward skip can be used to detect temporal relationships between features generated by 2D CNNs across non-consecutive frames. In yet another example, graph convolutional networks can be used as a complementary model to 2D CNNs. Graph convolutional networks can be used to generate relationships between concepts in a video based on spatial features generated by 2D CNNs.

[0039] If the most important categories are known for a particular video dataset, designing a machine learning model (e.g., a neural network) specifically for that category can be efficient. However, for analyzing video datasets that include various videos with different characteristics and containing information from different categories, relying on a single model can lead to excessive computational workload (e.g., when the model performs calculations attempting to detect categories not present in the sample videos), poor accuracy (e.g., when the model is not well-suited to detecting information from one or more categories present in the sample videos), or both. When encountering a new dataset that is not suitable for the model (e.g., because the model has not been trained to understand the types of content in the new dataset), a new target model can be developed to address the new problem presented by the new dataset. However, constantly developing new models can be expensive. Developing a new model may also not actually solve the overall video understanding problem, such as when the newly developed model is designed to solve a narrow problem or understand videos containing information from one or more different categories.

[0040] Table 1 below shows the relative performance of five different video understanding models, expressed in mean Average Precision (mAP). The contents of Table 1 provide examples illustrating the results of choosing one model over another for a video. In Table 1, the mAP values ​​of five different machine learning models (Models 1-5) are shown for the categories of action, attribute, concept, event, object, and scene. In some examples, the machine learning models may have specific perceptual biases applied, which can be done by limiting the prior assumptions of the model function by restricting the model's search space. In Table 1, one or more models with the highest mAP in each category (e.g., for the category "Event," Models 2 and 4 have similar values) are underlined. As shown, no single model has the highest accuracy across all categories. Therefore, applying any one of the models in Table 1 consistently to a video dataset containing video data covering multiple related categories will not achieve the best accuracy for that video dataset.

[0041] Model average action property concept event object Scene Inductive bias 1 39.6 50.1 33.8 26.5 39.2 33.1 55.2 - 2 51.5 51.1 44.0 51.8 <![CDATA[ 57.3 ]]> 51.3 53.5 Non-local attention 3 52.0 51.8 <![CDATA[ 45.7 ]]> 51.9 56.1 <![CDATA[ 53.2 ]]> 53.4 1D Convolution 4 52.2 51.8 45.3 <![CDATA[ 52.2 ]]> <![CDATA[ 57.4 ]]> 52.4 <![CDATA[ 54.0 ]]> Graphical convolution 5 46.7 <![CDATA[ 63.5 ]]> 35.3 44.8 50.3 42.7 43.4 3D Convolution

[0042] Table 1

[0043] In one illustrative example, the first model is HATNet as described in “Large-scale holistic video understanding” by Ali Diba et al. (2020), the entire contents of which are incorporated herein by reference and used for all purposes. In another illustrative example, the second, third, and fourth models share a common 2D CNN as the first stage, built on a ResNet-50 neural network architecture, such as that described in “Deep residual learning for image recognition” by Kaiming He et al. (2016), the entire contents of which are incorporated herein by reference and used for all purposes. The ResNet-50 2D CNN generates spatial features. For each of the second, third, and fourth models, the spatial features are further processed by a complementary model. For the second model, a non-local complementary model is applied to the features generated by the 2D CNN. For the third model, 1D convolutions can be applied to the features generated by the 2D CNN to generate temporal information from the spatial features (generated by the 2D CNN) evaluated over multiple frames. 1D convolutions can be applied to features generated by a 2D CNN across multiple frames to generate temporal information about the features in the time dimension (over time). For the fourth model, graph convolutions can be applied to features generated by a 2D CNN. The application of graph convolutions can lead to the generation of relational information from spatial features generated by a 2D CNN to the available classes that the neural network was trained to classify. In an illustrative example, the fifth model could be a 3D CNN using the Slowfast model described in "Slowfast networks for video recognition" by Christoph Feichtenhofer et al. in 2019.

[0044] This document describes systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively, “systems and techniques”) for performing holistic video understanding. As used herein, holistic video understanding refers to understanding the content of a video across many interrelated data categories contained within it. In some examples, the systems and techniques described herein are used to understand, analyze, and / or classify video. However, the systems and techniques described herein for understanding, analyzing, and / or classifying video can be applied to any sequence of frames or images.

[0045] As described in more detail below, the overall video understanding system may include one or more model decision engines (which may be machine learning models or machine learning systems), a feature extraction machine learning model (or machine learning system), and a supplementary machine learning model (or machine learning system). One or more model decision engines may be trained to classify input video data to determine which feature extraction models to use for a particular video or portion of a video (e.g., based on a combination of computational efficiency and accuracy of the feature extraction models combined with the supplementary models available for selection). In an illustrative example, the model decision engines may be trained to select the most accurate model while keeping the average computation for each video or portion of a video below 40 gigaFLOPs (GFLOPs). A portion of a video may include a video clip (e.g., a video clip υ as described below). ij During inference (after one or more model decision engines have been trained), the classification output by the model decision engines can be used to determine which feature extraction models(s) in the neural network will be used to process the input video or video segments. In some cases, the model decision engines can also be trained to learn features associated with the input video or frames of the input video. In some examples, video frames and labels can be used as training data to train the model decision engines, such as using one or more supervised or semi-supervised training techniques. During inference (e.g., after the model decision engines have been trained to extract features), the model decision engines can determine, in some cases, that the features they generate during the classification process will be provided to a supplementary model and subsequently used to classify the input video, without the need for any other available feature extraction models.

[0046] In some examples, the model decision engine is a neural network or a combination of multiple neural network architectures. In other examples, the model decision engine is a 2D CNN or any other suitable lightweight neural network architecture. As mentioned above, 2D CNNs are lightweight because they can identify objects in a video based on a single video frame, rather than performing functions (e.g., convolutions) simultaneously across multiple frames. Using a lightweight neural network architecture for the model decision engine can reduce the computational complexity introduced by leveraging the model decision engine in the overall video understanding system. Keeping the computational cost of including the model decision engine to a minimum (e.g., by using a lightweight neural network architecture) can lead to an overall increase in computational efficiency due to the computational savings from being able to select lightweight features for some input video to generate the model. In an illustrative example, the neural network of the model decision engine includes the MobileNetV2 neural network architecture, such as that described in Mark Sandler's "MobileNetV2: Inverted Residuals and Linear Bottlenecks" (2019), the entire contents of which are incorporated herein by reference and used for all purposes. In another illustrative example, the neural network of the model decision engine includes the ResNet-50 neural network architecture, as described in "Deep residual learning for image recognition" by Kaiming He et al. in 2016, the entire contents of which are incorporated herein by reference for all purposes.

[0047] In some examples, as described below, the model decision engine can be trained along with the feature extraction model and any complementary models in an end-to-end process. In some examples, as described below, the model decision engine can be trained in the final stage of a three-stage process. For example, a three-stage process may include: a first step of training the feature extraction model until a certain level of accuracy is reached; a second step of training the complementary model using features from the trained feature extraction model with frozen weights (e.g., where the weights of the trained feature extraction model are not changed); and a third step of training the model decision engine using the trained feature extraction model and the complementary model with frozen weights. In some cases, supervised learning with pseudo-labels is used to train the decision engine, which tells the decision engine which model is the most accurate. During training iterations, the decision engine attempts to maximize the accuracy of the selected most accurate model, where parameters (e.g., weights, biases, etc.) are tuned in each iteration based on a determined loss.

[0048] In an illustrative example, as described in more detail below, the second step of the three-stage process may include using cross-entropy loss (denoted as...). ) or binary cross-entropy loss (represented as The third step can include training the model decision engine using the following loss: in It is the cross-entropy (CE) loss (or softmax loss). It is a uniform loss, and It's FLOP loss. BCE loss. The decision engine is trained to maximize the accuracy of correct classification based on binary cross-entropy. FLOP loss. Training the decision engine more frequently to select effective models is beneficial because minimizing the expected FLOPs (i.e., the average value above the training data) that will be used is advantageous. The decision can be based on the decisions derived from the decision engine. V The beta (β) parameter controls or tunes the trade-off between accuracy and complexity or computational efficiency (e.g., measured by FLOPs). For example, if β is set to 0, the result is that the decision engine will learn to use the most accurate model. Setting the β parameter to a high value (e.g., close to 1, such as 0.75, 0.80, 0.85, etc.) will cause the decision engine to choose the computationally cheapest model. A uniform loss can be used. This ensures that the model decision engine doesn't always choose the same model that offers some kind of trade-off. The alpha (α) parameter is used to tune the uniform loss. An example of the optimal balance is achieved by selecting the most accurate model with an average of less than 40 GFLOPS.

[0049] By adaptively selecting among one or more machine learning models for processing video data, the systems and techniques described herein can provide more accurate and efficient overall video understanding. For example, as mentioned above, many video understanding systems apply a single, fixed machine learning model to analyze all videos, regardless of whether the data categories included in a particular input video are well-suited for analysis by that fixed model. By adaptively selecting among multiple different machine learning models to perform video understanding based on the characteristics of the specific video or video segment being analyzed, the system and techniques can analyze each video or video segment using a model that is highly suitable for the characteristics of that particular video or segment (e.g., the categories of data included within the video or segment). Furthermore, by avoiding unnecessary computations to computationally expensive models preferred for categories not present in the video or segment, the computational and power resources of the device performing the video understanding techniques can be reserved for other operations.

[0050] While this paper describes examples for classifying videos into specific categories (e.g., playing the piano, children playing soccer, etc.), the holistic video understanding system and techniques described herein can be used to generate other categories not specifically listed. In some examples, the holistic video understanding system and techniques described herein can be used to determine the activities of people or objects in other categories and classifications, such as a person sitting at a table in front of a computer, a person holding a mobile device, robotic equipment in a factory environment, and / or any other actions, events, attributes, concepts, and / or scenes.

[0051] The various aspects of the technology described herein will now be discussed with reference to the accompanying drawings. Figure 1 This is a block diagram illustrating an example of a holistic video understanding system 100. The holistic video understanding system 100 includes various components used to process video data (e.g., one or more videos) and detect or identify information categories (e.g., objects, actions, events, attributes, concepts, and scenes). As shown, the components of the holistic video understanding system 100 include a video retrieval unit 102, a storage device 104, a model decision engine 106, a feature extraction model 108, a supplementary model 110, and an output classifier 112.

[0052] The overall video understanding system may include or be part of the following: a mobile or fixed-line handset (e.g., a smartphone, cellular phone, etc.), a server computer (e.g., communicating with a vehicle computing system), a vehicle (e.g., a vehicle driver monitoring system (DMS)), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video streaming device, or any other suitable electronic device. In some examples, the overall video understanding system 100 may include one or more wireless transceivers (or separate wireless receivers and transmitters) for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, Bluetooth or other short-range communication, any combination thereof, and / or other communication. In some implementations, components of the overall video understanding system 100 (e.g., video retrieval unit 102, storage device 104, model decision engine 106, feature extraction model 108, supplementary model 110, and output classifier 112) may be part of the same computing device. In some implementations, components of the overall video understanding system 100 may be part of two or more separate computing devices. In some cases, the overall video understanding system 100 may be implemented as... Figure 12 This is a part of the computing system 1200 shown.

[0053] Although the overall video understanding system 100 is shown to include certain components, those skilled in the art will understand that the overall video understanding system 100 may include more than [other components]. Figure 1 The components shown may include more or fewer components. In some cases, additional components of the overall video understanding system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some cases, the overall video understanding system 100 may include components not shown in the original document. Figure 1 The system includes one or more sensors (e.g., one or more cameras, inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or one or more other software and / or hardware components. In some embodiments, additional components of the overall video understanding system 100 may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., digital signal processors (DSPs), microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), any combination thereof, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the overall video understanding system 100.

[0054] Video retriever 102 can retrieve frames (e.g., video frames, images, etc.) from storage device 104. Storage device 104 can include any suitable type of storage device (e.g., those described below). Figure 12 (This refers to any type of storage device described in storage device 1230). The video retrieved by video retriever 102 may include a sequence of frames. For example, a sequence of frames may include a set of consecutively captured images or other frame sequences. In an illustrative example, frames may include red-green-blue (RGB) images, luminance, chrominance-blue, chrominance-red (YCbCr or Y'CbCr) images, monochrome images, and / or any other suitable type of image. In some cases, the video retrieved by video retriever 102 may also receive audio components from storage device 104.

[0055] In some respects, video retriever 102 can segment the video retrieved from storage device 104 into evenly spaced portions or segments (e.g., each segment may include the same number of frames). The video segment υ is described below. ijExamples. In some examples, the video retrieval unit may also segment audio data into individual audio segments, such that the audio data matches the time associated with a corresponding frame sequence of a portion of the video. In some aspects, the video may be stored as video portions or segments within storage device 104, and the video retrieval unit may retrieve segments from storage device 104 rather than the entire video. In some aspects, video retrieval unit 102 may retrieve the complete video from storage device 104. In these aspects, the complete video may be divided into portions or segments elsewhere within the overall video understanding system 100 (e.g., before being input into model decision engine 106). In some aspects, the portion may include a subset of video frames, such as two frames, three frames, five frames, or any suitable number of frames. In some examples, the portion may be overlapping (e.g., the portion may share frames common to each other from the frame sequence). In one illustrative example where each video portion comprises three frames, the first portion of the first video may include the first, second, and third frames of the video, while the second portion of the first video may include the second, third, and fourth frames of the video.

[0056] Frames retrieved by video retrieval 102 can be provided as input to model decision engine 106. In some examples, the frames of the retrieved video may have a larger dimension than the dimension of the input layer (not shown) of model decision engine 106. In such cases, each input frame with a larger dimension can be normalized to a resolution that matches the input dimension or resolution to which model decision engine 106 is configured to process. For example, model decision engine 106 (or other components of the overall video understanding system 100) can normalize each frame with a larger dimension to the input dimension or resolution of model decision engine 106 by downsampling or scaling down each frame. In an illustrative example, the size of a video portion can be adjusted to match a 224×224 input resolution (to which model decision engine 106 is configured to process) by resizing the minimum dimension of the retrieved video frame to 224 and extracting a center crop (e.g., the center portion of the video frame). In some examples, downsampled, reduced-size, or otherwise resized frames may be stored in storage device 104 for use by subsequent stages of the overall video understanding system 100.

[0057] In some examples, the model decision engine 106 can be trained to determine which model(s) from the set of feature extraction models 108 should be used to process the input video or a portion thereof. In some cases, the model decision engine 106 can be trained based on a combination of the computational efficiency and accuracy of the feature extraction models 108 and the complementary model 110, which may also be selected by the model decision engine 106. For example, the model decision engine 106 can be trained to select between a first machine learning model and a second machine learning model (from feature extraction models 108 and / or complementary model 110) for processing the input video. In some examples, the model decision engine 106 can be trained to determine which "N" sets of feature extraction models 108 should be used to process the input video, where "N" is an integer greater than or equal to one. The number of feature extraction models 108 can be as many as required for a given application. Some example feature extraction models 108 may include 2D CNNs, 3D CNNs, audio processing models, optical flow networks, and any other neural network architectures that can be used to process video data. In some examples, model decision engine 106 may include a classification neural network trained to classify N output classes equal to the number of feature extraction models 108. For example, model decision engine 106 may process video data and determine probabilities for N classes based on the video data. Model decision engine 106 may select a given model (from feature extraction model 108) associated with the class having the highest probability value from the N classes. In some examples, the neural network of model decision engine 106 may include inner layers that generate one or more feature vectors from input video frames and fully connected layers that combine the features of one or more feature vectors into N classes. An example of training model decision engine 106 is provided below.

[0058] During inference (once the model decision engine 106 has been trained), the model decision engine 106 may receive one or more video frames representing an input video or video portion from the video retrieval unit 102 and / or from the storage device 104. In some examples, one or more frames representing the input video or video portion received by the model decision engine 106 from the video retrieval unit 102 and / or the storage device 104 are portions of the input video, as described above. In some examples, the model decision engine 106 may generate a classification of the input video or video portion (e.g., the class with the highest probability among N classes). In some cases, the model decision engine 106 may process each frame of the input video or video portion sequentially to generate a classification for each input video, video portion, or video frame. In some implementations, the model decision engine 106 may generate a classification from multiple sequential frames of the input video or video portion (e.g., by averaging features from multiple sequential frames and using a multilayer perceptron (MLP), fully connected layers, or other neural network components to process the averaged features to generate a classification).

[0059] In some examples, the classification generated by the model decision engine can be used to determine which of the feature extraction models 108 should be used for further processing of the input video (e.g., the entire input video). In some examples, the classification can be used to determine which model from the feature extraction models 108 should be used for further processing of a portion of the input video (e.g., one or more frames of the input video), where each portion of the video can receive a classification separate from the other portions of the video. For example, the first 90 frames of the video may contain static scenes, and action may begin in the video during the last 150 frames. In this example, the model decision engine 106 can determine (based on one or more classifications determined by the model decision engine 106) to use a 2D CNN to process the first 90 frames, and can determine to use a 3D CNN to process the last 150 frames. In some examples, the classification output by the model decision engine 106 can be stored in storage device 104.

[0060] In some examples, in addition to generating classifications, model decision engine 106 may also generate features based on the input video or video portions. For example, model decision engine 106 may include a 2D CNN that generates features from the input video. These features can then be fed into layers (e.g., fully connected layers or other layers) of model decision engine 106 to generate classifications (e.g., indicating which feature extraction model 108 should be used to process the input video or portions thereof, as described above). In some examples, features generated within model decision engine 106 may also be stored in storage device 104. Features generated by model decision engine 106 may be used as input to one or more supplementary models 110 for additional processing, or may be used directly to classify the video at output classifier 112. In some examples, features generated by model decision engine 106 may be stored in storage device 104.

[0061] Feature extraction model 108 can be trained to learn features associated with an input video or a portion of a video. In some examples, feature extraction model 108 may include at least two feature extraction models that model decision engine 106 can select from. In some examples, feature extraction model 108 may each have different neural network backbones that extract features from video data using different neural network architectures. For example, one feature extraction model from feature extraction model 108 may include a 2D CNN, which can be trained to extract spatial features from individual video frames (or images that are not part of the video). Another feature extraction model from feature extraction model 108 may include a 3D CNN, which can be trained to extract spatial and temporal features from a sequence of frames. In some examples, a 3D CNN can be trained to generate spatial and temporal features by performing convolutions on a portion of a video comprising three, five, or any other number of frames, providing an expected trade-off between temporal resolution and computational cost for the features generated by the 3D CNN. In some cases, another feature extraction model from feature extraction model 108 may be an audio model that can be trained to extract features from audio information associated with the input video. Another feature extraction model from feature extraction model 108 may include an optical flow model that can be trained to detect motion of objects or features (e.g., edges, shapes) across multiple video frames. In some examples, two or more feature extraction models from feature extraction model 108 may use similar modalities (e.g., 2D CNN, 3D CNN, or other CNNs) but operate with different architectures. Different architectures may have different computational costs. For example, two of feature extraction models 108 may both include 2D CNN architectures, where the first 2D CNN has lower accuracy in generating spatial features and lower computational cost compared to the second 2D CNN. Feature extraction model 108 can be used to extract features that can then be used to classify the input video. An example of training feature extraction model 108 is provided below.

[0062] During inference (once the feature extraction model 108 has been trained), the selected feature extraction model 108 (e.g., selected by the model decision engine 106) can receive one or more frames from the video retrieval unit 102, the storage device 104, and / or the model decision engine 106. As described above, the selected feature extraction model 108 can be activated based on the classification generated by the model decision engine 106. In some examples, the model decision engine 106 can directly activate the selected feature extraction model 108 (e.g., by issuing a command or signal instructing the selected feature extraction model 108 to begin). In some examples, the feature extraction model 108 can retrieve the classification output from the model decision engine 106 (e.g., directly from the model decision engine 106, from the storage device 104, etc.). The features extracted by the selected feature extraction model 108 can be provided as input to the supplementary model 110.

[0063] Depending on the type of model selected by the model decision engine 106, the selected feature extraction model 108 can process each frame of one or more frames consecutively (e.g., one frame at a time) or process multiple frames simultaneously. For example, when a 2D CNN is selected from the feature extraction model 108, the 2D CNN can extract spatial features from each frame (e.g., based on processing that frame individually). In another example, when a 3D CNN is selected from the feature extraction model 108, the 3D CNN can extract spatial and temporal features from multiple consecutive frames (e.g., based on processing multiple frames simultaneously). In some cases, choosing a 2D CNN may be advantageous. For example, compared to 3D CNNs or other models, 2D CNNs may have relatively low computational costs (e.g., measured by the number of floating-point operations (FLOPs) required to generate features using the model). In some examples, the ratio of computational costs between 2D CNNs and 3D CNNs can be a function of the number K of video frames processed simultaneously by the 3D CNN, where K can be equal to the depth of the 3D CNN kernel in the temporal dimension. In some cases, choosing a 3D CNN may be advantageous. For example, a 3D CNN can provide a strong temporal bias for understanding action in a specific video or portion of a video. In one example, when the video or portion being analyzed depicts a static scene (e.g., from a wall-mounted camera) and there is little or no motion in the scene, a 2D CNN can be selected by the model decision engine 106. In this example, the model decision engine 106 can determine characteristics of the video (e.g., the static scene) that make a 2D CNN suitable for the video. In another example, when the video or portion being analyzed depicts a dynamic scene (e.g., a scene captured by a static or moving camera including moving objects such as a sporting event), a 3D CNN can be selected by the model decision engine 106. In this example, the model decision engine 106 can determine characteristics of the video (e.g., motion within the scene) that make a 2D CNN suitable for the video. The ability of a 3D CNN to detect action by simultaneously extracting features from video data across multiple frames can lead to a more accurate understanding of video content compared to two 2D CNNs that can only extract features from one frame at a time.

[0064] In some cases, the output of the feature extraction model 108 may include a feature vector (or other feature representation) representing features extracted from each frame or from a sequence of frames. In some cases, a single feature vector may be extracted from a single input frame, where the feature vector represents the features of the input frame (e.g., for a 2D CNN that generates spatial features per frame). The features extracted from the input frames (e.g., represented as feature vectors) provide a representation of the frame, including information contained within the input frame. In some cases, a single feature vector may be extracted from multiple input frames, where the feature vector contains all the information associated with the features of multiple input frames (e.g., for a 3D CNN or optical flow network that generates spatial and temporal features over multiple frames). In an illustrative example, the feature vector generated by the feature extraction model per frame or for multiple frames may be a 1x2048 vector (indicating that the feature vector is a one-dimensional feature vector with a length of 2048 values). In some examples, storage device 104 may be used to store features extracted by the feature extraction model 108 from the input video or a portion of the video (e.g., by storing feature vectors extracted for each frame or group of frames). In some cases, feature representations other than feature vectors, such as tensors or other representations, can be used.

[0065] A supplementary model 110 can be trained to supplement the features extracted by the feature extraction model 108 with additional information that can be used for video classification. During inference (e.g., after training the supplementary model 110), the supplementary model 110 can receive features output by the feature extraction model 108 and can provide further processing. In some examples, a fully connected layer (not shown) can be included between the feature extraction model 108 and the supplementary model 110. In one example, as described above, the feature extraction model utilizing a 2D CNN can have a strong spatial bias in its generated features, but may not include any temporal information across multiple frames because the 2D CNN operates on only a single video frame at a time. In some examples, the supplementary model from the supplementary model 110 can be applied to the features generated by the 2D CNN for multiple frames to generate temporal information across multiple frames. In one example, the supplementary model can include a 1D convolutional neural network, which can be applied to the features generated by the 2D CNN (selected from the feature extraction model 108) for multiple frames to generate temporal information for those multiple frames. See below for reference. Figures 2A-2D An illustrative example describing a 1D convolutional network (as an example of a supplementary model).

[0066] Figures 2A-2C The illustration shows an example convolution operation that can be used as a supplementary model 110, which is applied to a system... Figure 1 Features generated by feature extraction model 108 in the overall video understanding system 100. Figures 2A-2C The diagram illustrates the combined feature vectors, including the combined feature vector 210 ( Figure 2A (middle), combined feature vector 212 ( Figure 2B (in the middle) and the combined eigenvector 214 ( Figure 2C (in Chinese). This includes a combination of five 1×2048 feature vectors from five consecutive frames output by a 2D CNN feature extraction model, as described above regarding... Figure 1 As described. The resulting combined feature vectors 210, 212, and 214 each have a dimension of 5 × 2048 (indicating that each combined feature vector 210, 212, and 214 includes features from five consecutive frames output from the 2D CNN). Figures 2A-2C In the illustrated example, each cube (whether black or white) in the combined feature vectors 210, 212, and 214 represents a value of the combined feature vector. Figures 2A-2D The black shading within the cubes of the combined feature vectors 210, 212, 214, and 216 illustrates the different convolutional filters that can be applied to the combined feature vectors (where the convolutional filters move across the data over time). Figure 2A In the example, the selected convolutional filter 211 is a 5×3 kernel that spans all five values ​​across the time dimension and across three channels (in the direction indicated by arrow 221). Figure 2A In the example, the kernel moves along the channel dimension (as shown by arrow 221) to perform convolution.

[0067] exist Figure 2B In the example, the selected convolutional filter 213 is a 3×1 kernel spanning three values ​​in the time dimension and one value in the channel dimension. The 3×1 kernel moves along the time and channel dimensions (as indicated by arrow 223 in the channel dimension and arrow 225 in the time dimension). In some cases, the 3×1 kernel first moves one “column” (corresponding to a stride of one) per channel in the time dimension to generate three new convolutional output values ​​for the first channel. In some cases, the 3×1 kernel then moves to the next “row” (corresponding to a stride of one) in the channel dimension of the combined feature vector. The 3×1 kernel can then repeat the three convolutions that have moved in the time dimension to generate three additional new convolutional output values ​​for the second channel. In some examples, this process can be repeated for each “row” in the channel dimension to generate a 3×2048 convolutional output. In some cases, an additional neural network layer can be used to generate a 1x2048 output from the 3x2048 convolutional output. In an illustrative example, an additional convolution (e.g., using a 3×1 kernel) that moves one “row” (corresponding to a stride of one) along the channel dimension each time can be applied to the 3×2048 convolution output to produce a 1×2048 combined feature vector. Figure 2B The convolutional filter shown can be considered a 2D convolution because the kernel moves along both the time and channel dimensions.

[0068] exist Figure 2C In the example, the selected convolutional filter 215 is a 3×2048 filter kernel spanning three values ​​in the time dimension and all 2048 values ​​in the channel dimension. The first 3×2048 kernel moves along the time dimension (as indicated by arrow 227), such as one cube at a time, two cubes at a time, etc. In an illustrative example, the first 3×2048 kernel creates a first convolutional output value based on the convolution of 3×2048 features at a first position. In such an example, the first 3×2048 kernel may move one “column” at a time in the time dimension and generate two additional values ​​for a total of 3×1 convolutional output values. In such an example, the process can be repeated with 2047 additional 3×2048 kernels to generate a 3×2048 combined convolutional output vector. In some examples, additional neural network layers (e.g., convolutional layers) may be used to generate a 1x2048 output from the 3x2048 convolutional output. In some implementations, Figure 2C The convolutional filter 215 described herein can be used to capture the recurrence pattern present in the temporal dimension of the data from the combined feature vector 214. Figure 2C The convolution shown can be considered a 1D convolution because each kernel only moves along the time dimension.

[0069] Figure 2D The diagram shows Figure 2AAn example of the output of a 1D convolution is shown. A convolutional filter 211 (as shown by the black cube in the combined feature vector 210) can be used to generate a new feature vector 216 with a dimension of 1×2048. As shown, the first element of the new feature vector 216 (as shown by the black cube 217) is generated based on the 5×3 convolutional filter kernel applied to the first three "rows" (in the time dimension) of the combined feature vector 210. Similarly, the second element of the new feature vector 216 can be generated by moving the 5×3 convolutional filter forward one row (e.g., in the direction of arrow 221) and applying convolution to each element of the combined feature vector 210. As the filter kernel moves in the channel dimension (as shown by arrow 221), each subsequent element of the new feature vector 216 can be generated based on the application of the 5×3 convolutional filter. The resulting new feature vector 216 can have a dimension of 1×2048, as described above, where each element of the new feature vector 216 represents the result of the convolutional filter applied to the combined feature vector 210. By combining features from multiple frames using this technique, temporal information about those features can be generated, even though the original features were extracted from a single frame without any temporal information. This technique offers the advantage of including temporal information for a more comprehensive understanding of the input video without introducing the full computational cost of performing 3D convolutions.

[0070] It should be understood that, despite the above regarding Figures 2A-2D The example is described based on a combination of feature vectors from the output of a 2D CNN, but Figures 2A-2D The convolutional filters described herein can be applied to any feature vector, regardless of the type of neural network from which it originates. Furthermore, although... Figures 2A-2C The example illustration depicts a feature vector with a combined dimension of 5×2048; however, those skilled in the art will understand that the described filter can be applied to feature vectors with different dimensions. For example, a feature vector representing a combination of only three frames of feature data could have a dimension of 3×2048. In another example, a feature vector representing a combination of five frames of feature data could have fewer features, such as 1024 features, and in this case, the combined feature vector could have a dimension of 5×1024.

[0071] Back Figure 1 As described above, an additional supplementary model 110 can be provided to further process the features from the feature extraction model 108. In some examples, the supplementary model (from supplementary model 110) applied to the features generated by the feature extraction model 108 may not apply additional inductive bias to the output of the feature extraction model 108. For example, as mentioned above... Figure 2DAs described, when a video does not contain motion (e.g., a static scene) or contains only a small amount of motion that is insufficient to benefit from the increased computational workload of a 3D CNN, a 1D convolutional supplementary model (selected from supplementary model 110) can be used to generate temporal information of the video based on features extracted by a 2D CNN feature extraction model (from feature extraction model 108). As an example, when the input video contains a static scene, a 1D convolutional supplementary model can be selected, such as a scene from an IP camera pointing to a fixed location where there may be incidental movement of objects or people to be analyzed within the scene. This approach reduces computational workload because temporal information can be processed from the spatial features of video frames extracted by a 2D CNN, rather than directly extracting both spatial and temporal information from video frames simultaneously (e.g., using a 3D CNN), which could potentially require processing significantly more data.

[0072] Another example of a complementary model is the nonlocal model (also known as an attention model or nonlocal attention model). Nonlocal models extract temporal information from features extracted by a 2D CNN feature extraction model by comparing features from each video frame with features from one or more other video frames in the set of video frames (e.g., every other video frame). For example, for a set of five video frames, features from the first frame can be compared separately with features from each of the second, third, fourth, and fifth frames to determine whether events or actions in distant frames (e.g., first and fifth frames, first and fourth frames, second and fifth frames, second and fourth frames, etc.) are related. For a large number of frames, the number of comparisons in a nonlocal complementary model can become very large. The benefit of nonlocal models is that they can detect relationships that models analyzing only information from closely grouped frames (e.g., consecutive frames) might not detect. For example, unlike 1D convolutional complementary models, the inductive bias of nonlocal models does not necessarily assume that related actions will be grouped together temporally. By attempting to determine these types of nonlocal relationships in a video, the overall video understanding system 100 can become more holistic based on its ability to analyze features over a wider range within the video (e.g., across more temporally separated frames). A potentially useful illustrative example of a nonlocal supplementary model 110 is a video of dominoes falling, where the effect of a domino falling on the last domino in the first frame may not be seen until many frames later. The relationship between the first and last domino falls can be detected not by a 3D CNN or a 2D CNN supplemented by 1D convolutions, but by a 2D CNN supplemented by a nonlocal model.

[0073] Another example complementary model that can be applied to the features extracted by the 2D CNN feature extraction model 108 is a graph convolutional network. In some examples, a graph convolutional network can be used as a complementary model (of complementary model 110) to associate features from the feature extraction model (from feature extraction model 108) with certain concepts using knowledge from the final classifier applied by classifier 112. In some examples, the graph convolutional network applies a concept or relation bias to the features it receives. In one example, the graph convolutional network may be able to infer the classification “children playing soccer” applied to a video or video segment based on the presence of features generated by the feature extraction model (e.g., ball, child, grass, etc.). In some implementations, this can be done by associating features with concepts and / or the output classifier or classification. In some cases, the complementary model may not impose any additional inductive bias on the features generated by the feature extraction model. For example, the complementary model can be used for the purpose of concatenating features generated by the feature extraction model to one of the classes that the output classifier 112 is trained to classify. In an illustrative example, the feature extraction model from feature extraction model 108 is a 3D CNN, and the complementary model 110 can be a multilayer perceptron (MLP) or a fully connected layer. For example, model decision engine 106 can select a 3D CNN from feature extraction model 108 with MLP complementary model 110 based on a video or video segment containing a large amount of motion (e.g., a video of a sports event where the camera is also in motion). In some examples, the output of complementary model 110 can be stored in storage device 104.

[0074] The output classifier 112 can be trained to classify the input video or video portion (or features associated with the input video or video portion) using one or more classifications. A training dataset including example videos and classification labels can be used to train the output classifier 112, such as using information about... Figure 10 and Figure 11 The described technique. During inference (e.g., after output classifier 112 has been trained), output classifier 112 may receive output from supplementary model 110 and may use that output to classify the input video or video portion. Example classifications may include “playing the piano,” “children playing soccer,” and any other classifications that output classifier 112 is trained to classify.

[0075] In some cases, the system, including model decision engine 106, feature extraction model 108, complementary model 110, and output classifier 112, can be trained in a single end-to-end training sequence. In one implementation of the end-to-end training sequence, a Gumbel Softmax layer (also referred to as a Gumbel layer) can facilitate end-to-end training. The Gumbel layer allows model decision engine 106 to make discrete decisions while still being differentiable, enabling a backpropagation training process to be performed. In some examples, a backpropagation training process can be performed to adjust the weights (and in some cases, other parameters such as biases) of the nodes of each neural network (e.g., model decision engine 106, feature extraction model 108, complementary model 110, and output classifier 112). (See also: ...) Figure 10 In more detail, backpropagation may include forward propagation, loss function, backward propagation, and weight updates. Forward propagation, loss function, backward propagation, and parameter updates may be performed for each training iteration (e.g., for each batch of data from the training set or for the entire training dataset). The backpropagation process may be repeated a certain number of iterations for each training dataset until the weights of the parameters of the model decision engine 106, feature extraction model 108, supplementary model 110, and output classifier 112 are accurately tuned. In some examples, the loss function used during end-to-end training may include cross-entropy (CE) loss, binary cross-entropy (BCE) loss, FLOP loss, uniform loss, any combination thereof, or any other suitable loss function. In some examples, the loss function used during end-to-end training may include a weighted combination of multiple loss functions (e.g., a combination of BCE loss and FLOP loss with different weights).

[0076] In some cases, it may be difficult to train the model decision engine 106 using an end-to-end training process across the entire video understanding system 100. For example, the model decision engine 106 might predict which model to choose based on past statistics of the feature extraction model 108, the complementary model 110, and the classifier 112, and these models change during each iteration of training due to the end-to-end nature of the training. In some examples, a three-stage training process can be used instead of end-to-end training of the entire video understanding system 100. For example, the feature extraction model 108 can be trained during the first stage of the three-stage training process. Similar to the end-to-end training process described above, a backpropagation training process can be performed to adjust the weights of each node in the feature extraction model 108 (and in some cases, other parameters such as biases). A loss function can be used to analyze the error in the output of the feature extraction model 108 relative to known outputs or ground truth (e.g., based on ground truth labels associated with the training data). In an illustrative example, the cross-entropy loss function can be used. Other loss functions can be used in some cases. After training the feature extraction model 108, the weights of the feature extraction model 108 (and other parameters such as bias in some cases) can be fixed (or "frozen").

[0077] In some examples, features extracted by the trained feature extraction model 108 can be used to train the supplementary model 110 in the second stage of the three-stage training process. In some cases, a backpropagation training process similar to that described for the feature extraction model 108 can be used to train the supplementary model. After training, the weights of the supplementary model 110 (and in some cases, other parameters, such as biases) can be fixed or frozen. In some examples, after training the feature extraction model 108 and the supplementary model 110, the model decision engine 106 can be trained in the final stage of the three-stage training process. In some cases, during the three-stage training process, the model decision engine 106 can be trained to learn which model(s) from the fixed set of models (e.g., feature extraction model 108 and supplementary model 110) should be selected to process the input video or video portion. In some examples, the same dataset can be used for all three training stages of the three-stage training process. See below for reference. Figure 3 The description provides additional details on training the overall video understanding system, including example loss functions that can be used to train the model decision engine 106.

[0078] Figure 3 This is an illustration of another example of the overall video understanding system 300. As shown, the components of the overall video understanding system 300 include a model decision engine 306, video processing models 313A, 313B to 313M, and a classifier 312. Figure 3 One or more components of the overall video understanding system 300 can be similar to Figure 1 Similar components of the overall video understanding system 100 and perform similar operations. For example, the model decision engine 306 and classifier 312 can be used with... Figure 1 The model decision engine 106 and output classifier 112 are similar and perform similar operations. Video processing models 313A, 313B to 313M are... Figure 1 Example of feature extraction model 108.

[0079] In some examples, the model decision engine 306 can be trained to determine which model(s) from video processing models 313A, 313B to 313M are used to process the input video 302 or a portion of the video (such as video clip υ as described below). ij In some cases, the model decision engine 306 can be trained to determine which model(s) to use based on a combination of computational efficiency and accuracy of video processing models 313A, 313B to 313M in processing input video 302. In some examples, the model decision engine 306 may include a classification neural network trained to classify N classes equal to the number of video processing models 313A, 313B to 313M. For example, the model decision engine 306 can process video data and determine probabilities for the N classes based on the processed video data. The model decision engine 306 can select the video processing model (from video processing models 313A, 313B to 313M) associated with the class having the highest probability value from the N classes. In some examples, the neural network of the model decision engine 306 may include inner layers that generate one or more feature vectors from input video 302 or video portions. In some cases, the neural network of the model decision engine 306 may also include fully connected layers that combine features from one or more feature vectors into N classes.

[0080] During inference (after training the model decision engine 306), the model decision engine 306 can determine which one or more of the video processing models 313A, 313B to 313M are used to process the input video 302 or a portion of the video. Return to Reference Figure 1 In some implementations, the model decision engine 306 may receive input video 302 or video portions from video retrieval unit 102 and / or storage device 104. In some examples, the input video may be decomposed into J equal-sized portions v. ij Return to reference. Figure 1In some cases, video processing models 313A, 313B through 313M may each include a feature extraction model (from feature extraction model 108) and a corresponding complementary model (from complementary model 110). As described above, based on processing the input video 302 or video portion, model decision engine 306 can generate a classification (e.g., N classes) representing the probability that each available model among video processing models 313A, 313B through 313M will produce the best output for a particular input video 302 or video portion (e.g., based on a combination of accuracy and computational effort). In some examples, model decision engine 306 may select a single video processing model (e.g., the video processing model associated with the class with the highest probability among the N classes) to process the input video 302 or video portion. In some cases, the classification generated by model decision engine 306 may be a one-hot representation, where the selected model has a value of 1, and the unselected model has a value of 0. In an illustrative example, for an overall video understanding system 300 comprising two video processing models, a classification vector [1 0] can indicate to the model decision engine 306 that it has selected a first video processing model from video processing models 313A, 313B to 313M to process the input video 302 or video portion v ij .

[0081] In some examples, model decision engine 306 may select two or more video processing models from video processing models 313A, 313B to 313M to process input video 302. For example, when there is no obviously optimal model to choose from video processing models 313A, 313B to 313M, model decision engine 306 may select two or more video processing models to process input video 302 or video portions. In some examples, the outputs from two or more selected video processing models from video processing models 313A, 313B to 313M may be combined before classifier 312 performs final classification of input video 302 or video portions. In an illustrative example, the outputs of two selected models may be combined by averaging. In some cases, classifier 312 may receive the outputs of selected video processing models (or combined outputs from two or more selected video processing models) from video processing models 313A, 313B to 313M. Using this output, classifier 312 may classify input video 302 or video portions. Example categories include "Playing the Piano," "Children Playing Soccer," "Making Sandwiches," and "Listening to Classical Music." (See also...) Figure 1 Storage device 104 can be used to store features extracted from frames of video 302 by one (or more) video processing models selected from video processing models 313A, 313B to 313M.

[0082] exist Figure 3 In an illustrative example of the training process for the overall video understanding system 300, a dataset D containing V videos can be used for the training process. For dataset D, υ ij (It can correspond to) Figure 3 The input video 302) represents video V (where video is represented as υ). i J uniformly spaced (and possibly overlapping) segments of a video (∈V). Video segment υ ij The portion of the input video 302 that can be processed by the overall video understanding system 300 (e.g., each video segment υ) is used as the input video 302. ij Examples (including a portion). Terminology This represents the neural networks that can be selected by the model decision engine 306 (e.g., Figure 3 The video processing models in the document are 313A, 313B to 313M. For a specific video segment υ ij The selected neural network (from video processing models 313A, 313B to 313M) can provide predictions. In some cases, each individual neural network can be trained separately using cross-entropy (CE) loss. as follows:

[0083]

[0084] in It is the expected value, and it applies to all fragments υ on dataset D. ij and associated tag y i Iteratively evaluate the loss function. In some cases, the binary cross-entropy (BCE) loss function can be used to train each individual neural network. In some respects, the BCE loss function can be used to train neural networks. This allows you to apply multiple category tags to a single video.

[0085] In an illustrative example, all the models The Adam optimizer, with an initial learning rate of 1e-4, can be trained for ten epochs. An epoch is a single pass through the entire dataset using the given model. In such an example, the model... This can be downgraded to 1e-5 and 1e-6 in Epoch II and Epoch VII, respectively. In an illustrative example, as described above, after resizing the minimum size of input video 302 to 224 and extracting the center crop, video clip υ ijThe input resolution is 224×224. In this illustrative example, during training, each video processing model 313A to 313M can end with a classification layer consisting of a dropout layer with p=0.5 and a linear layer for the class specific to the dataset.

[0086] As described above, in the three-stage training process, the feature extraction model 108 can be trained independently in the first stage. Once each feature extraction model 108 has been trained, a supplementary model 110 can be trained in the second stage using the trained feature extraction models 108 with fixed or frozen weights. Returning to... Figure 3 In some examples, video processing models 313A, 313B to 313M can be combined with feature extraction model 108 and supplementary model 110, such as Figure 1 As described. In such an example, the first training phase can be applied to the feature extraction model, while the second training phase can be applied to a supplementary model incorporated within the video processing model 313 using the aforementioned cross-entropy loss. In the third phase, the model decision engine 306 can be trained to predict which neural network (e.g., which of the video processing models 313A, 313B to 313M) will be applied to a specific video segment. The model decision engine 306 can be powered by the function π(υ). ij Mathematically represented. In some implementations, the model decision engine 306 selects which model to use to process the video or video portion, wherein the selection by the decision engine 306 can be determined by a ij =π(υ ij Defined by ) . Choice a from decision engine 306 ij (e.g., predictions of the selected model m) It can be represented by the following:

[0087]

[0088] in It is an indicator function, and It is the output classification of model m.

[0089] The basic fact label y of the model decision engine 306 i Through neural networks Running video clips of dataset D ij To infer, the accuracy of the model decision engine 306 can be evaluated using the following cross-entropy (CE) loss function (also known as softmax loss):

[0090]

[0091] In some examples, argmax m Return the neural network with the maximum negative cross-entropy (from neural network) The value of 1 for (selection) indicates a single heat source.

[0092] In addition to the aforementioned CE losses In addition, additional loss functions can be applied to optimize the decisions of the model decision engine 306. For example, a uniform loss term can also be used. A uniform loss term forces the model to make decisions on average across the total number of M available neural networks (e.g., Figure 3 Select each neural network from video processing models 313A, 313B to 313M. The number of equal occurrences is represented as follows:

[0093]

[0094] Finally, a loss function that addresses the efficiency of the entire system can be used, where the computational workload, represented by the total number of FLOPs, can be optimized. An example of such a loss function (called the FLOP loss) is as follows:

[0095]

[0096] The combined loss function of the three loss functions mentioned above can be used to train the model decision engine 106. The total loss function can be mathematically expressed as follows:

[0097]

[0098] Where α and β are terms that can be applied to the total tuning loss function. The weights of β. For example, increasing β can increase the degree to which the total loss function factor in the FLOP loss is increased, while setting β to zero will cause the total loss function to ignore the FLOP loss.

[0099] Figure 4 This is an illustration of an example of a holistic video understanding system 400. As shown, the components of the holistic video understanding system 400 include a model decision engine 406, a feature extraction model 408, a supplementary model 410, and a classifier 412. Figure 4 In the example, the feature extraction model includes a feature extraction component 429 (which in some cases may be part of the model decision engine 406), a first feature extraction model 431, and a second feature extraction model 433. The supplementary model 410 includes a first supplementary model 435, a second supplementary model 437, and a third supplementary model 439. The input video 402 processed by the model decision engine 306 may be the entire video or a portion of the video (e.g., from...). Figure 1 (Retrieved by video retriever 102 and / or storage device 104).

[0100] Figure 4 One or more components of the overall video understanding system 400 can be similar to Figure 1 The overall video understanding system 100 and / or any other example of the overall video understanding system described herein may use similar components and perform similar operations. For example, supplementary model 410 and classifier 412 may be used with Figure 1 The supplementary model 110 and the output classifier 112 are similar and perform similar operations. Figure 4 Feature extraction model 408 can be similar to Figure 1 Similar components of the overall video understanding system and perform similar parts, except as described below.

[0101] Similar to the above, the model decision engine 406 can be trained to determine which feature extraction model from the feature extraction model 408 is used to process the input video 402 or video portion. The model decision engine 406 can also be trained to extract features from the input video 402 or video portion. During inference (e.g., after the model decision engine 406 has been trained), the model decision engine can extract features from the input video 402 or video portion as a step in a classification process that determines which feature extraction model from the feature extraction model 408 is used to process the input video 402 or video portion. In some examples, when compared with features from other models from the feature extraction model 408, the model decision engine 406 can determine that the features generated by a given model from the feature extraction model 408 provide the best expected result (e.g., the best combination of accuracy and computational effort). As described above, in some embodiments, the feature extraction portion 429 can be part of the model decision engine 406 and can be considered one of the feature extraction models 408. Figure 4 As shown, each feature extraction model 408, including the feature extraction portion 429 of the model decision engine 406, can be associated with the features described above. Figure 1 The supplementary model 410 described is associated with this.

[0102] In an illustrative example of the overall video understanding system 400, model decision engine 406 includes a MobileNetV2 neural network, model 431 includes a ResNet-50 neural network, and model 433 includes a SlowFast neural network. In such an illustrative example, supplementary models 435 and 437 may each include 1D convolutional networks (e.g., as mentioned above regarding...). Figure 2A and Figure 2D(The 1D convolutional network described). Continuing with the same example, supplementary model 439 may include an MLP. The MobileNetV2 and ResNet-50 networks are 2D CNNs that extract spatial features from individual video frames. Spatial features from the MobileNetV2 and ResNet-50 networks can be supplemented with temporal information from the supplementary 1D convolutional network. The Slowfast network includes a 3D CNN that extracts features from 3D convolutions comprising multiple video frames, providing both spatial and temporal information. In such an example, the MobileNetV2 network (model decision engine 406) may be the least computationally intensive, the ResNet-50 network (model 431) may be more computationally intensive than MobileNetV2 but less than Slowfast, and Slowfast (model 433) may be the most computationally intensive. Examples of how model decision engine 406 can select each of the available feature extraction models (e.g., feature extraction part 429 and the first and second feature extraction models 431 and 433) (e.g., based on the content type in the input video 402 or a portion thereof) are referenced above. Figure 1 describe.

[0103] Figure 5 This is a diagram illustrating the overall video understanding system 500. As shown, the components of the overall video understanding system 500 include a model decision engine 506, a feature extraction model 508, a supplementary model 510, a combiner 511, and a classifier 512. Figure 5 One or more components of the overall video understanding system 500 can be similar to Figure 1 The overall video understanding system 100 and / or any other example of the overall video understanding system described herein, and similar components, perform similar operations. For example, feature extraction model 508, supplementary model 510, and classifier 512 can be used with... Figure 1 The feature extraction model 108, supplementary model 110, and output classifier 112 are similar and perform similar operations. The input video 502 can be processed by the model decision engine 506. (See reference...) Figure 1 As an example, the input video 502 can be retrieved from video retrieval device 102 or from storage device 104. Figure 4 As shown, the example overall video understanding system 400 includes a one-to-one (1:1) relationship between the feature extraction model 408 and the supplementary model 410. Figure 5 The overall video understanding system 500 illustrates a configuration in which each feature extraction model of the feature extraction model 508 may have a one-to-many (1:many) or many-to-many (many:many) relationship with the supplementary model 510.

[0104] In some examples, the model decision engine 506 can be trained to determine which feature extraction model(s) 508 will be used to process the input video 502 or video segment. The model decision engine 506 can also be trained to determine which supplementary model(s) from supplementary model 510 will be applied to the features generated by the feature extraction model(s) (selected from feature extraction model 508) to achieve the desired objective. One example of this objective is achieving an optimal balance between efficiency and accuracy. In some examples, each pairing of the selected feature extraction model(s) (from feature extraction model 508) and the selected supplementary model(s) (from supplementary model 510) can be similar to... Figure 3 One of the video processing models 313A, 313B to 313M.

[0105] The three-step training process described above can be used to train the overall video understanding system 500. For example, as described above, the model decision engine 506 can be trained in the third step after the feature extraction model 508 and the supplementary model 510 have been trained (with fixed or frozen weights after training). During inference (e.g., after the model decision engine 506 has been trained), the model decision engine 506 can base its decisions on the input video 502 or a portion of the input video 502 (e.g., a segment υ of the input video 502). ij This determines which feature extraction model(s) from feature extraction model 508 and which supplementary model(s) from supplementary model 510 will be used to process the input video 502 or video portion.

[0106] In some implementations, model decision engine 506 may output a classification that allows selection of a single feature extraction model 508 paired with one or more supplementary models from supplementary model 510. In one example, the model decision engine may select O (where O is one or more, two or more, or any other number) supplementary models to process the outputs of the feature extraction models from feature extraction model 508. In such an example, each of the O supplementary models from supplementary model 510 may receive features extracted by the selected feature extraction model from feature extraction model 508. Each of the O supplementary models may process the features and produce outputs, resulting in O total outputs (e.g., feature vectors) from the O supplementary models. In some examples, combiner 511 may receive O outputs and combine the O outputs into a single output (e.g., a single feature vector). In an illustrative example, combiner 511 may average the O outputs to combine them into a single output, which may be provided to classifier 512 for classifying input video 502 or video portions. In an illustrative example, combiner 511 can cascade O outputs to combine the O outputs into a single output, which can then be fed to classifier 512 to classify the input video 502 or a portion of the video.

[0107] Figure 5 This highlights an illustrative case where model decision engine 506 selects a single feature extraction model (e.g., feature extraction model 541) from feature extraction model 508. Features from the selected feature extraction model 541 are received by two selected complementary models (e.g., complementary model 543 and complementary model 545) from complementary model 510. Furthermore, in such an illustrative example, the output 547 of the selected complementary model 543 and the output 549 of the selected complementary model 545 are received by combiner 511. As described above, combiner 511 can combine the two outputs 547 and 549 (e.g., by averaging the values ​​of the feature vectors associated with outputs 547 and 549, by concatenating the values ​​of the feature vectors associated with output 547 and the feature vectors associated with output 549, or by otherwise combining outputs 547 and 549).

[0108] In some examples, model decision engine 506 may output a one-hot classification for processing input video 502 (e.g., selecting a single feature extraction model 508 paired with a single complementary model 510). In some implementations, when only one complementary model from complementary model 510 produces output, combiner 511 may pass the output directly (without changing the output) to classifier 512 for classification.

[0109] Figure 6This is an illustration of another example of the overall video understanding system 600. As shown, the components of the overall video understanding system 600 include a model decision engine 606, a feature extraction model, a supplementary model 610, a combiner 611, a classifier 612, and a second supplementary model decision engine 651. The feature extraction model includes a first feature extraction model 608A and a second feature extraction model 608B. The supplementary model 610 includes a first supplementary model 653, a second supplementary model 655, a third supplementary model 657, and a fourth supplementary model 659. Figure 6 One or more components of the overall video understanding system 600 can be similar to Figure 1 The overall video understanding system 100 and / or any other example of the overall video understanding system described herein, and similar components, perform similar operations. For example, the model decision engine 606, feature extraction models 608A and 608B, supplementary model 610, and classifier 612 can be used with... Figure 1 The feature extraction model 108, the supplementary model 110, and the output classifier 112 are similar and perform similar operations. In some cases, Figure 6 The combiner 611 of the overall video understanding system 600 can be similar to Figure 5 The overall video understanding system 500 combines the combiner 511 and performs similar operations.

[0110] Input video 602 can be processed by the model decision engine 606. (See reference) Figure 1 As an example, input video 602 or video portions can be retrieved from video retrieval device 102 or storage device 104. Figure 6 The concept of a secondary model decision engine 651 is introduced, which can be positioned between feature extraction models 608A and 608B and supplementary model 610. As shown, the secondary model decision engine 651 can receive features from the first feature extraction model 608A. The secondary model decision engine 651 can be trained to determine which of the supplementary models(s) coupled to the first feature extraction model 608A (e.g., supplementary models 653, 655, and 657) should be used to process the features generated by the first feature extraction model 608A. In some cases, the model decision engine 651 can be trained to determine, based on a desired combination of computational efficiency and accuracy, which of the supplementary models 653, 655, and 657 should be used to further process the features of the first feature extraction model 608A.

[0111] During inference (after the secondary model decision engine 651 has been trained), the secondary model decision engine 651 can receive features from the first feature extraction model 608A based on the decisions of the model decision engine 606. In one example, based on the characteristics of the features received from the first feature extraction model 608A, the secondary model decision engine 651 can determine which one or more of the supplementary models 653, 655, and / or 657 should be used to further process the features from the first feature extraction model 608A. In some examples, the secondary model decision engine 651 can select two or more of the supplementary models 653, 655, and / or 657. In an illustrative example, the secondary model decision engine 651 can select supplementary models 653 and 655. In such an example, each of the selected supplementary models 653 and 655 can receive features extracted by the first feature extraction model 608A from the input video 602, process the received features, and provide the corresponding output to the combiner 611. Combiner 611 can combine the outputs received from supplementary models 653 and 655 (e.g., by averaging, cascading, or otherwise combining the outputs). Combiner 611 can then pass the combined outputs to classifier 612 for classification of the video or video segments.

[0112] In another example where model decision engine 606 selects a second feature extraction model 608B to extract features from input video 602 or a video portion, model decision engine 651 may not perform any action on the input video 602 or the video portion. In such an example, the features generated by the second feature extraction model 608B can be output to a supplementary model 659 for further processing. The supplementary model 659 can then provide its output to a combiner 611. The combiner 611 can then forward its output to a classifier 612 to classify the input video 602 or the video portion.

[0113] Those skilled in the art will understand that the principle of utilizing the secondary model decision engine 651 is not limited to the specific topology of the two feature extraction models 608A and 608B, wherein three supplementary models 653, 655, and 657 are configured to receive input from the first feature extraction model 608A, and a supplementary model 659 is coupled to receive input from the second feature extraction model 608B. Any other suitable configuration can be used in conjunction with the model decision engine 651. For example, in some cases, the secondary model decision engine 651 can be used in any configuration where there are at least two supplementary models that can be used to process the features generated by one of the feature extraction models 608A and 608B. In some examples, multiple secondary model decision engines 651 can be used, wherein more than one of the feature extraction models 608A and 608B is coupled to two or more supplementary models 610.

[0114] Figure 7 This diagram illustrates an alternative configuration of the overall video understanding system 700. As shown, the components of the overall video understanding system 700 include a feature extraction model, a supplementary model 710, a combiner 711, and a classifier 712. The feature extraction model includes a first feature extraction model 708A and a second feature extraction model 708B. The supplementary model 710 includes a first supplementary model 753, a second supplementary model 755, a third supplementary model 757, and a fourth supplementary model 759. Figure 7 The components of the overall video understanding system 700 can be integrated with... Figure 6 The components of the overall video understanding system 600 and / or any other example of the overall video understanding system described herein are similar and perform similar operations. For example, each component of the overall video understanding system 700 may be similar to... Figure 6 The corresponding components are similar and perform similar operations. Figure 7 The overall video understanding system 700 does not include a model decision engine (e.g., model decision engine 606) or a secondary model decision engine (e.g., secondary model decision engine 651), but instead utilizes available feature extraction models 708A and 708B, as well as corresponding supplementary models 753, 755, 757, and 759, to process the input video 702 or video segments.

[0115] The input video 702 can be processed by a first feature extraction model 708A and a second feature extraction model 708B. (Reference) Figure 1 As an example, input video 702 can be retrieved from video retrieval device 102 or storage device 104. Supplementary models 753, 755, and 757 can process features (e.g., one or more feature vectors) output by the first feature extraction model 708A. Supplementary model 759 can process one or more features (e.g., feature vectors) output by the first feature extraction model 708B. Figure 7 In the example, combiner 711 combines (e.g., by averaging, cascading, etc.) all the outputs of models 753, 755, 757, and 759, and provides the combined output to classifier 712. Classifier 712 can classify the input video 702 or video portions based on the output from combiner 711 (e.g., the combined feature vectors).

[0116] exist Figure 7 In this configuration, regardless of the characteristics of the input video, the input video 702 is processed through all available paths (e.g., through all feature extraction models and all supplementary models 710). Compared to applying a single fixed model to the input video 702, Figure 7The overall video understanding system 700 shown can apply multiple feature extraction models (e.g., feature extraction model 708A and feature extraction model 708B) and multiple supplementary models 710 (each of which can be applied as described above). Figure 1 The different inductive biases described are holistic in meaning.

[0117] Table 2 below shows the experimental results comparing two configurations that utilize the four models 2-5 shown in Table 1 above and subsequently described. Table 2 compares the potential performance (measured in mAP) of selecting among models using a model decision engine with a strategy of averaging results from multiple models. For example, the first row of Table 2 shows similar... Figure 7 Experimental results of the overall video understanding system 700 configuration, which simultaneously processes input video through all four models 2-5. Figure 7 The combiner 711 averages the output of four models 2-5. In the experiment, the combined output was determined by... Figure 7 The classifier is 712.

[0118]

[0119] Table 2

[0120] The second row of Table 2 shows the experimental results used to select the best model or subset of models used to process each input video or video segment based on experimental data about which model(s) provided the best accuracy. The values ​​in the second row of Table 2 were obtained by selecting one or more models averaged in row 1 for each input video or video segment. As shown in the second row of Table 2, the average accuracy and accuracy across various categories (e.g., action, concept, event, object, and scene) are improved relative to the average shown in the first row. Table 2 illustrates the use of a model decision engine as described in this disclosure (e.g., ...). Figure 1 The model decision engine (106) in the model can improve accuracy compared to the method of selecting all models and averaging the results.

[0121] Figure 8 The illustration shows the comparison between using a reference model and methods that use only a fixed model and methods that randomly select between processing paths (e.g., each path could represent a pair of feature extraction models and complementary models). Figure 6 Experimental graphs showing the relative performance of the model decision engine 606 in the overall video understanding system 600. The mAP of the experimental results is plotted on the vertical axis, and the computational workload measured in GFLOPs per segment is plotted on the horizontal axis. In comparison... Figure 8In the experiments related to the curves in the paper, MobileNetV2 was used to implement the model decision engine 106. In the experiments, each configuration described in this paper was tested using the same video dataset. A 2D CNN implemented using ResNet-50 was used as the first feature extraction model 608A. The second model decision engine 651 was configured to apply all three complementary models 653, 655, and 657 to the output of the first feature extraction model 608A. For example, whenever the model decision engine 106 selected the first feature extraction model 608A to process a video (or a portion of a video, such as a video clip υ), it would apply the first feature extraction model 608A to the video. ij When processing features generated by feature extraction model 608A, all three of supplementary models 653, 655, and 657 are used. Combiner 611 combines the outputs from the three supplementary models (e.g., by averaging, concatenating, or otherwise combining features from the supplementary models). Combiner 611 provides the combined output to classifier 612 for classification. In experiments, 1D convolutions in the time dimension (e.g., regarding...) Figure 2A and Figure 2B The described model 653 was used as a supplementary model, the non-local time model was used as a supplementary model 655, and graph convolution was used as a supplementary model 657. A 3DCNN implemented using the Slowfast model was used as a second feature extraction model 608B, while an MLP was used as a supplementary model 659. In the experiments, whenever the model decision engine 606 selected the second feature extraction model 608B to process the video (or a portion of the video), the features of the second feature extraction model 608B were input into the supplementary model 659, the combiner 611 did not perform any operation (because only a single result was passed to its input), and the output of the supplementary model 659 was classified by the classifier 612.

[0122] Figure 8Data points 836A and 836G in the graph illustrate the results of the model decision engine 106 selecting between the first feature extraction model 608A and the second feature extraction model 608B. For example, data point 836A represents the result that the model decision engine 606 always selects the first feature extraction model 608A, while data point 836G represents the result that the model decision engine 106 always selects the second feature extraction model 608B. The set of data points 836A, 836B, 836C, 836D, 836E, 836F, and 836G represent the results of random selection between the first feature extraction model 608A and the second feature extraction model 608B according to the proportions [0.0, 0.1, 0.3, 0.5, 0.7, 0.9, 1.0], where 0.0 indicates that the first feature extraction model 608A is always selected, and 1.0 indicates that the second feature extraction model 608B is always selected. As shown in the figure, the mAP of data point 830 exceeds that of data points 836A and 836G, indicating that using model decision engine 106 improves accuracy compared to using only the first feature extraction model 608A or the second feature extraction model 608B. Based on the average improvement in accuracy, Figure 8 The graph in the image shows how the model decision engine 106 selects models based on the visual or temporal features of the video.

[0123] As described above, the holistic video understanding system (e.g., holistic video understanding systems 100, 300, 400, 500, and / or 600) and related techniques described herein allow the system to adaptively select video processing machine learning models to efficiently and accurately classify video content across multiple categories (e.g., actions, attributes, events, objects, and scenes). For example, using holistic video understanding system 100, a first video portion containing a static scene can be analyzed by a 2D CNN to extract spatial features from the first video portion for detecting objects in the scene, and a second video portion containing a sports event can be analyzed by a 3D CNN to extract spatial and temporal features from the second video portion to detect objects and actions associated with those objects. By leveraging model decision engine 106 (from feature extraction model 108) to select a feature extraction model that is suitable for the relevant video portion (e.g., the information type, motion, etc. of the relevant video portion) and has lower computational intensity than other available feature extraction models 108, the computational and power resources of the system (e.g., the computing system) can be saved for other operations. Besides saving system computational and power resources, the inclusion of various supplementary models 110 with different sensing biases can make the overall video understanding system 100 more holistic by providing models specifically designed to identify a wider range of categories. In contrast, a system that includes a fixed model for all input videos can perform well when the analyzed video contains data categories for which a fixed model is optimized, but may suffer from a loss of accuracy and / or unnecessary computational effort when the analyzed video contains information of different categories.

[0124] Figure 9 This is a flowchart illustrating an example of a process 900 using a holistic video understanding system to process video, based on some examples. At block 902, process 900 includes: obtaining a first video. In some examples, process 900 may obtain the first video from a storage device. In one illustrative example, the storage device includes... Figure 1 The storage device 104 and / or shown Figure 12 The storage device 1230 shown.

[0125] At block 904, process 900 includes: determining a first machine learning model (e.g., for processing at least a portion of the first video) from a set of machine learning models using a machine learning model decision engine. The first machine learning model may be determined based on one or more features of at least that portion of the first video. In an illustrative example, the model decision engine includes... Figure 1 The model decision engine 106 is shown. In some examples, this model decision engine includes neural networks (e.g., MobileNetV2 neural networks and / or ResNet-50 neural networks). In one illustrative example, the ensemble of machine learning models includes... Figure 1 The feature extraction model 108 is shown. In another illustrative example, this set of machine learning models includes... Figure 3 Models 313A, 313B to 313M are shown.

[0126] In some examples, the first machine learning model includes a convolutional neural network (CNN), as described above. For example, in some examples, the CNN includes a two-dimensional CNN (e.g., MobileNetV2 or ResNet-50), a three-dimensional CNN (e.g., Slowfast), or other CNNs. In some cases, the first machine learning model may include a first type of CNN that differs from the CNNs in at least some of the other machine learning models included in the set of machine learning models. In some examples, the one or more characteristics of at least that portion of the first video include spatial, temporal, and / or audio characteristics.

[0127] At block 906, process 900 includes: processing at least the portion of the first video using the first machine learning model. As described herein, processing at least the portion of the first video may include: extracting features from at least the portion of the first video, processing the features using one or more supplementary models, and classifying at least the portion of the first video, etc.

[0128] In some examples, process 900 may include: determining a classification of at least that portion of the first video based on processing the first portion using the first machine learning model. In some examples, the class determined during classification may include playing football, playing the violin, listening to music, etc. In an illustrative example, Figure 1 The output classifier 112 shown determines the classification. In some examples, process 900 includes determining a classification of multiple classes that are included in at least that portion of the first video, as described above.

[0129] In some examples, process 900 includes: extracting one or more first features from at least that portion of the first video using a first machine learning model. In some cases, the first machine learning model includes a neural network, such as a two-dimensional (2D) CNN. In this case, the 2D CNN can extract one or more first features from at least that portion of the first video in two spatial dimensions. In some implementations, the first machine learning model includes a 3D CNN. In such implementations, the 3D CNN can extract one or more first features from at least that portion of the video in two spatial dimensions and one temporal dimension.

[0130] Process 900 may include: processing the first one or more features extracted from at least that portion of the first video by the first machine learning model using a first supplementary model. In an illustrative example, the first supplementary model includes features included in... Figure 1 The supplementary model in the supplementary model 110 shown. In some cases, the first supplementary model includes a neural network, such as a one-dimensional (1D) CNN. For example, when the first supplementary model includes a 1D CNN, the first one or more features may include: a first feature vector based on at least that portion of the first video, a first frame, and a second feature vector based on at least that portion of the first video, a second frame. In this case, the 1D CNN can generate temporal information of at least that portion of the first video from at least the first feature vector and the second feature vector. In an illustrative example, the 1D CNN performs... Figure 2D The convolution shown and described above. In some cases, the first supplementary model includes a multilayer perceptron. In some implementations, the first supplementary model includes a graph convolutional network. In some cases, the first supplementary model includes a non-local model. In some examples, process 900 includes determining a first classification of at least that portion of the first video based on processing the first one or more features using the first supplementary model.

[0131] In some examples, process 900 includes: determining a second machine learning model (e.g., for processing at least another portion of the first video) from the set of machine learning models using the machine learning model decision engine. The second machine learning model may be determined based on one or more features of the at least another portion of the first video. In some cases, process 900 includes: processing the at least another portion of the first video using the second machine learning model. In one aspect, the second machine learning model may include a neural network (e.g., a CNN) different from one or more neural networks (e.g., a CNN) of the first machine learning model. In some cases, process 900 includes: extracting one or more second features from the at least another portion of the first video using the second machine learning model. Process 900 may include: processing the second or more features extracted from the at least another portion of the first video using a second supplementary model. In some examples, process 900 may determine a second classification of the at least another portion of the first video based on the one or more features from the second supplementary model.

[0132] In some examples, process 900 may obtain a second video. Process 900 may determine a second machine learning model (e.g., for processing at least a portion of the second video). The second machine learning model may be determined based on one or more features of at least that portion of the second video. In some examples, process 900 may obtain the second video from a storage device. In an illustrative example, the storage device includes... Figure 1 The storage device 104 and / or shown Figure 12 The storage device 1230 shown. Process 900 can use the second machine learning model to process at least that portion of the second video.

[0133] In some examples, process 900 includes: extracting first one or more features from at least that portion of the first video using the first machine learning model, in parallel with determining the first machine learning model from the set of machine learning models for processing at least that portion of the first video. In some cases, the machine learning model decision engine shares a common neural network with the first machine learning model. In one illustrative example, the model decision engine sharing a common neural network with the first machine learning model may include, for example... Figure 4 The model decision engine 406 and feature extraction component 429 are shown.

[0134] In some examples, the processes described herein (e.g., process 900 and / or other processes described herein) may be performed by a computing device or apparatus. In one example, one or more processes may be performed by... Figure 1 The overall video understanding system 100 executes this. In another example, one or more processes can be performed by... Figure 12 The computing system 1200 shown is executed. For example, it has... Figure 12 The computing device of the computing system 1200 shown may include components of the overall video understanding system 100 and can be implemented. Figure 9 The operation of process 900 and / or other processes described herein.

[0135] Computing devices may include any suitable device, such as mobile devices (e.g., mobile phones), desktop computing devices, tablet computing devices, wearable devices (e.g., VR headsets, AR headsets, AR glasses, network-connected watches or smartwatches, or other wearable devices), vehicles or vehicle computing devices (e.g., vehicle driver monitoring systems (DMS)), server computers, robotic devices, televisions, and / or any other computing device with the resource capability to perform the processes described herein (including process 900 and / or other processes described herein). In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0136] Components of a computing device may be implemented in a circuit. For example, a component may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0137] Process 900 is illustrated as a logic flowchart, where operations represent sequences of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which operations are described is not intended to be restrictive, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0138] Additionally, process 900 and / or other processes described herein may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, by hardware, or a combination thereof. As described above, the code may be stored, for example, on a computer-readable or machine-readable storage medium in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0139] As stated above, various aspects of this disclosure can be achieved using machine learning models or systems. Figure 10 This is an illustrative example of a deep learning neural network 1000 that can be used to implement the overall video understanding system described above. Input layer 1020 includes input data. In one illustrative example, input layer 1020 may include data representing pixels of an input video frame. Neural network 1000 includes multiple hidden layers 1022a, 1022b to 1022n. Hidden layers 1022a, 1022b to 1022n comprise "n" hidden layers, where "n" is an integer greater than or equal to one. The number of hidden layers can be as many as required for a given application. Neural network 1000 also includes an output layer 1021, which provides the output produced by the processing performed by hidden layers 1022a, 1022b to 1022n. In one illustrative example, output layer 1021 may provide a classification of objects in the input video frame. The classification may include classes that identify activity types (e.g., playing football, playing the piano, listening to the piano, playing the guitar, etc.).

[0140] Neural Network 1000 is a multi-layer neural network with interconnected nodes. Each node can represent a piece of information. The information associated with a node is shared between different layers, and each layer retains the information as it is processed. In some cases, Neural Network 1000 may include a feedforward network, in which case there are no feedback connections, and the network's output is fed back into itself. In some cases, Neural Network 1000 may include a recurrent neural network, which may have loops that allow information to be carried across nodes as input is read.

[0141] Information can be exchanged between nodes through node-to-node interconnects between layers. Nodes in input layer 1020 can activate the set of nodes in the first hidden layer 1022a. For example, as shown, each input node of input layer 1020 is connected to each node in the first hidden layer 1022a. Nodes in the first hidden layer 1022a can transform the information of each input node by applying an activation function to the input node information. The information derived from the transformation can then be passed to and can activate nodes in the next hidden layer 1022b, which can perform their own specified functions. Example functions include convolution, upsampling, data transformation, and / or any other suitable function. The output of hidden layer 1022b can then activate nodes in the next hidden layer, and so on. Finally, the output of hidden layer 1022n can activate one or more nodes in output layer 1021 that provide the output. In some cases, although a node in neural network 1000 (e.g., node 1026) is shown as having multiple output lines, the node has a single output, and all lines shown as outputs from the node represent the same output value.

[0142] In some cases, each node or the interconnection between nodes can have weights, which are a set of parameters derived from the training of the neural network 1000. Once the neural network 1000 is trained, it can be called a trained neural network, which can be used to classify one or more activities. For example, the interconnection between nodes can represent a piece of information about what the interconnected nodes have learned. The interconnections can have tunable digital weights, which can be tuned (e.g., based on the training dataset), allowing the neural network 1000 to adapt to the input and learn as more and more data is processed.

[0143] The neural network 1000 is pre-trained to process features from the data in the input layer 1020 using different hidden layers 1022a, 1022b to 1022n, in order to provide an output through the output layer 1021. In an example where the neural network 1000 is used to identify an activity being performed by a driver in a frame, the neural network 1000 can be trained using training data that includes frames and labels, as described above. For example, training frames can be input into the network, where each training frame has a label indicating the features in the frame (for a feature extraction machine learning system) or a label indicating the class of activity in each frame. In an example using object classification for illustrative purposes, a training frame could include an image of the number 2, in which case the image label could be [0 0 1 0 0 0 0 0 0 0].

[0144] In some cases, the neural network 1000 can use a training process called backpropagation to adjust the weights of its nodes. As described above, the backpropagation process can include forward pass, loss function, back pass, and weight update. For each training iteration, forward pass, loss function, back pass, and parameter update are performed. This process can be repeated a certain number of iterations for each training image set until the neural network 1000 is trained well enough that the layer weights are accurately tuned.

[0145] For an example of identifying objects within a frame, the forward pass may include passing training frames through a neural network 1000. The weights are initially randomized before training the neural network 1000. As an illustrative example, a frame may include a numerical array representing pixels of an image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and three color components (e.g., red, green, and blue, or luminance and two chromaticity components, etc.).

[0146] As described above, for the first training iteration of the neural network 1000, the output will likely include values ​​due to the weights randomly chosen during initialization, without prioritizing any particular class. For example, if the output is a vector with probabilities that an object belongs to a different class, the probability values ​​for each different class can be equal or at least very similar (e.g., for ten possible classes, each class could have a probability value of 0.1). Using the initial weights, the neural network 1000 cannot determine low-level features and therefore cannot accurately determine what the object's classification might be. A loss function can be used to analyze the error in the output. Any suitable loss function can be defined, such as cross-entropy loss. Another example of a loss function includes mean squared error (MSE), defined as... The loss can be set to equal E. total The value of .

[0147] The loss (or error) on the first training image will be high because the actual values ​​will differ significantly from the predicted output. The goal of training is to minimize the loss so that the predicted output matches the training labels. The Neural Network 1000 performs backpropagation by determining which input (weights) contributes most to the network's loss, and the weights can be adjusted to reduce and eventually minimize the loss. The derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight of a specific layer) can be calculated to determine the weights that contribute most to the network's loss. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. A weight update can be represented as... Where w represents the weight, w iLet represent the initial weights, and η represent the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes larger weight updates, while a lower value indicates smaller weight updates.

[0148] Neural Network 1000 can include any suitable deep network. An example includes a Convolutional Neural Network (CNN), which consists of an input layer and an output layer, with multiple hidden layers between them. The hidden layers of a CNN consist of a series of convolutional, non-linear, pooling (for downsampling), and fully connected layers. Neural Network 1000 can include any other deep network besides CNNs, such as autoencoders, deep belief networks (DBNs), recurrent neural networks (RNNs), etc.

[0149] Figure 11 This is an illustrative example of a Convolutional Neural Network (CNN) 1100. The input layer 1120 of the CNN 1100 includes data representing an image or frame. For example, the data could include a numerical array representing pixels of an image, where each number in the array includes a value from 0 to 255 describing the intensity of the pixel at that location in the array. Using the previous example above, this array could include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or luminance and two chromaticity components, etc.). The image can be processed through a convolutional hidden layer 1122a, an optional non-linear activation layer, a pooling hidden layer 1122b, and a fully connected hidden layer 1122c to obtain an output at the output layer 1124. While in Figure 11 Only one of each hidden layer is shown in the diagram, but those skilled in the art will understand that a CNN 1100 may include multiple convolutional hidden layers, non-linear layers, pooling hidden layers, and / or fully connected layers. As previously described, the output may indicate a single class of an object, or may include probabilities that best describe the class of an object in an image.

[0150] The first layer of CNN 1100 is a convolutional hidden layer 1122a. Convolutional hidden layer 1122a analyzes the image data input to layer 1120. Each node in convolutional hidden layer 1122a is connected to a region of the input image (pixels) called the receptive field. Convolutional hidden layer 1122a can be thought of as one or more filters (each filter corresponding to a different activation or feature map), where each convolutional iteration of the filter is a node or neuron in convolutional hidden layer 1122a. For example, the region of the input image covered by the filter at each convolutional iteration will be the filter's receptive field. In an illustrative example, if the input image consists of a 28×28 array and each filter (and its corresponding receptive field) is a 5×5 array, then there will be 24×24 nodes in convolutional hidden layer 1122a. Each connection between a node and its receptive field learns weights, and in some cases, an overall bias, allowing each node to learn to analyze its specific local receptive field in the input image. Each node in hidden layer 1122a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has a weight (digital) array and the same depth as the input. For the video frame example, the filter will have a depth of 3 (based on the three color components of the input image). An illustrative example of the filter array size is 5×5×3, corresponding to the size of the receptive field of the node.

[0151] The convolutional property of the convolutional hidden layer 1122a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 1122a can start at the top left corner of the input image array and can be convolved around the input image. As described above, each convolutional iteration of the filter can be considered as a node or neuron of the convolutional hidden layer 1122a. At each convolutional iteration, the value of the filter is multiplied by the corresponding number of original pixel values ​​of the image (e.g., a 5×5 filter array multiplied by a 5×5 array of input pixel values ​​at the top left corner of the input image array). The multiplications from each convolutional iteration can be summed to obtain the sum of that iteration or node. Next, the process continues at the next position in the input image based on the receptive field of the next node in the convolutional hidden layer 1122a. For example, the filter can be moved one step (called the stride) to the next receptive field. The stride can be set to 1 or other suitable amounts. For example, if the stride is set to 1, the filter will move 1 pixel to the right at each convolutional iteration. At each unique location of the input convolutional layer, the filter is processed to produce a number representing the filter result at that location, thereby determining a sum value for each node of the convolutional hidden layer 1122a.

[0152] The mapping from the input layer to the convolutional hidden layer 1122a is called an activation map (or feature map). An activation map includes the value of each node representing the filtering result at each location in the input volume. Activation maps can include an array comprising the sums of individual values ​​produced by each iteration of the filter over the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map would comprise a 24×24 array. The convolutional hidden layer 1122a can include several activation maps to identify multiple features in the image. Figure 11 The example shown includes three activation maps. Using three activation maps, the convolutional hidden layer 1122a can detect three different types of features, each of which is detectable across the entire image.

[0153] In some examples, nonlinear hidden layers can be applied after convolutional hidden layers 1122a. Nonlinear layers can be used to introduce nonlinearity into a system that has already computed linear operations. An illustrative example of a nonlinear layer is the Rectified Linear Unit (ReLU) layer. A ReLU layer applies the function f(x) = max(0,x) to all values ​​in the input volume, which changes all negative activations to 0. Therefore, ReLU can add nonlinear properties to CNN 1100 without affecting the receptive field of convolutional hidden layers 1122a.

[0154] A pooling hidden layer 1122b can be applied after the convolutional hidden layer 1122a (and, when used, after the non-linear hidden layer). The pooling hidden layer 1122b is used to simplify the information in the output of the convolutional hidden layer 1122a. For example, the pooling hidden layer 1122b can take each activation map from the output of the convolutional hidden layer 1122a and generate a compressed activation map (or feature map) using a pooling function. Max pooling is an example of a function performed by the pooling hidden layer. The pooling hidden layer 1122a can use other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. The pooling function (e.g., a max pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 1122a. Figure 11 In the example shown, three pooling filters are used for the three activation maps in the convolutional hidden layer 1122a.

[0155] In some examples, max pooling can be used by applying a max pooling filter (e.g., of 2×2 size) with a stride (e.g., equal to the dimension of the filter, such as stride 2) to the activation map output from the convolutional hidden layer 1122a. The output from the max pooling filter includes the maximum number in each sub-region around which the filter convolves. Using a 2×2 filter as an example, each unit in the pooling layer can summarize a region of 2×2 nodes from the previous layer (where each node is a value in the activation map). For example, the four values ​​(nodes) in the activation map will be analyzed by the 2x2 max pooling filter in each iteration of the filter, with the maximum of the four values ​​output as the "max" value. If such a max pooling filter is applied to the activation filter from the convolutional hidden layer 1122a having a dimension of 24×24 nodes, the output from the pooling hidden layer 1122b will be an array of 12×12 nodes.

[0156] In some examples, L2 norm pooling filters can also be used. L2 norm pooling filters involve computing the square root of the sum of squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (instead of computing the maximum value as done in max pooling), and using the computed value as the output.

[0157] Intuitively, pooling functions (e.g., max pooling, L2-norm pooling, or other pooling functions) determine whether a given feature is found anywhere in an image region, discarding precise location information. This can be done without affecting feature detection results, because once a feature is found, its precise location is less important than its approximate location relative to other features. Max pooling (and other pooling methods) offers the benefit of having fewer pooled features, thus reducing the number of parameters required in later layers of the CNN 1100.

[0158] The final connection layer in the network is a fully connected layer that connects each node from the pooling hidden layer 1122b to each output node in the output layer 1124. Using the example above, the input layer comprises 28×28 nodes encoding the pixel intensity of the input image, the convolutional hidden layer 1122a comprises 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for filtering) to three activation maps, and the pooling hidden layer 1122b comprises a layer of 3×12×12 hidden feature nodes based on applying a max-pooling filter to a 2×2 region across each of the three feature maps. Extending this example, the output layer 1124 may comprise ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 1122b is connected to each node of the output layer 1124.

[0159] The fully connected layer 1122c can take the output of the previous pooling hidden layer 1122b (which should represent the activation map of high-level features) and determine the features most relevant to a particular class. For example, the fully connected layer 1122c can determine the high-level features most relevant to a particular class and can include weights (nodes) of the high-level features. A product can be computed between the weights of the fully connected layer 1122c and the pooling hidden layer 1122b to obtain the probabilities for different classes. For example, if CNN 1100 is used to predict whether an object in a video frame is a person, high values ​​will be present in the activation map, which represents the high-level features of a person (e.g., the presence of two legs, the face at the top of the object, two eyes at the top left and top right of the face, the nose in the middle of the face, the mouth at the bottom of the face, and / or other features common to people).

[0160] In some examples, the output from output layer 1124 may include an M-dimensional vector (M = 10 in the previous example). M indicates the number of classes that CNN 1100 must choose from when classifying objects in an image. Other example outputs may also be provided. Each number in the M-dimensional vector can represent the probability that an object belongs to a certain class. In an illustrative example, if the 10-dimensional output vector represents objects of ten different classes as [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates a 5% probability that the image is a third class object (e.g., a dog), an 80% probability that the image is a fourth class object (e.g., a person), and a 15% probability that the image is a sixth class object (e.g., a kangaroo). The probability for a class can be thought of as the confidence that the object is part of that class.

[0161] Figure 12 This is an illustration of an example system used to implement certain aspects of this technology. Specifically, Figure 12 An example of computing system 1200 is shown, which can be any computing device, such as constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1205. Connection 1205 can be a physical connection using a bus, or a direct connection to processor 1210, such as in a chipset architecture. Connection 1205 can also be a virtual connection, a networking connection, or a logical connection.

[0162] In some embodiments, the computing system 1200 is a distributed system, wherein the functions described herein may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a plurality of such components, each performing some or all of the functions described for that component. In some embodiments, a component may be a physical or virtual device.

[0163] Example system 1200 includes at least one processing unit (CPU or processor) 1210 and connections 1205 that couple various system components, including system memories 1215 such as read-only memory (ROM) 1220 and random access memory (RAM) 1225, to processor 1210. Computing system 1200 may include a cache 1212 of high-speed memory that is directly connected to, adjacent to, or integrated as part of processor 1210.

[0164] Processor 1210 may include any general-purpose processor and hardware or software services, such as services 1232, 1234, and 1236 stored in storage device 1230, configured to control processor 1210 and dedicated processors, wherein software instructions are incorporated into the actual processor design. Processor 1210 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0165] To enable user interaction, the computing system 1200 includes an input device 1245, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. The computing system 1200 may also include an output device 1235, which can be one or more of a plurality of output mechanisms. In some instances, a multi-mode system allows the user to provide multiple types of input / output to communicate with the computing system 1200. The computing system 1200 may include a communication interface 1240, which typically monitors and manages user input and system output. This communication interface may use wired and / or wireless transceivers to perform or facilitate the receipt and / or transmission of wired or wireless communications, including transceivers utilizing audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, etc. Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs Wireless signal transmission Low Energy (BLE) wireless signal transmission The transceiver includes transceivers for wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), global microwave access interoperability (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or combinations thereof. The communication interface 1240 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of the computing system 1200 based on the reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There are no restrictions on operation on any particular hardware configuration, and therefore the basic features described herein can be readily replaced with improved hardware or firmware configurations as they are developed.

[0166] Storage device 1230 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as magnetic tape cassettes, flash memory cards, solid-state storage devices, digital multifunction disks, cassette tapes, floppy disks, flexible disks, hard disks, magnetic tapes, magnetic strips / strips, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, optical disc read-only memory (CD-ROM), rewritable optical disc (CD), digital video optical disc (DVD), Blu-ray disc (BDD), holographic disc, another optical medium, secure digital storage (SD) card, micro secure digital storage (microSD) card, etc. Cards, smart cards, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / pico SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase-change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette, and / or combinations thereof.

[0167] Storage device 1230 may include software services, servers, services, etc., which cause the system to perform functions when the code defining such software is executed by processor 1210. In some embodiments, hardware services that perform a particular function may include software components stored in a computer-readable medium that are connected to necessary hardware components such as processor 1210, connection 1205, output device 1235 to perform that function.

[0168] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include, but does not include, non-transitory media in which data can be stored, carrier waves and / or transient electronic signals propagated wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as CDs or DVDs, flash memory, memory, or memory devices. Code and / or machine-executable instructions may be stored on a computer-readable medium, which may represent processes, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments can be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted using any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0169] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0170] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that these embodiments can be practiced without these specific details. For clarity, in some instances, the technology may be presented as comprising individual functional blocks, including functional blocks comprising devices, device components, steps, or routines in a method embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the accompanying drawings and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as block diagrams to avoid obscuring the embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0171] The various embodiments described above can be illustrated as processes or methods depicted as flowcharts, diagrams, data flow graphs, structural diagrams, or block diagrams. Although a flowchart can describe operations as a sequential process, many operations can be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. When an operation of a process is completed, the process terminates, but may include additional steps not shown in the diagram. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0172] The processes and methods according to the examples above can be implemented using computer-executable instructions, which are stored in or available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a specific function or group of functions. Some of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include hard disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0173] Devices implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments for performing necessary tasks (e.g., computer program products) may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mount devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or insert cards. As a further example, such functionality may also be implemented on a circuit board between different chips or different processes that execute in a single device.

[0174] Instructions, media for transmitting the instructions, computing resources for executing them, and other structures for supporting these computing resources are example components for providing the functionality described in this disclosure.

[0175] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, while illustrative embodiments of this application have been described in detail herein, it should be understood that the concepts of the invention can be embodied and employed in various other ways and in various manners, and the appended claims are intended to be construed as including these variations beyond those limited by the prior art. Various features and aspects of the above-described applications can be used alone or in combination. Furthermore, embodiments can be used in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in a different order than that described.

[0176] Those skilled in the art will understand that, without departing from the scope of this specification, the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced by the less than or equal to (“≤”) and greater than or equal to (“≥”) symbols, respectively.

[0177] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0178] The phrase “coupled to” refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0179] The language used to express "at least one of" and / or "one or more of" in a claim, or other language, indicates that one or more members of that set (in any combination) satisfy the claim. For example, the language used to express "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the language used to express "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language used to express "at least one of" and / or "one or more of" does not limit the set to the items listed in the set. For example, the language used to express "at least one of A and B" or "at least one of A or B" can refer to A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0180] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this application.

[0181] The techniques described herein can also be implemented as electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication handsets, or integrated circuit devices having multiple uses, including applications in wireless communication handsets and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as a discrete but interoperable logic device. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product and may include encapsulation material. The computer-readable medium can include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, these technologies may be implemented at least in part by a computer-readable communication medium that carries or transmits program code in the form of instructions or data structures, and that the program code may be accessed, read and / or executed by a computer, such as a propagated signal or wave.

[0182] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0183] The illustrative aspects of this disclosure include:

[0184] Aspect 1: A method for processing video data, the method comprising: obtaining a first video; determining a first machine learning model from a set of machine learning models (e.g., for processing at least a portion of the first video) using a machine learning model decision engine, the first machine learning model being determined based on one or more features of at least a portion of the first video; and processing at least that portion of the first video using the first machine learning model.

[0185] Aspect 2: The method according to aspect 1 further includes: determining the classification of at least the portion of the first video based on processing at least the portion of the first video using the first machine learning model.

[0186] Aspect 3: The method according to any one of Aspect 1 or 2 further includes: extracting a first or more features from at least the portion of the first video using the first machine learning model; processing the first or more features using a first supplementary model; and determining a first classification of at least the portion of the first video based on processing the first or more features using the first supplementary model.

[0187] Aspect 4: The method according to any one of Aspect 3, wherein: the first supplementary model includes a one-dimensional convolutional neural network; the first one or more features include: a first feature vector based on a first frame of at least the portion of the first video and a second feature vector based on a second frame of at least the portion of the first video; and the first supplementary model generates temporal information of at least the portion of the first video from at least the first feature vector and the second feature vector.

[0188] Aspect 5: The method according to any one of Aspects 3 or 4, wherein the first supplementary model includes a multilayer perceptron.

[0189] Aspect 6: The method according to any one of Aspects 3 to 5, wherein the first supplementary model comprises a graph convolutional network.

[0190] Aspect 7: The method according to any one of Aspects 3 to 6, wherein the first supplementary model includes a non-local model.

[0191] Aspect 8: The method according to any one of Aspects 1 to 7 further includes: determining a second machine learning model from the set of machine learning models using the machine learning model decision engine, the second machine learning model being determined based on one or more features of at least another portion of the first video; and processing the at least other portion of the first video using the second machine learning model.

[0192] Aspect 9: The method according to aspect 8 further includes: extracting a second or more features from the at least other portion of the first video using the second machine learning model; processing the second or more features using a second supplementary model; and determining a second classification of the at least other portion of the first video based on processing the second or more features using the second supplementary model.

[0193] Aspect 10: The method according to any one of Aspects 1 to 9, wherein the first machine learning model comprises a two-dimensional CNN.

[0194] Aspect 11: The method according to aspect 10, wherein the 2D CNN extracts a first one or more features from at least that portion of the first video in two spatial dimensions.

[0195] Aspect 12: The method according to any one of Aspects 1 to 11, wherein the first machine learning model comprises a three-dimensional (3D) CNN.

[0196] Aspect 13: The method according to aspect 12, wherein the 3D CNN extracts one or more features from at least that portion of the first video in two spatial dimensions and one temporal dimension.

[0197] Aspect 14: The method according to any one of Aspects 1 to 13 further includes: obtaining a second video; determining a second machine learning model from a set of machine learning models using a machine learning model decision engine, the second machine learning model being determined based on one or more features of at least a portion of the second video; and processing at least that portion of the second video using the second machine learning model.

[0198] Aspect 15: The method according to any one of Aspects 1 to 14 further includes: extracting first or more features from at least the portion of the first video using the first machine learning model, in parallel with determining the first machine learning model from the set of machine learning models for processing at least the portion of the first video. In some cases, the machine learning model decision engine shares a common neural network with the first machine learning model.

[0199] Aspect 16: The method according to any one of aspects 1 to 15, wherein the one or more characteristics of at least that portion of the first video include spatial and temporal characteristics.

[0200] Aspect 17: The method according to any one of aspects 1 to 16, wherein at least one or more characteristics of the first video include audio characteristics.

[0201] Aspect 18: An apparatus for processing video data, comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: acquire a first video; determine a first machine learning model from a set of machine learning models (e.g., for processing at least a portion of the first video) using a machine learning model decision engine, the first machine learning model being determined based on one or more features of at least a portion of the first video; and process at least that portion of the first video using the first machine learning model.

[0202] Aspect 19: The apparatus according to aspect 18, wherein the one or more processors are configured to: determine the classification of at least the portion of the first video based on processing at least the portion of the first video using the first machine learning model.

[0203] Aspect 20: The apparatus according to aspect 19 or 20, wherein the one or more processors are configured to: extract a first one or more features from at least the portion of the first video using a first machine learning model; process the first one or more features using a first supplementary model; and determine a first classification of at least the portion of the first video based on processing the first one or more features using the first supplementary model.

[0204] Aspect 21: The apparatus according to aspect 20, wherein: the first supplementary model includes a one-dimensional convolutional neural network; the first one or more features include: a first feature vector based on a first frame of at least the portion of the first video and a second feature vector based on a second frame of at least the portion of the first video; and the first supplementary model generates temporal information of at least the portion of the first video from at least the first feature vector and the second feature vector.

[0205] Aspect 22: The apparatus according to any one of Aspects 20 or 21, wherein the first supplementary model includes a multilayer sensor.

[0206] Aspect 23: The apparatus according to any one of aspects 20 to 22, wherein the first supplementary model comprises a graph convolutional network.

[0207] Aspect 24: The apparatus according to any one of aspects 20 to 23, wherein the first supplementary model includes a non-local model.

[0208] Aspect 25: An apparatus according to any one of aspects 18 to 24, wherein the one or more processors are configured to: determine a second machine learning model from the set of machine learning models using the machine learning model decision engine, the second machine learning model being determined based on one or more features of at least another portion of the first video; and process the at least another portion of the first video using the second machine learning model.

[0209] Aspect 26: The apparatus according to aspect 25, wherein the one or more processors are configured to: extract a second or more features from the at least other portion of the first video using the second machine learning model; process the second or more features using a second supplementary model; and determine a second classification of the at least other portion of the first video based on the processing of the second or more features using the second supplementary model.

[0210] Aspect 27: The apparatus according to any one of aspects 18 to 26, wherein the first machine learning model comprises a two-dimensional CNN.

[0211] Aspect 28: The apparatus according to aspect 27, wherein the 2D CNN extracts a first one or more features from at least that portion of the first video in two spatial dimensions.

[0212] Aspect 29: The apparatus according to any one of aspects 18 to 28, wherein the first machine learning model comprises a three-dimensional (3D) CNN.

[0213] Aspect 30: The apparatus according to aspect 29, wherein the 3D CNN extracts a first one or more features from at least that portion of the first video in two spatial dimensions and one temporal dimension.

[0214] Aspect 31: An apparatus according to any one of aspects 18 to 30, wherein the one or more processors are configured to: obtain a second video; determine a second machine learning model from a set of machine learning models using a machine learning model decision engine, the second machine learning model being determined based on one or more features of at least a portion of the second video; and process at least that portion of the second video using the second machine learning model.

[0215] Aspect 32: An apparatus according to any one of aspects 18 to 31, wherein the machine learning model decision engine shares a common neural network with the first machine learning model, and wherein the one or more processors are configured to: extract a first one or more features from at least the portion of the first video using the first machine learning model in parallel with determining the first machine learning model from the set of machine learning models for processing at least the portion of the first video.

[0216] Aspect 33: The apparatus according to any one of aspects 18 to 32, wherein the characteristics of at least that portion of the first video include spatial and temporal characteristics.

[0217] Aspect 34: The apparatus according to any one of aspects 18 to 33, wherein one or more characteristics of at least the portion of the first video include audio characteristics.

[0218] Aspect 35: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any of the operations in aspects 1 to 34.

[0219] Aspect 36: An apparatus comprising components for performing any of the operations in aspects 1 to 34.

Claims

1. An apparatus for processing video data, comprising: Memory; as well as One or more processors are coupled to the memory, and the one or more processors are configured to: Get the first video; The machine learning model decision engine extracts a first plurality of features based on at least a portion of the first video; The machine learning model decision engine uses the first plurality of features to determine a first machine learning model from a set of machine learning models, the set of machine learning models including at least the first machine learning model and a second machine learning model, the first machine learning model having a different neural network architecture than the second machine learning model, and the first machine learning model being determined based on one or more characteristics of the first plurality of features; The first machine learning model is used to extract a second plurality of features based on at least said portion of the first video, wherein said second plurality of features include a first feature extracted from a first frame of said portion of the first video and a second feature extracted from a second frame of said portion of the first video, wherein said second frame is different from the first frame; as well as Process the second plurality of features.

2. The apparatus of claim 1, wherein the one or more processors are configured to: The classification of at least said portions of the first video is determined based on processing the second plurality of features.

3. The apparatus of claim 1, wherein the one or more processors are configured to: The second plurality of features are processed using a first supplementary model; and A first classification is determined based on processing the second plurality of features using the first supplementary model to determine at least the portion thereof of the first video.

4. The apparatus according to claim 3, wherein: The first supplementary model includes a one-dimensional convolutional neural network (CNN); The second plurality of features include: a first feature vector based on a first frame of at least said portion of the first video and a second feature vector based on a second frame of at least said portion of the first video; as well as The first supplementary model generates time information for at least the portion of the first video from at least the first feature vector and the second feature vector.

5. The apparatus of claim 3, wherein the first supplementary model comprises a multilayer sensor.

6. The apparatus of claim 3, wherein the first supplementary model comprises a graph convolutional network.

7. The apparatus of claim 3, wherein the first supplementary model comprises a non-local model.

8. The apparatus of claim 1, wherein the one or more processors are configured to: The machine learning model decision engine is used to extract a third plurality of features based on at least another portion of the first video; The machine learning model decision engine is used to determine the second machine learning model from the set of machine learning models based on the third plurality of features, wherein the second machine learning model is determined based on one or more characteristics of the third plurality of features; The second machine learning model is used to extract a fourth plurality of features based on at least another portion of the first video; as well as Process the fourth set of features.

9. The apparatus of claim 8, wherein the one or more processors are configured to: The fourth plurality of features are processed using a supplementary model; and The second classification of at least another portion of the first video is determined based on processing the fourth plurality of features using the supplementary model.

10. The apparatus of claim 1, wherein the first machine learning model comprises a two-dimensional 2D CNN.

11. The apparatus of claim 10, wherein the 2D CNN extracts first or more features from at least said portions of the first video in two spatial dimensions.

12. The apparatus of claim 1, wherein the first machine learning model comprises a three-dimensional 3D CNN.

13. The apparatus of claim 12, wherein the 3D CNN extracts first one or more features from at least said portions of the first video in two spatial dimensions and one temporal dimension.

14. The apparatus of claim 1, wherein the one or more processors are configured to: Obtain the second video; The machine learning model decision engine is used to extract a third set of features based on at least a portion of the second video; The machine learning model decision engine uses the third plurality of features to determine a third machine learning model from the set of machine learning models. The third machine learning model has a different architecture from the first machine learning model and the second machine learning model. The third machine learning model is determined based on one or more characteristics of the third plurality of features. The third machine learning model is used to extract a fourth plurality of features based on at least the portion of the second video; as well as Process the fourth set of features.

15. The apparatus of claim 1, wherein the machine learning model decision engine shares a common neural network architecture with the first machine learning model, and wherein the one or more processors are configured to: In parallel with determining the first machine learning model from the set of machine learning models for processing at least the portion of the first video, the second plurality of features are extracted from at least the portion of the first video using the first machine learning model.

16. The apparatus of claim 1, wherein one or more characteristics of the first plurality of features include spatial and temporal characteristics.

17. The apparatus of claim 1, wherein one or more of the characteristics of the first plurality of features include audio characteristics.

18. A method for processing video data, the method comprising: Get the first video; The machine learning model decision engine extracts a first plurality of features based on at least a portion of the first video; The machine learning model decision engine uses the first plurality of features to determine a first machine learning model from a set of machine learning models, the set of machine learning models including at least the first machine learning model and a second machine learning model, the first machine learning model having a different neural network architecture than the second machine learning model, and the first machine learning model being determined based on one or more characteristics of the first plurality of features; The first machine learning model is used to extract a second plurality of features based on at least said portions of the first video, wherein said second plurality of features include a first feature extracted from a first frame of said portion of the first video and a second feature extracted from a second frame of said portion of the first video; and Process the second plurality of features.

19. The method of claim 18, further comprising: The classification of at least said portions of the first video is determined based on processing the second plurality of features.

20. The method of claim 18, further comprising: The second plurality of features are processed using the first supplementary model; as well as A first classification is determined based on processing the second plurality of features using the first supplementary model to determine at least the portion thereof of the first video.

21. The method of claim 20, wherein: The first supplementary model includes a one-dimensional CNN; The second plurality of features includes: a first feature vector based on a first frame of the first video and a second feature vector based on a second frame of the first video; and The first supplementary model generates time information for at least the portion of the first video from at least the first feature vector and the second feature vector.

22. The method of claim 18, further comprising: The machine learning model decision engine is used to extract a third plurality of features based on at least another portion of the first video; The machine learning model decision engine is used to determine the second machine learning model from the set of machine learning models based on the third plurality of features, wherein the second machine learning model is determined based on one or more characteristics of the third plurality of features; The second machine learning model is used to extract a fourth plurality of features based on at least another portion of the first video; as well as Process the fourth set of features.

23. The method of claim 22, further comprising: The fourth set of features is processed using a supplementary model; as well as The second classification of at least another portion of the first video is determined based on processing the fourth plurality of features using the supplementary model.

24. The method of claim 18, further comprising: In parallel with determining the first machine learning model from the set of machine learning models for processing at least the portion of the first video, the first machine learning model is used to extract the second plurality of features from at least the portion of the first video, wherein the machine learning model decision engine shares a common neural network architecture with the first machine learning model.

25. The method of claim 18, wherein the first machine learning model comprises a 2D CNN.

26. The method of claim 25, wherein the 2D CNN extracts a first or more features from at least said portions of the first video in two spatial dimensions.

27. The method of claim 18, wherein the first machine learning model comprises a 3D CNN.

28. The method of claim 18, further comprising: Obtain the second video; The machine learning model decision engine is used to extract a third set of features based on at least a portion of the second video; The machine learning model decision engine determines a third machine learning model from the set of machine learning models based on the third plurality of features, wherein the third machine learning model is determined based on one or more characteristics of the third plurality of features; The third machine learning model is used to extract a fourth plurality of features based on at least the portion of the second video; as well as Process the fourth set of features.

29. The method of claim 18, wherein the one or more characteristics of the first plurality of features include spatial and temporal characteristics.

30. The method of claim 18, wherein one or more characteristics of the first plurality of features include audio characteristics.

31. An apparatus for processing video data, the apparatus comprising components for performing the method according to any one of claims 18 to 30.

32. A computer-readable medium having program code recorded thereon, wherein the program code is executable by one or more processors to cause the processors to perform the method according to any one of claims 18 to 30.

33. A computer program product comprising computer-readable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 18 to 30.

Images