Dynamic temporal fusion for video recognition

Abstract

Systems and techniques are described herein for performing dynamic temporal fusion of video classification, such as identification, detection, and / or other forms of classification. For example, a computing device may generate frame-level features obtained from a set of input frames via a first network. The computing device may generate, via a first multi-scale temporal feature fusion engine, a first local temporal context feature from a first adjacent subsequence of the set of input frames. The computing device may generate, via a second multi-scale temporal feature fusion engine, a second local temporal context feature from a second adjacent subset of the set of input frames. The computing device may further classify the set of input frames based on the first local temporal context features and the second local temporal context features.

Description

Technical Field

[0001] Aspects of the present disclosure generally relate to video recognition. For example, aspects of the present disclosure relate to systems and techniques for generating video representations that convey different temporal dynamics at different time periods of video recognition. Background Art

[0002] Convolutional neural networks (CNNs) can be used for a variety of recognition tasks. CNNs are network architectures used for deep learning that learn directly from data and are used to find patterns in images to identify objects, categories, or classes. For example, a CNN can be trained to identify the type of vehicle or animal that may be in an image. CNNs can also be used for video recognition. However, in the case of video recognition, temporal modeling is required that is not present in image recognition. Summary of the Invention

[0003] The following presents a simplified summary of one or more aspects disclosed herein. Therefore, the following summary should neither be considered an exhaustive overview of all contemplated aspects nor be considered to identify key or critical elements related to all contemplated aspects or to delineate the scope associated with any particular aspect. Therefore, the sole purpose of the following summary is to present certain concepts related to one or more aspects of the mechanisms disclosed herein in a simplified form prior to the detailed description presented below.

[0004] In some aspects, the technology described herein relates to an apparatus for performing video action classification, the apparatus comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory and configured to: generate frame-level features obtained from a set of input frames via a first network; generate first local temporal context features from a first adjacent subsequence of the set of input frames via a first multi-scale temporal feature fusion engine; generate second local temporal context features from a second adjacent subsequence of the set of input frames via a second multi-scale temporal feature fusion engine; and classify the set of input frames based on the first local temporal context features and the second local temporal context features.

[0005] In some aspects, the technology described herein relates to a method of classifying a video, the method comprising one or more of: generating frame-level features obtained from a set of input frames via a first network; generating first local temporal context features from a first adjacent subsequence of the set of input frames via a first multi-scale temporal feature fusion engine; generating second local temporal context features from a second adjacent subsequence of the set of input frames via a second multi-scale temporal feature fusion engine; and classifying the set of input frames based on the first local temporal context features and the second local temporal context features.

[0006] In some aspects, the technology described herein relates to a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to be configured to: generate frame-level features obtained from a set of input frames via a first network; generate first local temporal context features from a first adjacent subsequence of the set of input frames via a first multi-scale temporal feature fusion engine; generate second local temporal context features from a second adjacent subsequence of the set of input frames via a second multi-scale temporal feature fusion engine; and classify the set of input frames based on the first local temporal context features and the second local temporal context features. One or more of these operations may be performed by the non-transitory computer-readable medium.

[0007] In some aspects, the technology described herein relates to an apparatus for generating virtual content in a distributed system, the apparatus comprising one or more of: a component for generating frame-level features obtained from a set of input frames via a first network; a component for generating first local temporal context features from a first adjacent subsequence of the set of input frames via a first multi-scale temporal feature fusion engine; a component for generating second local temporal context features from a second adjacent subsequence of the set of input frames via a second multi-scale temporal feature fusion engine; and a component for classifying the set of input frames based on these first local temporal context features and these second local temporal context features.

[0008] In some aspects, the technology described herein relates to an apparatus for performing video classification, the apparatus comprising one or more of: a neural network configured to generate frame-level features in consecutive frames from a set of video frames; a first multiscale temporal feature fusion engine having a first kernel size configured to generate first local context features based on the frame-level features; a second multiscale temporal feature fusion engine having a second kernel size configured to generate second local context features based on the frame-level features; a first temporal relationship cross-transformer classifier configured to generate a first distance between a query video associated with the set of video frames and a set of support videos based on the first local context features; a second temporal relationship cross-transformer classifier configured to generate a second distance between the query video associated with the set of video frames and the sets of support videos based on the second local context features; and a computation engine configured to compute a final distance between the query video and the sets of support videos based on the first distance and the second distance.

[0009] In some aspects, the technology described herein relates to a method of performing video classification, the method comprising one or more of: generating frame-level features in consecutive frames from the set of video frames via a neural network configured to receive a set of video frames; generating first local context features based on the frame-level features via a first multi-scale temporal feature fusion engine having a first kernel size; generating second local context features based on the frame-level features via a second multi-scale temporal feature fusion engine having a second kernel size; generating a first distance between a query video associated with the set of video frames and a set of support videos via a first temporal relationship cross-transformer classifier and based on the first local context features; generating a second distance between the query video associated with the set of video frames and the sets of support videos via a second temporal relationship cross-transformer classifier and based on the second local context features; and calculating a final distance between the query video and the sets of support videos based on the first distance and the second distance.

[0010] In some aspects, the technology described herein relates to a non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to be configured to: generate frame-level features in consecutive frames from a set of video frames via a neural network configured to receive the set of video frames; generate first local context features based on the frame-level features via a first multi-scale temporal feature fusion engine having a first kernel size; generate second local context features based on the frame-level features via a second multi-scale temporal feature fusion engine having a second kernel size; generate a first distance between a query video associated with the set of video frames and a set of support videos via a first temporal relation cross-transformer classifier and based on the first local context features; generate a second distance between the query video associated with the set of video frames and the set of support videos via a second temporal relation cross-transformer classifier and based on the second local context features; and calculate a final distance between the query video and the set of support videos based on the first distance and the second distance. One or more of the above operations may be performed.

[0011] In some aspects, the technology described herein relates to an apparatus for generating virtual content in a distributed system, the apparatus comprising one or more of: a component for generating frame-level features in consecutive frames from a set of video frames via a neural network configured to receive a set of video frames; a component for generating first local context features based on the frame-level features via a first multi-scale temporal feature fusion engine having a first kernel size; a component for generating second local context features based on the frame-level features via a second multi-scale temporal feature fusion engine having a second kernel size; a component for generating a first distance between a query video associated with the set of video frames and a set of support videos via a first temporal relationship cross-transformer classifier and based on the first local context features; a component for generating a second distance between the query video associated with the set of video frames and the sets of support videos via a second temporal relationship cross-transformer classifier and based on the second local context features; and a component for calculating a final distance between the query video and the sets of support videos based on the first distance and the second distance.

[0012] In some aspects, one or more of the devices described herein are and / or include and / or are part of: an extended reality (XR) device or system (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a mobile device (e.g., a mobile phone or other mobile device), a wearable device, a wireless communication device, a camera, a personal computer, a laptop computer, a vehicle or a computing device or component of a vehicle, a server computer or server device (e.g., an edge or cloud-based server, a personal computer acting as a server device, a mobile device such as a mobile phone acting as a server device, an XR device acting as a server device, a vehicle acting as a server device, a network router, or other device acting as a server device), another device, or a combination thereof. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the devices described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0013] The features and technical advantages of the examples according to the present disclosure have been outlined quite broadly above so that the detailed description that follows may be better understood. Additional features and advantages will be described below. The concepts and specific examples disclosed may be readily used as a basis for modifying or designing other structures for achieving the same purposes of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein, both in terms of their organization and method of operation, and the associated advantages will be better understood by considering the following description in conjunction with the accompanying drawings. Each of the figures in the accompanying drawings is provided for the purpose of illustration and description and not as a definition of limitations to the claims.

[0014] Although various aspects are described in this disclosure by illustrating some examples, it will be understood by those skilled in the art that such aspects can be implemented in many different arrangements and scenarios. The technology described herein can be implemented using different platform types, devices, systems, shapes, sizes and / or packaging arrangements. For example, some aspects can be implemented via integrated chip implementations or other devices based on non-module components (e.g., end-user devices, vehicles, communication equipment, computing equipment, industrial equipment, retail / shopping equipment, medical equipment and / or artificial intelligence devices). Various aspects can be implemented in chip-level components, modular components, non-modular components, non-chip-level components, device-level components and / or system-level components. The equipment incorporating the various aspects and features described may include additional components and features for implementing and practicing the various aspects claimed and described. The various aspects described herein are intended to be practiced in various devices, components, systems, distributed arrangements and / or end-user devices of various sizes, shapes and compositions.

[0015] Other objects and advantages associated with the various aspects disclosed herein will be apparent to those skilled in the art based on the drawings and detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. This subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0016] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are presented to aid in describing various aspects of the present disclosure and are provided solely for illustration and not limitation of the aspects.

[0018] Figure 1 A multi-scale temporal feature fusion engine for a sequence of frame-level features of a video according to some examples is illustrated;

[0019] Figure 2illustrates the use of multiple multi-scale temporal feature fusion engines in a classifier for a sequence of frame-level features of a video according to some examples;

[0020] Figure 3 is a flowchart illustrating an example of a process for performing multi-scale temporal feature fusion according to some examples;

[0021] Figure 4 is a flow chart illustrating an example of a process for classifying a video using multiple multi-scale temporal feature fusion engines using different kernel sizes according to some examples; and

[0022] Figure 5 is a block diagram illustrating an example of a computing system according to some examples. DETAILED DESCRIPTION

[0023] For illustrative purposes, certain aspects of the present disclosure are provided below. Without departing from the scope of the present disclosure, alternative aspects may be designed. Additionally, well-known elements of the present disclosure will not be described in detail or will be omitted to avoid making the relevant details of the present disclosure difficult to understand. Some aspects described herein can be applied independently, and some of them can be applied in combination, which will be apparent to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it will be apparent that various aspects can be implemented without these specific details. Each drawing and description is not intended to be restrictive.

[0024] The following description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of the present disclosure. Instead, the following description of the exemplary aspects will provide those skilled in the art with a description that can be used to implement the exemplary aspects. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the scope of the present application as set forth in the appended claims.

[0025] Videos fed into convolutional neural networks (CNNs) for video recognition may include temporal dynamics as well as spatial appearance. Two-dimensional CNNs can process video representations, but such CNNs are typically applied to individual frames and are unable to model the temporal information of the video. Two-dimensional CNNs process data by sliding a kernel along two dimensions of the data (such as along image width and image height). Two-dimensional CNNs can use their kernels to extract spatial features (e.g., edges, color distribution, etc.) from the data. Three-dimensional CNNs can jointly learn spatial and temporal features, but the computational cost is very high, making deployment on edge devices (e.g., edge devices that provide access points to a service provider's core network, such as routers, switches, multiplexers, or other devices) difficult. Three-dimensional CNNs can be used with three-dimensional image data such as magnetic resonance imaging (MRI) data or video data. In a three-dimensional CNN, the kernel moves in three directions, and the input and output data of the three-dimensional CNN are four-dimensional. The art needs a new method for designing effective video representations to address these challenges.

[0026] Considering the challenges of using two-dimensional CNNs, spatial average pooling is often applied to aggregate width × height features into one-dimensional features at the final layer of the neural network. Spatial average pooling preserves overall image-level characteristics and also reduces feature complexity.

[0027] This paper describes systems and techniques for generating video representations that transmit different temporal dynamics in different time periods for video recognition. Video processing can make video representations more robust. In some cases, these systems and techniques can include extracting and merging temporal information at different frame rates. In some aspects, these systems and techniques provide effective video representations to enable video recognition to be performed by convolutional neural networks or other networks. In some examples, these systems and techniques introduce a time fusion (also referred to as a time module) engine that can be based on certain features (e.g., average pooling features) of a two-dimensional CNN.

[0028] Video input (e.g., a sequence of images or video frames) may include a sequence of one-dimensional features. In video, different temporal dynamics may be transmitted at different temporal granularities (e.g., different frame rates), which may make the video representation more robust. For example, fine-grained frame-level features may show less temporal dynamics than tuple-level (a set of adjacent frames) features. The systems and techniques described herein may extract and incorporate temporal information to exploit temporal dynamics from features at diverse temporal granularities.

[0029] In some cases, the systems and techniques described herein can improve the classification (e.g., recognition, detection, etc.) of a video based on the support videos. For example, a system can be trained using five categories from five support videos. The example method can be characterized as a five-way five-shot classification. The five support videos can include, for example, one video for vehicles, one video for animals, one video for buildings, one video for plants, and one video for tools. The input query video will be processed as described herein to classify the input query video into one of these categories. A distance value between the query video and the set of support videos can be calculated, and a classification probability can be calculated over the categories, which in some aspects can be associated with the negative value of the distance.

[0030] The systems and techniques can identify actions of interest identified by support videos in a test (query) video. In some cases, these systems and techniques can include a multi-scale temporal feature fusion (MSTFF) module (e.g., as described below with respect to Figure 1 ), where features describing local temporal context in a video are enhanced by collaboratively incorporating important information from frame-level features (e.g., in the absence of temporal context). In some examples, these systems and techniques can classify an input video by utilizing multiple MSTFF modules that vary the scope of local temporal context extraction (e.g., as described below with respect to Figure 2 The system can obtain discriminative video representations that can be used for few-shot tasks where the supporting video is insufficient to describe the action class (e.g., when there is not enough supporting video). For stable learning and performance improvement of models with MSTFF, these systems and techniques may include learning a local temporal context level auxiliary classifier in parallel with the main classifier (e.g., as described below with respect to Figure 2 described).

[0031] Figure 1 An example of an MSTFF module 100 for processing a sequence of frame-level features of a video is illustrated. Action recognition has been widely studied in deep learning, but most methods require large-scale video datasets as noted above. When dealing with fine-grained action categories, collecting a large number of videos labeled by various categories is time-consuming. In addition, the learned deep network may be faced with videos whose action categories have not been seen during training. Therefore, few-shot action recognition aims to recognize fine-grained actions in a test video based on scarce support videos whose action categories have not been seen in training. In other words, few-shot action recognition aims to recognize action categories from an input video using very few training samples or very few support videos used to train the model.

[0032] Since the support video is not sufficient to reliably represent the action category in a few-shot context, it is helpful to extract meaningful temporal information from the video to describe the action of interest. It is also helpful to reliably describe sub-actions in the video if the meaningful clues are included in parts of the video rather than on all frames in the video. However, there is a challenge since the speed as well as the start and end times of the actions vary from video to video. To address this challenge, tuples of frames of the query video and the support video can be matched at multiple cardinalities. To further improve this approach, the spatial context as well as the temporal context in the frame can be considered to generate better subsequence representations. In some cases, a hierarchical matching model can use coarse to fine cues in the spatial domain and the temporal domain, respectively.

[0033] The frame-level features from the backbone network (e.g., a 2D CNN) may already include some information about the spatial context. To exploit this information, the systems and techniques described herein provide a collaborative fusion of two different types of features at different temporal scales: frame-level features in a subsequence of video frames and features from the temporal local context.

[0034] In some aspects, these systems and techniques can be used Figure 1 The MSTFF module 100 shown. The MSTFF module 100 can generate robust temporal local context representations that retain important information in the frame-level features of the input video represented by the frame-level features X102. For example, the MSTFF module 100 can extract local temporal context information from adjacent frame-level features. The cross-attention module 116 can propagate highly correlated frame-level features to features including local temporal context. For example, through the cross-attention module 116, two features at different time scales are combined with high compatibility. Additionally, a local temporal context level auxiliary classifier (such as Figure 2 shown, but Figure 1 (not shown in ), this local temporal context-level auxiliary classifier leads to stable learning of the model and improves the few-shot action recognition performance, which is discarded in the testing phase.

[0035] The MSTFF module 100 illustrates a combination of a one-dimensional convolutional layer 104 (which in some aspects can be temporal in nature) and a cross-attention module 116. The one-dimensional convolutional layer 104 (e.g., with a kernel size k) can summarize information from k consecutive frames to produce an output U 114, which can also be referred to as a key 110. The cross-attention module 116 receives a query 112 (e.g., the value U can represent a coarse-grained feature) as well as a key 110 and a value 108 (e.g., a parameter X can represent a fine-grained feature) and can process the data by providing a weighted sum of the values ​​based on the relationship between the query 112 and the key 110. The described process can mean transferring knowledge of fine-grained features (before the one-dimensional convolutional layer 104) to coarse-grained features (after the one-dimensional convolutional layer 104). Skip connections can also be present to preserve information about the query features. The cross-attention module 116 can transmit information from two different temporal granularities (e.g., for the frame level and the tuple level).

[0036] Due to the challenges of processing videos in a few-shot regime, a lot of effort has been put into this problem. While in some cases memory networks are used to obtain keyframe representations, query and support videos of different lengths can be aligned. In other cases, monotonic temporal ordering is proposed to enhance temporal consistency between video pairs. The temporal relationship between the query and support videos has been modeled using temporal subsequences of multiple lengths. Query-specific action category prototypes are then generated and matched to query video features. The disclosed method can also provide further improvements by using richer video representations that can be obtained by using both spatial and temporal contexts as disclosed herein. Query support matching can also be performed using hierarchical contrastive learning at both the spatial and temporal levels to alleviate the complexity of spatiotemporal matching.

[0037] A video or query v can be represented by a sequence of T uniformly sampled frames. The backbone network (e.g., a convolutional network or Figure 1 Not shown but Figure 2 Any type of network 204 shown in FIG can generate frame-level features X = {x 1 ,…,x T}102, of which

[0038] For the frame-level features X 102, the MSTFF module 100 may apply a one-dimensional convolution layer 104 along the time axis (referred to as temporal one-dimensional convolution) to obtain local temporal context features from adjacent subsequences of frames by:

[0039] U=X⊙w k (1)

[0040] Where ⊙ represents the convolution operation along the time axis, w kRepresents the weights of the one-dimensional convolutional layer 104. k is w k (The kernel size of (k < T). The length of the sequence U is T′, where T′ < T. Then, U = {u 1 , …, u T′}114 represents the local temporal context information.

[0041] To propagate the important frame-level information in X to the local temporal context features, the system can apply the cross-attention module 116 to attend to U 114 (which can be represented as the key 110) via the frame-level feature X 102. This process can be referred to as cross-attention. The feature sequences U 114 and the frame-level feature X 102 can be projected onto the query 112 and the key-value pairs (e.g., the key 110 and the value 108).

[0042] Here, and

[0043] u t The cross-attention feature of 122 is calculated as follows:

[0044]

[0045] where the temperature τ is set to the square root of the dimension of the key to scale the dot product of the query 112 and the key 110. The calculation can be performed by a component or module 120, which can perform, for example, a bn-tanh (batch normalization tanh) operation. Tanh is the hyperbolic tangent function. Batch normalization can be performed before the tanh operation. In the first part of the batch normalization algorithm, the mean and standard deviation of the batch of activations can be calculated. The mean can be subtracted from each activation value, and then each activation value can be divided by the standard deviation of the batch. The expected value of any activation is now zero, which is the central value of the input to the tanh function, and the standard deviation of the activation is one, which means that most activation values will be between [-1, 1].

[0046] Additionally, to obtain the local temporal context from a more diverse perspective, the system also combines 122 with the adjacent frame-level features 118 by simply average pooling the frame-level feature X 102 along the time axis via the average pooling module 106 (e.g., by one-dimensional average pooling), which produces the frame-level feature 118.

[0047] Finally, the final enhanced local temporal context feature 126 is generated via the summation component 124 as follows:

[0048]

[0049] The method synergistically utilizes frame-level context and temporal-level context. The average pooling module 106 can provide information of temporally adjacent frames at a single temporal granularity. The cross-attended features (e.g., query 112, value 108, key 110) and the temporally average pooled features 118 can be summed by the summation component 124 to generate the final feature 126.

[0050] It should be noted that in equation (1), by changing k (kernel), the system can adjust the scope of local temporal context information. Experimentally, considering the diversity of action duration, two different k = {k1, k2} or Figure 2 Kernel values ​​shown.

[0051] Figure 2 The use of multiple multi-scale temporal feature fusion engines for a sequence of frame-level features of a video processed by classifier 200 is illustrated. To obtain an initial representation or T frame-level features 206 for the input query and support videos, the system may uniformly sample T frames 202 per video and employ a network 204. Network 204 may be any number of different networks, such as a backbone network including a two-dimensional convolutional neural network. The ResNet backbone is an example of a "residual network" backbone, which is a categorical neural network used for computer vision tasks. Other neural networks may also be used. In some aspects, network 204 may be a network pre-trained on an image dataset such as the ImageNet dataset. It may be a two-dimensional CNN without any temporal module and, therefore, may only perform frame-level information extraction on the sequence of input frames. In some aspects, network 204 may be a backbone network, such as a high-resolution network HRNet-48, as is known in the art. Network 204 extracts T frame-level features 206 for each video. Then, the corresponding MSTFF modules 208, 210 generate a sequence of T' features describing the local temporal context. The temporal context features 214, 216 obtained from the support video and the query video (combined into a pair of local temporal context features 212) are fed into the final corresponding classifiers 218, 220. Example classification modules include temporal relation matching (TRM) classification modules.

[0052] In some aspects, the last convolutional module of network 204 (e.g., such as HRNet-48) can be replaced with a temporal fusion engine (e.g., MSTFF module 100 or other baseline components).

[0053] During the testing phase of few-shot action recognition, an input test video can be classified into one of C categories, where each category is described by a small number of K support videos, and these categories have not been seen during training. The system (e.g., MSTFF module 100) can set K>1.

[0054] A meta-learning strategy can be used. In this strategy, the training set C train and test set C test The action categories in do not overlap. Then, in order to simulate the few-shot configuration of the support video and the query video that will occur in the test, the system (e.g., MSTFF module 100 or classifier 200) can use scenario training to simulate the few-shot task. For example, at a specific training iteration, train Randomly select N categories, and then K support videos and In some aspects, the system (e.g., MSTFF module) can represent the support set and query set as V s and The N-class classification module 224 may generate an output of the classifier 200 .

[0055] In contextual few-shot learning, auxiliary classifier 222 can be jointly trained to classify input queries into one of the ground-truth training categories, rather than the target N categories for a given context. This approach helps network 204 prevent overfitting and improves few-shot N-class classification performance.

[0056] In an alternative example, the system (e.g., MSTFF module) may include an auxiliary classifier 222 on top of the MSTFF module, which may include two MSTFF modules 208, 210, each using a corresponding kernel value k1, k2. In some aspects, the design of the auxiliary classifier may include a two-layer multilayer perceptron (MLP). Then, each enhanced temporal context feature Each enhanced temporal context feature in 214, 216 may be fed into an auxiliary classifier 222, and the auxiliary classifier 222 is learned to classify the data into |C train | categories. The ground truth labels can be shared by all (t, k) in the same video. From the auxiliary classifier 222 (e.g., a classifier based on local temporal context), each temporal context feature can better represent the action cue. In one aspect, the auxiliary classifier 222 is discarded in the deployed model.

[0057] In some aspects, the auxiliary classifier 222 classifies the input query video into one of the entire training class pool, which helps to stabilize the few-shot learning. The output can be a multi-class classification, such as Figure 2 Class c classification 226 is shown. In one aspect, auxiliary classifier 222 is not deployed in the test.

[0058] Each of the query video and the support videos may pass through the MSTFF modules 208, 210. Since the MSTFF modules 208, 210 have different k (kernel) values ​​as described above, the system may use a different classifier 218, 220 (e.g., a TRM classifier) ​​for each MSTFF module 208, 210. The respective classifiers 218, 220 may output the distance between the query video and the set of N support videos. The system obtains the final distance d between the query video and the support video by accumulating the distances from all k: d = d1 + d2. Then, the classification probability on category n is calculated as follows:

[0059]

[0060] It is proportional to the negative value of the distance. This value is used to finally classify the video into one of the categories.

[0061] In some aspects, the system can optimize the model by using the cross entropy loss for N-class classification as the main loss:

[0062]

[0063] The system can also use an additional cross-entropy loss to learn the auxiliary classifier:

[0064]

[0065] in is the auxiliary true value label and corresponds to the softmax probability Based on the |C train |Output to calculate. |N s |=T′1+T′2 where T′1 and T′2 are the number of final enhanced temporal context features of k1 and k2 respectively. Finally, the total loss is given by:

[0066] Figure 3 is a flow chart illustrating an example of a process 300 for performing multi-scale temporal feature fusion according to some examples disclosed herein. The operations of process 300 may be implemented as a process on one or more processors (e.g., Figure 5Process 300 may be performed by any device or group of devices. The operations of process 300 may be implemented as software components that execute and run on one or more processors (e.g., Figure 5 software components executed and run on the processor 510 and / or other processors).

[0067] At block 302, process 300 includes: one or more processors (e.g., Figure 5 The processor 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or subcomponents thereof, is configured to generate frame-level features obtained from the set of input frames (e.g., via the first network or via the computing system 500). In some aspects, the first network can be a two-dimensional convolutional neural network.

[0068] At block 304, process 300 includes one or more processors (e.g., Figure 5 The processor 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or subcomponents thereof, is configured to generate a first local temporal context feature from a first adjacent subsequence of the input frame set via the first multi-scale temporal feature fusion engine 208. The first multi-scale temporal feature fusion engine may apply a first kernel value to generate the first local temporal context feature, and wherein the second multi-scale temporal feature fusion engine 210 applies a second kernel value to generate the second local temporal context feature. In one aspect, the first adjacent subsequence of the input frame set may be equal to the second adjacent subsequence of the input frame set.

[0069] The process for generating first local temporal context features from a first adjacent subsequence of an input frame set via a first multi-scale temporal feature fusion engine may also include one or more of the following: generating first local temporal context features from the input frame set via a first convolutional neural network; generating a first cross-attention feature output based on the first local temporal context features via a first cross-attention module 116; generating a first average pooling dataset from the input frame set via a first average pooling module 106; and generating the first local temporal context features by adding the first cross-attention feature output to the first average pooling dataset.

[0070] In one aspect, the first convolutional neural network and the second convolutional neural network each perform a one-dimensional convolution using a corresponding kernel. In another aspect, the first convolutional neural network and the second convolutional neural network each perform a one-dimensional convolution using a corresponding kernel to aggregate information from k consecutive frames in the input frame set to generate a first local temporal context feature and a second local temporal context feature.

[0071] In one aspect, the first average pooling module 106 and the second average pooling module 106 each provide information of temporally neighboring frames at a single temporal granularity.

[0072] In one aspect, the first cross-attention module 116 generates a first cross-attended feature output based on the relationship between the query 112 and the keys 110 associated with the set of input frames, and wherein the second cross-attention module 116 generates a second cross-attended feature output based on the relationship between the query 112 and the keys 110 associated with the set of input frames.

[0073] At block 306, the process 300 may include one or more processors configured to: generate second local temporal context features from a second adjacent subsequence of the set of input frames via the second multi-scale temporal feature fusion engine 210. Generating the second local temporal context features from the second adjacent subsequence of the set of input frames via the second multi-scale temporal feature fusion engine 210 may further include: generating the second local temporal context features from the set of input frames via a second convolutional neural network; generating a second cross-attended feature output based on the first local temporal context features via the second cross-attention module 116; generating a second average pooling dataset 118 from the set of input frames via the second average pooling module 106; and generating the second local temporal context features by adding the second cross-attended feature output to the second average pooling dataset 118.

[0074] In one aspect, both the first cross-attention module 116 and the second cross-attention module 116 transfer data associated with fine-grained features before the one-dimensional convolution on the set of input frames to data associated with coarse-grained features after the one-dimensional convolution on the set of input frames. In another aspect, the first cross-attention module 116 and the second cross-attention module 116 each transfer information from two different time granularities.

[0075] In one aspect, the two different temporal granularities include a frame-level granularity and a tuple-level granularity.

[0076] At block 308, process 300 may include configuring one or more processors to classify the set of input frames based on the first local temporal context features and the second local temporal context features. The steps in block 308 may also include classifying the first local temporal context features and the second local temporal context features via an auxiliary classifier 222 during a training process. In one aspect, the auxiliary classifier 222 may include a two-layer multilayer perceptron (MLP).

[0077] An example apparatus for performing video action classification (e.g., recognition, detection, etc.) may include at least one memory and at least one processor coupled to the at least one memory. The at least one processor may be configured to: generate frame-level features obtained from a set of input frames via a first network; generate first local temporal context features from a first adjacent subsequence of the set of input frames via a first multi-scale temporal feature fusion engine 208; generate second local temporal context features from a second adjacent subsequence of the set of input frames via a second multi-scale temporal feature fusion engine 210; and classify the set of input frames based on these first local temporal context features and these second local temporal context features. Video action classification may involve recognition, detection, or other operations related to video action.

[0078] Another example apparatus for performing video classification may include a network 204 (e.g., a neural network) configured to receive a set of video frames and generate frame-level features in consecutive frames from the set of video frames; a first multiscale temporal feature fusion engine 208 having a first kernel size, configured to receive frame-level features and generate first local context features; a second multiscale temporal feature fusion engine 210 having a second kernel size, configured to receive frame-level features and generate second local context features; a first temporal relationship cross-transformer classifier 218 configured to receive the first local context features and generate a first distance between a query video associated with the set of video frames and a set of support videos; a second temporal relationship cross-transformer classifier 220 configured to receive the second local context features and generate a second distance between the query video associated with the set of video frames and a set of support videos; and a computation engine configured to compute a final distance between the query video and the sets of support videos based on the first distance and the second distance.

[0079] The use of the first temporally dependent cross-transformer classifier 218 and the second temporally dependent cross-transformer classifier 220 is by way of example only, as other classifiers may also be used.

[0080] Figure 4 is a flow chart illustrating an example of a process 400 for classifying a video using multiple multi-scale temporal feature fusion engines that use different kernel sizes. The process 400 may be performed by any device or group of devices. The operations of the process 400 may be implemented as a processor on one or more processors (e.g., Figure 5 A software component that is executed and runs on the processor 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or their subcomponents).

[0081] At block 402, the process 400 of performing video classification may include: one or more processors (e.g., Figure 5 The processor 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or subcomponents thereof, is configured to: generate frame-level features in consecutive frames from a set of video frames via a neural network configured to receive a set of video frames.

[0082] At block 404, process 400 may include: one or more processors (e.g., Figure 5 The processor 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or subcomponents thereof) is configured to generate first local context features based on frame-level features via a first multi-scale temporal feature fusion engine 208 with a first kernel size.

[0083] At block 406, process 400 may include: one or more processors (e.g., Figure 5 The processor 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or their subcomponents) is configured to: generate second local context features based on frame-level features via a second multi-scale temporal feature fusion engine 210 with a second kernel size.

[0084] At block 408, process 400 may include: one or more processors (e.g., Figure 5 The processor 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or subcomponents thereof) is configured to: generate a first distance between a query video and a support video set associated with a set of video frames via a first temporal relation cross transformer classifier 218 and based on a first local context feature.

[0085] At block 410, process 400 may include: one or more processors (e.g., Figure 5 The processor 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or subcomponents thereof) is configured to: generate a second distance between the query video and the support video set associated with the video frame set via the second temporal relationship cross transformer classifier 220 and based on the second local context features.

[0086] At block 412, process 400 may include: one or more processors (e.g., Figure 5 The processor 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or subcomponents thereof) is configured to calculate a final distance between the query video and the support video set based on the first distance and the second distance.

[0087] At block 414, process 400 may include: one or more processors (e.g., Figure 5 The processor 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or subcomponents thereof) is configured to perform optimization by calculating a main loss based on the first temporal relation cross transformer classifier 218 and the second temporal relation cross transformer classifier 220 and calculating an auxiliary loss based on the auxiliary classifier 220 used during the training process.

[0088] At block 416, process 400 may include: one or more processors (e.g., Figure 5 The processor 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or their subcomponents) is configured to output a c-class classification 226 via an auxiliary classifier 222 configured to receive the first local context feature and the second local context feature. The auxiliary classifier 222 may not be used after training.

[0089] In some aspects, a system may include a non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform any one or more of the operations disclosed herein. In another aspect, an apparatus for generating a video content classification may include one or more components for performing any one or more of the operations disclosed herein.

[0090] In some aspects, the processes described herein (e.g., process 300, process 400, and / or other processes described herein) may be performed by a computing device or apparatus (e.g., a network server, a client device, or any other device, etc.). Figure 5 510 and / or other processors, the MSTFF module 100 and / or the classifier 200, or their subcomponents). For example, as noted above, the processes 300 and 400 may be performed by a computer system. In another example, the processes 300 and / or 400 may be performed by a computer system having Figure 5 For example, a computing device having the computing system 500 shown in FIG. Figure 5 The wireless communication device of the computing architecture shown may include components of a computer system and may implement Figure 3 and / or Figure 4 operation.

[0091] In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, one or more network interfaces configured to communicate and / or receive data, any combination thereof, and / or other components. The one or more network interfaces may be configured to communicate and / or receive wired and / or wireless data, including data according to 3G, 4G, 5G, and / or other cellular standards, data according to WiFi (802.11x), data according to Bluetooth, and / or other standards. TM Standard data, data according to the Internet Protocol (IP) standard and / or other types of data.

[0092] Components of a computing device may be implemented in circuitry. For example, a component may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0093] Process 300 and process 400 are illustrated as logical flow diagrams, the operations of which represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0094] Additionally, process 300, process 400, and / or other processes described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed on one or more processors, through hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that may be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0095] Figure 5 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Figure 5 An example of a computing system 500 is illustrated, which can be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using a connection 505. Connection 505 can be a physical connection using a bus, or a direct connection into processor 510, such as in a chipset architecture. Connection 505 can also be a virtual connection, a networked connection, or a logical connection.

[0096] In some aspects, computing system 500 is a distributed system in which the functionality described herein can be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a plurality of such components that each perform some or all of the functionality for which the component is described. In some aspects, a component can be a physical or virtual device.

[0097] Example system 500 includes at least one processing unit (CPU or processor) 510 and connections 505 that communicatively couple various system components including system memory or cache 515, such as read-only memory (ROM) 520 and random access memory (RAM) 525, to processor 510. Computing system 500 may include cache 515 of high-speed memory directly connected to, in close proximity to, or integrated as part of processor 510.

[0098] Processor 510 may include any general-purpose processor and hardware or software services, such as services 532, 534, and 536 stored in storage device 530, configured to control processor 510 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 510 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0099] To enable user interaction, the computing system 500 includes an input device 545, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The computing system 500 can also include an output device 535, which can be one or more of a plurality of output mechanisms. In some cases, a multimodal system can enable a user to provide multiple types of input / output to communicate with the computing system 500.

[0100] The computing system 500 may include a communication interface 540, which generally governs and manages user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using wired and / or wireless transceivers, including using audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple TM Lightning TM Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs, 3G, 4G, 5G and / or other cellular data network wireless signal transmission, Bluetooth TM Wireless signal transmission, Bluetooth TM Low energy (BLE) wireless signal transmission, IBEACON TM The communication interface 540 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 500 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and thus the base features herein may be readily substituted for improved hardware or firmware arrangements as they are developed.

[0101] The storage device 530 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic stripe / magnetic stripe, any other magnetic storage medium, flash memory, a memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASH EPROM), a cache memory (e.g., a level 1 (L1) cache, a level 2 (L2) cache, a level 3 (L3) cache, a level 4 (L4) cache, a level 5 (L5) cache, other (L#) cache), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0102] The storage device 530 may include software services, servers, services, etc., which, when the code defining such software is executed by the processor 510, causes the system to perform a function. In some aspects, a hardware service that performs a particular function may include a software component for performing the function stored in a computer-readable medium connected to the necessary hardware components (such as the processor 510, the connection 505, the output device 535, etc.). The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. A computer-readable medium may include a non-transitory medium in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, a disk or tape, an optical storage medium (such as a compact disc (CD) or a digital versatile disc (DVD)), a flash memory, a memory, or a memory device. Computer-readable media can store thereon code and / or machine-executable instructions that can represent a process, function, subroutine, program, routine, subroutine, module, software package, category, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. can be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0103] This disclosure describes a method for few-shot action recognition. Figure 1 The MSTFF module 100 and Figure 2 The local temporal context feature level auxiliary classifier 222 is shown. Using the MSTFF module 100 effectively obtains richer video descriptors by synergistically combining frame-level features and local temporal context level features. Figure 2 The proposed (optional) auxiliary classifier 222 shown can prevent the model from overfitting to every training scenario and then increase the few-shot action recognition performance at test time.

[0104] Specific details are provided in the description above to provide a thorough understanding of the various aspects and examples provided herein, but those skilled in the art will recognize that the application is not limited thereto. Therefore, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including such variations unless limited by the prior art. The various features and aspects of the application described above can be used individually or in combination. In addition, without departing from the broader scope of the specification, the various aspects can be utilized in any number of environments and applications beyond those described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For illustrative purposes, each method is described in a specific order. It should be understood that, in alternative aspects, each method can be performed in a different order than described.

[0105] For clarity of explanation, in some instances, the present technology can be presented as including separate functional blocks, which include devices, device components, steps or routines in the method embodied in software or a combination of hardware and software. Additional components other than those components shown in the drawings and / or described herein can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other cases, well-known circuits, processes, algorithms, structures and techniques can be shown without unnecessary details to avoid confusing various aspects.

[0106] In addition, it will be appreciated by those skilled in the art that the various exemplary logic blocks, modules, circuits, and algorithmic steps described in conjunction with the various aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. Those skilled in the art can implement the described functions in different ways for each specific application, but such specific implementation decisions should not be interpreted as resulting in departure from the scope of this disclosure.

[0107] Various aspects may be described above as processes or methods, which may be depicted as flow charts, flowcharts, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe operations as a sequential process, many of the operations may be performed in parallel or concurrently. Furthermore, the order of the operations may be rearranged. A process is terminated when its operations are completed, but a process may have additional steps not included in the accompanying figures. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, termination of the process may correspond to the function returning to the calling function or main function.

[0108] The processes and methods according to the examples described above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, and the like.

[0109] In some aspects, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0110] Those skilled in the art will understand that information and signals can be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may, in some cases, be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or light particles, or any combination thereof, depending in part on the specific application, in part on the desired design, in part on the corresponding technology, etc.

[0111] The various illustrative logic blocks, modules, and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Examples of form factors include: a laptop computer, a smartphone, a mobile phone, a tablet device, or other small form factor personal computer, a personal digital assistant, a rack-mounted device, a stand-alone device, and the like. The functionality described herein may also be embodied in a peripheral device or add-in card. By way of further example, such functionality may also be implemented on circuit boards in different chips or different processes executed on a single device.

[0112] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0113] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium comprising program code, which includes instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, or the like. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0114] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Thus, the term "processor," as used herein, may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0115] One of ordinary skill in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the description.

[0116] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0117] The phrases “coupled to” or “communicatively coupled to” refer to any component being physically connected directly or indirectly to another component, and / or any component being in communication, directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0118] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or any repetition is information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0119] Illustrative aspects of the present disclosure include:

[0120] Aspect 1. A device for performing video action classification, the device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory and configured to: generate frame-level features obtained from an input frame set via a first network; generate first local temporal context features from a first adjacent subsequence of the input frame set via a first multi-scale temporal feature fusion engine; generate second local temporal context features from a second adjacent subsequence of the input frame set via a second multi-scale temporal feature fusion engine; and classify the input frame set based on the first local temporal context features and the second local temporal context features.

[0121] Aspect 2. An apparatus according to Aspect 1, wherein the first multi-scale temporal feature fusion engine applies a first kernel value to generate the first local temporal context feature, and wherein the second multi-scale temporal feature fusion engine applies a second kernel value to generate the second local temporal context feature.

[0122] Aspect 3. The apparatus according to aspect 1, wherein at least one processor is further configured to: classify the first local temporal context feature and the second local temporal context feature via an auxiliary classifier during a training process.

[0123] Aspect 4. The apparatus according to aspect 3, wherein the auxiliary classifier comprises a two-layer multi-layer perceptron (MLP).

[0124] Aspect 5. An apparatus according to any one of Aspects 1 to 5, wherein the first network comprises a two-dimensional convolutional neural network.

[0125] Aspect 6. An apparatus according to any one of Aspects 1 to 5, wherein at least one processor is further configured to: generate the first local temporal context feature from the first adjacent subsequence of the input frame set via the first multi-scale temporal feature fusion engine by the following operations: generate the first local temporal context feature from the input frame set via a first convolutional neural network; generate a first cross-attention feature output based on the first local temporal context feature via a first cross-attention module; generate a first average pooling dataset from the input frame set via a first average pooling module; and generate the first local temporal context feature by adding the first cross-attention feature output to the first average pooling dataset.

[0126] Aspect 7. An apparatus according to Aspect 6, wherein at least one processor is further configured to: generate the second local temporal context features from the second adjacent subsequence of the input frame set via the second multi-scale temporal feature fusion engine by the following operations: generate the second local temporal context features from the input frame set via a second convolutional neural network; generate a second cross-attention feature output based on the first local temporal context features via a second cross-attention module; generate a second average pooling dataset from the input frame set via a second average pooling module; and generate the second local temporal context features by adding the second cross-attention feature output to the second average pooling dataset.

[0127] Aspect 8. An apparatus according to any one of aspects 6 or 7, wherein the first contiguous subsequence of the input frame set is equal to the second contiguous subsequence of the input frame set.

[0128] Aspect 9. An apparatus according to Aspect 7, wherein the first cross-attention module generates the first cross-attention feature output based on the relationship between the query 112 and the key 110 associated with the input frame set, and wherein the second cross-attention module generates the second cross-attention feature output based on the relationship between the query 112 and the key 110 associated with the input frame set.

[0129] Aspect 10. An apparatus according to Aspect 9, wherein both the first cross-attention module and the second cross-attention module transfer data associated with fine-grained features before the one-dimensional convolution on the set of input frames to data associated with coarse-grained features after the one-dimensional convolution on the set of input frames.

[0130] Aspect 11. The apparatus according to aspect 10, wherein the first cross-attention module and the second cross-attention module each transmit information from two different time granularities.

[0131] Aspect 12. The apparatus according to aspect 11, wherein the two different time granularities include frame-level granularity and tuple-level granularity.

[0132] Aspect 13. The apparatus according to any one of aspects 7 to 12, wherein the first average pooling module and the second average pooling module each provide information of temporally adjacent frames at a single temporal granularity.

[0133] Aspect 14. An apparatus according to any one of Aspects 7 to 12, wherein the first convolutional neural network and the second convolutional neural network each perform a one-dimensional convolution using a corresponding kernel.

[0134] Aspect 15. An apparatus according to Aspect 14, wherein the first convolutional neural network and the second convolutional neural network each utilize the corresponding kernel to perform the one-dimensional convolution to summarize information in k consecutive frames in the input frame set to generate the first local temporal context feature and the second local temporal context feature.

[0135] Aspect 16. A method for classifying a video, the method comprising: generating frame-level features obtained from an input frame set via a first network; generating a first local temporal context feature from a first adjacent subsequence of the input frame set via a first multi-scale temporal feature fusion engine; generating a second local temporal context feature from a second adjacent subsequence of the input frame set via a second multi-scale temporal feature fusion engine; and classifying the input frame set based on the first local temporal context feature and the second local temporal context feature.

[0136] Aspect 17. A method according to Aspect 16, wherein the first multi-scale temporal feature fusion engine applies a first kernel value to generate the first local temporal context feature, and wherein the second multi-scale temporal feature fusion engine applies a second kernel value to generate the second local temporal context feature.

[0137] Aspect 18. The method according to aspect 16, wherein the method comprises: classifying the first local temporal context feature and the second local temporal context feature via an auxiliary classifier during a training process.

[0138] Aspect 19. The method according to aspect 18, wherein the auxiliary classifier comprises a two-layer multi-layer perceptron (MLP).

[0139] Aspect 20. A method according to any one of aspects 16 to 19, wherein the first network comprises a two-dimensional convolutional neural network.

[0140] Aspect 21. A method according to any one of Aspects 16 to 20, wherein generating the first local temporal context features from the first adjacent subsequence of the input frame set via the first multi-scale temporal feature fusion engine also includes: generating the first local temporal context features from the input frame set via a first convolutional neural network; generating a first cross-attention feature output based on the first local temporal context features via a first cross-attention module; generating a first average pooling dataset from the input frame set via a first average pooling module; and generating the first local temporal context features by adding the first cross-attention feature output to the first average pooling dataset.

[0141] Aspect 22. The method according to Aspect 21, wherein generating the second local temporal context features from the second adjacent subsequence of the input frame set via the second multi-scale temporal feature fusion engine further comprises: generating the second local temporal context features from the input frame set via a second convolutional neural network; generating a second cross-attention feature output based on the first local temporal context features via a second cross-attention module; generating a second average pooling dataset from the input frame set via a second average pooling module; and generating the second local temporal context features by adding the second cross-attention feature output to the second average pooling dataset.

[0142] Aspect 23. The method according to any one of aspects 20 to 22, wherein the first contiguous subsequence of the set of input frames is equal to the second contiguous subsequence of the set of input frames.

[0143] Aspect 24. A method according to Aspect 22, wherein the first cross-attention module generates the first cross-attention feature output based on the relationship between the query and the key associated with the set of input frames, and wherein the second cross-attention module generates the second cross-attention feature output based on the relationship between the query 112 and the key 110 associated with the set of input frames.

[0144] Aspect 25. A method according to Aspect 24, wherein both the first cross-attention module and the second cross-attention module transfer data associated with fine-grained features before the one-dimensional convolution on the set of input frames to data associated with coarse-grained features after the one-dimensional convolution on the set of input frames.

[0145] Aspect 26. The method according to aspect 25, wherein the first cross-attention module and the second cross-attention module each transmit information from two different time granularities.

[0146] Aspect 27. The method according to aspect 26, wherein the two different time granularities include frame-level granularity and tuple-level granularity.

[0147] Aspect 28. The method according to any one of aspects 22 to 27, wherein the first average pooling module and the second average pooling module each provide information of temporally neighboring frames at a single temporal granularity.

[0148] Aspect 29. A method according to any one of Aspects 22 to 27, wherein the first convolutional neural network and the second convolutional neural network each perform a one-dimensional convolution using a corresponding kernel.

[0149] Aspect 30. A method according to Aspect 29, wherein the first convolutional neural network and the second convolutional neural network each utilize the corresponding kernel to perform the one-dimensional convolution to summarize information in k consecutive frames in the input frame set to generate the first local temporal context feature and the second local temporal context feature.

[0150] Aspect 31. An apparatus for performing video classification, the apparatus comprising: a neural network configured to generate frame-level features in consecutive frames from a set of video frames; a first multi-scale temporal feature fusion engine having a first kernel size and configured to generate first local context features based on the frame-level features; a second multi-scale temporal feature fusion engine having a second kernel size and configured to generate second local context features based on the frame-level features; a first temporal relationship cross-transformer classifier configured to generate a first distance between a query video associated with the set of video frames and a set of support videos based on the first local context features; a second temporal relationship cross-transformer classifier configured to generate a second distance between a query video associated with the set of video frames and the set of support videos based on the second local context features; and a computing engine configured to calculate a final distance between the query video and the set of support videos based on the first distance and the second distance.

[0151] Aspect 32. An apparatus according to aspect 31, wherein the apparatus is optimized by calculating a main loss based on the first temporal relationship cross-transformer classifier and the second temporal relationship cross-transformer classifier and calculating an auxiliary loss based on an auxiliary classifier used during the training process.

[0152] Aspect 33. The apparatus according to aspect 31, further comprising: an auxiliary classifier, the auxiliary classifier being configured to receive the first local context feature and the second local context feature and output a multi-class classification.

[0153] Aspect 34. The apparatus according to aspect 33, wherein the auxiliary classifier is not used after training the apparatus.

[0154] Aspect 35. A method for performing video classification, the method comprising: generating frame-level features in consecutive frames from the video frame set via a neural network configured to receive a video frame set; generating first local context features based on the frame-level features via a first multi-scale temporal feature fusion engine having a first kernel size; generating second local context features based on the frame-level features via a second multi-scale temporal feature fusion engine having a second kernel size; generating a first distance between a query video associated with the video frame set and a support video set via a first temporal relationship cross-transformer classifier and based on the first local context features; generating a second distance between the query video associated with the video frame set and the support video set via a second temporal relationship cross-transformer classifier and based on the second local context features; and calculating a final distance between the query video and the support video set based on the first distance and the second distance.

[0155] Aspect 36. The method according to Aspect 35 further includes: performing optimization by calculating a main loss based on the first temporal relationship cross-transformer classifier and the second temporal relationship cross-transformer classifier and calculating an auxiliary loss based on an auxiliary classifier used during the training process.

[0156] Aspect 37. The method according to aspect 35, further comprising: outputting a multi-class classification via an auxiliary classifier configured to receive the first local context feature and the second local context feature.

[0157] Aspect 38. The method according to aspect 37, wherein the auxiliary classifier is not used after training.

[0158] Aspect 39. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of any one of aspects 16 to 30 or 35 to 38.

[0159] Aspect 40. An apparatus for generating a classification of video content, the apparatus comprising: one or more means for performing the operations of any one of aspects 16 to 30 or 35 to 38.

Claims

1. An apparatus for performing video action classification, the apparatus comprising: at least one memory; and at least one processor coupled to at least one memory and configured to: generating frame-level features obtained from a set of input frames via a first network; generating, via a first multi-scale temporal feature fusion engine, a first local temporal context feature from a first adjacent subsequence of the set of input frames; generating, via a second multi-scale temporal feature fusion engine, a second local temporal context feature from a second adjacent subsequence of the set of input frames; as well as The set of input frames is classified based on the first local temporal context features and the second local temporal context features.

2. The apparatus according to claim 1, wherein the first multi-scale temporal feature fusion engine applies a first kernel value to generate the first local temporal context feature, and wherein the second multi-scale temporal feature fusion engine applies a second kernel value to generate the second local temporal context feature.

3. The apparatus of claim 1 , wherein the at least one processor is further configured to: The first local temporal context features and the second local temporal context features are classified via an auxiliary classifier during a training process. The apparatus of claim 3 , wherein the auxiliary classifier comprises a two-layer multi-layer perceptron (MLP).

5. The apparatus of claim 1, wherein the first network comprises a two-dimensional convolutional neural network.

6. The apparatus of claim 1 , wherein the at least one processor is further configured to: generate the first local temporal context features from the first adjacent subsequence of the input frame set via the first multi-scale temporal feature fusion engine by: generating a first local temporal context feature from the set of input frames via a first convolutional neural network; generating, via a first cross-attention module, a first cross-attention feature output based on the first local temporal context features; generating a first average pooling dataset from the set of input frames via a first average pooling module; as well as The first local temporal context feature is generated by adding the first cross-attended feature output to the first average pooling dataset.

7. The apparatus of claim 6, wherein the at least one processor is further configured to: generate the second local temporal context features from the second adjacent subsequence of the input frame set via the second multi-scale temporal feature fusion engine by: generating second local temporal context features from the set of input frames via a second convolutional neural network; generating, via a second cross-attention module, a second cross-attention feature output based on the first local temporal context features; generating a second average pooling dataset from the set of input frames via a second average pooling module; as well as The second local temporal context features are generated by adding the second cross-attended feature output to the second average pooling dataset.

8. The apparatus of claim 6, wherein the first contiguous subsequence of the set of input frames is equal to the second contiguous subsequence of the set of input frames.

9. An apparatus according to claim 7, wherein the first cross-attention module generates the first cross-attention feature output based on the relationship between the query and the keys associated with the set of input frames, and wherein the second cross-attention module generates the second cross-attention feature output based on the relationship between the query and the keys associated with the set of input frames.

10. An apparatus according to claim 9, wherein both the first cross-attention module and the second cross-attention module transfer data associated with fine-grained features before the one-dimensional convolution on the set of input frames to data associated with coarse-grained features after the one-dimensional convolution on the set of input frames.

11. The apparatus of claim 10, wherein the first cross-attention module and the second cross-attention module each transmit information from two different time granularities.

12. The apparatus of claim 11, wherein the two different temporal granularities include a frame-level granularity and a tuple-level granularity.

13. The apparatus of claim 7, wherein the first average pooling module and the second average pooling module each provide information of temporally adjacent frames at a single temporal granularity.

14. The apparatus of claim 7, wherein the first convolutional neural network and the second convolutional neural network each perform one-dimensional convolution using a corresponding kernel.

15. The apparatus of claim 14, wherein the first convolutional neural network and the second convolutional neural network each perform the one-dimensional convolution using the corresponding kernel to aggregate information from k consecutive frames in the set of input frames to generate the first local temporal context features and the second local temporal context features.

16. A method for classifying a video, the method comprising: generating frame-level features obtained from a set of input frames via a first network; generating, via a first multi-scale temporal feature fusion engine, a first local temporal context feature from a first adjacent subsequence of the set of input frames; generating, via a second multi-scale temporal feature fusion engine, a second local temporal context feature from a second adjacent subsequence of the set of input frames; as well as The set of input frames is classified based on the first local temporal context features and the second local temporal context features.

17. The method of claim 16, wherein the first multi-scale temporal feature fusion engine applies a first kernel value to generate the first local temporal context feature, and wherein the second multi-scale temporal feature fusion engine applies a second kernel value to generate the second local temporal context feature.

18. The method according to claim 16, wherein the method further comprises: The first local temporal context features and the second local temporal context features are classified via an auxiliary classifier during a training process.

19. The method of claim 18, wherein the auxiliary classifier comprises a two-layer multi-layer perceptron (MLP).

20. The method of claim 16, wherein the first network comprises a two-dimensional convolutional neural network.

21. The method of claim 16, wherein generating the first local temporal context features from the first adjacent subsequence of the input frame set via the first multi-scale temporal feature fusion engine further comprises: generating a first local temporal context feature from the set of input frames via a first convolutional neural network; generating, via a first cross-attention module, a first cross-attention feature output based on the first local temporal context features; generating a first average pooling dataset from the set of input frames via a first average pooling module; as well as The first local temporal context feature is generated by adding the first cross-attended feature output to the first average pooling dataset.

22. The method of claim 21 , wherein generating the second local temporal context features from the second adjacent subsequence of the input frame set via the second multi-scale temporal feature fusion engine further comprises: generating second local temporal context features from the set of input frames via a second convolutional neural network; generating, via a second cross-attention module, a second cross-attention feature output based on the first local temporal context features; generating a second average pooling dataset from the set of input frames via a second average pooling module; as well as The second local temporal context features are generated by adding the second cross-attended feature output to the second average pooling dataset.

23. The method of claim 20, wherein the first contiguous subsequence of the set of input frames is equal to the second contiguous subsequence of the set of input frames.

24. A method according to claim 22, wherein the first cross-attention module generates the first cross-attention feature output based on the relationship between the query and the keys associated with the set of input frames, and wherein the second cross-attention module generates the second cross-attention feature output based on the relationship between the query and the keys associated with the set of input frames.

25. A method according to claim 24, wherein the first cross-attention module and the second cross-attention module both transfer data associated with fine-grained features before the one-dimensional convolution on the set of input frames to data associated with coarse-grained features after the one-dimensional convolution on the set of input frames.

26. The method of claim 25, wherein: The first cross-attention module and the second cross-attention module each deliver information from two different time granularities; The two different time granularities include frame-level granularity and tuple-level granularity; and The first average pooling module and the second average pooling module each provide information of temporally adjacent frames at a single temporal granularity.

27. The method of claim 22, wherein the first convolutional neural network and the second convolutional neural network each perform a one-dimensional convolution using a corresponding kernel.

28. The method of claim 27, wherein the first convolutional neural network and the second convolutional neural network each perform the one-dimensional convolution using the corresponding kernel to summarize information in k consecutive frames in the input frame set to generate the first local temporal context feature and the second local temporal context feature.

29. An apparatus for performing video classification, the apparatus comprising: a neural network configured to generate frame-level features in consecutive frames from a set of video frames; a first multi-scale temporal feature fusion engine having a first kernel size and configured to generate a first local context feature based on the frame-level feature; a second multi-scale temporal feature fusion engine having a second kernel size and configured to generate second local context features based on the frame-level features; a first temporal relational cross-transformer classifier configured to generate a first distance between a query video and a set of support videos associated with the set of video frames based on the first local contextual features; a second temporal relational cross-transformer classifier configured to generate a second distance between a query video associated with the set of video frames and the set of support videos based on the second local contextual features; and A calculation engine is configured to calculate a final distance between the query video and the set of supporting videos based on the first distance and the second distance.

30. A method for performing video classification, the method comprising: generating, via a neural network configured to receive a set of video frames, frame-level features in consecutive frames from the set of video frames; generating a first local context feature based on the frame-level feature via a first multi-scale temporal feature fusion engine having a first kernel size; generating second local context features based on the frame-level features via a second multi-scale temporal feature fusion engine having a second kernel size; generating, via a first temporal relation cross-transformer classifier and based on the first local contextual features, a first distance between a query video and a set of support videos associated with the set of video frames; generating, via a second temporal relation cross-transformer classifier and based on the second local context features, a second distance between a query video associated with the set of video frames and the set of support videos; as well as A final distance between the query video and the set of supporting videos is calculated based on the first distance and the second distance.

Images