Long duration structured video action segmentation

Abstract

The invention relates to long duration structured video action segmentation. The machine learning model may process the video and generate an output, such as an action segmentation that assigns portions of the video to a particular action, or an action classification that assigns an action category for each frame of the video. Some machine learning models may accurately predict short videos, but may not be particularly suitable for performing action segmentation on long-duration, structured videos. An efficient machine learning model may include a hybrid architecture involving a temporal convolutional network and a bidirectional graph neural network. A machine learning model may process a long duration structured video by using a temporal convolutional network as a first-track motion segmentation model to generate rich frame-level features. Frame-level features may be converted into a graph having forward and backward edges. The graph neural network may process the graph to refine the final fine-grained per-frame motion prediction.

Description

Technical Field

[0001] The present application relates generally to the field of machine learning, and more particularly to long-duration structured video action segmentation. Background Art

[0002] Machine learning models (e.g., deep learning models, deep neural networks, convolutional neural networks, transformer-based models, etc.) are used in various artificial intelligence and machine learning applications such as computer vision, speech recognition, and natural language processing. Machine learning models can receive and process inputs such as images, videos, audio, speech, text, etc. Machine learning models can generate outputs such as features and predictions based on the inputs. Summary of the invention

[0003] According to one aspect of the present application, a method is provided, comprising: inputting one or more frame-wise inputs associated with a sequence of video frames into a temporal convolutional network; generating one or more frame-level features based on the one or more frame-level inputs using the temporal convolutional network; generating a graph including one or more nodes and one or more edges based on the one or more frame-level features, wherein a node corresponds to a video frame and an edge connecting two nodes represents a connection between frame-level features of two video frames; inputting the graph into a graph neural network; and generating one or more predictions for the one or more nodes of the graph using the graph neural network.

[0004] According to another aspect of the present application, there is provided an apparatus comprising: one or more processors; and one or more storage devices storing a machine learning model, wherein the machine learning model has processing operations performed by the one or more processors, and the machine learning model comprises: a temporal convolutional network for receiving one or more frame-level inputs associated with a sequence of video frames and outputting one or more frame-level features; a converter for generating a graph comprising nodes and one or more edges based on the one or more frame-level features, wherein a node corresponds to a video frame and an edge connecting two nodes represents a connection between frame-level features of the two video frames; and a graph neural network for receiving the graph and outputting one or more predictions for the node. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To aid this description, similar reference numerals designate similar structural elements. The embodiments are illustrated in the accompanying drawings by way of example and not by way of limitation.

[0006] Figure 1An exemplary machine learning model for processing one or more inputs (e.g., a video having a sequence of image frames) and generating one or more outputs according to some embodiments of the present disclosure is illustrated.

[0007] Figure 2 Another exemplary machine learning model for processing one or more inputs (e.g., a video having a sequence of image frames) and generating one or more outputs according to some embodiments of the present disclosure is illustrated.

[0008] Figure 3 Illustrated are exemplary components of a temporal convolutional network according to some embodiments of the present disclosure.

[0009] Figure 4 Another exemplary component of a temporal convolutional network according to some embodiments of the present disclosure is illustrated.

[0010] Figure 5 Different types of edges of a bidirectional graph according to some embodiments of the present disclosure are illustrated.

[0011] Figure 6 A converter for generating a graph according to some embodiments of the present disclosure is illustrated.

[0012] Figure 7 An exemplary temporal skip connection according to some embodiments of the present disclosure is illustrated.

[0013] Figure 8 Illustrated is a graph neural network for generating predictions on a graph according to some embodiments of the present disclosure.

[0014] Fig. 9 An exemplary implementation of a graph neural network for generating predictions on a graph according to some embodiments of the present disclosure is illustrated.

[0015] Fig.10 The diagram illustrates training and computing the loss function of a temporal convolutional network according to some embodiments of the present disclosure.

[0016] Fig.11 The diagram illustrates training and computing the loss function of a graph neural network according to some embodiments of the present disclosure.

[0017] Fig.12 The flowchart of FIG. 1 shows a method for making predictions according to some embodiments of the present disclosure.

[0018] Fig.13 is a block diagram of an exemplary computing device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Overview

[0020] Machine learning models can be used in automation in various scenarios, such as robotics, industrial surveillance, industrial automation, and so on. In some real-world automation domains, videos can be processed to obtain frame-level action segmentation across complex, long-duration videos (e.g., videos capturing footage inside food processing facilities, videos capturing footage inside commercial kitchens, videos capturing footage inside pharmaceutical manufacturing facilities, and so on). Some automation domains may want to achieve fine-grained action prediction (e.g., a system that can accurately predict between dozens of action categories). Some automation domains may have long-tail data distributions. For example, in some manufacturing applications, the "idle" action category (e.g., nothing happened) may dominate or may be the dominant category. Some automation domains may have action categories that are difficult to distinguish. It is not easy to implement a machine learning model that can perform well. For example, different action categories may share strong visual similarities.

[0021] Machine learning models can process videos and generate outputs such as action segmentation, which assigns parts of a video to specific actions, or action classification, which assigns an action category to each frame of a video. Some machine learning models can make accurate predictions for short videos, but may not be particularly well suited for performing action segmentation on long-duration, structured videos.

[0022] An effective machine learning model may include a hybrid architecture involving a temporal convolutional network and a bidirectional graph neural network. The hybrid architecture can achieve good performance and strong robustness when generating fine-grained action predictions. The machine learning model can process a sequence of frames or a sequence of video frames (e.g., a long-duration structured video) and output a per-frame action category prediction.

[0023] A temporal convolutional network can be a lightweight and / or compact version of a temporal convolutional network. A temporal convolutional network can use a one-dimensional (1-D) convolution operator to capture long duration dependencies across the entire input (across time). A temporal convolutional network can predict structured processes. For example, in industrial manufacturing, a series of actions may be expected, such as action 1 → action 2 → action 3 →.... A temporal convolutional network can be used as part of a "first pass" action segmentation model to generate rich frame-level features. In some embodiments, the frame-level features are taken from the output of a layer (e.g., the second-to-last layer, the second-to-last layer, the middle layer, the hidden layer, or not the final output layer) in a temporal convolutional neural network. A temporal convolutional network can capture long duration dependencies across the entire input. A temporal convolutional network may include components that can perform dilation at different dilation rates to extract temporal information with a larger effective receptive field.

[0024] In some embodiments, the temporal convolutional network receives one or more frame-level inputs (e.g., a set of frame-level inputs, or several frame-level inputs) generated by a convolutional neural network processing a video having a sequence of frames. The convolutional neural network may include a three-dimensional convolutional neural network. The convolutional neural network may perform space-time convolution to extract space-time features for each frame.

[0025] The frame-level features may be converted into a graph having one or more forward edges and one or more backward edges. In some embodiments, the graph may have undirected edges. In some embodiments, the graph may have one or more forward edges, one or more backward edges, and one or more undirected edges. The graph may have nodes corresponding to frames in a sequence of frames. Frame-level features may be compared between frames to determine whether to connect two nodes corresponding to two frames with an edge. In some embodiments, values ​​may be assigned to edges based on the comparison. In some embodiments, time-hopping connections of a certain size may be inserted into the graph for additional information transfer between nodes. The size may be set based on user-adjustable variables, parameters, or hyperparameters.

[0026] Bidirectional graph neural networks can be sparse models. Graph neural networks can process graphs to refine the final fine-grained per-frame action predictions. Graph neural networks can utilize concurrent temporal forward and backward local message passing operations.

[0027] Although some examples described herein refer to machine learning models processing videos having temporal sequences of frames or image frames, machine learning models contemplated by the present disclosure may be used to process other types of input data involving temporal sequences of frames or temporal sequences of data capture, such as sequences of audio frames, sequences of sensor data capture, sequences of depth image frames, etc. In some cases, machine learning models contemplated by the present disclosure may be used to process temporal sequences of frames or data capture involving sensor data having different sensor modalities. In some cases, machine learning models contemplated by the present disclosure may be used to process temporal sequences of frames or data capture involving sensor data having views or fields of view.

[0028] Although some of the examples described herein refer to action segmentation and / or making action category predictions for each frame, the machine learning models contemplated by the present disclosure may be used to generate other types of predictions for frames, such as other classifications, active speaker identification, salient object identification, frame title determination, frame label determination, frame semantic determination, activity identification, rule violation determination, emergency situation determination, safety risk determination, forecasting, object localization, boundary prediction, and the like.

[0029] Using the hybrid architecture described in this paper, we implement machine learning models that can understand the meaning of long-duration structured videos that are difficult to classify. Understanding the meaning of video data can be a ubiquitous challenge for deployable AI systems (e.g., in the field of automation). Accurate, fine-grained action segmentation of real-world long-duration videos is a non-trivial task in computer vision. The hybrid architecture can leverage temporal convolutional networks to learn long-duration temporal dependencies and graph neural networks to learn from local evidence between nearby frames.

[0030] Exemplary machine learning models and their components

[0031] A machine learning model refers to a computer-implemented system that can perform one or more tasks. A machine learning model can receive input and generate output for the task at hand. Using and implementing a machine learning model may involve supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Machine learning models can be implemented in different ways. Machine learning models can include one or more of the following: artificial neural networks, decision trees, support vector machines, regression analysis, Bayesian networks, Gaussian processes, genetic algorithms, and the like.

[0032] Artificial neural networks may include one or more layers, modules, networks, blocks and / or operators that transform inputs into outputs. In some embodiments, layers, modules, networks, blocks and / or operators may include one or more processing units and / or one or more processing nodes. A processing unit may receive one or more inputs, perform a processing function or operation, and generate one or more outputs. Processing units may be interconnected to form a network. In some cases, processing units or nodes may be referred to as neurons. Different types of processing units or nodes may be distinguished by the processing functions / operations being performed by the processing unit or node. A processing unit may include one or more parameters. These parameters may be trained or learned. A processing unit may include one or more hyperparameters. Hyperparameters may be set or adjusted, adjusted or set by one or more users of a machine learning model.

[0033] One type of processing unit is a convolution block and / or operator. The processing unit applies a convolution operation to an input and generates an output. The convolution operation can extract features from the input and output the features as output. The convolution operation can transform the input and generate an output. The processing unit can convolve the input with a kernel to generate an output. The kernel can include a matrix. The kernel can encode a function or operation that can transform the input. The kernel can include values ​​or parameters that can be trained or learned. The processing unit can calculate an inner product (e.g., a dot product) using a sliding / moving window that captures a local area or patch of the input, and sums and / or accumulates the inner products to generate an output. As the sliding / moving window moves over the input matrix, the inner product can be continuously calculated over the input matrix. Convolution blocks and / or operators may be defined by the size of the kernel, such as 1x1 convolution (convolution operator with a kernel size of 1x1), 2x2 convolution (convolution operator with a kernel size of 2x2), 3x3 convolution (convolution operator with a kernel size of 3x3), 4x4 convolution (convolution operator with a kernel size of 4x4), 5x5 convolution (convolution operator with a kernel size of 5x5), etc. The distance that the window slides / moves may be set or defined by the span of the convolution operator. In some cases, the convolution block and / or operator may not apply padding, but use the input matrix as is. In some cases, the convolution block and / or operator may apply half padding and pad around a portion of the input matrix. In some cases, the convolution block and / or operator may apply full padding and pad around the input matrix. In some cases, the convolution block and / or operator may be defined by the dimension of the applied filter. For example, a 1-D convolution block and / or operator may apply a sliding convolution filter or kernel of size k (a hyperparameter) to a one-dimensional input. The values ​​in the sliding convolution filter or kernel may be trained and / or learned.

[0034] Exemplary layers, modules, blocks, and / or operators may include a dilated convolution block, which may increase the number of features that may be extracted at various scales. A dilated convolution module may expand a kernel by inserting gaps between weights in the kernel. A dilated convolution module may have a dilation rate or dilation factor that indicates the extent to which the kernel is widened. Parameters in the kernel may be trained or learned.

[0035] Another type of processing unit is a transformer unit or block. A transformer unit can be used in a transformer block. A transformer unit can implement an attention mechanism to extract dependencies between different parts of the input of the transformer unit. The transformer unit can receive an input and generate an output that represents the importance or attention of various parts of the input. The transformer unit can include query weights, key weights, and value weights as trainable or learnable parameters. The transformer unit can apply these parameters to extract relationship information between different parts of the input of the transformer unit.

[0036] Another type of processing unit is an activation unit or block. An activation block can implement or apply an activation function (e.g., a sigmoid function, a nonlinear function, a hyperbolic tangent function, a corrected linear unit, a leaky corrected linear unit, a parameter corrected linear unit, a sigmoid linear unit, an exponential linear unit, a scaled exponential linear function, a logical activation function, a Heaviside activation function, an identity function, a binary step function, a soft step function, a Gaussian error linear unit, a Gaussian function, a soft add function, etc.) to the input of the activation block and generate an output. An activation block can be used to map the input of a block to a value between 0 and 1. An activation block can be used to map the input of a block to 0 or 1. An activation block can introduce nonlinearity. An activation module can learn complex decision boundaries. One or more parameters of an activation function can be trained or learned.

[0037] Exemplary layers, modules, blocks, or operators may include an upsampling block. An upsampling block may increase the size of an input feature or feature map. An upsampling block may synthesize values ​​that may be added to an input feature or feature map to increase the size of an upsampled output feature or feature map.

[0038] An exemplary layer, module, block or operator may include a downsampling block. The downsampling block may perform downsampling on the features or feature maps generated at each stage, which may improve the operating efficiency of the machine learning model. The downsampling block may include a pooling layer, which may receive a feature map at its input and apply a pooling operation to the feature map. The output of the pooling layer may be provided or input to a subsequent stage for further processing. The pooling operation may reduce the size of the feature map while retaining its (important) characteristics. Therefore, the pooling operation may improve efficiency and may avoid over-learning. The pooling layer may perform a pooling operation by average pooling (calculating an average value for each patch on the feature map), maximum pooling (calculating a maximum value for each patch of the feature map), or a combination of both. The size of the output of the pooling layer is smaller than the size of the feature map provided as input to the pooling layer. In some embodiments, the pooling operation is 2×2 pixels applied with a span of 2 pixels, so that the pooling operation reduces the size of the feature map by a factor of 2, for example, the number of pixels or values ​​in the feature map is reduced to a quarter of the size. In some embodiments, a pooling layer applied to a 6×6 feature map results in a 3×3 output pooled feature map.

[0039] Exemplary layers, modules, blocks, or operators may include projection layers (sometimes referred to as 1x1 convolution blocks and / or operators). The projection layer may transform the input features into a new space, such as a space that is appropriate, information-rich, and / or useful for the task being performed (processed by the downstream module) by the downstream module. The projection layer may include a dense layer or a fully connected layer, in which each neuron (e.g., a node or processing unit in a neural network) is connected to each neuron in the previous layer. The projection layer may generate and / or output one or more new features (e.g., a new set of features) that are more abstract or advanced than the features in the input. The projection layer may implement one or more 1x1 convolution operations, in which the projection layer may convolve the input features with a filter of size 1x1 (e.g., with zero padding and a stride of 1). The projection layer may implement channel aggregation or feature map aggregation. The projection layer may reduce the dimensionality of the input features by aggregating features across channels. The projection layer may implement a 1x1 filter to create a linear projection of a stack of feature maps. The projection layer may implement a 1x1 filter to increase the number of feature maps. The projection layer may implement a 1x1 filter to reduce the number of channels. The projection layer may make the feature map compatible with subsequent processing layers, modules, blocks, or operators. The projection layer may ensure that an element-wise addition operation can be performed to add the output of the projection layer to another feature map. The projection layer may ensure that the dimensions of the output of the projection layer match the dimensions of the feature maps being element-wise added together. The parameters of the projection layer may be trained or learned.

[0040] An exemplary block may include an adder block. An adder block may perform element-wise addition of two or more inputs to generate an output. An adder block may be an exemplary block that may merge and / or combine two or more inputs together. Addition and summation may be synonyms. An adder block may be replaced by a concatenation block.

[0041] An exemplary block may include a multiplier block. A multiplier block may perform element-wise multiplication of two or more inputs to generate an output. A multiplier block may determine a Hadamard product.

[0042] Exemplary blocks may include concatenation blocks. A concatenation block may perform concatenation of two or more inputs to generate an output. A concatenation block may append vectors and / or matrices to the input to form new vectors and / or matrices. Vector concatenation may be appended to form larger vectors. Matrix concatenation may be performed horizontally, vertically, or in a merged manner. Horizontal matrix concatenation may be performed by concatenating matrices (of the same height) in the input in a width direction. Vertical matrix concatenation may be performed by concatenating matrices (of the same width) in the input in a height direction. A concatenation block may be an exemplary block that may merge and / or combine two or more inputs. When two or more inputs do not have the same dimensions, a concatenation block may be appropriate. When it is desired to keep two or more inputs unchanged or intact (e.g., without losing information), a concatenation block may be appropriate. A concatenation block may be replaced by an adder block.

[0043] Exemplary machine learning model for making predictions about videos

[0044] Figure 1 An exemplary machine learning model 100 for processing one or more inputs and generating one or more outputs according to some embodiments of the present disclosure is illustrated. Figure 2 An exemplary machine learning model 200 for processing one or more inputs and generating one or more outputs according to some embodiments of the present disclosure is illustrated. An example of one or more inputs may include an input video 102 having a time sequence of frames or image frames. For various examples herein, the input video 102 may have N frames, or frames corresponding to N different timestamps or time instances. An example of one or more outputs may include a prediction 112. The machine learning model 100 and / or the machine learning model 200 may include a temporal convolutional network (TCN) 106, a transformer 108, and a graph neural network (GNN) 110.

[0045] TCN 106 may capture long duration dependencies in the input (e.g., input video 102). TCN 106 may exploit the long duration dependencies and perform frame-level action classification, e.g., making predictions about each frame in input video 102. In some embodiments, TCN 106 may include multiple layers, or convolutional neural network layers. TCN 106 may include dilated layers or dilated convolutional layers, e.g., layer 1 1301, ..., layer L_X-1 130 L_X-1 , and layer L_X 130 L_X. L_X may represent the number of layers in the Xth stage in TCN 106. A dilated layer or dilated convolutional layer performs one or more dilated convolutions. In some cases, TCN 106 may include one or more stages, and each stage may have one or more layers or convolutional neural network layers. The number of layers in two different stages may be different, or may be the same. TCN 106 may apply one or more dilated convolutions with one or more (varying or different) dilation rates.

[0046] In some embodiments, the TCN 106 may include processing blocks. The processing blocks may form a layer of the TCN 106. A layer may include multiple processing blocks. The processing blocks of a layer may be connected to the processing blocks of a subsequent layer. The processing blocks of a layer may be connected to a subset of the processing blocks of a subsequent layer (e.g., not densely connected to all processing blocks of a subsequent layer). The processing blocks of a layer may be causally connected to the processing blocks of a subsequent layer to perform causal convolution operations (e.g., convolution operations may only depend on current input values ​​and / or past input values ​​in time). When the processing blocks are causally connected, the TCN 106 may receive and process frame-level input streams in real time. The processing blocks of a layer may be non-causally connected to the processing blocks of a subsequent layer to perform non-causal convolution operations (or convolution operations that are not aware of causal or temporal dependencies and may depend on current input values, future input values, and / or past input values ​​in time). When the processing blocks are non-causally connected, the TCN 106 may receive and process frame-level inputs offline (rather than in real time). In some embodiments, the processing modules may be arranged and / or connected together to form the TCN 106. The processing blocks may be implemented efficiently using relatively lightweight operators. Exemplary processing blocks may include one or more 1-D convolution operators, which may be computationally efficient. In some embodiments, the processing blocks may implement dilated convolution operations, such as 1-D dilated convolution operations. Exemplary processing blocks may include residual connections to improve feature processing efficiency. Figure 3-Figure 4 Exemplary components or building blocks of TCN 106 are illustrated. Fig.10 106 to describe additional details related to the training of TCN 106.

[0047] In some embodiments, a layer in TCN 106 may include processing blocks having a particular configuration on the layer. A layer may include processing blocks having one or more associated dilation rates on the layer. The dilated convolution operations performed by processing blocks arranged in a series of layers or successive layers may have different configurations and / or dilation rates. The dilation rate may vary depending on the layer (the position of the layer, such as the depth of the layer within TCN 106). Changing the configuration and / or dilation rate over a series of layers or successive layers may increase the effective receptive field of TCN 106. The larger receptive field of TCN 106 may allow TCN 106 to learn long-range temporal dependencies in input video 102.

[0048] In some embodiments, TCN 106 may include a single stage (X=1). In some embodiments, TCN 106 may include multiple (sequential) stages or X stages (X>1), for example, TCN stage 1 1201, ..., TCN stage X-1 1202. X-1 , and TCN stage X 120 X . The multi-stage architecture can refine the prediction process performed by TCN 106 and enhance the prediction performance of TCN 106. Each stage may include one or more layers. The implementation within two different stages may be different, or may be the same. Each stage may include a different number of layers. Each stage may change the expansion rate between layers in different ways. Each stage may include different implementations of processing blocks between layers. In some embodiments, TCN 106 may include one or more prediction stages (e.g., a low-level stage near the input) to process one or more frame-level inputs. TCN 106 may include one or more refinement stages (e.g., one or more higher-level stages near the output) to process the output of the prediction stage. In some embodiments, TCN 106 may include X=4 stages. In some embodiments, TCN 106 may include one prediction stage and 3 refinement stages.

[0049] The TCN 106 may receive and / or process one or more frame-level inputs 150 (e.g., a set of frame-level inputs 150, a number of frame-level inputs, or a per-frame input). One or more frame-level inputs 150 (e.g., a set of frame-level inputs 150) may be input into the TCN 106. The TCN 106 may include I In some cases, W I is equal to or greater than N, and TCN 106 can process all frames of the input. In some cases, TCN 106 processes the frame-level input in batches. I May be less than N, and may be equal to or greater than the batch size. TCN 106 may process fragments (eg, batches) of the input, rather than the entire input.

[0050] The frame-level input 150 may correspond to or be associated with a sequence of frames or image frames of a video (input video 102). In some embodiments, the frame-level input 150 associated with the frame sequence may include frame-level feature embedding or extracted features for each frame. The frame-level input 150 may include a first frame-level input including a first feature vector extracted from a first frame in the frame sequence, and a second frame-level input including a second feature vector extracted from a second frame in the frame sequence. In some embodiments, the frame-level input 150 associated with the frame sequence may include raw (unprocessed) data or information or filtered data or information for each frame. The frame-level input 150 may include a first frame-level input including a first frame in the frame sequence, and a second frame-level input including a second frame in the frame sequence.

[0051] The TCN 106 may extract and / or output one or more frame-level features (e.g., a set of frame-level features, a feature vector for each frame, a feature embedding for each frame, a feature array for each frame, etc.). The TCN 106 may generate one or more frame-level features, for example, based on one or more frame-level inputs 150. The frame-level features may include a set of features at each frame and / or each timestamp. The frame-level features may include, for example, a frame-level array or vector of values, logarithms, raw scores, or unnormalized scores associated with the task to be performed by the TCN 106. The size of the array or the length of the vector may correspond to the number of categories, C. These outputs may be the output of the next-to-last (second-to-last) layer in the TCN 106, or may be generated by the next-to-last layer in the TCN 106. An example of a second-to-last or next-to-last layer of the TCN 106 is layer L_X-1 130 L_X-1 In some cases, if desired, the one or more frame-level features may include outputs of other intermediate / hidden layers in TCN 106. A set or vector of frame-level values ​​may be referred to as one or more frame-level features of TCN 106, one or more frame-level sub-last layer embeddings, or one or more frame-level sub-last layer feature embeddings.

[0052] TCN 106 may include a final layer after the second final layer that converts the frame-level array or vector of values ​​into a frame-level prediction, e.g., an array or vector of frame-level probabilities indicating the likelihood that a given frame belongs to each class. The final layer may have processing blocks that each implement an activation function, e.g., a softmax activation, such that the sum of the frame-level probability array or vector is 1. The final layer of TCN 106 may have a W O Output. O Can be equal to W I An example of the last layer of TCN 106 is layer L_X 130 L_XThe output may produce an array or vector of one or more values ​​corresponding to one or more predictions for a frame (e.g., one or more class predictions for a frame, one or more forecast predictions for a frame, etc.). The size of the array or length of the vector may correspond to the number of categories, C. In some embodiments, the output from the last layer may be used by and / or input into the converter 108 to generate the graph 170. The output from the last layer may have less information than the output from the next last layer of the TCN 106.

[0053] The converter 108 may generate a graph 170 that may be processed by the GNN 110. One or more frame-level features 160 (e.g., a set of frame-level features 160) may be input into the converter 108. The converter 108 may receive one or more frame-level features 160 or information corresponding to different frames. The converter 108 may convert one or more frame-level features 160 into a graph 170. The converter 108 may generate a graph 170. The graph 170 may include one or more nodes and one or more edges connecting the nodes. A node may correspond to a frame. An edge may represent a connection between frame-level features of two frames. The graph 170 may encode or represent the existence and / or strength of a relationship or connection between different frames in the form of a graph. The converter 108 may compare different frame-level features and generate the graph 170 based on the comparison. In some cases, a node may correspond to a frame. In some cases, a node may correspond to each feature (or category) in each frame / timestamp. A node may have a corresponding value or set of values. An edge may encode information about whether and / or how two nodes are related to each other. An edge may represent a connection between frame-level features 160 of two frames. An edge may have a corresponding value. An edge may encode a value and / or have a value that may represent a comparison of frame-level features between two frames. Graph 170 may have different types of edges. Some edges may be directional (e.g., a forward edge corresponds to a connection forward in time, and a backward edge corresponds to a connection backward in time). Some edges may be non-directional. The directionality of an edge may encode a temporal relationship. In combination Figure 5-Figure 7 To describe additional details about graph 170 (including its nodes and edges).

[0054] In some embodiments, the TCN 106 may receive image frames of the input video 120 as frame-level inputs 150. The image frames of the input video 120 may be input into the TCN 106. In some embodiments, the machine learning model 100 and / or the machine learning model 200 may also include additional models, filters, or processing networks to process the image frames of the input video 120 to generate and / or output one or more frame-level features to be used as one or more frame-level inputs 150 of the TCN 106. In some cases, the input video 120 is provided and / or input into a convolutional neural network or a deep neural network to extract one or more frame-level features or frame-level feature embeddings. In some cases, the input video 120 is provided and / or input into a three-dimensional convolutional neural network (3D CNN) 104 for processing. The 3D CNN 104 may extract and / or generate one or more frame-level features or frame-level feature embeddings from information in the image space and information in the time space of the input video 130. The 3D CNN 104 may include operations in which a kernel may be moved in three directions (e.g., width and height directions of an image space, and a temporal direction of a temporal space) to perform a spatial-temporal convolution. Thus, the 3D CNN 104 may extract one or more frame-level features or frame-level feature embeddings to be used as one or more frame-level inputs 150 of the TCN 106, which may have spatial-temporal information about the input video 130. The 3D CNN 104 may process an input video 120 (e.g., a sequence of frames) and generate and / or output one or more frame-level inputs 150 at the input of the TCN 106. The input video 130 (e.g., a sequence of frames) may be input into the 3D CNN 104. The 3D CNN 104 may be used to generate one or more frame-level inputs 150 for the TCN 106. In some embodiments, the input video 130 may be provided and / or input to other kinds of models to extract one or more frame-level features or frame-level feature embeddings for use as one or more frame-level inputs 150 to the TCN 106. The 3D CNN 104 may extract and / or generate global frame-level features from the input video 120 (e.g., raw video). Examples may include: two-dimensional convolutional neural networks, recurrent neural networks (including long short-term memory networks), support vector machines, Markov models, statistical models, inter-frame motion estimation processes, visual transformer models, and the like.

[0055] exist Figure 1, the converter 108 may receive one or more frame-level features from the TCN 106, e.g., a second-last-layer feature embedding of the TCN 106. The one or more frame-level features from the TCN 106 (e.g., a set of frame-level features 160) may be input to the converter 108. The converter 108 may generate a graph 170 based on the one or more frame-level features from the TCN 106.

[0056] In some cases, converter 108 may receive additional frame-level information instead of, or in addition to, one or more frame-level features from TCN 106. Additional / other frame-level information may be input to converter 108. Figure 2 In the embodiment of the present invention, one or more frame-level feature embeddings 210 at the output of the 3D CNN 104 (e.g., one or more frame-level inputs 150 to the TCN 106) may be fused with one or more frame-level features 220 from the TCN 106 by a fusion block 202. The fusion block 202 may receive and / or fuse the one or more frame-level inputs 150 to the TCN 106 and the one or more frame-level features 220. The fusion block 202 may combine the one or more frame-level inputs 150 to the TCN 106 with the one or more frame-level features 220 in an appropriate manner. The fusion block 202 may perform fusion of features at the frame level or in units of frames. The output of the fusion block 202 may be input into the converter 108. The converter 108 may receive the output of the fusion block 202. The converter 108 may generate the map 170 based on one or more frame-level features from the TCN 106, and further based on one or more frame-level inputs 150 (e.g., one or more frame-level feature embeddings 210 at the output of the 3D CNN 104) to the TCN 106. The converter 108 may generate the map 170 based on the output of the fusion block 202 (e.g., one or more frame-level features 160).

[0057] The fusion block 202 may include a concatenation block to concatenate the frame-level feature embedding 210 (at the output of the 3D CNN 104 or its derivatives, such as the frame-level feature embedding of the output of the hidden or intermediate layer of the 3D CNN 104) with the frame-level features 220 (or its derivatives). The fusion block 202 may include a projection layer to transform the feature space of the concatenated features before outputting the concatenated features as one or more frame-level features 160. The fusion block 202 may include an adder block to sum the frame-level feature embedding 210 (or its derivatives) with the frame-level features 220 (or its derivatives) (element-wise). The fusion block 202 may include a neural network layer (which may be trained) to combine the frame-level feature embedding 210 (or its derivatives) with the frame-level features 220 (or its derivatives) and generate the frame-level features 160. The fusion block 202 may generate the (fused) frame-level features 160 for the converter 108 to convert into the map 170.

[0058] The combining or fusion operation performed by the fusion block 202 may utilize hyperparameters to set the respective weights of the one or more frame-level inputs 150 and the one or more frame-level features 220 to the TCN 106. The combining or fusion operation performed by the fusion block 202 may utilize trained and / or learned parameters to set the respective weights of the one or more frame-level inputs 150 and the one or more frame-level features 220 to the TCN 106. The fusion block 202 may output the one or more frame-level features 160 that will be used by the converter 108 to generate the map 170.

[0059] The GNN 110 may receive and / or process the graph 170 to further exploit the information encoded in the edges of the graph 170. The graph 170 may be input into the GNN 110. The GNN 110 may make and / or output one or more predictions 112 for nodes of the graph 170. The GNN 110 may generate one or more predictions 112 based on the graph 170. The GNN 110 may be used, for example, to generate one or more predictions 112 based on the graph 170. The GNN 110 may perform a message aggregation operation on the graph 170. The GNN 110 may include M (sequential) layers, for example, GNN layer 1 1401, ..., GNN layer M-1 1402. M-1 , GNN layer M 140 M . Combined Figure 8-Figure 9 and Fig.11 1 to describe additional details involving GNN 110. GNN 110 can ingest information extracted by TCN 106 to improve the results of TCN 106.

[0060] Predictions 112 may include category predictions for nodes. For example, a node may correspond to a frame of input video 102, and predictions 112 may include one or more category predictions for each node (e.g., identifying the category to which the frame most likely belongs, including a vector of one or more probabilities that the frame belongs to one or more categories). In some cases, a node may correspond to each feature (or category) in each frame-level feature for each frame / timestamp, and predictions 112 may include, for each node, a probability that the frame belongs to a particular category or a decision as to whether the frame belongs to a particular category.

[0061] An exemplary implementation of a temporal convolutional network

[0062] Figure 3 FIG. 1 illustrates a temporal convolutional network (e.g., Figure 1 An exemplary component 300 of the TCN 106). Figure 4 FIG. 1 illustrates a temporal convolutional network (e.g., Figure 1 Another exemplary component 300 of the TCN 106 of FIG. 106 is shown. Specifically, Figure 3 Processing block 302 is illustrated, and Figure 4 Different processing blocks 402 are illustrated. Component 300 illustrates how to construct layers and optionally stages of processing blocks in a TCN to extract temporal information with a larger effective receptive field. Component 300 can be replicated and / or repeated to match the number of inputs the TCN is receiving (e.g., as combined with Figure 1-Figure 2 As mentioned above, W I =W O ). A set of blocks such as processing block 302 in component 300 may be replicated and / or repeated to form successive layers in stage X of the TCN. A set of blocks such as processing block 402 in component 300 may be replicated to form successive layers in stage X of the TCN. The layer number for stage X of the TCN may be L_X (uppercase). l_x (lowercase) may represent a specific layer number and / or position (in terms of layers) of a processing block in stage X of the TCN.

[0063] The input 310 may be one of the one or more frame-level inputs to the TCN (or a single input). The input 310 (e.g., frame-level input) may be processed by a 1x1 convolution operator 342 (e.g., a convolution operator with a kernel size of 1x1). The 1x1 convolution operator 342 may be applied to normalize the feature dimension, or may be applied to normalize the feature dimension of the input 310. The 1x1 convolution operator 342 may adjust the dimension of the input 310 to match the number of feature maps in the TCN. For one or more frame-level inputs to the TCN, a 1x1 convolution operator 342 may be provided for each input to form a 1x1 convolution layer. With WI A TCN with 1 input can have W forming a 1x1 convolutional layer. I 342 1x1 convolution operators.

[0064] The output 312 of the 1x1 convolution operator 342 may be provided as an input to the processing block 302 and / or the processing block 402. The processing block 302 and / or the processing block 402 may be implemented as a residual processing block, for example, which may facilitate gradient flow. The processing block 302 and / or the processing block 402 may include an expansion convolution block. The processing block 302 may be replicated / repeated to form an expansion layer or expansion convolution layer in the TCN. The processing block 302 may be used as a building block to provide more than one expansion layer or expansion convolution layer in the TCN. The processing block 402 may be replicated / repeated to form an expansion layer or expansion convolution layer in the TCN. The processing block 402 may be used as a building block to provide more than one expansion layer or expansion convolution layer in the TCN. The expansion layer or expansion convolution layer may have the same number of processing blocks, such as the processing block 302 or the processing block 402. The dilated layer or dilated convolution layer may have the same type or variant of a dilated processing block or a dilated convolution processing block, such as processing block 302 or processing block 402 .

[0065] refer to Figure 3 , the processing block 302 has an input path 370 that passes the output 312 of the 1x1 convolution operator 342 as the output of the input path 370 to the adder block 328. The processing block 302 may include a residual path 380 to perform variable dilation depending on the position of the processing block in the stage or within the TCN (e.g., l_x). Specifically, the residual path 380 may include a 2 l_x The invention also provides a dilated convolution block 322 with a dilation rate of , an activation block 324 for receiving and / or processing the output of the dilated convolution block 322, and a 1x1 convolution block for receiving and / or processing the output of the activation block 324. l_x The dilation rate of may depend on the position of the dilated convolutional layer l_x where processing block 302 is located (within the Xth stage of the TCN). In some embodiments, a TCN with multiple (consecutive) dilated convolutional layers may implement processing block 302 to double the dilation rate at each layer l_x, e.g., 2 l_x =1, 2, 4, ..., 256, 512, 1024. As the position of the dilated convolutional block becomes deeper or higher in the TCN or the stage of the TCN, the dilation rate may be doubled or increased exponentially based on another factor.

[0066] refer to Figure 4, the processing block 402 has an input path 370 that passes the output 312 of the 1x1 convolution operator 342 as the output of the input path 370 to the adder block 328. The processing block 402 may include a residual path 450 to perform variable dilation depending on the position (e.g., l_x) of the processing block in the stage or within the TCN. The residual path 450 may perform multiple dilations (e.g., double dilation) and may have multiple (or double) dilation rates. For example, the residual path 450 may include a 2 L_X-l_x The first dilated convolution block 424 of the first dilation rate of 2 l_x The first dilated convolution block 424 and the second dilated convolution block 422 may be expanded in parallel, but at different expansion rates. The first expansion rate may be different from the second expansion rate. L_X-l_x The first dilation rate may depend on the position of the dilated convolutional layer l_x where the processing block 402 is located (within the Xth stage of the TCN). The second dilation rate 2 l_x The processing block 402 may depend on the position of the dilated convolution layer l_x where the processing block 402 is located. In some embodiments, a TCN with multiple (consecutive) double dilated convolution layers may implement the processing block 402 to double the second dilation rate at each layer l_x, for example, 2 l_x =1,2,4,…,256,512,1024, and at each layer l_x the first dilation rate is halved, e.g., 2 L_X-l_x =1024,512,256,…,8,4,2,1. As the position of the second dilated convolution block becomes deeper in the TCN or the stage of the TCN, the second dilation rate may be doubled or increased exponentially based on another factor. As the position of the first dilated convolution block becomes deeper in the TCN or the stage of the TCN, the first dilation rate may be halved or decreased exponentially based on another factor.

[0067] In some embodiments, the output of processing block 302 or processing block 402 may optionally be processed by and / or input to 1x1 convolution operator 330. The output of 1x1 convolution operator 330 may be provided as input to the next stage in TCN. 1x1 convolution operator 330 may be applied to obtain frame-level probabilities of different output categories (rather than features). O A TCN with 1 output can have W formed as a 1x1 convolutional layer O 1x1 convolution operator 330.

[0068] Alternatives to TCN 106 for Extracting One or More Frame-Level Features

[0069] Figure 1-Figure 2 The TCN 106 shown in can be implemented to perform action segmentation and / or frame-level action classification. TCN 106 can extract frame-level features and / or frame-level feature embedding that can be used to form predictions. TCN 106 can examine temporal relationships at different temporal resolutions (e.g., long and short). In some embodiments, at least a portion of the functionality of TCN 106 can be replaced or supplemented by another different type of model to perform action segmentation and / or frame-level action classification. Examples of another type of model include recursive neural networks, Markov models, conditional random fields, encoder-decoder temporal convolutions, transformer-based action segmentation models, encoder-decoder structured transformer-based models, and graph-based temporal reasoning models.

[0070] Convert frame-level features into graphs

[0071] Information and data can be structured as graphs. Graphs can be used to encode the relationships between different pieces of information. Pieces of information can be encoded as nodes of a graph. The relationship between two pieces of information can be encoded as edges of a graph. Back to Reference Figure 1-Figure 2 , the converter 108 may have the technical task of converting information (e.g., one or more frame-level features produced by one or more upstream models) into a graph 170 that can be processed by the GNN and / or can be input into the GNN.

[0072] Figure (for example, Figure 1-Figure 2 170) may include one or more nodes and one or more edges connecting the nodes. In some embodiments, a node may correspond to a frame or a timestamp. A node may correspond to a frame, or a timestamp. An edge may encode or represent a relationship between frames or timestamps. An edge may encode a relationship between two frames or two timestamps. In some embodiments, a node may correspond to each feature (or category) in each frame / timestamp. One or more nodes may represent one or more features (or one or more categories) of a frame or timestamp. One or more frame-level features corresponding to different frames may have vectors of feature embeddings corresponding to different frames. A node may correspond to a feature embedding in a vector at a specific frame / timestamp. A node may have or encode a corresponding value or set of values.

[0073] Figure 5Different types of edges of a bidirectional graph according to some embodiments of the present disclosure are illustrated. An edge may encode or represent information about whether and / or how two nodes are related to each other. Since a node may correspond to a specific frame or timestamp (frame and timestamp may be synonymous), and different nodes may represent different points in time, the nodes may be connected by directed or undirected edges to indicate the temporal relationship between the nodes. A graph may have one or more graphical types of edges. In some embodiments, nodes may represent different features or feature embeddings at the same frame or timestamp, and the nodes may be connected by undirected edges. A graph may be divided into different graphs, each with a single type of edge.

[0074] exist Figure 5 In the top portion of the graph, nodes corresponding to different frames or timestamps can be connected with a forward edge. A forward edge can connect a node corresponding to a previous timestamp with a node corresponding to a subsequent timestamp to encode a relationship between the two nodes. When processing a bidirectional graph, message passing can occur along the direction of the forward edge, for example, from a node corresponding to a previous timestamp going forward to a node corresponding to a subsequent timestamp. Message passing in the reverse direction can be prohibited.

[0075] exist Figure 5 In the middle part of a graph, nodes corresponding to different frames or timestamps can be connected with a backward edge. A backward edge can connect a node corresponding to a subsequent timestamp with a node corresponding to a previous timestamp to encode a relationship between the two nodes. When processing a bidirectional graph, message passing can occur along the direction of the backward edge, for example, from a node corresponding to a subsequent timestamp going backwards to a node corresponding to a previous timestamp. Message passing in the opposite direction can be prohibited.

[0076] exist Figure 5 The bottom part of the graph, nodes corresponding to the same frame or timestamp can be connected with undirected edges. Undirected edges can connect nodes corresponding to a specific timestamp with nodes corresponding to the same timestamp to encode the relationship between the two nodes. When processing a bidirectional graph, message passing can occur in either direction or in both directions between two nodes. In some embodiments, a single node can be provided for each frame or timestamp, and the graph can have forward edges, backward edges, and no undirected edges. In some embodiments, one or more nodes can be provided for each frame or timestamp, and the graph can have forward edges, backward edges, and undirected edges (edges connecting one or more nodes at a given frame or timestamp). In some embodiments, if undirected message passing is required between two nodes corresponding to different frames / timestamps, undirected edges can connect nodes corresponding to different frames / timestamps.

[0077] Typed edges can help capture relationships and interactions between frames and / or features in different frames and enrich the information stored in graph 170.

[0078] Figure 6 The diagram is shown in Figure 1. Figure 1-Figure 2 170) of the converter (e.g., Figure 1-Figure 2 108). The edge may encode and / or represent the relationship and / or interaction between nodes. As shown, the converter 108 may receive one or more frame-level features 160 (e.g., features or feature embeddings corresponding to each frame or each timestamp). One or more frame-level features 160 may be input into the converter 108. As an example, one or more frame-level features 160 may include a first frame-level feature 602 corresponding to a first frame. The first frame-level feature 602 may include a first vector (or array) having features and / or feature embeddings. One or more frame-level features 160 may include a second frame-level feature 604 corresponding to a second frame (e.g., at a subsequent time from the first frame). The second frame-level feature 604 may include a second vector (or array) having features and / or feature embeddings. The first and second vectors may have the same size or have the same number of elements in the vectors. One or more frame-level features 160 may include additional frame-level features corresponding to other frames. The converter 108 may generate a graph 170 based on one or more frame-level features 160. The converter 108 may assign a node to each frame-level feature or feature embedding for each frame or each timestamp. The converter 108 may assign a first node to the first frame-level feature 602. The converter 108 may assign a second node to the second frame-level feature 604.

[0079] The converter 108 may assign or configure edges to temporally connect nodes corresponding to frame-level features at different frames / timestamps. In order to determine the relationship and / or interaction between frame-level features at different frames / timestamps, the converter 108 may perform a pairwise comparison of frame-level features at different frames / timestamps. For example, the converter 108 may compare a first frame-level feature 602 corresponding to a first frame with a second frame-level feature 604 corresponding to a second frame. Comparing the first frame-level feature corresponding to the first frame with the second frame-level feature may extract and / or generate information about the relationship and / or interaction between the first frame and the second frame. The information may indicate that a certain relationship is present or exists between the first frame-level feature of the first frame and the second frame-level feature of the second frame. The information may indicate the strength or degree of the relationship between the first frame-level feature of the first frame and the second frame-level feature of the second frame.

[0080] The converter 108 may connect the first node corresponding to the first frame in the graph and the second node corresponding to the second frame in the graph using a first edge based on the comparison or contrast. The first edge may encode or represent the presence or existence of a relationship between the first node and the second node. The first edge may encode or represent the strength or degree of the relationship between the first node and the second node. The converter 108 may assign a first value (e.g., a weight) to the first edge based on the comparison or contrast, which may represent the strength of the relationship and / or interaction between the first node and the second node. The first edge may be directional (e.g., forward or backward). The converter 108 may connect the first node to the second node with a forward edge (e.g., forward in time). The converter 108 may connect the second node to the first node with a backward edge (e.g., backward in time). The converter 108 may connect the first node and the second node with an undirected edge. The converter 108 may perform one or more comparisons for each pair of frame-level features in the one or more frame-level features 160 to generate a graph 170.

[0081] The comparison or contrast may include determining the distance between the first frame-level feature 602 and the second frame-level feature 604 (e.g., calculating the Euclidean distance of the two vectors). The distance or a derivative thereof may be assigned as a first value of the first edge. If the distance is smaller, the value may be higher (e.g., the value and the distance may have an inverse relationship). The comparison or contrast may also include determining whether the distance is close enough to assign an edge between the two nodes (e.g., comparing the distance to a threshold). The comparison or contrast may determine whether the first frame-level feature 602 and the second frame-level feature 604 are similar enough, or whether they match or correspond to each other enough. The comparison or contrast may include counting the number of features or feature embeddings with the same or sufficiently close / matching values ​​(e.g., comparing the features or feature embedding pairs at the same position in the first frame-level feature 602 and the second frame-level feature 604). The comparison or contrast may also include determining whether the count is high enough to assign an edge between the two nodes (e.g., comparing the count to a threshold). The count or a derivative thereof may be assigned as the first value of the first edge.

[0082] In some embodiments, the comparison or contrast may include comparing features or feature embedding pairs at the same location in the first frame-level features 602 and the second frame-level features 604. In some cases, the converter 108 may assign an edge to each pair of values ​​that are the same or sufficiently close / matching.

[0083] Individual features or feature embeddings in the frame-level features (e.g., a vector of features or feature embeddings for each frame) can correspond to different categories (e.g., action categories). For example, the first value of the first position in the vector of features or feature embeddings of the first frame (e.g., the first frame-level features 602) may correspond to a first category, or provide an indication of whether the first frame belongs to the first category. The second value of the second position in the vector of features or feature embeddings of the first frame (e.g., the first frame-level features 602) may correspond to a second category, or provide an indication of whether the first frame belongs to the second category. Nodes may be assigned to individual features or feature embeddings for each frame. In some cases, separate nodes may be assigned to individual features or feature embeddings in each frame-level feature in one or more frame-level features 160 (rather than one node per frame).

[0084] A node may be assigned to each individual feature or feature embedding in each frame. Nodes may be assigned only to individual features or feature embeddings that are sufficiently present or have a sufficiently strong signal in the vector (e.g., features or feature embeddings that have a non-zero value or a sufficiently high value). A node may be assigned to each of the top K features or feature embeddings with the highest value in the vector. For example, one or more nodes may be assigned to features in the first frame-level features 602. One or more nodes may be assigned to features in the second frame-level features 604.

[0085] The converter 108 can connect nodes representing features or feature embeddings in the same frame / timestamp with undirected edges. If the features or feature embeddings at the same position in the vector exist in two different frames, the converter 108 can connect nodes representing features / feature embeddings in a specific position (e.g., a specific category) in the vector with edges. The converter 108 can connect nodes based on a comparison of a pair of values ​​corresponding to a specific feature or feature embedding at the same position in the vector, and assign edges based on the comparison. The comparison may include whether the comparison values ​​are close enough. The comparison may include determining the absolute value difference between the values, and comparing the difference with a threshold. The difference may be assigned as the value of the edge. If the difference is small, the value of the edge can be higher (e.g., the value and the difference can have an inverse relationship). The converter 108 can connect nodes using forward edges (e.g., forward in time). The converter 108 can connect nodes using backward edges (e.g., backward in time). The converter 108 can connect nodes using undirected edges. For all features or feature embeddings, the converter 108 may assign an edge to each pair of nodes that corresponds to a particular feature or feature embedding (eg, at a particular position in a vector) to produce the graph 170 .

[0086] Figure 7The diagram illustrates exemplary time skip connections according to some embodiments of the present disclosure. In some cases, the converter may add one or more time skip connections to the graph. A time skip connection may connect two nodes that are separated by at least one frame and / or at least one timestamp. The number of frames or timestamps separating two nodes may be considered the size of the time skip connection. Edge 702 is an example of a time skip connection connecting node 2 and node 7, which are separated by 5 frames. Time skip connections or intermediate-level time skip connections may allow messages to be passed between non-adjacent nodes corresponding to non-adjacent frames. Edge 704 is not a time skip connection because edge 404 connects adjacent nodes corresponding to adjacent frames that are not separated by any frames (e.g., node 1 is adjacent to node 2). Additional message passing channels such as time skip connections may allow graph neural networks to better synthesize intermediate temporal information to improve prediction robustness, especially in the case of over-segmented and video frames representing action transitions. Converters (e.g., such as Figure 1-Figure 2 and Figure 6 The converter 108 shown in ) can artificially add one or more time jump connections to a node. When there is no relationship between two nodes that are τ frames or timestamps apart, the converter can add one or more edges connecting the two nodes. τ can represent the size of the time jump connection. The edge can be a forward edge. The edge can be a backward edge. The edge can be an undirected edge. The converter can add one or more edges in the graph according to a predefined value τ. τ can be a hyperparameter that can be adjusted or regulated or modulated by the user. The hyperparameter τ can define the size of one or more (artificially) added time jump connections. In some cases, τ=10. Nodes representing video frames that are 10 frames apart may include edge connections.

[0087] Example implementation of a graph neural network

[0088] Figure 8 The diagram illustrates that, according to some embodiments of the present disclosure, GNN 110 receives graph 170 and generates predictions 112 on graph 170. Graph 170 may be input into GNN 110. As graph 170 is processed by GNN 110, nodes and their corresponding values ​​may be updated using information from neighboring nodes. Edges may serve as channels for message passing or information sharing between nodes connected by the edges. Edges may have corresponding values, which may weight the information or messages passed between nodes. After being updated by GNN 110, the value a corresponding to a node may be updated to the value a corresponding to a node. j (j represents a node index in the graph 170) may represent a prediction 112 made by the GNN 110. In some cases, the graph 170 has one node for each frame or timestamp, and the GNN 110 may generate one or more predictions for each frame or timestamp.

[0089] Fig. 9An exemplary implementation of a GNN 110 for generating predictions 112 on a graph according to some embodiments of the present disclosure is illustrated. Figure 6 and Figure 8 The graph 170 shown in FIG. 170 may be converted or transformed into two or more separate graphs. The GNN 110 may process these separate graphs in parallel. The graph may be converted into a forward graph 832 having only nodes and forward edges. The graph may be converted into a backward graph 834 having only nodes and backward edges. The graph may be converted into an undirected graph 836 having only nodes and undirected edges.

[0090] The GNN 110 may include M layers, for example, GNN layer 1 1401, ..., GNN layer M-1 140 M-1 , GNN layer M 140 M . Each layer may implement a different type of processing or message aggregation (e.g., convolution type) for each graph. Each layer may implement a mix or combination of types of processing or message aggregation (e.g., convolution type) to extract or generate information from each graph. Each layer may include (parallel) processing blocks dedicated to processing a specific graph. For example, GNN layer 1 1401 may include a type 1 convolution operator 802 for processing the forward graph 832 F The GNN layer 1 1401 may include a type 1 convolution operator 802 for processing the backward graph 834 B The GNN layer 1 1401 may include a type 1 convolution operator 802 for processing an undirected graph 836 U . GNN layer M-1 140 M-1 A type 2 convolution operator 812 may be included for processing information extracted upstream from the forward graph 832 F . GNN layer M-1 140 M-1 A type 2 convolution operator 812 may be included for processing information extracted upstream from the backward graph 834 B . GNN layer M-1 140 M-1 A type 2 convolution operator 812 may be included for processing information extracted upstream from the undirected graph 836 U . GNN layer M 140 M A type 3 convolution operator 822 may be included for processing information extracted upstream from the forward graph 832 F . GNN layer M 140 M A type 3 convolution operator 822 may be included for processing information extracted upstream from the backward graph 834 B . GNN layer M 140 M A type 3 convolution operator 822 may be included for processing information extracted upstream from the undirected graph 836 UThe type of processing or message aggregation provided in a layer (e.g., type of convolution) may not be limited to 3 different types.

[0091] In some embodiments, one or more intermediate layers (e.g., including GNN layer 2 to GNN layer M-1 (not GNN layer 11401, nor GNN layer M 140 M ) in a layer set) can be shared between different graphs. For example, the type 2 convolution operator 812 can be shared F , Type 2 convolution operator 812 B and type 2 convolution operator 812 U Sharing parameters and / or weights may mean that the parameters and / or weights are the same and during training, the parameters and / or weights are updated to have the same value. Sharing parameters and / or weights may cause message bottlenecks. Sharing parameters and / or weights may help achieve generalization and avoid overfitting.

[0092] Different types of processing or message aggregation (e.g., convolution types) may include edge convolution / aggregation, attention convolution / aggregation, and sampling and aggregation convolution / aggregation.

[0093] Edge convolution / aggregation can learn pairwise interactions between nodes. Edge convolution / aggregation can model global and local structures by applying a channel-wise symmetric aggregation operation to edge features associated with all edges emanating from each node. Edge convolution / aggregation may have many parameters and may be very expressive. Edge convolution / aggregation may be appropriate in the first layer (e.g., GNN layer 1 1401) in GNN110. In some cases, all edges emanating from a node are aggregated.

[0094] Attention convolution / aggregation can exploit the self-attention mechanism between a node and its neighborhood. Attention convolution / aggregation can apply multi-head attention, which can improve its modeling ability. Attention convolution / aggregation may be particularly suitable for layers that serve as message bottlenecks (e.g., middle layers in GNN 110, GNN layer M-1 140). M-1 , etc.). Attention convolution / aggregation may aggregate edges emanating from nodes based on the importance of the edges (eg, the aggregation may be weighted by the importance of the edges).

[0095] The sampling and aggregation convolution / aggregation may have a computationally efficient form. The sampling and aggregation convolution / aggregation may apply a learnable linear transformation with a nonlinear activation function, which may be suitable for task-dependent aggregation. The sampling and aggregation convolution / aggregation may be particularly suitable for the last layer in the GNN 110 (e.g., GNN layer M 140 M). Sampling and Aggregation Convolution / Aggregation can aggregate edges emanating from nodes based on learned parameters about the node’s neighborhood.

[0096] In some embodiments, M=3. The first layer may implement edge convolution / aggregation. The second layer may implement focused convolution / aggregation. Parameters or weights in the second layer may be shared between different graphs. The third layer may implement sampling and aggregation convolution / aggregation.

[0097] In some embodiments, M=3. The first layer may implement sampling and aggregation convolution / aggregation. The second layer may implement attention convolution / aggregation. Parameters or weights in the second layer may be shared between different graphs. The third layer may implement sampling and aggregation convolution / aggregation.

[0098] Operators from the last GNN layer (e.g., type 3 convolution operator 822 F , Type 3 convolution operator 822 B and type 3 convolution operator 822 U ) may include individual graphs. The outputs (e.g., individual graphs) may be summed at a summer 850. Summer 850 may sum, combine, or aggregate individual values ​​or sets of values ​​generated by different operators for a given node (e.g., node by node). Values ​​corresponding to the nodes, such as action prediction pairs, may be provided to a prediction layer 860. Prediction layer 860 may apply a trained / learned activation function to generate predictions 112.

[0099] Predictions 112 may include per-node predictions. Predictions 112 may include per-frame predictions. Predictions 112 may include per-feature embedding / class per-frame predictions.

[0100] In some cases, summer 850 and prediction layer 860 may be replaced by a trained / learned layer that can fuse the various values ​​generated by different operators for a given node into a prediction for the given node. The trained / learned layer can be a projection layer. The trained / learned layer can have respective weights or parameters according to which the various values ​​can be fused, combined, summed, or aggregated. The trained / learned layer can apply a weighted average function.

[0101] In some cases, summer 850 and prediction layer 860 may be replaced by a processing layer that processes the individual values ​​produced by different operators for a given node and generates a prediction for each node. The processing layer may apply a maximum or maximum function. The processing layer may apply an average function. The processing layer may calculate the average of the individual values.

[0102] Training and calculating loss function

[0103] The training dataset may include an input video, where each frame of the training video has one or more labeled categories.

[0104] Fig.10 1 illustrates training and computing a loss function for a temporal convolutional network according to some embodiments of the present disclosure. A training video may be provided as an input video 102. The final output of the TCN 106 may be compared with the labels in the training data. The loss function may include a cross entropy loss L CE and the time smoothing loss L T-MSE . The cross entropy loss can measure whether the final output of TCN 106 is correct (using the labels in the training data). The temporal smoothness loss can measure the jitter or smoothness of the final output of TCN 106 over time (and penalize the model for changing classification too frequently, as class predictions for action segmentation are expected to be smooth). The different contributions to the loss function from cross entropy and temporal smoothness can be balanced or set by α. An optimizer (e.g., implementing Adam optimization) can update the parameters of TCN 106 to minimize the loss function. The parameters of 3DCNN 104 can be fixed, or pre-trained.

[0105] Fig.11 The diagram illustrates training and computing a loss function for a graph neural network according to some embodiments of the present disclosure. The training video may be provided to a pre-trained model (e.g., Figure 1-Figure 2 104 and TCN 106) to generate one or more frame-level features 160. One or more frame-level features 160 may be converted into a graph 170 by a converter 108. Graph 170 may be ingested by GNN 110. The final output of GNN 110 may be compared with the labels in the training data. The loss function may include a cross entropy loss L CE and the time smoothing loss L T-MSE . The cross entropy loss can measure whether the final output of GNN 110 is correct (using the labels in the training data). The temporal smoothness loss can measure the jitter or smoothness of the final output of GNN 110 over time (and penalize the model for changing classification too frequently, as class predictions for action segmentation are expected to be smooth). The different contributions to the loss function from cross entropy and temporal smoothness can be balanced or set by α. An optimizer (e.g., implementing Adam optimization) can update the parameters of GNN 110 to minimize the loss function.

[0106] Another exemplary method for making or generating one or more predictions

[0107] Fig.12 The flowchart of FIG. 1200 shows a method 1200 for making predictions according to some embodiments of the present disclosure. A computing device, such as Fig.13 The computing device 1300 in the embodiment of the present invention can be used to perform the method 1200. Figure 1-Figure 2Method 1200 may be performed using one or more components shown. At 1202, one or more frame-level inputs associated with a sequence of frames are input into a temporal convolutional network (e.g., TCN 106). The temporal convolutional network may process the one or more frame-level inputs. At 1204, using the temporal convolutional network, one or more frame-level features may be generated based on the one or more frame-level inputs. The temporal convolutional network may extract or generate one or more frame-level features. At 1206, a converter (e.g., converter 108) may generate a graph including one or more nodes and one or more edges based on the one or more frame-level features. A node may correspond to a frame. One or more nodes may correspond to one or more frames. An edge may represent a connection between frame-level features of two frames. One or more edges may represent one or more connections, relationships, and / or interactions between frame-level features of two frames. At 1208, the graph may be input into a graph neural network (e.g., GNN 110). At 1210, using the graph neural network, one or more predictions may be generated for one or more nodes of the graph. The graph neural network may extract or generate one or more predictions based on the graph.

[0108] Exemplary Computing Devices

[0109] Fig.13 1 is a block diagram of an apparatus or system (e.g., an exemplary computing device 1300) according to some embodiments of the present disclosure. One or more computing devices 1300 may be used to implement the functionality described herein in conjunction with the accompanying drawings. Several components are illustrated in the accompanying drawings that may be included in computing device 1300, but any one or more of these components may be omitted or repeated as appropriate for the application. In some embodiments, some or all of the components included in computing device 1300 may be attached to one or more motherboards. In some embodiments, some or all of these components are manufactured onto a single system-on-a-chip (SoC) die. In addition, in various embodiments, computing device 1300 may not include Fig.13 , and computing device 1300 may include interface circuitry for coupling with the one or more components. For example, computing device 1300 may not include display device 1306, and may include display device interface circuitry (e.g., connectors and driver circuitry) to which display device 1306 may be coupled. In another set of examples, computing device 1300 may not include audio input device 1318 or audio output device 1308, and may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which audio input device 1318 or audio output device 1308 may be coupled.

[0110] The computing device 1300 may include a processing device 1302 (e.g., one or more processing devices, one or more processing devices of the same type, one or more processing devices of different types). The processing device 1302 may include an electronic circuit that processes electronic data from a data storage element (e.g., a register, a memory, a resistor, a capacitor, a qubit unit) to transform the electronic data into other electronic data that can be stored in the register and / or the memory. Examples of the processing device 1302 may include a central processing unit (CPU), a graphical processing unit (GPU), a quantum processor, a machine learning processor, an artificial intelligence processor, a neural network processor, an artificial intelligence accelerator, an application specific integrated circuit (ASIC), an analog signal processor, an analog computer, a microprocessor, a digital signal processor, a field programmable gate array (FPGA), a tensor processing unit (TPU), a data processing unit (DPU), and the like.

[0111] The computing device 1300 may include a memory 1304, which may itself include one or more memory devices, such as volatile memory (e.g., DRAM), non-volatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid-state memory, and / or a hard drive. The memory 1304 includes one or more non-transitory computer-readable storage media. In some embodiments, the memory 1304 may include a memory that shares a die with the processing device 1302. In some embodiments, the memory 1304 includes one or more non-transitory computer-readable media storing instructions that are executable to perform the operations described herein in conjunction with the accompanying figures, such as Figure 12-Figure 15. Exemplary components that may be encoded as instructions and stored in memory 1304 are depicted. Memory 1304 may store instructions encoding one or more exemplary components. The instructions stored in one or more non-transitory computer-readable media may be executed by processing device 1302. In some embodiments, memory 1304 may store data, such as data structures, binary data, bits, metadata, files, binary large objects, etc., as described herein in conjunction with the accompanying drawings. Exemplary data that may be encoded and stored in memory 1304 are depicted. Memory 1304 may store one or more of the depicted data.

[0112] In some embodiments, memory 1304 may store one or more machine learning models (and / or portions thereof). Memory 1304 may store input data, output data, intermediate outputs, intermediate inputs. Memory 1304 may store instructions for performing one or more operations of a machine learning model. Memory 1304 may store one or more parameters used by a machine learning model. Memory may store information encoding how processing units are connected to each other. Examples of machine learning models or components of machine learning models may include Figure 1 Machine learning models 100, Figure 2 , a machine learning model 200, a 3D CNN 104, a TCN 106, a fusion block 202, a converter 108, a GNN 110, and a prediction 112.

[0113] In some embodiments, computing device 1300 may include communication device 1312 (e.g., one or more communication devices). For example, communication device 1312 may be configured to manage wired and / or wireless communications for transmitting data to and from computing device 1300. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc. that can communicate data through a non-solid medium using modulated electromagnetic radiation. The term does not mean that the associated devices do not contain any wires, although in some embodiments they may not. The communication device 1312 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards, including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), Long-Term Evolution (LTE) project and any amendments, updates and / or revisions (e.g., LTE-Advanced project, ultramobile broadband (UMB) project (also known as "3GPP2"), etc.). Broadband Wireless Access (BWA) networks compatible with IEEE 802.16 are generally referred to as WiMAX networks, which stands for Worldwide Interoperability for Microwave Access, and is a certification mark for products that have passed compliance and interoperability testing of the IEEE 802.16 standard. The communication device 1312 can operate according to the Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications Syste (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA) or LTE network.The communication device 1312 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication device 1312 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO) and its derivatives, and any other wireless protocols designated as 3G, 4G, 5G and above. In other embodiments, the communication device 1312 may operate in accordance with other wireless protocols. The computing device 1300 may include an antenna 1322 to facilitate wireless communications and / or receive other wireless communications (e.g., radio frequency transmissions). The computing device 1300 may include a receiver circuit and / or a transmitter circuit. In some embodiments, the communication device 1312 may manage wired communications, such as electrical, optical, or any other appropriate communication protocol (e.g., Ethernet). As described above, the communication device 1312 may include multiple communication chips. For example, the first communication device 1312 may be dedicated to shorter-range wireless communications, such as Wi-Fi or Bluetooth, while the second communication device 1312 may be dedicated to longer-range wireless communications, such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO or other. In some embodiments, the first communication device 1312 may be dedicated to wireless communications, while the second communication device 1312 may be dedicated to wired communications.

[0114] Computing device 1300 may include power supply / power circuitry 1314. Power supply / power circuitry 1314 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of computing device 1300 to an energy source (e.g., DC power, AC power, etc.) separate from computing device 1300.

[0115] Computing device 1300 may include a display device 1306 (or corresponding interface circuitry, as described above). Display device 1306 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.

[0116] Computing device 1300 may include audio output device 1308 (or corresponding interface circuitry, as described above). Audio output device 1308 may include any device that generates audible indications, such as a speaker, headphones, or earbuds.

[0117] Computing device 1300 may include audio input device 1318 (or corresponding interface circuitry, as described above). Audio input device 1318 may include any device that generates a signal representing sound, such as a microphone, a microphone array, or a digital musical instrument (e.g., a musical instrument with a musical instrument digital interface (MIDI) output).

[0118] Computing device 1300 may include GPS device 1316 (or corresponding interface circuitry, as described above). GPS device 1316 may communicate with a satellite-based system and may receive the location of computing device 1300, as is known in the art.

[0119] The computing device 1300 may include a sensor 1330 (or one or more sensors). The computing device 1300 may include corresponding interface circuits, as described above. The sensor 1330 may sense physical phenomena and convert the physical phenomena into electrical signals that can be processed by, for example, the processing device 1302. Examples of the sensor 1330 may include: a capacitive sensor, an inductive sensor, a resistive sensor, an electromagnetic field sensor, a light sensor, a camera, an imager, a microphone, a pressure sensor, a temperature sensor, a vibration sensor, an accelerometer, a gyroscope, a strain sensor, a moisture sensor, a humidity sensor, a distance sensor, a range sensor, a time of flight sensor, a pH sensor, a particle sensor, an air quality sensor, a chemical sensor, a gas sensor, a biosensor, an ultrasonic sensor, a scanner, and the like.

[0120] Computing device 1300 may include additional output devices 1310 (or corresponding interface circuitry, as described above). Examples of other output devices 1310 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, a tactile output device, a gas output device, a vibration output device, a lighting output device, a home automation controller, or an additional storage device.

[0121] The computing device 1300 may include additional input devices 1320 (or corresponding interface circuits, as described above). Examples of other input devices 1320 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device (e.g., a mouse), a stylus, a touchpad, a barcode reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0122] The computing system 1300 may have any desired form factor, such as a handheld or mobile computing system (e.g., a cellular phone, a smart phone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, a personal digital assistant (PDA), an ultra-mobile personal computer, a remote control, a wearable device, headwear, glasses, footwear, electronic clothing, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, an Internet of Things device, or a wearable computing system. In some embodiments, the computing device 1300 may be any other electronic device that processes data.

[0123] Select Example

[0124] Example 1 provides a method, comprising inputting one or more frame-level inputs associated with a sequence of video frames into a temporal convolutional network; generating one or more frame-level features based on the one or more frame-level inputs using the temporal convolutional network; generating a graph including one or more nodes and one or more edges based on the one or more frame-level features, wherein a node corresponds to a video frame and an edge connecting two nodes represents a connection between frame-level features of two video frames; inputting the graph into a graph neural network; and generating one or more predictions for one or more nodes of the graph using the graph neural network.

[0125] Example 2 provides a method as described in Example 1, wherein the one or more frame-level inputs associated with a sequence of video frames include a first frame-level input, which includes a first feature vector extracted from a first frame in the sequence of video frames; and a second frame-level input, which includes a second feature vector extracted from a second frame in the sequence of video frames.

[0126] Example 3 provides a method as described in Example 1 or 2, wherein the one or more frame-level inputs associated with the sequence of frames include a first frame-level input, which includes a first frame in the sequence of video frames; and a second frame-level input, which includes a second frame in the sequence of video frames.

[0127] Example 4 provides a method as described in any of Examples 1-3, further comprising inputting the sequence of video frames into a three-dimensional convolutional neural network; and generating the one or more frame-level inputs using the three-dimensional convolutional neural network.

[0128] Example 5 provides a method as described in any of Examples 1-4, wherein generating the one or more frame-level features based on the one or more frame-level inputs includes normalizing feature dimensions of the one or more frame-level inputs.

[0129] Example 6 provides a method as described in any of Examples 1-5, wherein generating the one or more frame-level features based on the one or more frame-level inputs includes applying one or more dilated convolutions having one or more dilation rates.

[0130] Example 7 provides a method as described in any of Examples 1-6, wherein the one or more frame-level features are generated by the penultimate layer of the temporal convolutional network.

[0131] Example 8 provides a method as described in any of Examples 1-7, wherein generating the graph further comprises generating the graph based on the one or more frame-level inputs.

[0132] Example 9 provides a method as described in any one of Examples 1-8, wherein generating the graph includes comparing a first frame-level feature corresponding to a first video frame with a second frame-level feature corresponding to a second video frame; and based on the comparison, connecting a first node in the graph corresponding to the first video frame with a second node in the graph corresponding to the second video frame using a first edge.

[0133] Example 10 provides a method as described in any of Examples 1-9, wherein generating the graph includes comparing a first frame-level feature corresponding to a first video frame with a second frame-level feature corresponding to a second video frame; connecting a first node in the graph corresponding to the first video frame and a second node in the graph corresponding to the second video frame using a first edge; and assigning a first value to the first edge based on the comparison.

[0134] Example 11 provides a method as described in any of Examples 1-10, wherein generating the one or more predictions includes performing a message aggregation operation on the graph.

[0135] Example 12 provides an apparatus comprising one or more processors; and one or more storage devices storing a machine learning model, the machine learning model having processing operations performed by the one or more processors, the machine learning model comprising a temporal convolutional network to receive one or more frame-level inputs associated with a sequence of video frames and output one or more frame-level features; a converter to generate a graph comprising nodes and one or more edges based on the one or more frame-level features, wherein the nodes correspond to video frames and the edges connecting two nodes represent connections between frame-level features of the two video frames; and a graph neural network to receive the graph and output one or more predictions for the nodes.

[0136] Example 13 provides an apparatus as described in Example 12, wherein the machine learning model also includes a three-dimensional convolutional neural network to receive the sequence of video frames and output the one or more frame-level inputs.

[0137] Example 14 provides an apparatus as described in Example 12 or 13, wherein the temporal convolutional network includes one or more prediction stages to process the one or more frame-level inputs; and one or more refinement stages to process the output of the prediction stage.

[0138] Example 15 provides an apparatus as described in any of Examples 12-14, wherein the temporal convolutional network includes one or more convolution operators to receive one or more of the frame-level inputs, the convolution operator having a kernel size of 1x1.

[0139] Example 16 provides an apparatus as described in any of Examples 12-15, wherein the temporal convolutional network includes multiple dilated convolutional layers, each layer including multiple processing blocks.

[0140] Example 17 provides an apparatus as described in Example 16, wherein a processing block among the multiple processing blocks of a dilated convolution layer among the multiple dilated convolution layers includes an input path; a residual path, which includes a dilated convolution block with a dilation rate; an activation block to process the output of the dilated convolution block; and a convolution block to process the output of the activation block, the convolution block having a kernel size of 1x1; and an adder block to sum the output of the input path and the output of the residual path.

[0141] Example 18 provides an apparatus as described in Example 17, wherein the expansion rate depends on the expanded convolution layer in which the processing block is located.

[0142] Example 19 provides an apparatus as described in Example 16, wherein a processing block among the multiple processing blocks of a dilated convolution layer among the multiple dilated convolution layers includes an input path; a residual path, which includes a first dilated convolution block with a first dilation rate; a second dilated convolution block with a second dilation rate; a concatenation block to concatenate the output of the first dilated convolution block and the output of the second dilated convolution block; an activation block to process the output of the concatenation block; and a convolution block to process the output of the activation block, the convolution block having a kernel size of 1x1; and an adder block to sum the output of the input path and the output of the residual path.

[0143] Example 20 provides an apparatus as described in Example 19, wherein the first expansion rate depends on the expanded convolution layer in which the processing block is located.

[0144] Example 21 provides an apparatus as described in Example 19 or 20, wherein the second expansion rate depends on the expanded convolutional layer in which the processing block is located.

[0145] Example 22 provides an apparatus as described in any of Examples 19-21, wherein the first expansion rate is different from the second expansion rate.

[0146] Example 23 provides an apparatus as described in any of Examples 12-22, wherein the temporal convolutional network comprises a plurality of layers and the one or more frame-level features are generated by the penultimate layer of the temporal convolutional network.

[0147] Example 24 provides an apparatus as described in any of Examples 12-23, wherein the machine learning model also includes a fusion block to receive and fuse the one or more frame-level inputs and the one or more frame-level features, and the converter also receives an output of the fusion block.

[0148] Example 25 provides an apparatus as described in any of Examples 12-24, wherein the graph has one or more leading edges and one or more trailing edges.

[0149] Example 26 provides an apparatus as described in any of Examples 12-25, wherein the graph has one or more forward edges, one or more backward edges, and one or more undirected edges.

[0150] Example 27 provides an apparatus as described in any of Examples 12-26, wherein the edges of the graph encode values ​​representing a comparison of frame-level features between two video frames.

[0151] Example 28 provides an apparatus as described in any of Examples 12-27, wherein: the graph includes one or more time-hopping connections, wherein the time-hopping connection connects two nodes that are separated by at least one timestamp.

[0152] Example 29 provides an apparatus as described in Example 28, wherein: the size of the time skip connection is a user-adjustable hyperparameter.

[0153] Example 30 provides one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: process one or more frame-level inputs associated with a sequence of video frames by a temporal convolutional network; extract one or more frame-level features by the temporal convolutional network; generate a graph comprising one or more nodes and one or more edges based on the one or more frame-level features, wherein a node corresponds to a video frame and an edge connecting two nodes represents a connection between frame-level features of two video frames; process the graph by a graph neural network; and generate one or more predictions for the nodes by the graph neural network.

[0154] Example 31 provides one or more non-transitory computer-readable media as described in Example 30, wherein the instructions, when executed by the one or more processors, cause the one or more processors to also perform any one of the methods provided in Examples 2-11.

[0155] Example 32 provides an apparatus comprising a device for performing or for performing any of the computer-implemented methods provided in Examples 1-11.

[0156] Changes and other notes

[0157] The above description of the illustrated implementations of the present disclosure, including those described in the Abstract, is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Although specific implementations of the present disclosure and examples thereof are described herein for illustrative purposes, as will be appreciated by those skilled in the relevant art, various equivalent modifications are possible within the scope of the present disclosure. These modifications may be made to the present disclosure in accordance with the above detailed description.

[0158] For purposes of illustration, specific numbers, materials, and configurations are described to provide a thorough understanding of the illustrative implementations. However, it will be apparent to those skilled in the art that the present disclosure may be implemented without the specific details, and / or may be implemented with only some of the described aspects. In other cases, well-known features are omitted or simplified so as not to obscure the illustrative implementations.

[0159] In addition, reference is made to the accompanying drawings, which form a part of this document, in which embodiments that can be implemented are shown by way of illustration. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be understood in a limiting sense.

[0160] Various operations may be described in turn as a plurality of discrete actions or operations in a manner that is most helpful for understanding the disclosed subject matter. However, the order of description should not be interpreted as meaning that these operations must be dependent on the order. Specifically, these operations may not be performed in the order presented. The operations described may be performed in an order different from the described embodiments. Various additional operations may be performed or the operations described may be omitted in additional embodiments.

[0161] For purposes of this disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B or C" or the phrase "A, B and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). The term "between" when used in reference to a measurement range includes both ends of the measurement range.

[0162] The specification uses the phrases "in one embodiment" or "in an embodiment", each of which refers to one or more identical or different embodiments. The terms "including", "comprising", "having", etc., used for the embodiments of the present disclosure, are synonymous. The present disclosure may use perspective-based descriptions, such as "above", "below", "top", "bottom", and "side", to illustrate various features of the drawings, but these terms are only for ease of discussion and do not imply a desired or required orientation. The drawings are not necessarily drawn to scale. Unless otherwise indicated, the use of ordinal adjectives "first", "second", and "third", etc. to describe common objects simply indicates that different instances of similar objects are referenced, and is not intended to imply that the objects described in this way must be in a given sequence in time, space, ranking, or in any other way.

[0163] In the detailed description that follows, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0164] The terms "substantially," "close," "approximately," "near," and "approximately" generally refer to within + / - 20% of a target value as described herein or known in the art. Similarly, terms indicating the orientation of various elements, such as, for example, "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between elements, generally refer to within + / - 5-20% of a target value as described herein or known in the art.

[0165] Furthermore, the terms "comprises," "includes," "has," or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, or apparatus. Additionally, the term "or" refers to an inclusive "or" rather than an exclusive "or."

[0166] The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for all of the desirable attributes disclosed herein. The details of one or more implementations of the subject matter described in this specification are set forth in the specification and the drawings.

Claims

1. A method comprising: inputting one or more frame-level inputs associated with a sequence of video frames into a temporal convolutional network; generating, using the temporal convolutional network, one or more frame-level features based on the one or more frame-level inputs; generating a graph including one or more nodes and one or more edges based on the one or more frame-level features, wherein a node corresponds to a video frame, and an edge connecting two nodes represents a connection between frame-level features of two video frames; Inputting the graph into a graph neural network; and Using the graph neural network, one or more predictions are generated for the one or more nodes of the graph.

2. The method of claim 1, wherein: The one or more frame-level inputs associated with a sequence of video frames include: a first frame-level input comprising a first feature vector extracted from a first frame in the sequence of video frames; and A second frame-level input includes a second feature vector extracted from a second frame in the sequence of video frames.

3. The method according to claim 1 or 2, wherein: The one or more frame-level inputs associated with a sequence of video frames include: a first frame-level input comprising a first frame in a sequence of video frames; and A second frame level input comprises a second frame in the sequence of video frames.

4. The method according to claim 1 or 2, further comprising: Inputting the sequence of video frames into a three-dimensional convolutional neural network; and The one or more frame-level inputs are generated using the three-dimensional convolutional neural network.

5. The method according to claim 1 or 2, wherein: Generating the one or more frame-level features based on the one or more frame-level inputs comprises: The feature dimensions of the one or more frame-level inputs are normalized.

6. The method according to claim 1 or 2, wherein: Generating the one or more frame-level features based on the one or more frame-level inputs comprises: Apply one or more dilated convolutions with one or more dilation rates.

7. The method according to claim 1 or 2, wherein: The one or more frame-level features are generated by the penultimate layer of the temporal convolutional network.

8. The method according to claim 1 or 2, wherein: Generating the graph further comprises: The map is further generated based on the one or more frame-level inputs.

9. The method according to claim 1 or 2, wherein: Generating the graph includes: comparing a first frame-level feature corresponding to the first video frame with a second frame-level feature corresponding to the second video frame; and Based on the comparison, a first node in the graph corresponding to the first video frame and a second node in the graph corresponding to the second video frame are connected using a first edge.

10. The method according to claim 1 or 2, wherein: Generating the graph includes: comparing a first frame-level feature corresponding to the first video frame with a second frame-level feature corresponding to the second video frame; connecting a first node in the graph corresponding to the first video frame and a second node in the graph corresponding to the second video frame using a first edge; and A first value is assigned to the first edge based on the comparison.

11. The method according to claim 1 or 2, wherein: Generating the one or more predictions comprises: A message aggregation operation is performed on the graph.

12. An apparatus comprising: one or more processors; as well as one or more storage devices storing a machine learning model having processing operations performed by the one or more processors, the machine learning model comprising: a temporal convolutional network for receiving one or more frame-level inputs associated with a sequence of video frames and outputting one or more frame-level features; a converter for generating a graph including nodes and one or more edges based on the one or more frame-level features, wherein a node corresponds to a video frame, and an edge connecting two nodes represents a connection between the frame-level features of the two video frames; and A graph neural network is configured to receive the graph and output one or more predictions for the nodes.

13. The device of claim 12, wherein: The machine learning model also includes: A three-dimensional convolutional neural network is used to receive the sequence of video frames and output the one or more frame-level inputs.

14. The device according to claim 12 or 13, wherein: The temporal convolutional network comprises: one or more prediction stages for processing the one or more frame-level inputs; and One or more refinement stages for processing the output of the prediction stage.

15. The device according to claim 12 or 13, wherein: The temporal convolutional network comprises: One or more convolution operators, for receiving one or more of the frame-level inputs, the convolution operator having a kernel size of 1x1.

16. The device according to claim 12 or 13, wherein: The temporal convolutional network comprises: Multiple dilated convolutional layers, each layer consists of multiple processing blocks.

17. The device of claim 16, wherein: A processing block in a plurality of processing blocks of a dilated convolutional layer in the plurality of dilated convolutional layers comprises: Input path; The residual path includes: Dilated convolutional blocks with dilation rate; an activation block for processing the output of the dilated convolution block; and a convolution block for processing the output of the activation block, the convolution block having a kernel size of 1x1; and An adder block is configured to sum the output of the input path and the output of the residual path.

18. The device of claim 17, wherein: The dilation rate depends on the dilated convolutional layer where the processing block is located.

19. The device of claim 16, wherein: A processing block in a plurality of processing blocks of a dilated convolutional layer in the plurality of dilated convolutional layers comprises: Input path; The residual path includes: a first dilated convolutional block having a first dilation rate; a second dilated convolutional block having a second dilation rate; A concatenation block, used to concatenate the output of the first dilated convolution block and the output of the second dilated convolution block; an activation block for processing the output of the concatenation block; and a convolution block for processing the output of the activation block, the convolution block having a kernel size of 1x1; and An adder block is configured to sum the output of the input path and the output of the residual path.

20. The device of claim 19, wherein: The first dilation rate depends on the dilated convolutional layer in which the processing block is located.

21. The apparatus of claim 19, wherein: The second dilation rate depends on the dilated convolutional layer in which the processing block is located.

22. The apparatus of claim 19, wherein: The first expansion rate is different from the second expansion rate.

23. The device of claim 12 or 13, wherein: The temporal convolutional network comprises a plurality of layers, and the one or more frame-level features are generated by a penultimate layer of the temporal convolutional network.

24. The device of any one of claims 12 to 23, wherein: The machine learning model also includes a fusion block for receiving and fusing the one or more frame-level inputs and the one or more frame-level features, and the converter is also used to receive an output of the fusion block.

25. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: Processing one or more frame-level inputs associated with a sequence of video frames using a temporal convolutional network; Extracting one or more frame-level features using the temporal convolutional network; A graph including one or more nodes and one or more edges is generated based on the one or more frame-level features, wherein: The nodes correspond to video frames, and the edge connecting two nodes represents the connection between the frame-level features of the two video frames; processing the graph using a graph neural network; and One or more predictions are generated for the one or more nodes using the graph neural network.

Images