Temporal self-similarity matrix based class-agnostic repeated counting in videos

Abstract

Techniques are disclosed that enable the use of recurrent neural networks to process videos that capture periodic activity to generate periodic outputs (e.g., a period length of the periodic activity captured in the video and / or a frame-by-frame periodic indication of the video that captures periodic activity). Various implementations include a class-agnostic recurrent network that can be used to generate periodic outputs for a variety of periodic activities. Additional or alternative implementations include generating synthetic recurrent videos that can be used to train the recurrent network.

Description

Background Technology

[0001] Repetitive actions and processes are ubiquitous in everyday life. A person chewing food while stirring sugar in a cup of coffee and tapping food to background music is simultaneously performing at least three periodic activities. Periodic activities can be found in biological processes, manufacturing processes, human activities (e.g., exercise), object manipulation, and / or additional or alternative activities. Given the ubiquity of repetition, understanding repetition in video can be helpful to a variety of computational systems. For example, identifying repetition is useful for systems because (1) there is usually an intention or driving reason behind something happening multiple times; (2) the same event can be observed again, but with subtle variations; (3) these repetitions may lead to gradual changes in the scene; (4) they can provide well-defined units of action (e.g., temporally segmentable subsequences of action); and / or for other or alternative reasons. Summary of the Invention

[0002] The techniques disclosed herein relate to using repetitive networks to process video capturing periodic activities to generate periodic outputs, including (a) the period length of the periodic activity and / or (b) per-frame periodicity classification (e.g., an indication of whether a periodic activity was captured in a particular frame of the video). In some implementations, the repetitive network can be used to generate periodic outputs in a class-agnostic manner. In other words, the same repetitive network can be used to generate periodic outputs for a variety of periodic activities. For example, the repetitive network can be used to generate periodic outputs based on video capturing a human jumping firecrackers (i.e., a first-class periodic activity), and the same repetitive network can be used to generate periodic outputs based on additional video capturing bird wing flapping (e.g., a second-class periodic activity).

[0003] In some implementations, the repetition network may include an encoder part, which can be used to generate an coded representation of the video. In some of these implementations, the coded representation of the video may include the coded representation of each frame of the video. A temporal self-similarity matrix can be generated based on the coded representation of the video. In some implementations, the self-similarity matrix may represent shared parameterization from which the number of repetitions of an activity can be determined. For example, the temporal self-similarity matrix can be generated by determining the pairwise similarity between all pairs of coded video frames. Additionally or alternatively, the periodic predictor model part of the repetition network can be used to process the temporal self-similarity matrix to generate a per-frame period length estimate and / or a per-frame binary periodicity classification.

[0004] According to the implementations described herein, additional or alternative implementations are for generating synthetic repeating videos that can be used to train a repeating network. In some implementations, synthetic repeating videos can be generated based on unlabeled video. In some implementations, synthetic repeating videos can be generated based on video captured using an image sensor (e.g., a camera and / or additional or alternative image sensors) and / or based on synthetically generated video generated in a simulation. A portion of the unlabeled video (e.g., clips, video frame sequences, etc.) can be selected. The selected portion can be repeated to generate repeating active portions of the synthetic repeating video. Additional or alternatively, the selected portion can be reversed, and the selected portion and the reversed selected portion can be repeated to simulate an action that is also performed in reverse in the synthetic repeating video (e.g., jumping with firecrackers). In some implementations, additional portions of the video can be prepended to the beginning of the synthetic repeating video (e.g., one or more frames immediately preceding the selected clip can be prepended to the beginning of the synthetic repeating video). Additional or alternatively, one or more other portions of the video can be appended to the synthetic training video (e.g., one or more frames immediately following the selected clip can be appended to the end of the synthetic repeating video).

[0005] Therefore, various implementations illustrate techniques for training class-agnostic repeating networks that can be used to generate periodic outputs by processing videos capturing any periodic activity. In contrast, conventional techniques require training separate networks to generate periodic outputs for each type of periodic activity. For example, in conventional techniques, one network is trained to generate periodic outputs by capturing videos of people doing push-ups, and a second network is trained to generate periodic outputs by capturing videos of birds flying. The technique described in this paper can use the same network for any type of periodic activity captured in the training video. Training a single model instead of multiple models saves computational resources (e.g., processor cycles, memory, battery power, etc.).

[0006] Additionally or alternatively, various implementations illustrate techniques for generating synthetic training data from unlabeled videos and using this synthetic training data to train repetitive models. Synthetic training data can be generated without requiring a human reviewer to identify one or more portions of the training video capturing periodic activities. Large amounts of synthetic training data can be generated quickly and inexpensively, capturing a wide variety of periodic activities. In some cases, based on synthetic training data capturing a broader range of periodic activities, repetitive models can be trained faster (i.e., with fewer training instances). In other words, the diversity of periodic activities captured in training instances can be used to train repetitive models with fewer training instances.

[0007] The above description is provided only as an overview of some implementations disclosed herein. These and other implementations of the technology are disclosed in more detail below.

[0008] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. Attached Figure Description

[0009] Figure 1 The illustration shows examples of using repeating networks to process video frame sequences to generate period lengths and / or periodic classifications according to various implementations disclosed herein.

[0010] Figure 2 The illustration shows example environments in which various implementation methods disclosed in this article can be implemented.

[0011] Figure 3 The illustration shows synthetic repeating video sequences based on various implementations disclosed herein.

[0012] Figure 4A This is a flowchart illustrating an example process for generating synthetic training data according to various implementations disclosed in this paper.

[0013] Figure 4B This is a flowchart illustrating another example process for generating synthetic training data according to the various implementations disclosed in this paper.

[0014] Figure 5 This is a flowchart illustrating an example process of training a repeating network according to various implementations disclosed in this paper.

[0015] Figure 6 This is a flowchart illustrating an example process of generating the period length of periodic activities and / or the periodic classification per frame using a repeating network according to various implementations disclosed herein.

[0016] Figure 7 The illustration shows an example architecture of a computing device. Detailed Implementation

[0017] The need to understand periodic videos is pervasive. Videos of biological processes, manufacturing processes, people exercising, and objects being manipulated are just a few examples, where each domain would benefit if it could automatically estimate the counts and frequencies of repetitions in the videos. The technique described in this paper addresses a method for estimating the periodicity of actions repeated in videos. In some implementations, the key to this method is constraining the periodicity prediction module to use temporal self-similarity as an intermediate representation bottleneck, which allows generalization to repetitions not visible in field videos. In other implementations, the model can be trained using a synthetic dataset generated from a large collection of unlabeled videos by sampling short clips of varying lengths and repeating them with different periods and counts. This combination of synthetic data and a robust yet constrained model allows for the prediction of periods in a class-agnostic manner.

[0018] Imagine the most mundane scene—a person eating alone in a café. They might be stirring sugar in their coffee while chewing their food, and tapping their feet to background music. This person is performing at least three cyclical activities simultaneously. Repetitive actions and processes are ubiquitous in our daily lives. In fact, biological life is driven by many repetitive phenomena that occur like clockwork in a cell. Given the pervasiveness of repetition, a video repetition understanding module could benefit any perceptual system designed to observe our world over extended periods.

[0019] Repetition can be interesting for several reasons: (1) there may be an intention or driving force behind something happening multiple times; (2) the same event can be observed again, but with subtle variations; (3) these repetitions may lead to gradual changes in the scene; and (4) they may provide us with explicit units of action, i.e., subsequences of actions that can be segmented over time (e.g., if you are cutting onions, the unit of action is the manipulation action repeated to produce additional slices). For these reasons, some agents that interact with the world may benefit from such a system. Furthermore, repetition counting may be relevant to many computer vision applications, such as counting the number of times exercise is performed, measuring biological events (such as heartbeats), etc.

[0020] However, research into understanding periodic videos has been limited, likely due to a lack of large, labeled datasets of repetitive videos. Action recognition benefits greatly from the availability of dynamically scaled datasets. One aspect enabling the large-scale collection of such data is the availability of video-related keywords / text. Unfortunately, videos are rarely annotated with notes related to repetitive activities, as text is more likely to describe semantic content. For this reason, the technique described in this paper involves generating a large dataset of repetitions in videos and labeling it with the number of repetitions present in each video.

[0021] However, due to the fine-grained temporal nature of the task, labeling videos using repetition counting can be tedious and expensive. To increase the amount of training data, the techniques described in this paper involve creating synthetic repeating videos by repeating clips from existing videos with different periods. Since we are creating these videos, we also annotate the periods and number of repetitions in the videos, which can be used to train models using supervised learning. However, such synthetic videos may fail to capture all the nuances of real repeating videos and may be prone to overfitting by large-capacity deep learning models. To address this issue, a data augmentation strategy for synthetic videos is proposed so that models trained on them can be transferred to real videos with repetitions.

[0022] In some implementations, a single model can be used for many categories of periodic videos, and indeed, for video categories not seen during training. In some implementations, this is achieved by using an intermediate representation that encourages generalization to unseen classes. This representation—a temporal self-similarity matrix—can be used to predict the periodicity of actions repeated in a video. This temporal self-similarity matrix representation can be used on different types of repetitive videos, thus achieving the desired generalization. For example, whether it's a person doing push-ups or a child swinging on a playground, the self-similarity matrix is ​​a shared parameter from which the number of repetitions can be inferred. In some implementations, this (extreme) bottleneck (reducing the number of channels in the feature map from 512 to 1) can also help generalize from synthetic data to real data. Another advantage of this representation is that model interpretability can be incorporated into the network architecture, as the network predicts the periodicity only from the self-similarity matrix, rather than inferring it from underlying high-dimensional features.

[0023] Some implementations can focus on (i) repetition counting, which identifies the number of repetitions in a video. This problem can be reformulated as first estimating the period length of each frame and then converting them into repetition counts. Alternatively, some implementations can focus on (ii) periodicity detection, which identifies whether the current frame is part of a repetitive temporal pattern. This can be treated as a per-frame binary classification problem.

[0024] Extracting periodicity (e.g., for periodic motion detection) and periodicity by leveraging autocorrelation in time series is a well-studied problem. Periodicity estimation in videos is performed using periodograms based on autocorrelation or wavelet transforms of handcrafted features derived from optical flow. Extracted periodic motion supports multiple tasks, including 3D reconstruction and bird classification. Periodicity has been used in various applications, including temporal pattern classification.

[0025] Temporal self-similarity matrices can be useful representations for human action recognition and gait analysis because they are robust to large viewpoint changes when paired with appropriate feature representations. Unsupervised methods have been proposed to identify periodic segments in videos using specialized filters based on self-similarity matrices derived from improved dense trajectories. Unlike these methods, temporal self-similarity matrices (TSMs) can be intermediate layers in neural network architectures, acting as information bottlenecks, according to some implementations disclosed in this paper.

[0026] The use of synthetic training data in computer vision is becoming increasingly common. Pasting object patches onto real images has proven effective as training data for object detection and human pose estimation. Mixing multiple videos or images together is extremely useful for generating synthetic training data for specific tasks and for standardizing deep learning models. When synthetic data for training repetition counting was first proposed, it introduced a dataset of synthetic repetition patterns and used it to train a deep learning-based counting model. However, the data used for training consisted of hand-designed random patterns that did not appear realistic. These patterns might not be diverse enough to capture all the nuances of repetition in real videos. Instead, the implementation disclosed in this paper creates a synthetic training dataset of real video repetitions from existing video datasets.

[0027] Counting objects and people in images is an active area in computer vision. However, video repeat counting has attracted less community attention in the era of deep learning. Existing techniques have been used to predict cycles (cycle lengths), although these techniques do not use TSM (Time-of-Sight Counting).

[0028] Repetition counting and periodicity detection can be temporally fine-grained tasks, such as temporal action localization, per-frame phase classification, and future prediction. In some implementations, interfaces previously used for collecting action localization datasets can be leveraged to create, for example, a repetition dataset called Countix. In some implementations, the range of periodic segments in the video and the number of repetitions within each segment can be labeled in the repetition dataset, rather than labeling semantic segments.

[0029] In some implementations, the repetitive network model architecture can consist of two learning components: an encoder and a periodic predictor, with a temporally self-similar layer between these components. In some implementations, it can be assumed that the video V = [v1, v2, ..., v...]. N The sequence consists of N frames. The latent embedding X = [x1, x2, ..., xn] for each frame can be obtained by feeding the video V to the encoder φ. N ] T That is, X = φ(V). Alternatively or additionally, the embedding X can be used to compute the pairwise similarity S between all embedding pairs. ijTo obtain the self-similar matrix S. Furthermore, in some implementations, S can be fed into a periodic predictor module that can output a period length estimate l = ψ(S) per frame and / or a binary periodicity classification p = τ(S) per frame.

[0030] In some implementations, the encoder may include three main components: (1) a convolutional feature extractor, (2) a temporal context, and (3) dimensionality reduction. In some implementations, the convolutional feature extractor may utilize the ResNet-50 architecture as our base convolutional neural network (CNN) to extract features from individual frames of the input video. i Extract 2D convolutional features. These frames can be 12×12×3 in size. In some implementations, the output of the conv4_block3 layer can be used to obtain larger spatial 2D feature maps. The size of each frame feature obtained can be 7×7×1024.

[0031] These convolutional features can be added to the features of each frame through layers of 3D convolution. In some implementations, 512 filters of size 3×3×3 with ReLU activation can be used. Temporal context can help model short-term motion and enable the model to distinguish between similar frames that capture different motions (e.g., a hand moving up or down during exercise).

[0032] Alternatively or additionally, the dimensionality of the extracted spatiotemporal features can be reduced, for example, by using global 2D max pooling on the spatial dimension, which can generate features corresponding to each frame v in the video. i The corresponding embedding vector x i Folding spatial dimensions can eliminate the need for tracking regions of interest as explicitly done in previous methods.

[0033] In obtaining each frame v i Potential embeddings x i Then, the embedding can be computed for x. i and x j All pairwise similarities S between ij =Sim(x) i ,x j We construct a self-similarity matrix S using the formula ), where Sim(·) is the similarity function. In some implementations, the negative of the squared Euclidean distance can be used as the similarity function, such that Sim(a,b) = -||ab||. 2 Then, a row-by-row softmax operation can be performed.

[0034] In some implementations, the TSM has a channel that can act as an information bottleneck in the middle of the network and can provide regularization. The TSM can also be interpretable, which can bring further insights into the predictions made by the model.

[0035] In some implementations, the final module of the repetitive network is a periodic predictor. This module can accept a self-similar matrix S = [s1, s2, ..., s...]. N ] T , where each row s i This is a self-similar representation of each frame. In some implementations, this module can generate a period length estimate l = ψ(S) for each frame and / or a binary periodic classification p = τ(S) for each frame. Both l and p can be vectors, and their corresponding elements can be the predictions for each frame (i.e., l...). i It is the prediction period length of the i-th frame.

[0036] It should be noted that the predictors ψ and τ can share a common architecture and weights until the final classification stage. In some implementations, the shared processing pipeline begins with 32 3×3 2D convolutional filters, followed by a transformer layer (e.g., multi-head attention, followed by a fully connected layer with learned positional embeddings). In some implementations, a four-head transformer with 512 dimensions can be used. After the shared pipeline, two classifiers can be used: a period length classifier ψ and a periodicity classifier τ. In some implementations, the period length classifier and / or the periodicity classifier can consist of two fully connected layers of size 512.

[0037] In some implementations, the periodic classifier τ outputs a periodic classification p for each frame. i Furthermore, it can be optimized using binary classification loss (e.g., binary cross-entropy). In some implementations, the period length estimator ψ outputs a period length estimate l per frame. i ∈L, where class is the discrete period length Where N is the number of input frames. Multi-class classification objectives (e.g., softmax cross-entropy) can be used to optimize the model. In some implementations, N = 64 can be used. In some of these implementations, to predict larger period lengths, different frame rates can be used to sample the input video, as described below.

[0038] In some implementations, periodic segments of the test video can be used to obtain an estimate of the period length per frame. The repetition count can be determined by dividing the number of frames with the average estimated period length by the test video. In some implementations, the model only accepts N = 64 frames of input. Therefore, computing a robust estimate of the period length per frame may require additional operations during inference: (1) multi-clip evaluation and / or (2) multi-velocity evaluation. In multi-clip evaluation, K = 15 windows of N = 64 frames can be sampled (e.g., randomly sampled) from the input video. The period length estimates from the K windows can be averaged to predict the number of repetitions of the test video. Additionally or alternatively, in some implementations, the model can predict period lengths up to 32. In multi-velocity evaluation, to cover longer period lengths, the input video can be sampled at different frame rates (e.g., the video can be played at speeds of 1×, 2×, 3×, 4×, etc.). The frame rate with the highest score can be selected for predicting the output.

[0039] In some implementations, supervised methods for periodic estimation involve collecting a large training set of periodic videos and annotating the start and end of each period in all repetitive actions. However, collecting such a dataset can be challenging due to the fine-grained nature of the task. In some implementations, a training strategy can be employed that utilizes repetitions synthesized in the field using unlabeled videos.

[0040] In some implementations, synthetic repeating videos can be created based on large sets of unlabeled videos. The advantage of using real videos to create synthetic data is that the generated training data is likely to be closer to real repeating videos compared to using synthetic modalities. Another advantage of using real videos is that using large datasets ensures that the model sees a vast diversity of data. This allows for the training of large, complex models capable of handling real repeating videos.

[0041] In some implementations, the synthetic periodic video generation pipeline may begin by sampling a random video V from a video dataset (e.g., a video dataset without any labels). Alternatively or additionally, clips C of random length P frames may be sampled from V. Clip C may be repeated K times (where K > 1) to simulate a video with repetition. In some implementations, the clips may be reversed and concatenated before repetition to simulate an action performed in reverse within that period (e.g., jumping on firecrackers, the motion of a yo-yo, etc.). In some implementations, the repeating frames may be prepended and / or appended with other non-repetitive segments from V, which may be exactly before and after C, respectively. The lengths of these non-repetitive segments may be randomly chosen and may be zero. This operation can include both periodic and non-repetitive segments in the generated video. Finally, a period length label P is assigned to each frame in the repeating portion of the generated video. Periodic labels may also be generated, indicating whether the frame is inside or outside the repeating portion of the generated video.

[0042] In some implementations, synthetic video generation can include camera motion enhancement. While predicting the view of an arbitrarily moving camera without knowing the 3D structure, occlusions, and lighting in the scene may be impractical, it can be approximated using affine image transformations. This can be considered by observing the affine motion of the camera on the video, which can include temporally smooth variations in rotation, translation, and scaling. In some implementations, the training loss decreases rapidly when training without these enhancements, but the model may not transfer to realistic, repetitive videos. In some implementations, camera motion enhancement can be a crucial part of training an effective model using synthetic videos.

[0043] In some implementations, to achieve camera motion enhancement, parameters for various motion types can be changed continuously over time as the video progresses. For example, rotation angles can change smoothly over time. This ensures that the video remains temporally consistent even after enhancement. In some implementations, temporal enhancement parameters can drive the viewframe for each motion type. This may result in less approximate repetition in repeating segments of the video.

[0044] In some implementations, this method can be implemented in TensorFlow. The encoder can be initialized using weights from ResNet-50 checkpoints pre-trained from ImageNet. In some implementations, the ADAM optimizer and a batch size of 5 videos (64 frames per video) can be used with a 6×10... -6 The learning rate is set to train the model for 400K steps.

[0045] Now turn to the attached diagram. Figure 1The illustration shows an example 100 of using a repetitive network to process a sequence of video frames to generate period length and / or periodicity classification according to various implementations disclosed herein. The video frame sequence 102 can be processed using encoder 104 to generate an encoded video frame sequence 106. In some implementations, the video frame sequence 102 may capture periodic activities (e.g., a bird flapping its wings, a person's heart beating, a human stirring sugar in a cup of coffee, etc.). In some of these implementations, the video frame sequence 102 may be captured directly using a visual sensor (e.g., a camera). Alternatively or additionally, the video frame sequence 102 may be generated by performing preprocessing on the original video (e.g., preprocessing the original video captured using a visual sensor). In some implementations, the original video may include multiple moving objects (e.g., the video may capture two or more people exercising, two or more horses galloping, a human exercising and a horse galloping, one or more additional or alternative periodic activities, and / or one or more additional non-periodic activities). For example, an image processing system can be used to preprocess the raw video to track individual objects in each video frame (e.g., a bird flapping its wings).

[0046] Encoder 104 can encode each frame of video frame sequence 102. In some implementations, encoder 104 may include a convolutional feature extractor, a temporal context part, a dimensionality reduction part, and / or one or more additional or alternative parts. For example, the convolutional neural network part of the encoder can be used to process video frame sequence 102 to generate two-dimensional convolutional features for each video frame. Alternatively, the three-dimensional convolutional neural network part of the encoder can be used to process the two-dimensional convolutional features for each video frame to generate temporal context features for each video frame. Furthermore, the max-pooling part of the encoder can be used to process the temporal context features for each video frame to generate encoded video frames.

[0047] A temporal self-similarity matrix generator 108 can be used to process the encoded video frame sequence 106 to generate a temporal self-similarity matrix 110. In some implementations, the temporal self-similarity matrix 110 is the pairwise similarity between pairs of encoded video frames in the encoded video frame sequence 106. For example, the temporal self-similarity matrix generator 108 can be used to generate the self-similarity matrix 110 by generating negative squared Euclidean distances between all pairs of encoded video frames in the encoded video frame sequence 106. In some of these implementations, generating the negative squared Euclidean distances between all pairs of encoded video frames can follow a line-by-line Softmax operation.

[0048] The periodic predictor model 112 can be used to process the temporal self-similarity matrix 110 to generate the period length of periodic activity captured in the video frame sequence 102, and / or a frame-by-frame periodic classification 114 indicating whether each video frame in the video frame sequence 102 captures periodic activity.

[0049] Figure 2 A block diagram of example environment 200 is shown, in which the implementations disclosed herein can be carried out. Example environment 200 includes computation system 202, which may include encoder engine 204, temporal self-similarity matrix engine 206, periodic predictor engine 208, and / or one or more additional engines (not depicted). Additionally or alternatively, computation system 202 may be associated with repeating network 210, training engine 212, training instances 214, training instance engine 216, and / or one or more additional or alternative components (not depicted).

[0050] In some implementations, computing system 202 may include user interface input / output devices (not depicted), which may include, for example, a physical keyboard, a touchscreen (e.g., implementing a virtual keyboard or other text input mechanism), a microphone, a camera, a display screen, and / or one or more speakers. The user interface input / output devices may be integrated with one or more computing systems 202 of a user. For example, a user's mobile phone may include user interface input / output devices; a standalone digital assistant hardware device may include user interface input / output devices; a first computing device may include one or more user interface input devices, and a separate computing device may include one or more user interface output devices; and so on. In some implementations, all or aspects of computing system 202 may be implemented on a computing system that also includes user interface input / output devices.

[0051] Some non-limiting examples of computing system 202 include one or more of the following: desktop computing devices, laptop computing devices, stand-alone hardware devices at least partially dedicated to automated assistants, tablet computing devices, mobile phone computing devices, vehicle computing devices (e.g., in-vehicle communication and entertainment systems, in-vehicle navigation systems), or wearable devices for users including computing devices (e.g., watches for users with computing devices, glasses for users with computing devices, virtual or augmented reality computing devices). Additional and / or alternative computing systems may be provided. Computing system 202 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by computing system 202 may be distributed across multiple computing devices. For example, computing programs running on one or more computers at one or more locations may be coupled to each other via a network.

[0052] like Figure 2 As shown, training instance engine 216 can be used to generate training instance 214. In some implementations, training instance engine 216 can be used to generate synthetic training instance 214 using unlabeled video. For example, training instance engine 216 can be based on the methods described herein. Figure 4A Process 4A and / or Figure 4B The process 4B generates synthetic training instances. In some implementations, the training instance engine 216 can be based on synthetic videos (e.g., here). Figure 3 The synthetic video described in the text generates training instance 214.

[0053] Training engine 212 can be used to train repetitive network 210. In some implementations, training engine 212 can process one or more training instances 214 to generate a training loss, wherein the training loss can be used to update one or more parts of repetitive network 210 (e.g., via backpropagation). For example, training instance 214 may include training videos that capture training periodic activities and basic fact periodic data, wherein the basic fact periodic data may include the period length of the training periodic activities, frame-by-frame periodicity indications of the training videos, and / or the number of repetitions of the periodic activities in the training videos.

[0054] Encoder engine 204 can be used to process a sequence of video frames using the encoder portion of repeating network 210 to generate an encoded sequence of video frames. For example, encoder engine 204 can be used to process video frame sequence 102 using encoder 104 to generate an encoded video frame sequence 106, such as... Figure 1 As shown.

[0055] The temporal self-similarity matrix engine 206 can be used to process encoded video frame sequences, such as those generated by encoder engine 204, to generate a temporal self-similarity matrix. In some implementations, the temporal self-similarity matrix can be the pairwise similarity of each pair of encoded video frames in the encoded sequence of video frames. As an illustrative example, the temporal self-similarity matrix engine 206 can use the temporal self-similarity matrix generator 108 to process the encoded video frame sequence 106 to generate a temporal self-similarity matrix 110, such as... Figure 1 As shown.

[0056] The periodic predictor engine 208 can be used with the periodic predictor model portion of the repetition network 210 to generate (a) the period length of periodic activity captured in a video frame sequence (i.e., a video frame sequence processed using encoder engine 204) and / or a frame-by-frame periodic classification of the video frame sequence. As an illustrative example, the periodic predictor engine 208 can be used to process the temporal self-similarity matrix 110 using the periodic predictor model 112 to generate pairwise length and / or frame-by-frame periodic classifications 114, such as... Figure 1As shown.

[0057] Figure 3 Examples of synthesized videos 302, 304, and 306 generated based on the original video sequence 300 are illustrated. In the illustrated example, the original video sequence 300 includes a sequence of eight video frames: “A”, “B”, “C”, “D”, “E”, “F”, “G”, and “H”.

[0058] The synthesized video sequence 302 is an example synthesized video sequence generated based on the original video sequence 300 by selecting three frames from the original video sequence "C", "D", and "E". The selected three frames from "C", "D", and "E" are repeated four times to represent that the activity is repeated three times. Furthermore, the two frames from the original videos "A" and "B" immediately preceding the selected three frames from "C", "D", and "E" are prepended to the beginning of the synthesized video sequence 302. Similarly, the two frames from the original videos "F" and "G" immediately following the selected three frames from "C", "D", and "E" are appended to the end of the synthesized video sequence 302. In some implementations, this can be done based on... Figure 4A The process 400 generates a synthetic video sequence 302.

[0059] The synthesized video sequence 304 is another sample synthesized video sequence generated based on the original video sequence 300 by selecting a three-frame sequence of "C", "D", and "E". Furthermore, the selected sequence is an inverted "D" and "C". The selected three-frame sequence and the inverted sequence "C", "D", "E", "D", "C" are repeated twice in the synthesized video sequence 304 to represent periodic activities, such as jumping with firecrackers, which are reversed during repetition. The two-frame portion of "A" and "B" immediately preceding the selected three-frame portion of "C", "D", and "E" is prepended to the beginning of the synthesized video sequence 304. Similarly, the two-frame portion of the original video "F" and "G" immediately following the selected three-frame portion of "C", "D", and "E" is appended to the end of the synthesized video sequence 304. In some implementations, it can be based on... Figure 4B The process 450 generates a synthetic video sequence 304.

[0060] Synthetic video sequence 304 is another sample synthetic video sequence generated based on the original video sequence 300. Synthetic video sequence 304 is generated by selecting a four-frame sequence of “C”, “D”, “E”, and “F”. Furthermore, the selected four-frame sequence is inverted to “E”, “D”, and “C”. The selected four-frame sequence and the inverted sequence are repeated four times in synthetic video sequence 306 to represent a periodic activity including sequence inversion (e.g., jumping firecrackers). The video frame “C” immediately preceding the selected four-frame sequence is prepended to the beginning of synthetic video sequence 306. Furthermore, the video frame “G” immediately following the selected four-frame sequence is appended to the end of synthetic video sequence 306. In some implementations, it can be based on… Figure 4B The process 450 generates a synthetic video sequence 306.

[0061] The original video sequence 300 and the synthesized video sequences 302, 304, and 306 are merely illustrative examples of synthesized repeating videos generated according to the implementation described herein. For example, additional or alternative video sequences (e.g., video sequences with fewer or more video frames) can be used as the original video sequence. One or more additional or alternative synthesized video sequences can be generated based on the original video sequence. For example, different numbers of video frames can be selected to represent periodic activities. Additionally or alternatively, different numbers of video frames (including no video frames) can be selected to prepend and / or append to the synthesized video.

[0062] Figure 4A This is a flowchart illustrating an example process 400 for generating synthetic training instances according to the implementation disclosed herein. For convenience, the operations of the flowchart are described with reference to the system performing the operations. This system may include various components of various computer systems, such as one or more components of computing system 202 and / or computing system 710. Furthermore, although the operations of process 400 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0063] In box 402, the system selects a video. In some implementations, the selected video is an unlabeled video. In some implementations, the selected video is a real video captured using one or more visual sensors (compared to a video generated in a simulation). For example, the selected video may include videos of a dog eating, a human heart beating, a butterfly flying, a robot performing a task, and / or additional or alternative activities.

[0064] In box 404, the system selects a portion of the video to represent periodic activity. For example, the system can select a single video frame, two video frames, three video frames, four video frames, and / or additional or alternative numbers of video frames.

[0065] In box 406, the system appends the selected portion of the video to the training video.

[0066] In box 408, the system determines whether to append additional cycles of the repetitive activity to the training video. For example, the system may determine whether the training video meets one or more conditions, such as whether the training video includes a repetition threshold, whether the training video exceeds a threshold size, and / or whether additional or alternative conditions are met. If the system determines to append additional cycles of the repetitive activity to the video, the system returns to box 406, appends additional instances of the selected portion of the video to the training video, and then returns to box 408. If not, the system proceeds to box 410.

[0067] In box 410, the system advances an appendix of the selected video to the beginning of the training video. In some implementations, the appendix of the selected video may be one or more video frames immediately preceding the selected video portion at box 404. Alternatively, the system may not advance any appendix of the selected video to the beginning of the training video.

[0068] In box 412, the system appends another portion of the selected video to the end of the training video. In some implementations, this other portion of the selected video may be one or more video frames immediately following the selected video portion at box 404. Alternatively, the system may not append any additional portion of the selected video to the end of the training video.

[0069] In box 414, the system stores the training video as a training instance and (1) the length of the selected portion of the video (e.g., the number of video frames in the selected portion of the video at box 404); (2) the number of repetitions of the selected portion of the video (e.g., the number of cycles appended to the training video at box 406); and / or (3) a frame-by-frame periodicity indication (e.g., whether a video frame in the training video is the selected portion of the video representing repeated activity at box 404, or whether a video frame is preceded by the training video at box 410 and / or appended to the training video at box 412).

[0070] In box 416, the system determines whether to generate additional training instances. In some implementations, the system may determine whether to generate additional training instances based on whether one or more conditions are met, such as whether a threshold number of training instances have already been generated, whether there are additional videos to select at box 402, and / or whether one or more additional or alternative conditions have been met. If the system determines to generate additional training instances, it returns to box 402 and selects the additional video before the video based on the additional selection enters boxes 404, 406, 408, 410, and 412. If the system determines not to generate additional training instances, the process ends.

[0071] Figure 4B This is a flowchart illustrating an additional example process 450 for generating synthetic training instances according to the implementation disclosed herein. For convenience, the operations of the flowchart are described with reference to the system performing the operations. This system may include various components of various computer systems, such as one or more components of computing system 202 and / or computing system 710. Furthermore, although the operations of process 450 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0072] In box 452, the process selects a video. In some implementations, the selected video is an unlabeled video. In some implementations, the selected video is a real video captured using one or more visual sensors (compared to a video generated in a simulation). For example, the selected video may include videos of a dog eating, a human heart beating, a butterfly flying, a robot performing a task, and / or additional or alternative activities.

[0073] In box 454, the system selects a portion of the video. For example, the system can select a single video frame, two video frames, three video frames, four video frames, and / or an additional or alternative number of video frames.

[0074] In box 456, the system appends the selected portion of the video to the training video.

[0075] In box 458, the system inverts the selected portion of the video and appends the inverted portion of the selected video to the training video.

[0076] In box 460, the system determines whether to append additional cycles of the repetitive and reversed activities to the training video. For example, the system may determine whether the training video meets one or more conditions, such as whether the training video includes a repetition threshold, whether the training video exceeds a threshold size, and / or whether additional or alternative conditions are met. If the system determines to append additional cycles of the repetitive activities to the video, the system returns to box 456 to append additional instances of the selected portion of the video to the training video, proceeds to box 458 to append the reversed portion of the selected portion of the video to the training video, and then returns to box 460. If not, the system proceeds to box 462.

[0077] In box 462, the system moves an additional portion of the video to the beginning of the training video. In some implementations, the additional portion of the selected video may be one or more video frames immediately preceding the selected video portion at box 454. Alternatively, the system may not move any additional portion of the selected video to the beginning of the training video.

[0078] In box 464, the system appends another portion of the video to the end of the training video. In some implementations, this other portion of the selected video may be one or more video frames immediately following the selected video portion at box 454. Alternatively, the system may not append any additional portion of the selected video to the end of the training video.

[0079] In box 466, the system stores the training video as a training instance and (1) the length of the selected portion of the video (e.g., the number of video frames in the selected portion of the video at box 405); (2) the number of repetitions of the selected portion of the video (e.g., the number of cycles appended to the training video at boxes 456 and 458); and / or (3) a frame-by-frame periodicity indication of the training video (e.g., whether a video frame in the training video is the video portion selected at box 454 representing repeated activity, or whether a video frame is prepended to the training video at box 462 and / or appended to the training video at box 464).

[0080] In box 468, the system determines whether to generate additional training instances. In some implementations, the system may determine whether to generate additional training instances based on whether one or more conditions are met, such as whether a threshold number of training instances have already been generated, whether there is an additional video to be selected at box 452, and / or whether one or more additional or alternative conditions are met. If the system determines to generate additional training instances, it returns to box 52 and selects the additional video before proceeding to boxes 454, 456, 458, 460, 462, and 464 based on the additional selected video. If the system determines not to generate additional training instances, the process ends.

[0081] Figure 5 This is a flowchart illustrating a process 500 for training a repeating network according to the implementation disclosed herein. For convenience, the operations of the flowchart are described with reference to the system performing the operations. This system may include various components of various computer systems, such as one or more components of computing system 202 and / or computing system 710. Furthermore, although the operations of process 500 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0082] In box 502, the system selects a training instance, including: a training video capturing periodic training activities and a basic fact periodic output. In some implementations, the basic fact periodic output may include (1) the period length of the periodic training activities, (2) the number of repetitions of the periodic training activities in the training video, and / or (3) a frame-by-frame periodicity indication of the training video. In some implementations, it may be based on... Figure 4A Process 400 or Figure 4B The process generates 450 training instances.

[0083] In box 504, the system uses the encoder portion of the repeating network to process the training video to generate an encoded sequence of video frames.

[0084] In box 506, the system generates a temporal self-similarity matrix based on the encoded sequence of video frames. In some implementations, the temporal self-similarity matrix represents the pairwise similarity of each pair of encoded video frames in the sequence. In some implementations, the temporal self-similarity matrix can be generated by determining a distance metric between each pair of encoded video frames. For example, the distance metric between each pair of encoded video frames can be an Euclidean distance metric, a squared Euclidean distance metric, a negative squared Euclidean distance metric, a Gaussian distance metric, and / or a cosine similarity metric. In some implementations, the temporal self-similarity matrix can be generated based on processing the distance metric for each pair of encoded video frames. For example, the temporal self-similarity matrix metric can be generated by processing the distance metric for each pair of encoded video frames using a row-by-row Softmax procedure.

[0085] In box 508, the system uses the periodic predictor model portion of the repeating network to process the temporal self-similarity matrix to generate the predicted periodic output. In some implementations, the predicted periodic output may include (1) the predicted period length of the periodic training activity, (2) the predicted number of repetitions of the periodic training activity in the training video, and / or (3) the predicted frame-by-frame periodicity indication of the training video.

[0086] In box 510, the system generates a loss based on (1) the predicted periodic output and (2) the periodic output portion of the basic facts of the training instances.

[0087] In box 512, the system updates one or more parts of the repeating network based on the generated loss (e.g., via backpropagation).

[0088] In box 514, the system determines whether to perform additional training on the repeated network. In some implementations, the system may determine to perform more training if one or more additional unprocessed training instances exist and / or if other criteria / standards have not yet been met. These other criteria / standards may include, for example, whether a threshold number of epochs has been reached and / or whether a threshold training duration has been reached. If the system determines to perform additional training, it returns to box 502, selects additional training instances, and proceeds to boxes 504, 506, 508, 510, and 512 based on the additional training instances. If not, the process ends. Although process 500 is described with respect to non-batch learning techniques, batch learning may be used additionally and / or alternatively.

[0089] Figure 6This is a flowchart illustrating a process 600, implemented according to the method disclosed herein, that uses a repeating network to process a sequence of video frames to generate (a) the period length of periodic activity captured in the video frame sequence and / or (b) a frame-by-frame periodic classification of the video frames. For convenience, the operations of the flowchart are described with reference to the system performing the operations. This system may include various components of various computer systems, such as one or more components of computing system 202 and / or computing system 710. Furthermore, although the operations of process 600 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0090] In box 602, the system uses the encoder portion of a repetitive network to process a sequence of video frames capturing periodic activity to generate an encoded video frame sequence. In some implementations, one or more image processing procedures can be used to preprocess the video frame sequence. For example, when multiple moving objects are captured in a video (e.g., a video capturing two people exercising, a video capturing two robots performing a task, etc.), image processing procedures can be used to track individual moving objects in the video frame sequence.

[0091] In box 604, the system generates a temporal self-similarity matrix based on the sequence of encoded video frames.

[0092] In box 606, the system uses the periodic predictor model portion of the repetition network to process the temporal self-similarity matrix to generate (a) the period length of the periodic activity and / or (b) a frame-by-frame periodicity classification of the video frame sequence. In some implementations, the system may determine the number of times the periodic activity repeats in the video frame sequence based on the period length of the periodic activity and / or based on the frame-by-frame periodicity classification of the video frame sequence.

[0093] Figure 7 This is a block diagram of an example computing device 710 that can be optionally used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client computing device and / or other components may include one or more components of the example computing device 710.

[0094] Computing device 710 typically includes at least one processor 714 that communicates with multiple peripheral devices via a bus subsystem 712. These peripheral devices may include a storage subsystem 724 (including, for example, a memory subsystem 725 and a file storage subsystem 726), a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices allow users to interact with computing device 710. The network interface subsystem 716 provides an interface to an external network and couples to corresponding interface devices in other computing devices.

[0095] User interface input device 722 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen integrated into a display, audio input devices such as a voice recognition system, a microphone, and / or other types of input devices. Generally, the term "input device" is used to encompass all possible types of devices and methods for inputting information onto computing device 710 or a communication network.

[0096] User interface output device 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (“CRT”), a flat panel device such as a liquid crystal display (“LCD”), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, for example, via an audio output device. Generally, the term “output device” is used to encompass all possible types of devices and methods for outputting information from computing device 710 to a user or another machine or computing device.

[0097] Storage subsystem 724 stores programs and data constructs that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include programs and data constructs that perform operations... Figure 4A , Figure 4B , Figure 5 and / or Figure 6 One or more processes and implementations Figure 2 The logic for the selection of various components described in the document.

[0098] These software modules are typically executed by processor 714 alone or in combination with other processors. The memory 725 used in storage subsystem 724 may include multiple memories, including main random access memory (“RAM”) 730 for storing instructions and data during program execution and read-only memory (“ROM”) 732 for storing fixed instructions. File storage subsystem 726 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing certain implementations of functionality may be stored by file storage subsystem 726 within storage subsystem 724 or in other machines accessible to processor 714.

[0099] The bus subsystem 712 provides a mechanism for enabling various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0100] The computing device 710 can be of various types, including workstations, servers, computing clusters, blade servers, server groups, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, to illustrate some implementation methods, Figure 7 The description of the computing device 710 depicted is intended only as a specific example. Many other configurations of the computing device 710 may have... Figure 7 The computing device depicted in the text has more or fewer components.

[0101] In situations where the systems described herein collect or may use personal information about users (or, generally referred to herein as "participants"), users may be given the opportunity to control whether a program or function collects user information (e.g., information about a user's social networks, social behaviors or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. Furthermore, certain data may be processed in one or more ways before being stored or used, thereby erasing personally identifiable information. For example, a user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized (e.g., down to the city, zip code, or state level) when geographic location information is obtained, making it impossible to determine the user's specific geographic location. Therefore, users can control how information about themselves is collected and / or used.

[0102] A method implemented by one or more processors is provided, the method comprising: processing a sequence of video frames capturing periodic activity using an encoder portion of a repetitive network to generate an encoded video frame sequence. The method further comprises: generating a temporal self-similarity matrix based on the encoded video frame sequence, the temporal self-similarity matrix indicating pairwise similarity between encoded video frames in the encoded video frame sequence. The method further comprises: processing the temporal self-similarity matrix using a periodic predictor model portion of the repetitive network to generate (a) the periodic length of the periodic activity in the video frame sequence and / or (b) a per-frame periodicity classification of the video frame sequence.

[0103] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0104] In some implementations, the method further includes processing an additional sequence of video frames capturing additional periodic activity using the encoder portion of the repeating network to generate an additional sequence of additional encoded video frames. The method further includes generating an additional temporal self-similarity matrix based on the encoded video frame additional sequence, the additional temporal self-similarity matrix indicating pairwise similarity between additional encoded video frames in the additional sequence of additional encoded video frames. The method further includes processing the additional temporal self-similarity matrix using a periodic predictor model of the repeating network to generate (a) the additional period length of the additional periodic activity in the video frame additional sequence and / or (b) an additional per-frame periodicity classification of the video frame additional sequence.

[0105] In some implementations, processing the video frame sequence capturing the periodic activity using the encoder portion of the repeating network to generate the encoded video frame sequence includes: for each video frame in the video frame sequence, processing the video frame using the convolutional neural network portion of the encoder to generate two-dimensional convolutional features of the video frame. The method further includes: processing the two-dimensional convolutional features of the video frame using the three-dimensional convolutional neural network portion of the encoder to generate temporal context features of the video frame. The method further includes: processing the temporal context features of the video frame using the max-pooling portion of the encoder to generate corresponding encoded video frames in the encoded video frame sequence.

[0106] In some implementations, generating the temporal self-similarity matrix, which indicates pairwise similarity between encoded video frames in the encoded video frame sequence, includes: for each pair of encoded video frames, determining a distance metric between the pair of encoded video frames in the encoded video frame sequence. The method further includes: for each pair of encoded video frames, generating the temporal self-similarity matrix based on the distance metric between the pair of encoded video frames. In some versions of those implementations, determining the distance metric between the pair of encoded video frames includes: for each pair of encoded video frames, determining an Euclidean distance, squared Euclidean distance, negative squared Euclidean distance, Gaussian distance, and / or cosine similarity metric between the pair of encoded video frames in the encoded video frame sequence. In some versions of those implementations, generating the temporal self-similarity matrix for each pair of encoded video frames based on the distance metric between the pair of encoded video frames includes: generating the temporal self-similarity matrix by processing the distance metric between the pair of encoded video frames using a row-by-row Softmax procedure for each pair of encoded video frames.

[0107] In some implementations, processing the temporal self-similarity matrix using the periodic predictor model to generate the periodicity length of the periodic activity includes: processing the temporal self-similarity matrix using a two-dimensional convolutional neural network portion to generate a convolutional output. The method further includes: processing the convolutional output using a transformer portion to generate a transformer output. The method also includes: processing the transformer output using a periodicity length classifier to generate the periodicity length of the periodic activity.

[0108] In some implementations, the video frame sequence that captures the periodic activity is a randomly sampled video frame sequence from the video that captures the periodic activity.

[0109] In some implementations, the repetitive network is trained before processing the video frame sequence capturing the periodic activity using the encoder portion of the repetitive network. Training the repetitive network includes, for each of a plurality of training instances, processing a training sequence of training video frame portions of training instances capturing periodic training activities using the encoder portion of the repetitive network to generate a training sequence of encoded training video frames. The method further includes generating a training temporal self-similarity matrix indicating pairwise similarity between encoded training video frames in the training sequence of encoded training video frames. The method further includes processing the training sequence of encoded training video frames using the periodic predictor model portion of the repetitive network to generate (a) the training period length of the training periodic activity and / or (b) a training per-frame periodic classification of the training sequence of video frames. The method further includes determining a loss based on (1) the training period length of the training periodic activity and / or the training per-frame periodic classification of the training sequence of video frames and (2) the corresponding ground truth portion of the training instances. The method further includes updating one or more portions of the repetitive network based on the determined loss. In some versions of those implementations, the training sequence of at least one training video frame portion of the training instance is a synthetic repeating video.

[0110] In some versions of those implementations, the synthetic repeating video is generated based on unlabeled video, and wherein generating the synthetic repeating video based on unlabeled video includes: selecting a sequence of video frames from the unlabeled video. The method further includes: for each of a plurality of iterations, generating the synthetic training video by appending the selected video frame sequence to the synthetic repeating video in the stated order, wherein the selected video frame sequence is the synthetic repeating video at the initial iteration. In some versions of those implementations, the synthetic repeating video is generated based on unlabeled video, and wherein generating the synthetic repeating video based on unlabeled video includes: selecting a sequence of video frames from the unlabeled video. The method further includes: for each of a plurality of iterations, generating the synthetic training video by appending the selected video frame sequence to the synthetic repeating video in the stated order, and appending the selected video frame sequence to the synthetic repeating video in the reverse order, wherein the video frame sequence selected in the stated order and the video frame sequence selected in the reverse order are the synthetic repeating video at the initial iteration.

[0111] In some implementations, generating the synthetic repeating video further includes: selecting an appended sequence of video frames from the unlabeled video, wherein the appended sequence immediately precedes the selected video frame sequence. The method further includes: prepending the selected appended sequence to the beginning of the synthetic repeating video. In some implementations, generating the synthetic repeating video further includes: selecting another video frame sequence from the unlabeled video, wherein the other video frame sequence immediately follows the selected video frame sequence. The method further includes: appending the selected other video frame sequence to the end of the synthetic repeating video.

[0112] In some versions of those implementations, generating the synthetic repeating video further includes processing each frame of the synthetic repeating video using an affine image transformation. In some versions of those implementations, the affine image transformation includes rotation, translation, and / or scaling.

[0113] In some implementations, the method further includes: generating a repetition count of the number of repetitions in the video frame sequence based on the period length of the periodic activity and / or the periodicity classification of each frame of the video frame sequence.

[0114] In some implementations, a method implemented by one or more processors is provided, the method comprising: training a repetitive network to generate (a) the period length of periodic activities in the video frame sequence and / or (b) the per-frame periodic classification of the video frame sequence, wherein training the repetitive network comprises: for each of a plurality of training instances, processing a training sequence of training video frame portions of training instances capturing periodic training activities using an encoder portion of the repetitive network to generate a training sequence of encoded training video frames. The method further comprises: generating a training temporal self-similarity matrix indicating pairwise similarity between encoded training video frames in the training sequence of encoded training video frames. The method further comprises: processing the training sequence of encoded training video frames using a periodic predictor model portion of the repetitive network to generate (a) the training period length of the training periodic activities and / or (b) the training per-frame periodic classification of the training sequence of video frames. The method further comprises: determining a loss based on (1) the training period length of the training periodic activities and / or the training per-frame periodic classification of the training sequence of video frames and (2) the corresponding ground truth portions of the training instances. The method further includes updating one or more parts of the repeated network based on the determined loss.

[0115] In some implementations, a method implemented by one or more processors is provided, the method comprising: generating a synthetic repeating video, wherein generating the synthetic repeating video comprises: selecting a sequence of video frames from an unlabeled video. The method further comprises: for each of a plurality of iterations, generating the synthetic training video by: (a) appending the selected video frame sequence to the synthetic repeating video in the stated order, wherein the selected video frame sequence is the synthetic repeating video at the initial iteration, or (b) appending the selected video frame sequence to the synthetic repeating video in reverse order, wherein the video frame sequence selected in the stated order and the video frame sequence selected in the reverse order are the synthetic repeating video at the initial iteration.

[0116] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0117] In some implementations, the method further includes: training a model using the synthetic training video to generate (a) the period length of periodic activity in the video frame appended sequence and / or (b) the per-frame periodic classification of the video frame appended sequence.

[0118] In some versions of those implementations, training the model using the synthetic training video to generate (a) the period length of periodic activity in the additional sequence of video frames and / or (b) the per-frame periodic classification of the additional sequence of video frames includes: processing the synthetic training video using the model to generate (a) candidate period lengths of the training period lengths in the synthetic training video and / or (b) candidate per-frame periodic classifications of the synthetic training video. The method further includes: determining a loss based on (1) the candidate period lengths and / or the candidate per-frame periodic classifications of the synthetic training video and (2) the corresponding ground truth representations of the synthetic training video. The method further includes: updating one or more parts of the model based on the determined loss.

[0119] In some implementations, a method implemented by one or more processors is provided, the method comprising: processing a sequence of video frames capturing periodic activity using an encoder portion of a repetitive network to generate an encoded video frame sequence. The method further comprises: for each pair of encoded video frames, determining a distance metric between the pairs of encoded video frames in the encoded video frame sequence. The method further comprises: for each pair of encoded video frames, generating a matrix based on the distance metric between the video frames. The method further comprises: processing the matrix using a periodic predictor model portion of the repetitive network to generate (a) the period length of the periodic activity in the video frame sequence and / or (b) a per-frame periodic classification of the video frame sequence.

[0120] Furthermore, some implementations include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause the execution of any methods described herein. Some implementations also include one or more transient or non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any methods described herein.

Claims

1. A method using a repetitive network implemented by one or more processors, the repetitive network being trained to generate periodic outputs, the method comprising: Processing a sequence of video frames capturing periodic activity using an encoder to generate a sequence of encoded video frames, wherein processing the sequence of video frames capturing periodic activity using the encoder portion of the repeating network to generate the sequence of encoded video frames includes: For each video frame in the sequence of video frames: The video frames are processed using the convolutional neural network portion of the encoder to generate two-dimensional convolutional features for the video frames; The two-dimensional convolutional features of the video frame are processed using the three-dimensional convolutional neural network portion of the encoder to generate the temporal context features of the video frame; and The temporal context features of the video frames are processed using the max-pooling portion of the encoder to generate corresponding encoded video frames for the sequence of encoded video frames; A temporal self-similarity matrix is ​​generated at one layer based on the sequence of encoded video frames. This temporal self-similarity matrix indicates the pairwise similarity between encoded video frames in the sequence, wherein this layer acts as an information bottleneck between the encoder and the periodic predictor model portion of the repeating network; and The temporal self-similarity matrix is ​​processed using the periodic predictor model portion of the repeating network to generate the periodic output, wherein the periodic output includes (a) the period length of periodic activity in the sequence of video frames, and / or (b) the periodic classification of each frame of the sequence of video frames.

2. The method according to claim 1, further comprising: The encoder portion of the repeating network is used to process additional sequences of video frames that capture additional periodic activity, in order to generate additional sequences of additional encoded video frames. An additional temporal self-similarity matrix is ​​generated from the additional sequence of encoded video frames, the additional temporal self-similarity matrix indicating the pairwise similarity between additional encoded video frames in the additional sequence of additional encoded video frames; and The additional temporal self-similarity matrix is ​​processed using the periodic predictor model of the repeating network to generate (a) the additional period length of the additional periodic activity in the additional sequence of the video frame, and / or (b) the additional per-frame periodic classification of the additional sequence of the video frame.

3. The method according to claim 1, wherein, Generating the temporal self-similarity matrix, which indicates the pairwise similarity between the coded video frames in the sequence of coded video frames, includes: For each pair of coded video frames in the sequence of coded video frames: Determine the distance metric between the encoded video frames of the pair; and For each pair of coded video frames, the temporal self-similarity matrix is ​​generated based on the distance metric between the coded video frames in the pair.

4. The method according to claim 3, wherein, Determining the distance metric between the encoded video frames of the pair includes: For each pair of coded video frames in the sequence of coded video frames: Determine the Euclidean distance, squared Euclidean distance, negative squared Euclidean distance, Gaussian distance, and / or cosine similarity metric between the encoded video frames of the pair.

5. The method according to claim 3, wherein, Generating the temporal self-similarity matrix for each pair of coded video frames based on the distance metric between the coded video frames in the pair includes: The temporal self-similarity matrix is ​​generated by processing the distance metric between the encoded video frames of the pair using a line-by-line Softmax process for each pair of encoded video frames.

6. The method according to claim 1, wherein, Processing the time self-similarity matrix using the cycle predictor model to generate the cycle length of the cycle activity includes: The temporal self-similarity matrix is ​​partially processed using a two-dimensional convolutional neural network to generate convolutional output; The convolutional output is processed using the transformer section to generate the transformer output; and The converter output is processed using a cycle length classifier to generate the cycle length of the periodic activity.

7. The method according to claim 1, wherein, The sequence of video frames that capture the periodic activity is a sequence of randomly sampled video frames from the video that captures the periodic activity.

8. The method according to claim 1, further comprising: Before processing the sequence of video frames capturing the periodic activity using the encoder portion of the repeating network, the repeating network is trained, wherein training the repeating network includes: For each of the multiple training instances: The encoder portion of the repetitive network is used to process the training sequence of training video frames that capture training instances of periodic training activities to generate a training sequence of encoded training video frames. Generate a training temporal self-similarity matrix, the training temporal self-similarity matrix indicating the pairwise similarity between encoded training video frames in a training sequence of encoded training video frames; and The periodic predictor model of the repetitive network is used to process the training sequence of encoded training video frames to generate (a) the training period length of the periodic training activity, and / or (b) the training sequence of the video frames for training frame-by-frame periodic classification. The loss is determined based on the training period length of the periodic training activity described in (1) and / or the training sequence of video frames for periodic classification of each frame, and the corresponding ground truth portion of the training instance described in (2); and Update one or more parts of the repeating network based on the determined loss.

9. The method according to claim 8, wherein, The training sequence of the training video frame portion of at least one of the training instances is a synthetic repeating video.

10. The method according to claim 9, wherein, Generating the synthetic repeating video based on unlabeled video, wherein generating the synthetic repeating video based on unlabeled video includes: A sequence of video frames selected from the unlabeled video; and For each of the multiple iterations: The synthetic repeating video is generated by sequentially appending a sequence of selected video frames to the synthetic repeating video, wherein the selected sequence of video frames is the synthetic repeating video at the initial iteration.

11. The method according to claim 9, wherein, Generating the synthetic repeating video based on unlabeled video, wherein generating the synthetic repeating video based on unlabeled video includes: A sequence of video frames selected from the unlabeled video; For each of the multiple iterations: The synthetic repeating video is generated by appending a sequence of selected video frames in sequence to the synthetic repeating video and appending a sequence of selected video frames in reverse order to the synthetic repeating video, wherein the sequence of selected video frames in sequence and the sequence of selected video frames in reverse order are the synthetic repeating video in the initial iteration.

12. The method according to claim 10, wherein, Generating the synthetic repeating video further includes: An additional sequence of video frames is selected from the unlabeled video, wherein the additional sequence of video frames immediately precedes the sequence of the selected video frames; and The additional sequence of the selected video frames is prepended to the beginning of the synthesized repeating video.

13. The method according to claim 10, wherein, Generating the synthetic repeating video further includes: Select another sequence of video frames from the unlabeled video, wherein the other sequence of video frames immediately follows the sequence of the selected video frames; and Another sequence of the selected video frames is appended to the end of the synthesized repeating video.

14. The method of claim 10, wherein, Generating the synthetic repeating video further includes: Each frame of the synthetic repeating video is processed using affine image transformation.

15. The method according to claim 14, wherein, The affine image transformations include rotation, translation, and / or scaling.

16. The method according to any one of claims 1-15, further comprising: Based on the period length of the periodic activity and / or the periodic classification of each frame in the sequence of video frames, a repetition count is generated for the number of repetitions in the sequence of video frames.

17. A method implemented by one or more processors, the method comprising: Training a repetition network to generate (a) the period length of periodic activity in a sequence of video frames, and / or (b) a per-frame periodic classification of the sequence of video frames, wherein training the repetition network includes: For each of the multiple training instances: The encoder portion of the repetitive network is used to process the training sequence of training video frames capturing training instances of periodic training activities to generate a training sequence of encoded training video frames. The process of using the encoder portion of the repetitive network to process the training sequence of the training video frames to generate the training sequence of encoded training video frames includes: For each video frame in the training sequence of the training video frames: The training video frames are processed using the convolutional neural network portion of the encoder to generate two-dimensional convolutional features for the training video frames. The three-dimensional convolutional neural network portion of the encoder is used to process the two-dimensional convolutional features of the training video frames to generate the temporal context features of the training video frames; and The temporal context features of the training video frames are processed using the max-pooling portion of the encoder to generate corresponding encoded video frames for the sequence of encoded training video frames. Generate a training temporal self-similarity matrix, the training temporal self-similarity matrix indicating the pairwise similarity between encoded training video frames in a training sequence of encoded training video frames; and The periodic predictor model of the repetitive network is used to process the training sequence of encoded training video frames to generate (a) the training period length of the periodic training activity, and / or (b) the training sequence of the video frames for training frame-by-frame periodic classification. The loss is determined based on the training period length of the periodic training activity described in (1) and / or the training sequence of video frames for periodic classification of each frame, and the corresponding ground truth portion of the training instance described in (2); and Update one or more parts of the repeating network based on the determined loss.

18. A method implemented by one or more processors, the method comprising: The encoder portion of the repeating network processes a sequence of video frames capturing periodic activity to generate a sequence of encoded video frames, wherein processing the sequence of video frames capturing the periodic activity using the encoder portion of the repeating network to generate the sequence of encoded video frames includes: For each video frame in the sequence of video frames: The video frames are processed using the convolutional neural network portion of the encoder to generate two-dimensional convolutional features for the video frames; The two-dimensional convolutional features of the video frame are processed using the three-dimensional convolutional neural network portion of the encoder to generate the temporal context features of the video frame; and The temporal context features of the video frames are processed using the max-pooling portion of the encoder to generate corresponding encoded video frames for the sequence of encoded video frames; For each pair of coded video frames in the sequence of coded video frames: Determine the distance metric between the encoded video frames of the pair; For each pair of coded video frames, a matrix is ​​generated based on the distance metric between the video frames; and The matrix is ​​processed using the periodic predictor model portion of the repeating network to generate (a) the period length of periodic activity in the sequence of video frames, and / or (b) the periodic classification of each frame in the sequence of video frames.

19. A computer program product comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to perform the method according to any one of claims 1-18.

20. A computing system, the computing system comprising: One or more memory units; as well as One or more processors coupled to the one or more memories, the one or more processors executing instructions stored in the one or more memories to cause the one or more processors to perform the method according to any one of claims 1 to 18.

21. A computer-readable storage medium storing instructions executable by one or more processors of a computing system to perform the method according to any one of claims 1 to 18.

Images