Video processing using delta distillation

Abstract

This document provides techniques and apparatus for processing video content using artificial neural networks. An exemplary method generally includes the following steps: receiving a video data stream comprising at least a first frame and a second frame; extracting a first feature from the first frame using a teacher neural network; determining the difference between the first frame and the second frame; extracting a second feature from the difference between at least the first frame and the second frame using a student neural network; generating a feature map of the second frame based on the sum of the first and second features; and generating an inference for at least the second frame of the video data stream based on the generated feature map of the second frame.

Description

[Technical Field]

[0001] This patent application claims the benefit and priority of U.S. Patent Application No. 18 / 054,274, filed November 10, 2022, and U.S. Provisional Patent Application No. 63 / 264,072, filed November 15, 2021, each of which is incorporated herein by reference in its entirety.

[0002] The various aspects of this case relate to the processing of video content. [Previous Technology]

[0003] Artificial neural networks can be used to perform various operations on video content or other content including spatial and temporal components. For example, artificial neural networks can be used to compress video content into a smaller representation to improve storage and transmission efficiency, and to match the intended use of the video content (e.g., appropriate data resolution for the display size of the device). Compression of the content can be performed using lossy techniques so that the decompressed version of the data is an approximation of the compressed original data, or it can be performed using lossless techniques that result in the decompressed version of the data being equivalent to (or at least visually equivalent to) the original data. In another instance, artificial neural networks can be used to detect objects in video content. Object detection can include, for example, subject pose estimation for identifying moving subjects in the video content and predicting how the subjects will move in the future; object classification for identifying objects of interest in the video content; and so on.

[0004] Generally speaking, the temporal component of video content can be represented by different frames within the video content. An artificial neural network can process each frame in the video content independently through each layer of the network. Therefore, the cost of video processing via an artificial neural network can increase at a rate different from (and higher than) the rate of information growth in the video content. That is, there may be small variations between successive frames in the video content, as only a small amount of data changes during the time elapsed between different frames. However, since neural networks typically process each frame independently, artificial neural networks often process repetitive data between frames (e.g., unchanged parts of the scene), which is highly inefficient. [Summary of the Invention]

[0005] Some models provide a method for processing video content using artificial neural networks. An exemplary method generally includes the following steps: receiving a video data stream including at least a first frame and a second frame; extracting a first feature from the first frame using a teacher neural network; determining the difference between the first frame and the second frame; extracting a second feature from the difference between at least the first frame and the second frame using a student neural network; generating a feature map of the second frame based on the sum of the first feature and the second feature; and generating an inference for at least the second frame of the video data stream based on the generated feature map of the second frame.

[0006] Some models provide a method for training an artificial neural network to process video content. An exemplary method generally includes the following steps: receiving a training dataset comprising a plurality of video samples. Each video sample may include a plurality of frames. Training a teacher neural network based on the training dataset, and training a student neural network based on predicted differences between feature maps of successive frames in each video sample and actual differences between feature maps of successive frames in each video sample. Deploying the teacher neural network and the student neural network.

[0007] Other configurations include: a processing system configured to perform the foregoing methods and the methods described herein; a non-transitory computer-readable medium including instructions which, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and the methods described herein; a computer program product implemented on a computer-readable storage medium including code for performing the foregoing methods and the methods further described herein; and a processing system including components for performing the foregoing methods and the methods further described herein.

[0008] The following description and related figures illustrate in detail certain illustrative features of one or more states.

Implementation Method

[0019] The various forms of this case provide technology for efficiently processing video content using artificial neural networks.

[0020] As discussed, artificial neural networks can be used to perform various inference operations on video content. These inferences can be used for, for example, various compression schemes, object detection, computer vision operations, various image processing and modification operations (e.g., magnification, noise reduction, etc.), and so on. However, artificial neural networks can process each video frame in the video content independently. Therefore, such artificial neural networks may not utilize the various redundancies across frames in the video content.

[0021] Training neural networks and using trained neural networks to perform inference can be a computationally complex task, and the computational complexity scales with the accuracy of the neural network. That is, an accurate model may be more computationally complex to train and use, while a less accurate model may be less computationally complex to train and use. To allow for improvements in computational complexity while preserving the accuracy of artificial neural networks, data redundancy can be employed. For example, channel redundancy can allow weights to be pruned based on various error terms so that weights with minimal or no impact on inference are removed from the trained neural network (e.g., for a specific channel, such as a color channel in image data); quantization can be used to represent weights with smaller bit widths; and singular value decomposition can be used to approximate the weight matrix with a more compact representation. In another instance, spatial redundancy can be used to leverage similarities in the spatial domain. In yet another instance, knowledge distillation can be used, where a student neural network is trained to match the feature outputs of a teacher neural network. However, such techniques may not utilize temporal redundancy in video content or other content that includes a temporal component.

[0022] Various embodiments of this invention provide techniques for training neural networks using temporal redundancy in video content or other content including time components, and for using the neural network to generate inferences about the video content or other content including time components. Such temporal redundancy can be represented by differences or increments between successive portions of the video content or other content having time components. By training the neural network based on increments distilled from successive portions of the video content or other content having time components and using the trained neural network to perform inferences, various embodiments of this invention can reduce the amount of data used in training the neural network and performing inferences using the neural network. This can accelerate the process of training the neural network and performing inferences using the neural network, reduce the number of processing loops and the memory used in such operations, reduce the amount of power used in training and inference operations, etc. Exemplary Incremental Distillation in Artificial Neural Networks

[0023] Figure 1 illustrates an exemplary distillation architecture 100 including a teacher neural network and a student neural network. In the teacher-student neural network paradigm, the teacher neural network can be a larger network than the student neural network, and the data learned by the teacher neural network can be used to train the student network using data distillation techniques, as discussed in further detail below.

[0024] As illustrated, the distillation architecture 100 includes a teacher neural network 110 and a student neural network 120. The teacher neural network 110 typically includes a layers, while the student neural network 120 includes b layers (which may be greater than a, as discussed in further detail below). Input 105 may be provided to both the teacher neural network 110 and the student neural network 120, and the distillation loss 115 between features is generated by the teacher neural network 110 and the student neural network 120.

[0025] The teacher neural network 110 can be represented as a general backbone consisting of L parameter blocks according to the following formula:

[0026] Each parameter block maps the input to the output z. Generally, for the l-th parameter block, the output produced by the block can be represented by the following formula: where x represents the input to the parameter block and represents the learnable parameters of the parameter block.

[0027] Given a backbone, one technique for reducing the computational cost of training and performing inference includes knowledge distillation. In this example, each of the backbones can be viewed as a teacher neural network providing a target feature map, which can be used as a supervisory signal for a separate student neural network with learnable parameters. The student neural network can be designed to have lower computational complexity and thus lower computational cost than the teacher neural network, for example, by reducing the number of parameters, reducing the number of layers, reducing the amount of data processed in the student neural network by scaling down or other input scaling, etc. The distillation objective used in training the student neural network can be optimized, for example, by seeking the expected norm of the error between and according to the following formula:

[0028] For data with a time component (such as video content), each can be considered as mapping the input xt to the output zt at time t. Furthermore, according to the Taylor expansion of the function, the current output can be represented as an additive update of the previous output (e.g., as an additive update of zt-1). Thus, the output zt can be represented by the equation, and consequently, the increment (or difference) between zt and zt-1 can be represented by the equation. Furthermore, it can be noted that for many functions, f' can have fewer parameters than f, which may indicate that the change between the frames at time t and time t-1 is likely more compressible and can lie in similar regions in the feature space.

[0029] Figure 2 illustrates an example in which the teacher neural network 210 and the student neural network 220 are trained to distill the differences (or increments) between different video frames that are separated in time. As illustrated, the teacher neural network 210 (represented by the function f discussed above) and the student neural network 220 (represented by the function g discussed above) can receive two video frames xt-1 and xt (e.g., the current frame with timestamp t and the previous frame with timestamp t-1) as inputs. The teacher neural network 210 can generate feature maps of both inputs xt-1 and xt, and generate a difference feature map (represented as ) at time t representing the difference between inputs xt-1 and xt. The student neural network 220 can similarly receive two video frames xt-1 and xt as inputs, and can be trained to predict the difference between the feature maps generated for video frames xt-1 and xt, which is represented as . In order to allow the student neural network 220 to produce accurately (given that the output of the teacher neural network 210 is regarded as true data), the student neural network 220 can be trained to minimize the loss function between the actual difference and the predicted difference.

[0030] Since the frame xt can be represented as the sum of the frame xt-1 and the additive increment, the feature map zt can similarly be represented as the sum of the feature map zt-1 (e.g., the feature map of the frame xt-1) and the additive increment according to the following equation: , where represents the additive increment, which characterizes the output change for a given time change of the input xt-1. More generally, the additive increment can be defined as a function of the current input and the previous input, which can be approximated by the student neural network 220 using parameters according to the following equation: .

[0031] Figure 3 illustrates an example 300 in which a neural network including a teacher neural network 310 and a student neural network 320 is trained based on the changes between frames in the video content of the various types of this case, and the neural network is used for inference.

[0032] As illustrated, the teacher neural network 310, represented by the function fl, can be trained to generate a feature map z for any given input video frame x. The feature map z generated by the teacher neural network 310 can be considered as ground truth data for training the student neural network 320. The teacher neural network 310 can receive frame x as input, while the student neural network 320 can receive the differences between different frames as input. At a given time t+n, the differences between the frames used as input to the student neural network can be represented as follows, where t represents the timestamp associated with the underlying frame from which inference is performed (e.g., a key frame in a video stream), and n represents the difference between timestamp t and the timestamp associated with frame x t+n.

[0033] The student neural network 320 can be represented as one or more linear blocks, wherein the student neural network receives residual values ​​between different samples (e.g., different video frames) in the temporal data as input. In such a case, the student neural network 320 can be represented by an equation, which can be a first-order term in the approximation and can be a non-zero term. The derivative is constant in the case of linearity. In another instance, the student neural network 320 can be represented by one or more nonlinear blocks. In such a case, the student neural network can receive both the previous input and the residual values ​​as input at time t.

[0034] Generally, the structure of the student neural network 320 can be selected based on a channel reduction strategy or a space reduction strategy. In the channel reduction strategy, the student neural network 320 can be structurally a mirror image of the teacher neural network 310, but can have fewer channels than the teacher neural network 310. Several point-by-point spirals can be introduced into the student neural network 320 as the first and last layers, which can reduce and increase the number of channels, respectively. In the space reduction strategy, the student neural network 320 can be similar to the teacher neural network 310. However, the student neural network 320 can operate with a smaller spatial resolution on the input video frame, which can be achieved via a point-by-point step spiral layer (e.g., a spiral layer using a 1×1 core with intervals between different parts of the input video frame) introduced as the first layer in the student neural network 320 and a primitive shuffling upsampling layer introduced as the last layer in the student neural network 320.

[0035] Within the backbone model implemented by the teacher neural network 310, different layers can be compressible to varying degrees. For example, some layers may be incompressible (or non-distillable) because such layers might completely impair the performance of the teacher neural network 310. Therefore, a student neural network can be selected from two candidate networks, where a student neural network operates at the same computational cost as the teacher neural network (e.g., distilling the increments between frames without compression) and is computationally cheaper by a certain objective factor. A learnable bias can be introduced, representing the more suitable network among the two candidate networks. This learnable bias can be learned using, for example, gradient descent via gradient descent after reparameterizing the estimated gradient using Gumbel-softmax.

[0036] The incremental distillation block 315 can be used to optimize a video model for a specific application. A task-specific objective function can be defined, where the parameters of the backbone model implemented by the teacher neural network 310 represent the parameters. During training, optimization can be performed on training video clips, where both the teacher neural network 310 and the student neural network 320 contribute to prediction. The teacher neural network 310 contributes to predicting the output z (e.g., feature map) of the initial frame, and the student neural network 320 contributes to predicting the output z of the remaining frames.

[0037] To optimize incremental distillation (e.g., via incremental distillation block 315), each block can be designated as a teacher, which supervises the student's learning by providing a target increment to the corresponding student block. Therefore, the incremental distribution loss can be minimized at incremental distillation block 315. The incremental distribution loss can be defined by the objective between the actual change of zt and the change modeled by the student block and can be expressed as follows:

[0038] A complexity objective can be further introduced to facilitate the use of low-computational-cost candidate networks (e.g., ) as student neural networks where possible. In general, the cost function can be optimized because the nonregularization bias term converges to the selection of the least compressed student network (e.g., ), which is because the least compressed student has higher capacity and better incremental distillation capability. The cost function can be expressed as: where represents the complexity measure of the student neural network, and represents the classification distribution on the candidate network, which can be obtained by providing it as input to the softmax layer. Minimizing (or at least reducing) and can guide the student network toward searching for candidate networks that produce the best trade-off between cost and performance. Finally, the overall objective to be optimized when selecting the student neural network can be expressed as: , where and are the hyperparameters of and , and the summation is applied to each block L of the teacher neural network 310.

[0039] Generally, during the training of the student neural network 320, the distilled increments will converge to the teacher increments. Since the gradient of the task loss propagates back through the teacher neural network 310 over time, these task loss gradients can provide a representation of the first frame that can be additively updated by the student neural network 320. Thus, the task loss gradients can prompt the teacher to provide a more easily updated representation, thereby improving temporal consistency within the network. Furthermore, the technique described herein can transform the backbone implemented in the teacher neural network 310 into a recurrent model, as the teacher neural network can propagate its output from one time point to another, further improving temporal consistency in the pipeline comprising the teacher neural network 310 and the student neural network 320.

[0040] Figure 4 illustrates an example 400 of using a neural network to perform inferences about video content based on the changes between frames in the video content according to the various states of this case.

[0041] To perform inference on the video content, a teacher neural network 410 and one or more student neural networks 420 may be used to process frames in the video content. Generally, the output generated by the teacher neural network 410 and(the student neural networks) 420 for a given input of video data can be represented by the following formula:

[0042] The teacher neural network 410 can process a specified initial frame, such as a key frame, and define subsequent frames of that key frame based on the difference relative to the key frame. Meanwhile, the student neural networks 420 can process other frames in the video content at least partially based on the increment between the frame and the previous frame. That is, for a frame xt at time t, the teacher neural network 410 can generate a feature map zt. For a frame xt+1 at time t+1, the increment can be calculated as the difference between frame xt and xt+1. The student neural network 420 can then generate an approximate feature map based on the feature map and the feature map of frame xt. The approximate feature map of frame xt+1 can be calculated as the sum of the feature maps. Similarly, the student neural network 420 can generate an approximate feature map for the difference between frames xt+1 and xt+2 as a sum of the features maps. This process can continue for any number of video frames x. For example, incremental distillation can be used to generate feature maps for frames until a new key frame is encountered in the video content. This new key frame can be processed using teacher neural network 410, and subsequent frames until the next key frame can be processed using student neural network 420. An exemplary method for video processing using incremental distillation and teacher and student neural networks is described below.

[0043] Figure 5 illustrates an exemplary operation 500, which, according to certain aspects of this case, can be executed by a system to train a neural network to perform inferences about video content based on increments distilled from successive video frames (or differences between frames in the video content). Operation 500 can be executed, for example, by a computing device (e.g., the processing system 700 illustrated in Figure 7) that can train a machine learning model and deploy that model to another device for generating inferences from video data.

[0044] As illustrated, operation 500 begins at block 510, where a training dataset is received. Generally, the training dataset may include a plurality of video samples, and each of the plurality of video samples may include a plurality of frames. Within the plurality of frames of each video sample, one or more frames may be designated as key frames, and frames following a first key frame and preceding a second key frame may be defined based on differences relative to the first key frame.

[0045] In block 520, the teacher neural network is trained based on the training dataset. Generally, when training the teacher neural network, it can be trained to generate feature maps for each frame in each of the plurality of video samples. The teacher neural network can be represented, for example, as the backbone of L parameter blocks, and the output z can be generated according to the following formula: where x represents the input (e.g., a frame in a video sample).

[0046] In block 530, a student neural network is trained. Generally, the student neural network can be trained based on the predicted difference between the feature maps of successive frames in each video sample and the actual difference between the feature maps of successive frames in each video sample.

[0047] In some cases, both the teacher and student neural networks can be trained to minimize the same task-specific objective function. The task-specific objective function can be, for example, an objective function based on each of the multiple parameters of the model implemented by the teacher neural network. The task-specific objective function can be defined based on a weighted incremental distribution loss term and a weighted cost term. For example, the task-specific objective function can be expressed by an operational formula. The incremental distribution loss term can be associated with the difference between the actual and predicted changes in the feature maps generated for successive frames in video sampling, and the cost term can be associated with a complex metric of the student neural network.

[0048] In some cases, training the student neural network may include training the student neural network to minimize the incremental distillation loss. The incremental distillation loss can typically represent the difference between the actual difference between the outputs generated for successive frames in the training dataset (e.g., generated by the teacher neural network) and the predicted difference between the outputs generated for successive frames in the video samples (e.g., generated by the student neural network).

[0049] In some cases, training the student neural network may include: training the student neural network to minimize a cost function defined by a complexity metric based on the student neural network and a classification distribution over a plurality of candidate models. For example, training the student neural network may be based on a complexity metric of a first neural network that matches the teacher neural network in computational complexity and a second neural network with reduced complexity.

[0050] In block 540, the teacher neural network and the student neural network are deployed. For example, the teacher neural network and the student neural network may be deployed to devices that perform inferences on the captured video data, such as user equipment (UE) in a wireless network, vehicles with autonomous driving capabilities that operate at least partially based on computer vision capabilities, etc. Such inferences may include, for example, encoding content into a latent space for compression, object detection in the video content, subject pose estimation and motion prediction, semantic segmentation of the video content into different segments, etc.

[0051] Figure 6 illustrates an exemplary operation 600, according to certain aspects of this invention, for performing inference of video content using a neural network trained to process video content incrementally based on successive video frames. Operation 600 may be performed, for example, by a device on which a teacher neural network and a student neural network are deployed (e.g., the processing system 800 illustrated in Figure 8), such as a user equipment (UE), autonomous vehicle, etc.

[0052] As illustrated, operation 600 may begin at block 610, where a video data stream is received. Generally, the video data stream may include key frames and one or more non-key frames. Key frames may be frames used by a neural network as the initial basis for performing inference, and inferences about non-key frames may be performed based on differences between successive frames.

[0053] In block 620, a teacher neural network can be used to extract a first feature from the first frame. As discussed, the first frame can be a key frame or an initial frame from which other frames in the video content are derived (e.g., defined by the differences applied to the first frame).

[0054] In block 630, the difference between the first frame and the second frame is determined. Generally, the difference between the first frame and the second frame may include information related to changes in the primitives of each of the first frame and the second frame, such that the combination of the first frame and the determined difference results in the second frame.

[0055] In block 640, a student neural network is used to extract a second feature from at least the difference between the first frame and the second frame. Generally, the second feature can be an approximation of the difference between the first feature and the feature that the teacher neural network would extract for the second frame. In some cases, the second feature may be further extracted from the first frame in conjunction with the difference between the first frame and the second frame.

[0056] In block 650, a feature map of the second frame is generated based on the sum of the first features and the second features. As discussed, the first features can typically be the entire set of features extracted from the first frame, and the second features can be the set of features extracted from the differences between the first and second frames. Since the second frame can be represented as the first frame and the sum of the differences between the first and second frames, the feature map representing the second frame can similarly be represented as the sum of the first features (extracted from the first frame) and the second features (extracted from the differences between the first and second frames).

[0057] In block 660, an inference for at least the second frame of the video stream is generated based on the generated feature map of the second frame. In some embodiments, generating the inference may include identifying one or more objects in the second frame. In some embodiments, generating the inference may include estimating the pose and / or predicting the motion of a subject in the video stream. The pose estimation and motion prediction may then be used, for example, to control an autonomous vehicle to react to the predicted motion of a subject recorded in the video content to avoid a collision between the autonomous vehicle and the subject. In another embodiment, generating the inference may include semantically segmenting the video stream into a plurality of segments. For example, the video stream may be segmented into one or more segments associated with different subjects captured in the video stream. In another instance, the video stream may be segmented into foreground content and background content, which allows analysis of certain content and ignoring other content until that content becomes foreground content (if it exists).

[0058] In yet another example, generating the inference may include mapping the second frame to a code among a plurality of codes in the latent space. One or more modifications may be performed on the second frame based on the code to which it is mapped in the latent space. For example, the second frame, or a portion of the second frame (e.g., a part of interest, such as foreground content, a specific object, etc.), may be modified. Modifications may include resolution changes (enlarging and / or reducing to preserve detail after modifying the second frame), noise reduction, and other modifications that may be performed on the image or a portion of the image.

[0059] In some cases, the teacher neural network may be a linear network comprising a plurality of linear blocks, as discussed above. The student neural network may be configured to decompose the weights into ranks lower than the weight rank of the teacher neural network. The student neural network may include a plurality of grouped spiral layers. In the case where the teacher neural network is a linear network, the student neural network may be able to generate a second feature (e.g., based on the difference between the previous frame and the current frame) without needing to receive the previous frame to generate the second feature.

[0060] In some cases, the teacher neural network may be a nonlinear network comprising a plurality of nonlinear blocks. In such cases, the student neural network may be a network having one or more of the following relative to the teacher neural network: a reduced number of channels, a reduced spatial resolution, or a reduced quantization. As discussed, in the case where the teacher neural network is a nonlinear network, the student neural network may generate a second feature (e.g., based on previous frames and the difference between the previous frame and the current frame) (e.g., based on).

[0061] In some embodiments, the student neural network may be a neural network trained to minimize a loss function based on the difference between the actual change in the feature map between the first and second frames and the predicted change in the feature map between the first and second frames. This loss function may be further based on a cost function defined according to a complexity metric of the student neural network and a classification distribution over a plurality of candidate models. The plurality of candidate models may include, for example, student neural networks operating at the same computational cost as the teacher neural network (e.g., distilling the increments between frames without compression) and student neural networks that are computationally cheaper than the teacher neural network by a certain target factor.

[0062] In some cases, the difference between the second frame and a third frame in the video data stream can be determined. A third feature can be extracted from at least the difference between the second and third frames using a student neural network. A feature map of the third frame can be generated based on the sum of the second and third features, and an inference for the third frame can be generated based on the generated feature map of the third frame. An exemplary processing system for performing video processing using incremental distillation in a neural network.

[0063] Figure 7 illustrates an exemplary processing system 700 for training machine learning models to perform inferences about video content (or other content with a temporal component) using incremental distillation, teacher neural networks, and student neural networks, as described herein, for example, with respect to Figure 5.

[0064] The processing system 700 includes a central processing unit (CPU) 702, which in some instances may be a multi-core CPU. Instructions executed at the CPU 702 may be loaded, for example, from program memory associated with the CPU 702 or from memory 724.

[0065] The processing system 700 also includes additional processing elements customized for specific functions, such as a graphics processing unit (GPU) 704, a digital signal processor (DSP) 706, a neural processing unit (NPU) 708, a multimedia element 710, and a wireless connectivity element 712.

[0066] An NPU (such as the NPU 708) is typically configured to implement dedicated circuitry for implementing control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. An NPU may sometimes be alternatively referred to as a neural signal processor (NSP), tensor processing unit (TPU), neural network processor (NNP), intelligent processing unit (IPU), visual processing unit (VPU), or graphics processing unit.

[0067] NPUs (such as NPU 708) are configured to accelerate the execution of common machine learning tasks such as image classification, machine translation, object detection, and various other predictive models. In some instances, multiple NPUs may be generated on a single chip (such as a system-on-a-chip (SoC)), while in other instances, an NPU may be part of a dedicated neural network accelerator.

[0068] The NPU can be optimized for training or inference, or in some cases configured to balance performance between training and inference. For an NPU capable of performing both training and inference, these two tasks may typically still be performed independently.

[0069] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves a highly computationally intensive operation that involves taking an existing dataset (often labeled or sublabeled), performing repeated computations on the dataset, and then adjusting model parameters (such as weights and biases) to improve model performance. Generally, optimization based on incorrect predictions involves backpropagating through the layers of the model and determining gradients to reduce prediction errors.

[0070] NPUs designed to accelerate inference are typically configured to operate on the full model. Such NPUs can thus be configured to take new data segments as input and process those segments rapidly via an already trained model to produce model outputs (e.g., inference).

[0071] In one implementation, the NPU 708 is part of one or more of the CPU 702, GPU 704 and / or DSP 706.

[0072] In some instances, the wireless connectivity element 712 may include sub-elements for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity element 712 is further coupled to one or more antennas 714.

[0073] In some instances, one or more processors of the processing system 700 may be based on the ARM or RISC-V instruction set.

[0074] The processing system 700 may also include one or more sensor processing units 716 associated with any type of sensor, one or more image signal processors (ISP) 718 associated with any type of image sensor, and / or may include navigation elements 720 that include satellite-based positioning system elements (e.g., GPS or GLONASS) and inertial positioning system elements.

[0075] The processing system 700 may also include one or more input and / or output devices 722, such as a screen, a touch-sensitive surface (including a touch-sensitive display), a physical button, a speaker, a microphone, etc.

[0076] Memory 724 represents one or more static and / or dynamic memory, such as dynamic random access memory (DRAM), static memory based on flash memory, etc. In this example, memory 724 includes computer executable elements that can be executed by one or more of the aforementioned elements of the processing system 700.

[0077] Specifically, in this example, memory 724 includes a training dataset receiving element 724A, a teacher neural network training element 724B, a student neural network training element 724C, and a neural network deployment element 724D. The illustrated elements and other elements not illustrated may be configured to perform various states of the methods described herein.

[0078] Generally speaking, the processing system 700 and / or its components may be configured to perform the methods described herein.

[0079] It is worth noting that, in other embodiments, the components of the processing system 700 may be omitted, such as when the processing system 700 is a server computer. For example, in other embodiments, the multimedia component 710, the wireless connectivity component 712, the sensor processing unit 716, the ISP 718, and / or the navigation component 720 may be omitted. Furthermore, the components of the processing system 700 may be distributed, such as those used to train a model and use the model to generate inferences.

[0080] Figure 8 illustrates an exemplary processing system 800 for performing inference of video content (or other content with a time component) using incremental distillation, a teacher neural network, and a student neural network, as described herein, for example, with respect to Figure 6.

[0081] The processing system 800 includes a central processing unit (CPU) 802, which in some instances may be a multi-core CPU. The processing system 800 also includes additional processing elements tailored for specific functions, such as a graphics processing unit (GPU) 804, a digital signal processor (DSP) 806, and a neural processing unit (NPU) 808. The CPU 802, GPU 804, DSP 806, and NPU 808 may be similar to the CPU 702, GPU 704, DSP 706, and NPU 708 discussed above with respect to Figure 7.

[0082] In some instances, the wireless connectivity element 812 may include sub-elements for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G LTE), fifth-generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity element 812 may be further connected to one or more antennas (not shown).

[0083] In some instances, one or more processors of the processing system 800 may be based on the ARM or RISC-V instruction set.

[0084] The processing system 800 also includes memory 824, which represents one or more static and / or dynamic memories, such as dynamic random access memory, static memory based on flash memory, etc. In this example, memory 824 includes computer executable elements that can be executed by one or more of the aforementioned processors of the processing system 800.

[0085] Specifically, in this example, memory 824 includes a video data streaming receiving element 824A, a feature extraction element 824B, a difference determination element 824C, a feature map generation element 824D, an inference generation element 824E, and a neural network element 824F (such as neural networks 310 and 320 described above with respect to FIG3 or neural networks 410 and 420 described above with respect to FIG4). The illustrated elements and other elements not illustrated may be configured to perform various forms of the methods described herein.

[0086] Generally speaking, the processing system 800 and / or its components may be configured to perform the methods described herein.

[0087] It is worth noting that, in other embodiments, various components of the processing system 800 may be omitted, such as when the processing system 800 is a server computer. For example, in other embodiments, multimedia element 810, wireless connectivity element 812, sensor 816, ISP 818, and / or navigation element 820 may be omitted. Exemplary Terms

[0088] The implementation details of each state in this case are described in the following numbered clauses.

[0089] Clause 1: A method comprising the steps of: receiving a video data stream including at least a first frame and a second frame; extracting a first feature from the first frame using a teacher neural network; determining a difference between the first frame and the second frame; extracting a second feature from the difference between at least the first frame and the second frame using a student neural network; generating a feature map of the second frame based on the sum of the first feature and the second feature; and generating an inference for at least the second frame of the video data stream based on the generated feature map of the second frame.

[0090] Clause 2: The method of Clause 1, wherein the first frame includes key frames in the video stream and the second frame includes non-key frames in the video stream.

[0091] Clause 3: The method of Clause 1 or 2 further includes the following steps: determining the difference between the second frame and the third frame in the video data stream; using the student neural network to extract a third feature from at least the difference between the second frame and the third frame; generating a feature map of the third frame based on the sum of the second features and the third features; and generating an inference of the third frame for the video data stream based on the generated feature map of the third frame.

[0092] Clause 4: The method of any of Clauses 1 to 3, wherein the teacher neural network includes a linear network.

[0093] Clause 5: The method of any of Clauses 1 to 4, wherein the student neural network is configured to decompose the weights into ranks lower than the weight ranks in the teacher neural network.

[0094] Clause 6: The method of any of Clauses 1 to 5, wherein the student neural network comprises one or more grouped spiral layers.

[0095] Clause 7: The method of any of Clauses 1 to 6, wherein the teacher neural network includes a nonlinear network.

[0096] Clause 8: The method of any of Clauses 1 to 7, wherein the student neural network includes a network having one or more of a reduced number of channels, a reduced spatial resolution, or a reduced quantization relative to the teacher neural network.

[0097] Clause 9: The method of any of Clauses 1 to 8, wherein the second feature is further extracted from the first frame in combination with the difference between the first frame and the second frame.

[0098] Clause 10: The method of any of Clauses 1 to 10, wherein the student neural network comprises a neural network trained to minimize a loss function based on the difference between the actual change in the feature map between the first frame and the second frame and the predicted change in the feature map between the first frame and the second frame.

[0099] Clause 11: The method of Clause 10, wherein the loss function is further based on a cost function defined according to a complexity metric of the student neural network and a classification distribution over a plurality of candidate models.

[0100] Clause 12: The method of any of Clauses 1 to 11, wherein the inference is generated by: one or more objects in the second frame that identify the video data stream.

[0101] Article 13: The method of any of Articles 1 to 12, wherein the inference is generated by estimating at least one of the pose of a subject in the video data stream or predicting motion.

[0102] Clause 14: The method of any of Clauses 1 to 13, wherein generating the inference includes: semantically segmenting the video stream into a plurality of segments associated with different subjects captured in the video stream.

[0103] Clause 15: The method of any of Clauses 1 to 14, wherein generating the inference includes: mapping the second frame to a code among a plurality of codes in the latent space, and wherein the method further includes the step of: modifying the second frame based on the code to which the second frame is mapped in the latent space.

[0104] Clause 16: A method comprising the steps of: receiving a training data set comprising a plurality of video samples, each of the plurality of video samples comprising a plurality of frames; training a teacher neural network based on the training data set; training a student neural network based on a predicted difference between feature maps of successive frames in each video sample and an actual difference between feature maps of successive frames in each video sample; and deploying the teacher neural network and the student neural network.

[0105] Clause 17: The method of Clause 16, wherein the teacher neural network and the student neural network are trained to minimize the same task-dependent objective function, and the task-dependent objective function comprises a function defined based on a weighted incremental distribution loss term associated with the difference between actual and predicted changes in feature maps generated for successive frames in video sampling and a weighted cost term associated with a complexity metric of the student neural network.

[0106] Clause 18: The method of Clause 16 or 17, wherein training the student neural network comprises: training the student neural network to minimize the loss between the actual difference between the outputs generated for successive frames in a video sample in the training dataset and the predicted difference between the outputs generated for successive frames in a video sample.

[0107] Clause 19: The method of any of Clauses 16 to 18, wherein training the student neural network comprises: training the student neural network to minimize a cost function defined on a complex metric of the student neural network and a classification distribution on a plurality of candidate models.

[0108] Article 20: The method of any of Articles 16 to 19, wherein the teacher neural network includes a linear network.

[0109] Clause 21: The method of any of Clauses 16 to 20, wherein the student neural network includes a network configured to decompose weights into rank lower than the weight rank in the teacher neural network.

[0110] Clause 22: The method of any of Clauses 16 to 21, wherein the teacher neural network includes a nonlinear network.

[0111] Clause 23: A processing system comprising: memory including computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method pursuant to any of Clauses 1-22.

[0112] Clause 24: A processing system comprising components for performing a method pursuant to any one of Clauses 1-22.

[0113] Clause 25: A non-transitory computer-readable medium comprising computer-executable instructions, which, when executed by one or more processors of a processing system, cause the processing system to perform a method according to any one of Clauses 1-22.

[0114] Clause 26: A computer program product implemented on a computer-readable storage medium, the computer-readable storage medium including code for performing a method pursuant to any of Clauses 1-22. Additional considerations

[0115] The foregoing description is provided to enable any person skilled in the art to practice the various forms described herein. The examples discussed herein are not intended to limit the scope, applicability, or forms set forth in the claims. Various modifications to these forms will be readily understood by those skilled in the art, and the universal principles defined herein can be applied to other forms. For example, the function and arrangement of the discussed elements can be changed without departing from the scope of this application. Various procedures or elements can be appropriately omitted, substituted, or added to the various examples. For example, the described method can be performed in a different order than that described, and various steps can be added, omitted, or combined. Moreover, features described with reference to some examples can be combined in some other examples. For example, any number of forms set forth herein can be used to implement an apparatus or method of practice. Furthermore, the scope of this application is intended to cover such apparatuses or methods practiced using other structures, functions, or structures and functions that supplement or differ from the various forms set forth herein. It should be understood that any form of the case disclosed herein may be implemented by one or more elements of the claim.

[0116] As used herein, the term "exemplary" means "serving as an example, instance, or illustration." Any state described herein as "exemplary" need not be construed as superior to or better than other states.

[0117] As used herein, the phrase “at least one of” a list of items refers to any combination of such items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, cc, and ccc, or any other ordering of a, b, and c).

[0118] As used herein, the term "decision" encompasses a wide variety of actions. For example, "decision" can include calculation, computation, processing, derivation, research, examination (e.g., examining in a table, database, or other data structure), identification, and similar actions. Moreover, "decision" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and similar actions. Similarly, "decision" can also include parsing, selecting, choosing, building, and similar actions.

[0119] The methods disclosed herein include one or more steps or actions for implementing the methods. These method steps and / or actions can be interchanged without departing from the scope of the claim. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claim. Furthermore, the various operations of the above methods can be performed by any suitable component capable of performing the corresponding function. Such components may include various hardware and / or software elements and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations illustrated in the figures are present, these operations may have corresponding paired component functional elements with similar numbering.

[0120] The following claims are not intended to be limited to the various forms shown herein, but should be granted the full scope consistent with the language of the claims. Within the claims, references to singular elements are not intended to mean "one and only one" (unless specifically stated so), but rather "one or more." Unless specifically stated otherwise, the term "some / a certain" refers to one or more. No element of the claims should be interpreted in accordance with Article 18, Paragraph 8 of the Implementing Regulations of the Patent Law, unless the element is explicitly described using the phrase "component for..." or, in the case of a method claim, the element is described using the phrase "step for...". Elements of the various forms described throughout this case, all structural and functional equivalents known now or hereafter to the person of ordinary skill, are explicitly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be donated to the public, whether or not such disclosure is explicitly stated in the scope of the patent application. [Simplified Explanation of the Diagram]

[0009] The accompanying drawings illustrate certain features of the one or more states and are therefore not considered to limit the scope of this case.

[0010] Figure 1 illustrates an exemplary distillation architecture including a teacher neural network and a student neural network.

[0011] Figure 2 illustrates an example of incremental distillation used to distill the changes between frames using a teacher neural network and a student neural network.

[0012] Figure 3 illustrates an example of training a neural network, including teacher and student neural networks, based on the changes between frames in the video content according to the various states of this case.

[0013] Figure 4 illustrates an example of using a neural network to perform inferences about video content based on the changes between frames in the video content according to the various states of this case.

[0014] Figure 5 illustrates exemplary operations according to the various forms of this case for training a neural network to perform inference of video content based on increments distilled from successive video frames.

[0015] Figure 6 illustrates an exemplary operation according to the present case for using a neural network trained to perform inference of video content based on incremental processing of video content distilled from successive video frames.

[0016] Figure 7 illustrates an exemplary system on which various forms of this case can be implemented.

[0017] Figure 8 illustrates another exemplary system on which various forms of this case can be implemented.

[0018] To facilitate understanding, the same element symbols are used where possible to designate common elements shared by all figures. Elements and features of one pattern can be beneficially incorporated into other patterns without further elaboration. [Biomaterial Storage]

[0122] Domestic storage information (please note in order of storage institution, date, and number) None Foreign storage information (please note in order of storage country, institution, date, and number) None

Claims

1. A processor-implemented method comprising the steps of: receiving a video data stream including at least a first frame and a second frame; extracting a first feature from the first frame using a teacher neural network; determining a difference between the first frame and the second frame; extracting a second feature from the difference between at least the first frame and the second frame using a student neural network; generating a feature map of the second frame based on a summation of the first feature and the second feature; and generating an inference for at least the second frame of the video data stream based on the generated feature map of the second frame; wherein the student neural network includes a network having one or more of a reduced number of channels, a reduced spatial resolution, or a reduced quantization relative to the teacher neural network.

2. The method of claim 1, wherein the first frame includes a key frame in the video data stream, and wherein the second frame includes a non-key frame in the video data stream.

3. The method of claim 1 further includes the following steps: determining a difference between the second frame and a third frame in the video data stream; extracting a third feature from the difference between at least the second frame and the third frame using the student neural network; generating a feature map of the third frame based on a sum of the second features and the third features; and generating an inference for the third frame in the video data stream based on the generated feature map of the third frame.

4. The method of request item 1, wherein the teacher neural network comprises a linear network.

5. The method of request 1, wherein the student neural network is configured to: decompose the weights into a rank lower than that of a weight in the teacher neural network.

6. The method of claim 1, wherein the student neural network includes one or more grouped spiral layers.

7. The method of claim 1, wherein the teacher neural network includes a nonlinear network.

8. The method of claim 1, wherein the second feature is further extracted from the first frame in conjunction with the difference between the first frame and the second frame.

9. The method of claim 1, wherein the student neural network includes a neural network trained to minimize a loss function based on a difference between an actual change in a feature map between the first frame and the second frame and a predicted change in the feature map between the first frame and the second frame.

10. The method of claim 9, wherein the loss function is further based on a cost function defined according to a complex metric of the student neural network and a classification distribution over a plurality of candidate models.

11. The method of claim 1, wherein the step of generating the inference includes the following steps: identifying one or more objects in the second frame of the video data stream.

12. The method of claim 1, wherein the step of generating the inference includes the steps of: estimating at least one of a pose or a predicted motion of a subject in the video data stream.

13. The method of claim 1, wherein the step of generating the inference includes the following steps: semantically segmenting the video stream into a plurality of segments associated with different subjects captured in the video stream.

14. The method of claim 1, wherein the step of generating the inference includes the step of mapping the second frame to one of a plurality of codes in a latent space, and wherein the method further includes the step of modifying the second frame based on the code to which the second frame is mapped in the latent space.

15. A processor-implemented method comprising the steps of: receiving a training dataset comprising a plurality of video samples, each of the plurality of video samples comprising a plurality of frames; training a teacher neural network based on the training dataset; training a student neural network based on a predicted difference between feature maps of successive frames in each video sample and an actual difference between feature maps of such successive frames in each video sample; and deploying the teacher neural network and the student neural network; wherein the student neural network comprises a network having one or more of a reduced number of channels, a reduced spatial resolution, or a reduced quantization relative to the teacher neural network.

16. As in request item 15, wherein: The teacher neural network and the student neural network are trained to minimize the same task-dependent objective function, which includes a weighted incremental distribution loss term associated with a difference between the actual and predicted changes in the feature maps generated for successive frames in a video sample and a weighted cost term associated with a complexity metric of the student neural network.

17. The method of claim 15, wherein the step of training the student neural network includes the following steps: training the student neural network to minimize a loss between an actual difference between outputs generated for successive frames in a video sample in the training dataset and a predicted difference between outputs generated for successive frames in the video sample.

18. The method of claim 15, wherein the step of training the student neural network includes the following steps: training the student neural network to minimize a cost function defined on a complex metric of the student neural network and a classification distribution on a plurality of candidate models.

19. The method of claim 15, wherein the teacher neural network comprises a linear network.

20. The method of claim 15, wherein the student neural network includes a network configured to decompose weights into a rank lower than that of the teacher neural network.

21. A processing system, comprising: It contains a memory that stores executable instructions; A processor configured to execute executable instructions such that the processing system: receives a video data stream comprising at least a first frame and a second frame; extracts a first feature from the first frame using a teacher neural network; determines a difference between the first frame and the second frame; extracts a second feature from the difference between at least the first frame and the second frame using a student neural network; generates a feature map of the second frame based on a summation of the first and second features; and generates an inference for at least the second frame of the video data stream based on the generated feature map of the second frame; wherein the student neural network comprises a network having one or more of a reduced number of channels, a reduced spatial resolution, or a reduced quantization relative to the teacher neural network.

22. The processing system of claim 21, wherein the processor is further configured such that the processing system: determines a difference between the second frame and a third frame in the video data stream; extracts a third feature from the difference between at least the second frame and the third frame using the student neural network; generates a feature map of the third frame based on a summation of the second features and the third features; and generates an inference for the third frame in the video data stream based on the generated feature map of the third frame.

23. The processing system of request item 21, wherein the teacher neural network includes a linear network.

24. The processing system of claim 21, wherein the teacher neural network includes a nonlinear network.

25. The processing system of claim 21, wherein the second feature is further extracted from the first frame in conjunction with the difference between the first frame and the second frame.

26. The processing system of claim 21, wherein the student neural network includes a neural network trained to minimize a loss function based on a difference between an actual change in a feature map between the first frame and the second frame and a predicted change in the feature map between the first frame and the second frame.

27. The processing system of request item 26, wherein the loss function is further based on a cost function defined according to a complex metric of the student neural network and a classification distribution on a plurality of candidate models.

28. A processing system, comprising: It contains a memory that stores executable instructions; And a processor configured to execute the executable instructions such that the processing system: receives a training dataset comprising a plurality of video samples, each of the plurality of video samples comprising a plurality of frames; trains a teacher neural network based on the training dataset; trains a student neural network based on a predicted difference between feature maps of successive frames in each video sample and an actual difference between feature maps of successive frames in each video sample; and deploys the teacher neural network and the student neural network; wherein the student neural network comprises a network having one or more of a reduced number of channels, a reduced spatial resolution, or a reduced quantization relative to the teacher neural network.

29. The processing system of claim 28, wherein, in order to train the student neural network, the processor is configured such that the processing system: trains the student neural network to minimize a cost function defined on a complex metric of the student neural network and a classification distribution over a plurality of candidate models.

Images