Video Processing Using Delta Distillation

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for processing video content using artificial neural networks. An exemplary method generally includes receiving a video data stream including at least a first frame and a second frame. A first feature is extracted from the first frame using a teacher neural network. A difference between the first frame and the second frame is determined. A second feature is extracted from the difference between at least the first frame and the second frame using a student neural network. A feature map for the second frame is generated based on a summation of the first feature and the second feature. An inference is generated for at least a second frame of the video data stream based on the generated feature map for the second feature.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 264,072, filed November 15, 2021, which claims the benefit of and priority to U.S. Provisional Patent Application No. 18 / 054,274, filed November 10, 2022, each of which is incorporated by reference in its entirety herein. [Background technology]

[0002] Aspects of the present disclosure relate to processing video content.

[0003] Artificial neural networks can be used to perform various operations on video content or other content that includes spatial and temporal components. For example, artificial neural networks can be used to compress video content into a smaller size representation to improve storage and transmission efficiency and match the intended use of the video content (e.g., a suitable resolution of the data relative to the size of a device's display). This compression of content can be performed using lossy techniques, such that the decompressed version of the data is an approximation of the original data that was compressed, or by using lossless techniques, which result in the decompressed version of the data being equivalent (or at least visually equivalent) to the original data. In another embodiment, artificial neural networks can be used to detect objects in video content. Object detection can include, for example, subject pose estimation, which is used to identify moving objects in video content and predict how the objects will move in the future, object classification, etc., to identify objects of interest in video content.

[0004] In general, a temporal component of a video content can be represented by different frames in the video content. The artificial neural network can process the frames in the video content independently through each layer of the artificial neural network. Therefore, the cost of processing the video through the artificial neural network can grow at a different (and higher) rate than the rate of growth of the information in the video content. That is, between successive frames in the video content, there may be small changes between each frame because only a small amount of data may change during the time that elapses between different frames. However, because the neural network generally processes each frame independently, the artificial neural network generally processes repeated data between frames (e.g., parts of a scene that do not change), which is highly inefficient. Summary of the Invention

[0005] Certain aspects provide a method for processing video content using an artificial neural network. An exemplary method generally includes receiving a video data stream including at least a first frame and a second frame. A first feature is extracted from the first frame using a teacher neural network. A difference between the first frame and the second frame is determined. A second feature is extracted from the difference between at least the first frame and the second frame using a student neural network. A feature map for the second frame is generated based on a summation of the first feature and the second feature. An inference is generated for at least a second frame of the video data stream based on the generated feature map for the second feature.

[0006] Certain aspects provide a method for training an artificial neural network to process video content. An exemplary method generally includes receiving a training dataset including a plurality of video samples. Each video sample may include a plurality of frames. A teacher neural network is trained based on the training dataset, and a student neural network is trained based on predicted differences between feature maps for successive frames in each video sample and actual differences between feature maps for successive frames in each video sample. The teacher neural network and the student neural network are deployed.

[0007] Other aspects provide a processing system configured to perform the aforementioned method as well as methods further described herein; a non-transitory computer readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned method as well as methods further described herein; a computer program product embodied on the computer readable storage medium including code for performing the aforementioned method as well as methods further described herein; and the processing system comprising means for performing the aforementioned method as well as methods further described herein.

[0008] The following description and the annexed drawings set forth in detail certain illustrative features of the one or more aspects.

[0009] The accompanying drawings illustrate particular features of one or more embodiments and therefore should not be considered as limiting the scope of the disclosure. [Brief description of the drawings]

[0010] [Figure 1] 1 illustrates an exemplary distillation architecture, including a teacher neural network and a student neural network. [Diagram 2] 1 illustrates an example of delta distillation used to distill changes between frames using a teacher neural network and a student neural network. [Diagram 3] 1 illustrates an example of training neural networks, including a teacher neural network and a student neural network, based on frame-to-frame changes in video content according to aspects of the present disclosure. [Figure 4] 1 illustrates an example of using a neural network to perform inference regarding video content based on frame-to-frame changes in the video content, according to aspects of the present disclosure. [Diagram 5] 1 illustrates example operations for training a neural network to perform inference on video content based on deltas distilled from successive video frames, in accordance with aspects of this disclosure. [Figure 6] 1 illustrates example operations for performing inference on video content using a neural network trained to process the video content based on deltas distilled from successive video frames in accordance with aspects of the present disclosure. [Figure 7] 1 illustrates an example system in which aspects of the present disclosure may be implemented. [Figure 8] 1 illustrates another example system in which aspects of the present disclosure may be implemented.

[0011] For ease of understanding, wherever possible, like reference numerals have been used to designate like elements common to the figures, and it is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further detail. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Aspects of the present disclosure provide techniques for efficiently processing video content using artificial neural networks.

[0013] As discussed, artificial neural networks can be used to perform various inference operations on video content. These inferences can be used, for example, in various compression schemes, object detection, computer vision operations, various image processing and modification operations (e.g., size enlargement, noise removal, etc.), etc. However, artificial neural networks may process each video frame in the video content independently. Therefore, these artificial neural networks may not exploit various redundancies across frames in the video content.

[0014] Training a neural network and performing inference using the trained neural network can be a computationally complex task, with the degree of computational complexity corresponding to the accuracy of the neural network. That is, a more accurate model may be more computationally complex to train and use, while a less accurate model may be less computationally complex to train and use. To allow for improved computational complexity while retaining accuracy in the artificial neural network, redundancy in the data can be exploited. For example, channel redundancy can allow for pruning weights based on various error terms, such that weights that have minimal or no impact on inference (e.g., for a particular channel, such as a color channel in image data) are removed from the trained neural network, quantization can be used to represent weights using smaller bit widths, and singular value decomposition can be used to approximate the weight matrix with a more compact representation. In another example, spatial redundancy can be used to exploit similarities in the spatial domain. In yet another example, knowledge distillation can be used, where a student neural network is trained to match the feature output of a teacher neural network. However, these techniques cannot exploit temporal redundancy in video content or other content that contains a temporal component.

[0015] Aspects of the present disclosure provide techniques that exploit temporal redundancies in video content or other content with a temporal component to train neural networks and use the neural networks to generate inferences for video content or other content with a temporal component. These temporal redundancies can be represented by differences or deltas between successive portions of video content or other content with a temporal component. By training a neural network based on deltas distilled from successive portions of video content or other content with a temporal component and using the trained neural network to perform inferences, aspects of the present disclosure can reduce the amount of data used in training the neural network and performing inferences using the neural network. This can speed up the process of training the neural network and performing inferences using the neural network, thereby reducing the number of processing cycles and memory used in these operations, reducing the amount of power used in the training and inference operations, etc.

[0016] Exemplary Delta Distillation in Artificial Neural Networks 1 illustrates an example distillation architecture 100 that includes a teacher neural network and a student neural network. In the teacher-student neural network paradigm, the teacher neural network can be a larger network than the student neural network, and data learned by the teacher neural network can be used to train the student network using data distillation techniques, as discussed in more detail below.

[0017] As shown, distillation architecture 100 includes a teacher neural network 110 and a student neural network 120. Teacher neural network 110 typically includes a layers, and student neural network 120 includes b layers (which may be greater than a, as discussed in more detail below). Inputs 105 may be provided to both teacher neural network 110 and student neural network 120, which generate inter-feature distillation losses 115.

[0018] The teacher neural network 110 can be expressed as a global backbone F as a composition of L parameter blocks according to the following equation:

[0019]

number

[0020] Each parameter block can map an input to an output z. In general, for the l-th parameter block, the output generated by this block can be expressed according to the following equation:

[0021]

number

[0022] Given a backbone F, one technique for reducing the computational cost of performing training and inference involves knowledge distillation. In this example, for each

[0023]

number

[0024]

number

[0025]

number

[0026]

number

[0027]

number

[0028] For data that has a time component, such as video content,

[0029]

number

[0030]

number

[0031]

number

[0032] 2 illustrates one embodiment in which a teacher neural network 210 and a student neural network 220 are trained to distill the difference (or delta) between different video frames that are separated in time. As shown, both the teacher neural network 210, represented by the function f discussed above, and the student neural network 220, represented by the function g discussed above, receive as input two video frames x t-1 and x t (e.g., a current frame with timestamp t, and a previous frame with timestamp t-1). The teacher neural network 210 receives both inputs x t-1 and x t Generate a feature map for Δz t The input x is expressed as t-1 and x t The student neural network 220 can similarly generate a difference feature map at time t that represents the difference between two video frames x t-1 and x tcan receive as input a video frame x t-1 and x t It can be trained to predict the difference between feature maps generated with respect to

[0033]

number

[0034]

number

[0035]

number

[0036]

number

[0037] Frame x t frame x t-1 and an additive delta, so the feature map z t Similarly, the feature map z t-1 (For example, frame x t-1 can be expressed as the sum of (feature maps with respect to ) plus an additive delta:

[0038]

number

[0039]

number

[0040]

number

[0041] FIG. 3 illustrates an example implementation 300 of training and using a neural network, including a teacher neural network 310 and a student neural network 320, based on frame-to-frame changes in video content according to aspects of the present disclosure.

[0042] As shown, the function f l A teacher neural network 310, represented by x, can be trained to generate a feature map z for any given input video frame x. The feature map z generated by the teacher neural network 310 can be treated as ground truth data for training a student neural network 320. The teacher neural network 310 can receive frame x as input, and the student neural network 320 can receive the difference Δx between different frames as input. At a given time t+n, the difference between frames used as input to the student neural network is

[0043]

number

[0044] The student neural network 320 may be represented as one or more linear blocks, where the student neural network receives as input residual values ​​between different samples in the time data (e.g., different video frames). In such a case, the student neural network 320 may be represented as

[0045]

number

[0046]

number

[0047]

number

[0048]

number

[0049] In general, the structure of the student neural network 320 can be selected based on a channel reduction strategy or a spatial reduction strategy. In a channel reduction strategy, the student neural network 320 can mirror the teacher neural network 310 in structure, but can have fewer channels than the teacher neural network 310. A number of pointwise convolutions can be introduced into the blocks in the student neural network 320 as the first and last layers, which can reduce and expand the number of channels, respectively. In a spatial reduction strategy, the student neural network 320 can be similar to the teacher neural network 310. However, the student neural network 320 can operate using a smaller spatial resolution on the input video frames, which can be achieved through a pointwise strided convolution layer (e.g., a convolution layer using a 1×1 kernel with intervals between different parts of the input video frame) introduced as the first layer in the student neural network 320 and a pixel shuffle upsampling layer introduced as the last layer in the student neural network 320.

[0050] Within the backbone model F implemented by the teacher neural network 310, different layers may be compressible to different degrees. For example, some layers may not be compressible (or distillable) because they may completely impair the performance of the teacher neural network 310. Therefore, the student neural network

[0051]

number

[0052]

number

[0053]

number

[0054]

number

[0055]

number

[0056] The delta distillation block 315 can be used to optimize the video model for a particular application. task (Θ), where Θ={θ 1 ,...,θ L} represent the parameters of the backbone model F implemented by the teacher neural network 310. During training, L task can be optimized on a training video clip, where both the teacher neural network 310 and the student neural network 320 contribute to the prediction. The teacher neural network 310 may contribute to predicting the output z (e.g., feature map) for the initial frame, and the student neural network 320 may contribute to predicting the output z for the remaining frames.

[0057] To optimize delta distillation (e.g., via delta distillation block 315), a target delta Δz t Block students

[0058]

number

[0059]

number

[0060]

number

[0061]

number

[0062]

number

[0063] When possible, we choose candidate networks that are computationally inexpensive (e.g.,

[0064]

number

[0065]

number

[0066]

number

[0067]

number

[0068]

number

[0069]

number

[0070] In general, in the course of training the student neural network 320, the distilled deltas will converge to the teacher's deltas. As the gradients of the task loss are backpropagated over time to the teacher neural network 310, these task loss gradients can provide a representation for the first frame that can be additively updated by the student neural network 320. Therefore, the task loss gradients can prompt the teacher to provide a representation that is easier to update, which can improve temporal consistency within the network. Furthermore, the techniques described herein can convert the backbone F implemented in the teacher neural network 310 into a recurrent model, which further improves temporal consistency in the pipeline including the teacher neural network 310 and the student neural network 320, since the teacher neural network can propagate the output from one time point to another.

[0071] FIG. 4 illustrates an example implementation 400 of using a neural network to perform inference regarding video content based on frame-to-frame changes in the video content according to aspects of the present disclosure.

[0072] To perform inference on the video content, frames in the video content can be processed using a teacher neural network 410 and one or more student neural networks 420. In general, for a given input of video data, the output generated by the teacher neural network 410 and the student neural network 420 can be expressed according to the following equation:

[0073]

number

[0074] The teacher neural network 410 can process a designated initial frame, such as a key frame, from which subsequent frames are defined in terms of their deltas relative to the key frame, while the student neural network 420 can process other frames in the video content based at least in part on the delta between the given frame and a previous frame. t With respect to t At time t+1, frame x t+1 With respect to delta Δx t+1 A, frame x t and frame x t+1 The student neural network 420 then calculates Δx t+1 Feature map for

[0075]

number

[0076]

number

[0077]

number

[0078]

number

[0079]

number

[0080]

number

[0081] Exemplary Method for Video Processing Using Delta Distillation and Teacher and Student Neural Networks 5 illustrates example operations 500 that may be performed by a system to train a neural network to perform inferences on video content based on deltas distilled from successive video frames (or differences between frames in the video content) in accordance with certain aspects of the present disclosure. Operations 500 may be performed by a computing device (e.g., processing system 700 shown in FIG. 7) that may, for example, train a machine learning model and deploy the machine learning model to another device for use in generating inferences from video data.

[0082] As shown, the operation 500 begins at block 510 where a training data set is received. In general, the training data set may include a plurality of video samples, and each video sample of the plurality of video samples may include a plurality of frames. One or more of the plurality of frames for each video sample may be designated as a key frame, and a frame after a first key frame and before a second key frame may be defined based on a difference relative to the first key frame.

[0083] At block 520, a teacher neural network is trained based on the training data set. In general, when training the teacher neural network, the teacher neural network may be trained to generate a feature map for each frame in each of a plurality of video samples. The teacher neural network may be represented, for example, as a backbone F of L parameter blocks and may generate an output z according to the following equation:

[0084]

number

[0085] A student neural network is trained at block 530. In general, the student neural network may be trained based on predicted differences between feature maps for successive frames in each video sample and actual differences between feature maps for successive frames in each video sample.

[0086] In some aspects, the teacher neural network and the student neural network can be trained to minimize the same task-specific objective function. The task-specific objective function can be, for example, an objective function based on each of a plurality of parameters of the model implemented by the teacher neural network. The task-specific objective function can be defined based on a weighted delta-distributed loss term and a weighted cost term. For example, the task-specific objective function can be expressed as the formula

[0087]

number

[0088]

number

[0089]

number

[0090] In some aspects, training the student neural network may include training the student neural network to minimize a delta distillation loss, which may generally represent the difference between an actual difference between outputs generated for successive frames in a video sample in a training dataset (e.g., generated by a teacher neural network) and a predicted difference between outputs generated for successive frames in that video sample (e.g., generated by a student neural network).

[0091] In some aspects, training the student neural network may include training the student neural network to minimize a cost function defined based on a complexity measure for the student neural network and a category distribution across the multiple candidate models. For example, training the student neural network may be based on a complexity measure for a first neural network that matches the teacher neural network in computational complexity and a second neural network that has reduced complexity.

[0092] At block 540, the teacher neural network and the student neural network are deployed. For example, the teacher neural network and the student neural network can be deployed to devices that perform inferences on captured video data, such as user equipment (UE) in a wireless network, vehicles with autonomous driving capabilities that operate at least in part based on computer vision capabilities, etc. These inferences can include, for example, encoding content into a latent space for compression, object detection in the video content, subject pose estimation and movement prediction, semantic segmentation of the video content into various segments, etc.

[0093] 6 illustrates example operations 600 for performing inference on video content using a neural network trained to process the video content based on deltas distilled from successive video frames, according to certain aspects of the present disclosure. The operations 600 may be performed by a device (e.g., processing system 800 shown in FIG. 8) on which the teacher neural network and the student neural network are deployed, such as, for example, a user equipment (UE), an autonomous vehicle, etc.

[0094] As shown, the operation 600 may begin at block 610 where a video data stream is received. In general, the video data stream may include key frames and one or more non-key frames. A key frame may be a frame that is used by a neural network as an initial reference from which inference is performed, and inference may be performed on the non-key frames based on differences between successive frames.

[0095] At block 620, a teacher neural network can be used to extract first features from a first frame. The first frame can be a key frame or an initial frame from which other frames (e.g., defined in terms of differences to apply to the first frame) in the video content are derived, as discussed.

[0096] A difference between the first frame and the second frame is determined at block 630. In general, the difference between the first frame and the second frame may include information about the change in each pixel between the first frame and the second frame such that a combination of the first frame and the determined difference results in the second frame.

[0097] At block 640, a second feature is extracted from at least the difference between the first frame and the second frame using the student neural network. In general, the second feature may be an approximation of the difference between the first feature and the feature that would have been extracted by the teacher neural network for the second frame. In some aspects, the second feature may be further extracted from the first frame in conjunction with the difference between the first frame and the second frame.

[0098] At block 650, a feature map is generated for the second frame based on the sum of the first feature and the second feature. As discussed, the first feature can generally be a set of features extracted in its entirety from the first frame, and the second feature can be a set of features extracted from the difference between the first and second frames. Because the second frame can be represented as the sum of the first frame and the difference between the first and second frames, the feature map representing the second frame can similarly be represented as the sum of the first feature (extracted from the first frame) and the second feature (extracted from the difference between the first and second frames).

[0099] At block 660, an inference is generated for at least a second frame of the video data stream based on the generated feature map for the second frame. In some aspects, generating an inference may include identifying one or more objects in the second frame. In some aspects, generating an inference may include estimating a pose and / or a predicted movement of a subject in the video data stream. The pose estimate and predicted movement may then be used in controlling the autonomous vehicle to react to the predicted movement of the subject recorded in the video content, for example, to avoid a collision between the autonomous vehicle and the subject. In another aspect, generating an inference may include semantically segmenting the video data stream into a number of segments. For example, the video data stream may be segmented into one or more segments associated with various subjects captured in the video stream. In another example, the video data stream may be segmented into foreground and background content, which may allow certain content to be analyzed and other content to be ignored until such time as the content becomes foreground content (if present).

[0100] In yet another example, generating an inference may include mapping the second frame to a code from a plurality of codes in the latent space. Based on the code in the latent space to which the second frame is mapped, one or more modifications may be performed on the second frame. For example, the second frame or a portion of the second frame (e.g., a portion of interest, such as foreground content, a particular object, etc.) may be modified. The modifications may include changing the resolution (enlarging and / or reducing the size such that details are preserved after modifying the second frame), denoising, and other modifications that may be performed on the image or portions of the image.

[0101] In some aspects, the teacher neural network can be a linear network including multiple linear blocks, as discussed above. The student neural network can be configured to decompose the weights to a lower rank than the rank of the weights in the teacher neural network. The student neural network can include multiple group convolutional layers. When the teacher neural network is a linear network, the student neural network can calculate the weights based on the difference between the previous frame and the current frame (e.g., Δx t =x t -x t-1 ), a second feature (e.g., Δz t ), and it may not be necessary to receive the previous frame to generate the second feature.

[0102] In some aspects, the teacher neural network may be a nonlinear network including multiple nonlinear blocks. In such a case, the student neural network may be a network having one or more of a reduced number of channels, reduced spatial resolution, or reduced quantization compared to the teacher neural network. As discussed, when the teacher neural network is a nonlinear network, the student neural network may be a network having one or more of a reduced number of channels, reduced spatial resolution, or reduced quantization compared to the teacher neural network.t-1 , and based on the difference between the previous frame and the current frame (e.g., Δx t =x t -x t-1 ), a second feature (e.g., Δz t ) can be generated.

[0103] In some aspects, the student neural network can be a neural network trained to minimize a loss function based on the difference between the actual change in the feature map between the first frame and the second frame and the predicted change in the feature map between the first frame and the second frame. The loss function can be further based on a cost function defined based on a measure of the complexity of the student neural network and a category distribution across the multiple candidate models. The multiple candidate models can be, for example, a student neural network that operates at the same computational cost as the teacher neural network (e.g., distills deltas between frames without compression).

[0104]

number

[0105]

number

[0106] In some aspects, a difference between a second frame and a third frame in the video data stream can be determined. A student neural network can be used to extract a third feature from the difference between at least the second frame and the third frame. A feature map can be generated for the third frame based on a summation of the second feature and the third feature, and an inference can be generated for the third frame based on the generated feature map for the third frame.

[0107] Exemplary Processing System for Video Processing Using Delta Distillation in Neural Networks FIG. 7 illustrates an example processing system 700 for training a machine learning model to perform inference on video content (or other content having a temporal component) using delta distillation, a teacher neural network, and a student neural network, e.g., as described herein with respect to FIG. 5.

[0108] The processing system 700 includes a central processing unit (CPU) 702, which in some embodiments may be a multi-core CPU. Instructions executed in the CPU 702 may be loaded from a program memory associated with the CPU 702, or may be loaded from a memory 724, for example.

[0109] The processing system 700 also includes additional processing components tailored to particular functions, such as a graphics processing unit (GPU) 704, a digital signal processor (DSP) 706, a neural processing unit (NPU) 708, multimedia components 710, and wireless connectivity components 712.

[0110] NPUs, such as NPU 708, are generally specialized circuits configured to implement control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs may alternatively be referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), vision processing units (VPUs), or graph processing units.

[0111] An NPU, such as NPU 708, is configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some embodiments, multiple NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other embodiments, an NPU may be part of a dedicated neural network accelerator.

[0112] NPUs can be optimized for training or inference, or in some cases can be configured to balance performance between both. For NPUs capable of performing both training and inference, the two tasks can still generally be performed independently.

[0113] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters, such as weights and biases, to improve model performance. In general, optimization based on mispredictions involves backpropagating through layers of the model to determine gradients to reduce prediction errors.

[0114] NPUs designed to accelerate inference are generally configured to operate on complete models, such that such NPUs can be configured to input new data and rapidly process that data through models that have already been trained to produce model outputs (e.g., inferences).

[0115] In one implementation, the NPU 708 is part of one or more of the CPU 702, GPU 704, and / or DSP 706.

[0116] In some embodiments, the wireless connectivity component 712 may include sub-components for, for example, third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 712 is further coupled to one or more antennas 714.

[0117] In some embodiments, one or more of the processors of processing system 700 may be based on the ARM or RISC-V instruction set.

[0118] The processing system 700 may also include one or more sensor processing units 716 associated with any type of sensor, one or more image signal processors (ISPs) 718 associated with any type of image sensor, and / or navigation components 720, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0119] Processing system 700 may also include one or more input and / or output devices 722, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0120] Memory 724 represents one or more static and / or dynamic memories, such as dynamic random access memory (DRAM), flash-based static memory, etc. In this example, memory 724 includes computer-executable components capable of being executed by one or more of the aforementioned components of processing system 700.

[0121] Specifically, in this example, memory 724 includes a training data set receive component 724A, a teacher neural network train component 724B, a student neural network train component 724C, and a neural network deploy component 724D. The illustrated components, and other components not illustrated, can be configured to perform various aspects of the methods described herein.

[0122] In general, the processing system 700 and / or its components may be configured to perform the methods described herein.

[0123] Notably, in other aspects, elements of processing system 700 may be omitted, such as when processing system 700 is a server computer, etc. For example, multimedia component 710, wireless connectivity component 712, sensor processing unit 716, ISP 718, and / or navigation component 720 may be omitted in other aspects. Furthermore, elements of processing system 700 may be distributed, such as for training models and generating inferences using the models.

[0124] FIG. 8 illustrates an example processing system 800 for performing inference on video content (or other content having a temporal component) using delta distillation, a teacher neural network, and a student neural network, e.g., as described herein with respect to FIG. 6 .

[0125] The processing system 800 includes a central processing unit (CPU) 802, which in some embodiments may be a multi-core CPU. The processing system 800 also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 804, a digital signal processor (DSP) 806, and a neural processing unit (NPU) 808. The CPU 802, GPU 804, DSP 806, and NPU 808 may be similar to the CPU 702, GPU 704, DSP 706, and NPU 708 discussed above with respect to FIG.

[0126] In some embodiments, the wireless connectivity component 812 may include sub-components for, for example, third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 812 may further be connected to one or more antennas (not shown).

[0127] In some embodiments, one or more of the processors of processing system 800 may be based on the ARM or RISC-V instruction set.

[0128] Processing system 800 also includes memory 824, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, memory 824 includes computer-executable components capable of being executed by one or more of the aforementioned processors of processing system 800.

[0129] Specifically, in this example, memory 824 includes a video data stream receiving component 824A, a feature extraction component 824B, a difference determination component 824C, a feature map generating component 824D, an inference generating component 824E, and a neural network component 824F (such as neural networks 310 and 320 described above with respect to FIG. 3, or neural networks 410 and 420 described above with respect to FIG. 4). The illustrated components, and other components not shown, may be configured to perform various aspects of the methods described herein.

[0130] In general, the processing system 800 and / or its components may be configured to perform the methods described herein.

[0131] In particular, in other aspects, elements of processing system 800 may be omitted, such as when processing system 800 is a server computer, etc. For example, multimedia components 810, wireless connectivity components 812, sensors 816, ISP 818, and / or navigation components 820 may be omitted in other aspects.

[0132] Illustrative clauses Details of the implementation of various aspects of the disclosure are set forth in the following numbered clauses.

[0133] Clause 1: A method comprising: receiving a video data stream including at least a first frame and a second frame; extracting a first feature from the first frame using a teacher neural network; determining a difference between the first frame and the second frame; extracting a second feature from the difference between at least the first frame and the second frame using a student neural network; generating a feature map for the second frame based on a summation of the first features and the second features; and generating an inference for at least a second frame of the video data stream based on the generated feature map for the second frame.

[0134] Clause 2: The method of clause 1, wherein the first frame comprises a key frame in the video data stream and the second frame comprises a non-key frame in the video data stream.

[0135] Clause 3: The method of clause 1 or 2, further comprising determining a difference between a second frame and a third frame in the video data stream, extracting a third feature from at least the difference between the second frame and the third frame using a student neural network, generating a feature map for the third frame based on a summation of the second feature and the third feature, and generating an inference for the third frame of the video data stream based on the generated feature map for the third frame.

[0136] Clause 4: Any of the methods of clauses 1 to 3, wherein the teacher neural network comprises a linear network.

[0137] Clause 5: The method of any of clauses 1-4, wherein the student neural network is configured to decompose weights to a lower rank than the rank of the weights in the teacher neural network.

[0138] Clause 6: Any of the methods of clauses 1 to 5, wherein the student neural network includes one or more group convolutional layers.

[0139] Clause 7: Any of the methods of clauses 1 to 6, wherein the teacher neural network includes a nonlinear network.

[0140] Clause 8: Any of the methods of clauses 1-7, wherein the student neural network includes a network having one or more of a reduced number of channels, reduced spatial resolution, or reduced quantization compared to the teacher neural network.

[0141] Clause 9: The method of any of clauses 1-8, wherein a second feature is further extracted from the first frame together with a difference between the first frame and the second frame.

[0142] Clause 10: Any of the methods of clauses 1-10, wherein the student neural network includes a neural network trained to minimize a loss function based on a difference between an actual change in the feature map between the first frame and the second frame and a predicted change in the feature map between the first frame and the second frame.

[0143] Clause 11: The method of clause 10, further based on a cost function, where the loss function is defined based on a measure of complexity of the student neural network and a categorical distribution across multiple candidate models.

[0144] Clause 12: The method of any of clauses 1-11, wherein generating an inference includes identifying one or more objects within a second frame of the video data stream.

[0145] Clause 13: The method of any of clauses 1-12, wherein generating an inference includes estimating at least one of a pose or a predicted movement of a subject in the video data stream.

[0146] Clause 14: Any of the methods of clauses 1-13, wherein generating inferences includes semantically segmenting the video data stream into a plurality of segments associated with different subjects captured in the video data stream.

[0147] Clause 15: The method of any one of clauses 1-14, wherein generating an inference includes mapping the second frame to a code from a plurality of codes in a latent space, the method further including modifying the second frame based on the code in the latent space to which the second frame is mapped.

[0148] Clause 16: A method, comprising: receiving a training dataset including a plurality of video samples, each video sample of the plurality of video samples including a plurality of frames; training a teacher neural network based on the training dataset; training a student neural network based on predicted differences between feature maps for successive frames in each video sample and actual differences between feature maps for successive frames in each video sample; and developing the teacher neural network and the student neural network.

[0149] Clause 17: The method of clause 16, wherein the teacher neural network and the student neural network are trained to minimize the same task-specific objective function, the task-specific objective function including a function defined based on a weighted delta-distribution loss term associated with the difference between actual and predicted changes in feature maps generated for successive frames in the video sample, and a weighted cost term associated with a complexity measure for the student neural network.

[0150] Clause 18: The method of clause 16 or 17, wherein training the student neural network includes training the student neural network to minimize a loss between actual differences between outputs generated for successive frames in the video samples in the training dataset and predicted differences between outputs generated for successive frames in the video samples.

[0151] Clause 19: Any of the methods of clauses 16-18, wherein training the student neural network includes training the student neural network to minimize a cost function defined based on a complexity measure for the student neural network and a categorical distribution across the multiple candidate models.

[0152] Clause 20: Any of the methods of clauses 16 to 19, wherein the teacher neural network includes a linear network.

[0153] Clause 21: The method of any of clauses 16-20, including a network in which the student neural network is configured to decompose weights to a rank lower than the rank of the weights of the teacher neural network.

[0154] Clause 22: Any of the methods of clauses 16 to 21, wherein the teacher neural network includes a nonlinear network.

[0155] Clause 23: A processing system comprising a memory containing computer-executable instructions and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any of clauses 1 to 22.

[0156] Clause 24: A processing system comprising means for carrying out the method according to any of clauses 1 to 22.

[0157] Clause 25: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform any of the methods of clauses 1 to 22.

[0158] Clause 26: A computer program product embodied on a computer-readable storage medium comprising code for performing the method according to any of clauses 1 to 22.

[0159] Additional Considerations The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The embodiments discussed herein are not intended to limit the scope, applicability, or aspects described in the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, changes may be made in the function and arrangement of the elements discussed without departing from the scope of the disclosure. Various embodiments may omit, substitute, or add various procedures or components as appropriate. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some embodiments may be combined in some other embodiments. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects described herein. Furthermore, the scope of the disclosure is intended to encompass apparatuses or methods that are practiced using other structures, functions, or structures and functions in addition to or other than the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim.

[0160] As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0161] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. By way of example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination having multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other permutation of a, b, and c).

[0162] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching a table, database, or another data structure), ascertaining, and the like. "Determining" may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. "Determining" may also include resolving, selecting, choosing, establishing, and the like.

[0163] The methods disclosed herein include one or more steps or actions for achieving the method. The steps and / or actions of the methods may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order of specific steps and / or actions and / or the use of those steps and / or actions may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. Those means may include various hardware and / or software components, including but not limited to circuits, application specific integrated circuits (ASICs), or processors, and / or various hardware and / or software modules. In general, when operations are illustrated in figures, those operations may have corresponding equivalent means-plus-function components that are similarly numbered.

[0164] The following claims are not intended to be limited to the embodiments set forth herein, but are to be accorded the full scope consistent with the language of the claims. Within the claims, reference to an element in the singular is not intended to mean "only one" unless expressly stated as such, but rather "one or more." Unless otherwise expressly stated, the term "several" refers to one or more. No element of a claim is to be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase "means for" or, in the case of a method claim, unless the element is recited using the phrase "step for." All structural and functional equivalents to the elements of the various embodiments described throughout this disclosure that are known or later become known to those of skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is expressly recited in the claims.

Claims

1. A processor-implemented method for generating inferences for video content relating to object detection and computer vision operations, or image processing and modification operations, comprising: receiving a video data stream including at least a first frame and a second frame; extracting first features from the first frame using a teacher neural network; determining a difference between the first frame and the second frame; extracting second features from the difference between at least the first frame and the second frame using a student neural network; generating a feature map for the second frame based on a sum of the first feature and the second feature; generating an inference about at least the second frame of the video data stream based on the generated feature map for the second frame, the inference relating to an object detection and computer vision operation or an image processing and modification operation; performing tasks of object detection and computer vision operations or image processing and modification operations and outputting the results of said tasks; Including, the student neural network comprises a network having one or more of a reduced number of channels, reduced spatial resolution, or reduced quantization compared to the teacher neural network; the student neural network is trained to minimize a loss function based on the difference between an actual change in a feature map between the first frame and the second frame and a predicted change in the feature map between the first frame and the second frame.

2. The method of claim 1 , wherein the first frame comprises a key frame in the video data stream and the second frame comprises a non-key frame in the video data stream.

3. determining a difference between the second frame and a third frame in the video data stream; extracting a third feature from the difference between at least the second frame and the third frame using the student neural network; generating a feature map for the third frame based on a sum of the second feature and the third feature; generating an inference about the third frame of the video data stream based on the generated feature map for the third frame; The method of claim 1 further comprising:

4. The method of claim 1 , wherein the teacher neural network comprises a linear network or wherein the teacher neural network comprises a non-linear network.

5. the student neural network is configured to decompose weights to a rank lower than the rank of weights in the teacher neural network; The method of claim 1 , wherein the student neural network includes one or more group convolutional layers.

6. The method of claim 1 , wherein the second feature is further extracted from the first frame along with the difference between the first frame and the second frame.

7. The method of claim 1 , wherein the loss function is further based on a cost function defined based on a measure of complexity of the student neural network and a categorical distribution across multiple candidate models.

8. generating the inference includes identifying one or more objects within the second frame of the video data stream; or generating the inference includes estimating at least one of a pose or a predicted movement of a subject in the video data stream; or generating the inferences includes semantically segmenting the video data stream into a plurality of segments associated with different objects captured in the video data stream; or 10. The method of claim 1 , wherein generating the inference comprises mapping the second frame to a code from a plurality of codes in a latent space, the method further comprising modifying the second frame based on the code in the latent space to which the second frame is mapped.

9. A method executed by a processor for training a neural network for object detection and computer vision operations, or image processing and modification operations, comprising: receiving a training dataset including a plurality of video samples, each video sample of the plurality of video samples including a plurality of frames; training a teacher neural network based on the training data set; training a student neural network based on predicted differences between feature maps for successive frames within each video sample and actual differences between the feature maps for said successive frames within each video sample; deploying the teacher neural network and the student neural network to generate inferences on video content for object detection and computer vision operations, or image processing and modification operations; Including, the student neural network comprises a network having one or more of a reduced number of channels, reduced spatial resolution, or reduced quantization compared to the teacher neural network; The method, wherein training the student neural network includes training the student neural network to minimize a loss function based on a difference between actual changes in a feature map between successive frames in video samples in the training dataset and predicted changes in the feature map between the successive frames.

10. the teacher neural network and the student neural network are trained to minimize the same task-specific objective function; the task-specific objective function includes a function defined based on a weighted delta-distribution loss term associated with the difference between actual and predicted changes in feature maps generated for successive frames in a video sample, and a weighted cost term associated with a complexity measure for the student neural network; 10. The method of claim 9.

11. 10. The method of claim 9, wherein training the student neural network comprises training the student neural network to minimize a loss between actual differences between outputs generated for successive frames in video samples in the training data set and predicted differences between the outputs generated for the successive frames in the video samples.

12. 10. The method of claim 9, wherein training the student neural network comprises training the student neural network to minimize a cost function defined based on a complexity measure for the student neural network and a category distribution across a plurality of candidate models.

13. The method of claim 9 , wherein the teacher neural network comprises a linear network.

14. 10. The method of claim 9, wherein the student neural network comprises a network configured to decompose weights to a rank lower than the rank of weights of the teacher neural network.

15. 1. A processing system comprising: a memory having executable instructions stored therein; a processor configured to execute said executable instructions, said executable instructions causing said processing system to perform the method of any one of claims 1 to 14.

Images