Frame Alignment in Video Processing Using Self-Attention

Abstract

A processor-implemented method for processing a video includes receiving the video as an input at an artificial neural network (ANN). The video includes a sequence of frames. A set of features is extracted for a current frame of the video and a previous frame of the video. The set of features includes a set of supporting features for a set of pixels of the previous frame to be aligned with a set of reference features of the current frame. A similarity between the supporting feature for each pixel in the set of pixels of the set of supporting features of the previous frame and a corresponding reference feature of the current frame is calculated. An attention map is generated based on the similarity. An output including a reconstruction of the current frame is generated based on the attention map.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001]

[0001] This application claims priority to U.S. patent application Ser. No. 17 / 696,797, entitled "EFFICIENT SELF-ATTENTION FOR VIDEO PROCESSING," filed on March 16, 2022, which claims the benefit of U.S. Provisional Patent Application Ser. No. 63 / 162,478, entitled "EFFICIENT SELF-ATTENTION FOR VIDEO PROCESSING," filed on March 17, 2021, the disclosures of which are expressly incorporated by reference in their entireties. [Technical field]

[0002] Aspects of the present disclosure relate generally to neural networks, and more particularly, to video processing using self-attention. [Background technology]

[0003]

[0003] An artificial neural network may comprise an interconnected group of artificial neurons (e.g., neuron models). An artificial neural network may be a computing device or may be represented as a method executed by a computing device.

[0004]

[0004] A neural network is composed of operands that consume tensors and produce tensors. Neural networks can be used to solve complex problems, but the time it takes for the network to complete a task can be long, as the network size and the number of calculations that can be performed to generate a solution can be large. Furthermore, these tasks can be performed on mobile devices that may have limited computing power, so the computational cost of deep neural networks can be an issue.

[0005]

[0005] A convolutional neural network is a type of feed-forward artificial neural network. A convolutional neural network may include a collection of neurons, each with a receptive field, that collectively tile the input space. Convolutional neural networks (CNNs), such as deep convolutional neural networks (DCNs), have many applications. In particular, these neural network architectures are used in a variety of technologies, such as image recognition, pattern recognition, speech recognition, autonomous driving, video processing, and other classification tasks.

[0006]

[0006] Video processing involves obtaining information from adjacent frames. Temporal information obtained from adjacent frames can contribute to video frame reconstruction quality, but it comes at the cost of significant computational overhead. The challenges presented through increased computational overhead are further exacerbated in resource-limited devices, such as smartphones and other mobile devices. Summary of the Invention

[0007]

[0007] The present disclosure is set out in respective independent claims. Some aspects of the disclosure are set out in dependent claims.

[0008]

[0008] In one aspect of the disclosure, a processor-implemented method for processing a video is presented. The processor-implemented method includes receiving a video as an input at an artificial neural network (ANN), the video comprising a sequence of frames. The processor implementation also includes extracting a set of features for each of a current frame of the video and a prior frame of the video. The set of features includes a set of support features for a set of pixels of the current frame to be aligned with a set of reference features of the current frame. The processor implementation additionally includes calculating a similarity between the support feature for each pixel in the set of pixels of the set of support features of the prior frame and a corresponding reference feature of the current frame. The processor implementation further includes generating an attention map based on the similarity. The method also includes generating an output comprising a reconstruction of the current frame based on the attention map.

[0009]

[0009] Another aspect of the disclosure is directed to an apparatus for processing a video. The apparatus includes means for receiving a video as an input at an artificial neural network (ANN). The video includes a sequence of frames. The apparatus also includes means for extracting a set of features for each of a current frame of the video and a previous frame of the video. The set of features includes a set of support features for a set of pixels of the current frame to be aligned with the set of reference features of the current frame. The apparatus additionally includes means for calculating a similarity between the support feature for each pixel in the set of pixels of the set of support features of the previous frame and the corresponding reference feature of the current frame. The apparatus further includes means for generating an attention map based on the similarity. The apparatus also includes means for generating an output comprising a reconstruction of the current frame based on the attention map.

[0010]

[0010] In another aspect of the present disclosure, a non-transitory computer-readable medium having recorded thereon a non-transitory program code for processing a video is disclosed. The program code, when executed by a processor, includes program code for receiving a video as an input in an artificial neural network (ANN). The video includes a sequence of frames. The program code also includes program code for extracting a set of features for each of a current frame of the video and a previous frame of the video. The set of features includes a set of support features for a set of pixels of the current frame to be aligned with the set of reference features of the current frame. The program code additionally includes program code for calculating a similarity between the support feature for each pixel in the set of pixels of the set of support features of the previous frame and the corresponding reference feature of the current frame. The program code further includes program code for generating an attention map based on the similarity. The program code also includes program code for generating an output comprising a reconstruction of the current frame based on the attention map.

[0011]

[0011] Another aspect of the present disclosure is directed to an apparatus for processing a video. The apparatus includes a memory and one or more processors coupled to the memory. The processor(s) is configured to receive a video as an input in an artificial neural network (ANN). The video includes a sequence of frames. The processor(s) is also configured to extract a set of features for each of a current frame of the video and a previous frame of the video. The set of features includes a set of support features for a set of pixels of the current frame to be aligned with the set of reference features of the current frame. The processor(s) is additionally configured to calculate a similarity between the support feature for each pixel in the set of pixels of the set of support features of the previous frame and the corresponding reference feature of the current frame. The processor(s) is further configured to generate an attention map based on the similarity. Furthermore, the processor(s) is configured to generate an output comprising a reconstruction of the current frame based on the attention map.

[0012]

[0012] Additional features and advantages of the present disclosure are described below. It should be understood by those skilled in the art that the present disclosure may be easily utilized as a basis for modifying or designing other structures for carrying out the same purpose of the present disclosure. It should also be recognized by those skilled in the art that such equivalent structures do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features believed to be characteristic of the present disclosure, both in terms of its construction and method of operation, together with further objects and advantages, will be better understood from the following description when considered in conjunction with the accompanying drawings. It should be expressly understood, however, that each of the drawings is provided for the purpose of illustration and description only, and is not intended as a definition of the limits of the present disclosure.

[0013]

[0013] The features, nature, and advantages of the present disclosure will become more apparent from the detailed description set forth below when considered in conjunction with the drawings in which like reference characters refer to like throughout. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 illustrates an example implementation of a system on a chip (SoC) including a central processing unit (CPU) in accordance with certain aspects of the present disclosure. [Figure 2A]

[0015] FIG. 2A is a diagram illustrating a neural network according to an embodiment of the present disclosure. [Figure 2B] FIG. 2B is a diagram illustrating a neural network according to an embodiment of the present disclosure. [Figure 2C] FIG. 2C is a diagram illustrating a neural network according to an embodiment of the present disclosure. [Figure 2D]

[0016] FIG. 2D is a diagram illustrating an example deep convolutional network (DCN) according to an embodiment of the present disclosure. [Diagram 3]

[0017] FIG. 3 is a block diagram illustrating an example deep convolutional network (DCN) according to an aspect of the present disclosure. [Figure 4]

[0018] FIG. 4 is a block diagram illustrating an example software architecture that may modularize artificial intelligence (AI) functions. [Diagram 5]

[0019] FIG. 5 is a block diagram illustrating an example frame of video for processing according to an aspect of this disclosure. [Figure 6]

[0020] FIG. 6 is a block diagram illustrating an example alignment of frames using self-attention, according to an embodiment of the present disclosure. [Figure 7A]

[0021] FIG. 7A illustrates an example architecture for aligning frames using self-attention, according to an aspect of the present disclosure. [Figure 7B] FIG. 7B illustrates an example architecture for aligning frames using self-attention, according to an aspect of the present disclosure. [Figure 8]

[0022] FIG. 8 is a diagram illustrating an example alignment of frames using query selection according to an aspect of the disclosure. [Figure 9]

[0023] FIG. 9 is a flow diagram illustrating a method for processing video according to an aspect of the disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015]

[0024] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the described concepts may be practiced. The detailed description includes specific details intended to provide a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0016]

[0025] Based on the present teachings, one skilled in the art should understand that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether implemented in combination with any other aspect of the present disclosure or implemented independently. For example, an apparatus may be implemented or a method may be performed using any number of the aspects described. In addition, the scope of the present disclosure is intended to cover such an apparatus or method that is implemented using other structures, functions, or structures and functions in addition to or other than the various aspects of the present disclosure described. It should be understood that any aspect of the present disclosure disclosed may be embodied by one or more elements of a claim.

[0017]

[0026] The word "exemplary" is used to mean "serving an example, instance, or illustration." Any aspect described as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0018]

[0027] Although specific aspects are described, many variations and permutations of these aspects are within the scope of the present disclosure. Although some benefits and advantages of the preferred aspects are described, the scope of the present disclosure is not intended to be limited to specific benefits, applications or purposes. Rather, the aspects of the present disclosure are intended to be broadly applicable to different technologies, system configurations, networks and protocols, some of which are illustrated by way of example in the drawings and the following description of the preferred aspects. The detailed description and drawings are merely illustrative of the present disclosure, rather than limiting, the scope of the present disclosure being defined by the appended claims and their equivalents.

[0019]

[0028] Neural networks can be used to solve complex problems. However, the time it takes for a network to complete a task can be long, as the network size and the number of calculations that can be performed to generate a solution can be large. Furthermore, these tasks can be performed on mobile devices that may have limited computing power, so the computational cost of deep neural networks can be an issue.

[0020]

[0029] Convolutional neural networks (CNNs), such as deep convolutional neural networks (DCNs), have many applications. In particular, these neural network architectures are used in a variety of technologies, including, for example, video processing.

[0021]

[0030] Video processing involves obtaining information from neighboring frames. In the presence of misregistration (e.g., due to local or global motion), convolution kernels may struggle to exploit temporal information. Traditional approaches rely on expansive operators, which lead to significant computational cost, delay, and energy consumption.

[0022]

[0031] To address these and other challenges, aspects of the present disclosure are directed to an attention-based alignment model. The attention-based alignment model exploits the locality of motion between adjacent supporting frames. According to aspects of the present disclosure, since pixels undergo displacement between successive frames, the attended context may be limited to a local spatial neighborhood. Additionally, since pixels may not undergo the same amount of displacement from one frame to the next, the alignment operation may be limited to regions that undergo significant change under the control of a gating module. The gating module may learn a binary function over the pixels that indicates where alignment may be skipped based on the input.

[0023]

[0032] Additionally, aspects of the present disclosure are broadly applicable in the field of video processing, including but not limited to video-to-video processing. For example, aspects of the present disclosure may be applied to improve super-resolution in video (e.g., smartphone sensors) and other video enhancements such as frame alignment, motion compensated denoising, deblocking, deblurring, and stabilization in video.

[0024]

[0033] 1 illustrates an example implementation of a system-on-chip (SoC) 100, which may include a central processing unit (CPU) 102 or a multi-core CPU configured for video processing using an artificial neural network (e.g., a neural end-to-end network). Variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a weighted neural network), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with the CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, or distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from a program memory associated with the CPU 102 or from the memory block 118.

[0025]

[0034] The SoC 100 may also include a GPU 104, a DSP 106, connectivity blocks 110, which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, and the like, as well as additional processing blocks tailored to specific functions, such as a multimedia processor 112 that may detect and recognize gestures. In one implementation, the NPU 108 is implemented in the CPU 102, the DSP 106, and / or the GPU 104. The SoC 100 may also include a sensor processor 114, an image signal processor (ISP) 116, and / or a navigation module 120, which may include a global positioning system.

[0026]

[0035] The SoC 100 may be based on an ARM instruction set. In one aspect of the disclosure, the instructions loaded into the general-purpose processor 102 may include code for receiving a video as an input in an artificial neural network (ANN). The video comprises a sequence of frames. The general-purpose processor 102 may also include code for extracting a set of features of a current frame of the video and a previous frame of the video. The set of features includes a set of support features for a set of pixels of the previous frame to be aligned with the set of reference features of the current frame. The general-purpose processor 102 may additionally include code for calculating a similarity between a support feature for each pixel in the set of pixels of the set of support features of the previous frame and a corresponding reference feature of the current frame. The general-purpose processor 102 may also include code for generating an attention map based on the similarity. Additionally, the general-purpose processor 102 may include code for generating an output comprising a reconstruction of the current frame based on the attention map.

[0027]

[0036] Deep learning architectures can perform object recognition tasks by learning to represent inputs at successively higher levels of abstraction at each layer, thereby building up useful feature representations of the input data. In this way, deep learning addresses the major bottleneck of traditional machine learning. Before the advent of deep learning, machine learning approaches to object recognition problems relied heavily on human engineered features, possibly in combination with shallow classifiers. A shallow classifier could be, for example, a two-class linear classifier in which a weighted sum of feature vector components could be compared to a threshold to predict which class an input belongs to. Human engineered features could be templates or kernels tuned to a particular problem domain by an engineer with domain expertise. Deep learning architectures, in contrast, can learn to represent features similar to those that a human engineer could design, albeit through training. Furthermore, deep networks can learn to represent and recognize new types of features that humans may not have considered.

[0028]

[0037] Deep learning architectures may learn a hierarchy of features. For example, when presented with visual data, a first layer may learn to recognize relatively simple features, such as edges, in the input stream. In another example, when presented with auditory data, the first layer may learn to recognize spectral power at specific frequencies. A second layer, taking the output of the first layer as input, may learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases.

[0029]

[0038] Deep learning architectures can perform particularly well when applied to problems that have a natural hierarchical structure. For example, motorized vehicle classification can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways in higher layers to recognize cars, trucks, and planes.

[0030]

[0039] Neural networks can be designed with a variety of connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, with each neuron in a given layer transmitting to neurons in higher layers. Hierarchical representations can be built in successive layers of feedforward networks, as described above. Neural networks can also have recurrent connections or feedback connections (also called top-down). In recurrent connections, the output from a neuron in a given layer can be transmitted to another neuron in the same layer. Recurrent architectures can be useful for recognizing patterns that span more than one of the input data chunks that are given to the neural network in a sequence. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be useful when recognition of higher-level concepts can help distinguish certain lower-level features of the input.

[0031]

[0040] The connections between layers of a neural network can be fully connected or locally connected. FIG. 2A illustrates an example of a fully connected neural network 202. In the fully connected neural network 202, a neuron in a first layer may transmit its output to all neurons in a second layer, such that each neuron in the second layer receives input from all neurons in the first layer. FIG. 2B illustrates an example of a locally connected neural network 204. In the locally connected neural network 204, a neuron in a first layer may be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in a layer has the same or similar connectivity pattern, but with connection strengths (e.g., 210, 212, 214, and 216) that may have different values. The connectivity patterns of local connections may give rise to spatially distinct receptive fields in the upper layers, since upper layer neurons in a given region may receive inputs that are tuned through training to the properties of a restricted subset of the total inputs to the network.

[0032]

[0041] One example of a locally connected neural network is a convolutional neural network. Figure 2C illustrates an example of a convolutional neural network 206. The convolutional neural network 206 is configured such that the connection strengths (e.g., 208) associated with the inputs for each neuron in the second layer are shared. Convolutional neural networks may be well suited for problems in which the spatial location of the inputs is meaningful.

[0033]

[0042] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D illustrates a detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capture device 230, such as an on-board camera. The DCN 200 of the current example may be trained to identify traffic signs and numbers provided on traffic signs. Of course, the DCN 200 may be trained for other tasks, such as identifying lane markings or identifying traffic signals.

[0034]

[0043] The DCN 200 may be trained using supervised learning. During training, the DCN 200 may be presented with an image, such as the image 226 of a speed limit sign, and then a "forward pass" may be computed to generate the output 222. The DCN 200 may include a feature extraction section and a classification section. Upon receiving the image 226, the convolution layer 232 may apply a convolution kernel (not shown) to the image 226 to generate the first set of feature maps 218. As an example, the convolution kernel for the convolution layer 232 may be a 5×5 kernel that generates a 28×28 feature map. In this example, four different convolution kernels were applied to the image 226 in the convolution layer 232 because four different feature maps are generated in the first set of feature maps 218. The convolution kernel may also be referred to as a filter or a convolution filter.

[0035]

[0044] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220, such as 14×14, is smaller than the size of the first set of feature maps 218, such as 28×28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0036]

[0045] In the example of FIG. 2D , the second set of feature maps 220 are convolved to generate a first feature vector 224. Moreover, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature in the second feature vector 228 may include a number that corresponds to a possible feature of the image 226, such as “sign,” “60,” and “100.” A softmax function (not shown) may convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of the DCN 200 is the probability that the image 226 contains one or more features.

[0037]

[0046] In this example, the probability in output 222 for "sign" and "60" is higher than the probability of others of output 222, such as "30", "40", "50", "70", "80", "90", and "100". Before training, the output 222 generated by DCN 200 is likely to be incorrect. Thus, an error may be calculated between output 222 and a target output. The target output is the ground truth of image 226 (e.g., "sign" and "60"). The weights of DCN 200 may then be adjusted so that the output 222 of DCN 200 is more closely aligned with the target output.

[0038]

[0047] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error would increase or decrease if the weights were adjusted. In the top layer, the gradient may correspond directly to the values ​​of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the values ​​of the weights and the calculated error gradients of the upper layers. The weights may then be adjusted to reduce the error. This method of adjusting weights may be called "back propagation" because it involves a "backward pass" through the neural network.

[0039]

[0048] In practice, the error gradient of the weights may be calculated over a small number of examples such that the calculated gradient approximates the true error gradient. This approximation method may be called stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the overall system stops decreasing or until the error rate reaches a target level. After training, the DCN may be presented with new images and a forward pass through the network may result in an output 222, which may be considered the inference or prediction of the DCN.

[0040]

[0049] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of a training dataset. DBNs can be obtained by stacking up layers of Restricted Boltzmann Machines (RBMs). RBMs are a type of artificial neural network that can learn probability distributions over a set of inputs. RBMs are often used in unsupervised learning because they can learn probability distributions without information about the class into which each input should be categorized. Using a hybrid unsupervised and supervised paradigm, the bottom RBM of a DBN can be trained in an unsupervised manner and act as a feature extractor, and the top RBM can be trained in a supervised manner (with a joint distribution of the target class and the inputs from the previous layer) and act as a classifier.

[0041]

[0050] Deep Convolutional Networks (DCNs) are networks of convolutional networks, constructed with additional pooling and normalization layers. DCNs have reached state-of-the-art performance for many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many exemplars and are used to modify the network weights by using gradient descent.

[0042]

[0051] The DCN may be a feedforward network. In addition, as described above, the connections from neurons in the first layer of the DCN to groups of neurons in the next higher layer are shared across the neurons in the first layer. The feedforward and shared connections of the DCN may be exploited for high speed processing. The computational burden of the DCN may be much less than that of a similarly sized neural network with, for example, recurrent or feedback connections.

[0043]

[0052] The processing of each layer of a convolutional network can be considered to be a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, a convolutional network trained on that input can be considered to be three-dimensional, with two spatial dimensions along the axes of the image and a third dimension capturing color information. The output of the convolutional connections can be considered to form a feature map in a subsequent layer, with each element of the feature map (e.g., 220) receiving input from each of the multiple channels and from a range of neurons in the previous layer (e.g., feature map 218). Values ​​in the feature map can be further processed using nonlinearities, such as rectification, max(0,x). Values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction. Normalization, corresponding to whitening, can also be applied through lateral inhibition between neurons in the feature map.

[0044]

[0053] The performance of deep learning architectures can improve as more labeled data points become available or as computational power increases. Modern deep neural networks are routinely trained with computational resources thousands of times greater than those available to the typical researcher only 15 years ago. New architectures and training paradigms can further boost deep learning performance. Rectified linear units can reduce a training challenge known as vanishing gradients. New training techniques can reduce over-fitting and thus enable larger models to achieve better generalization. Encapsulation techniques can abstract data in a given receptive field, further boosting overall performance.

[0045]

[0054] 3 is a block diagram illustrating a deep convolutional network 350. The deep convolutional network 350 may include multiple different types of layers based on connectivity and weight sharing. As shown in FIG. 3, the deep convolutional network 350 includes convolutional blocks 354A, 354B. Each of the convolutional blocks 354A, 354B may be composed of a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a max pooling layer (MAX POOL) 360.

[0046]

[0055] The convolution layer 356 may include one or more convolution filters, which may be applied to the input data to generate feature maps. Although only two convolution blocks 354A, 354B are shown, the disclosure is not so limited, and instead, any number of convolution blocks 354A, 354B may be included in the deep convolution network 350 according to design preference. The normalization layer 358 may normalize the output of the convolution filters. For example, the normalization layer 358 may provide whitening or lateral inhibition. The max pooling layer 360 may provide downsampling aggregation across space for local invariance and dimensionality reduction.

[0047]

[0056] For example, parallel filter banks of the deep convolutional network may be loaded on the CPU 102 or GPU 104 of the SoC 100 to achieve high performance and low power consumption. In alternative embodiments, the parallel filter banks may be loaded on the DSP 106 or ISP 116 of the SoC 100. In addition, the deep convolutional network 350 may access other processing blocks that may be present on the SoC 100, such as the sensor processor 114 and the navigation module 120 dedicated to sensors and navigation, respectively.

[0048]

[0057] The deep convolutional network 350 may also include one or more fully connected layers 362 (FC1 and FC2). The deep convolutional network 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of the deep convolutional network 350, there are weights (not shown) to be updated. The output of each of these layers (e.g., 356, 358, 360, 362, 364) may serve as an input of a subsequent one of these layers (e.g., 356, 358, 360, 362, 364) in the deep convolutional network 350 to learn a hierarchical feature representation from the input data 352 (e.g., image, audio, video, sensor data, and / or other input data) provided at the beginning of the convolution block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 may be a set of probabilities, where each probability is the probability that the input data contains a feature from the set of features.

[0049]

[0058] 4 is a block diagram illustrating an example software architecture 400 that may modularize artificial intelligence (AI) functions. Using this architecture, applications may be designed that may cause various processing blocks (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) of a system on chip (SoC) 420 to support adaptive rounding as disclosed for post-training quantization for AI applications 402 in accordance with aspects of the present disclosure.

[0050]

[0059] The AI ​​application 402 may be configured to invoke functions defined in user space 404, which may provide, for example, scene detection and recognition indicative of the location in which the device is currently operating. The AI ​​application 402 may configure the microphone and camera differently depending, for example, on whether the recognized scene is an office, an auditorium, a restaurant, or an outdoor environment such as a lake. The AI ​​application 402 may make requests to compiled program code associated with libraries defined in the AI ​​function application programming interface (API) 406. The requests may ultimately rely on the output of a deep neural network configured to provide an inference response based, for example, on the video and positioning data.

[0051]

[0060] Further accessible to the AI ​​application 402 may be a runtime engine 408, which may be compiled code of a runtime framework. The AI ​​application 402 may cause the runtime engine to request inference, for example, at specific time intervals or triggered by an event detected by the application's user interface. When caused to provide an inference response, the runtime engine may in turn signal an operating system in the operating (OS) space, such as kernel 412, running on the SoC 420. The operating system may in turn cause successive relaxations of quantization to be performed on the CPU 422, DSP 424, GPU 426, NPU 428, or some combination thereof. The CPU 422 may be accessed directly by the operating system, and other processing blocks may be accessed through drivers, such as drivers 414, 416, or 418 for the DSP 424, GPU 426, or NPU 428, respectively. In an illustrative example, the deep neural network may be configured to run on a combination of processing blocks, such as CPU 422, DSP 424, and GPU 426, or may be run on NPU 428.

[0052]

[0061] An application 402 (e.g., an AI application) may be configured to invoke functions defined in user space 404, which may provide, for example, scene detection and recognition indicative of the location in which the device is currently operating. The application 402 may configure the microphone and camera differently depending on whether the recognized scene is, for example, an office, an auditorium, a restaurant, or an outdoor environment such as a lake. The application 402 may make a request to compiled program code associated with a library defined in SceneDetect application programming interface (API) 406 to provide an estimate of the current scene. The request may ultimately rely on the output of a differential neural network configured to provide a scene estimate based on, for example, video and positioning data.

[0053]

[0062] Further accessible to the application 402 may be a runtime engine 408, which may be compiled code of a runtime framework. The application 402 may cause the runtime engine to request scene estimation, for example, at specific time intervals or triggered by an event detected by the application's user interface. Upon being caused to estimate the scene, the runtime engine may in turn signal an operating system 410, such as kernel 412, running on the SoC 420. The operating system 410 may in turn cause calculations to be performed on the CPU 422, DSP 424, GPU 426, NPU 428, or some combination thereof. The CPU 422 may be accessed directly by the operating system, and other processing blocks may be accessed through drivers, such as drivers 414-418 for the DSP 424, for the GPU 426, or for the NPU 428. In an illustrative example, the differential neural network may be configured to run on a combination of processing blocks, such as the CPU 422 and the GPU 426, or may be run on the NPU 428.

[0054]

[0063] FIG. 5 is a diagram illustrating an example frame of a video 500 for processing according to an aspect of the disclosure. Referring to FIG. 5, a pair of sequential frames (e.g., t and t+1) of the video 500 are shown. Frames t and t+1 each include a grid of pixels. Multiple separate objects 502, 504 are represented in the frames. For example, object 502 is represented by a square and object 504 is represented by a triangle. Each of the objects 502, 504 is moving independently as represented by shifting the square for object 504 and the triangle for object 504 to a different set of pixels illuminated for each object from frame t to a subsequent time step (t+1). That is, the shift in pixels for object 502 may be different from the shift in pixels for object 504. Conventional convolution kernels (e.g., w) cannot efficiently model multiple independent motions.

[0055]

[0064] To address the scenario illustrated in frames t and t+1 of a video, aspects of the present disclosure are directed to video processing using self-attention. Self-attention is used to align frames (e.g., t and t+1) in a video. Attention can be viewed as emphasizing or weighting more heavily a portion of an input, such as a sequence or image, than other regions. In the context of images, attention may emphasize certain regions at high resolution, while other areas, such as the background, may be observed at low resolution. Aspects of the present disclosure exploit spatial and temporal redundancy in videos to improve processing efficiency.

[0056]

[0065] According to an aspect of the present disclosure, an alignment operator based on pixel-to-pixel self-attention is provided. The alignment operator takes three input tensors, t sup , uref , and u sup It can be defined as a function of the unaligned feature tensor t sup represents the frame or feature to be aligned. The reference feature tensor u ref represents the frame or feature that should be used for the reference coordinate system for alignment. The supporting feature tensor u sup is the reference feature tensor u ref The input tensor represents the spatially-distributed support features for use in estimating motion for a self-attention query (q ref ), key(k sup ), and value (v sup ) can be associated with embeddings:

[0057]

number

[0058] where each φ represents a pixel-wise projection function separately parameterized on q, k, and v, followed by reshaping, and d k and d v identifies the key and value dimensions, and k h and k w Let h and w denote the height and width of the local window centered on each query, respectively, and the feature tensors (e.g., u ref , u sup , and t sup ) are the height and width of the

[0059]

[0066] Reference feature tensor u ref Each reference pixel in is a pixel in the query vector q ref The support feature tensor u sup q ref Center each query on key k supEach query for a pixel is subjected to a local search in a window of keys, which is sup Each query may be matched by dot product with each of the keys in its window to provide a similarity-based search within the keys. The similarities may be normalized by applying a softmax operation to generate an attention mask or alignment operator. In other words, an alignment operator (which may also be called a warping operator) is a linear algorithm that uses a supporting feature tensor u sup and the reference feature tensor u ref We estimate the warp between the misaligned feature tensor t sup , the reference feature tensor u ref The alignment operator can be defined as follows:

[0060]

number

[0061] where σ denotes softmax normalization.

[0062]

[0067] The alignment operator is a motion estimation part given by the softmax operation.

[0063]

number

[0064] The motion estimation may include ref and support key k sup It can be in the form of a dense hw × hw matrix, with elements that are the pairwise similarities between

[0065]

number

[0066] may operate over the entire spatial extent of the frame and therefore allocate the same amount of computation to every pixel.

[0067]

number

[0068] is the reference feature (e.g., u ref ) and supporting features (e.g., u sup , and t sup ) and the aligned feature map

[0069]

number

[0070] Thus, the registration operator can provide a motion estimation

[0071]

number

[0072] According to the support value v sup can be converted.

[0073] In some aspects, the query selection may be for one or more portions of a window, rather than the entire spatial range. The query may be dynamically selected based on the input. A parametric binary function g may determine, for each pixel, whether to perform self-attention motion compensation or directly utilize context information without alignment, as follows:

[0074]

number

[0075] where m is the mask and g is the gate function.

[0076]

number

[0077] is the Hadamard product operator.

[0078]

[0069] Figure 6 is a diagram illustrating an example alignment of a sequence of frames 600 using self-attention according to an aspect of the present disclosure. Referring to Figure 6, a reference frame 602a and a support frame 602b are shown. The reference frame 602a may be low-resolution features of a current frame time t. The support frame 602b may be low-resolution features of a previous frame at time t-1. The reference frame 602a and the support frame 602b may be sourced from an intermediate layer of a neural network, for example. An unaligned high-resolution frame 602c is also shown, which may have high-resolution features of a previous frame (t-1) processed in a subsequent layer of the neural network.

[0079]

[0070] An alignment operator 616 may be used to estimate the motion between the previous frame (t-1) and the current frame (t). Attention weights 608a-608d may be generated using an attention function 614. The attention function 614 may apply a normalization function (e.g., softmax normalization) to the product of the key and the query. Each pixel 604a-604d of the reference frame 602a may represent a query (low-resolution feature at time step t) in frame t. Each query (e.g., 604a-604d) has a corresponding window of the key 606a-606d to the support frame 602b (e.g., the previous frame at time step t-1). Each of the windows of the key 606a-606d may be considered as a neighborhood of pixels surrounding the corresponding pixel location of the query 604a-604d. The window of keys 606a-606d may define a search space of pixels in the support frame 602b. Thus, each query 604a-604d is subjected to a local search within the window of keys 606a-606d in the support frame 602b. That is, each pixel (e.g., query) 604a-604d of the reference frame 602a is compared with each of the pixels in the corresponding window of keys 606a-606d in the support frame 602b. For example, the similarity between a query in the reference frame 602a and all keys in their respective windows in the support frame 602b may be calculated by taking the dot product of the query (e.g., 604a-604d) with all keys (e.g., 606a-606d) in their respective windows.

[0080] As shown in FIG. 6, the similarity may be normalized to generate a set of attention weights 608a-608d corresponding to the keys 606a-606d, respectively. The set of attention weights 608a-608d may represent an attention map 608. In turn, the attention weights 608a-608d, which may represent the attention map 608, may be applied to respective values ​​610a-610d of the unaligned high resolution frame 602c. The values ​​610a-610d may represent weighted sums of pixels corresponding to respective windows (e.g., neighborhoods of pixels) of the keys 606a-606d. The unaligned high resolution frame 602c may have been generated in a previous iteration and may represent features of the neural network at a previous time step. Each of the weighted sums may be taken as a value (e.g., 610a-610d) of the unaligned high resolution frame 602c. A reconstructed video frame 602d having improved alignment and with high resolution features at time t may be generated according to the alignment operator 616 based on the calculated products of the values ​​(e.g., 610a-610d) and the respective attention maps 608a-608d. In some aspects, the reconstructed video frame 602d may be output and displayed.

[0081] 7A and 7B illustrate example architectures 700 and 750 for aligning frames using self-attention, according to aspects of the present disclosure. With reference to FIG. 7A, the example architecture 700 is a recurrent neural network model. The example architecture 700 receives a video as an input. Frames t and t-1 of the video are subjected to a feature block 702. The feature block 702 may include, for example, a set of convolutional layers (not shown), each of which may function similarly to the convolutional layer 356 shown in FIG. 3. The feature block 702 may process frames t and t-1 to generate respective features, feature t-1 and feature t. These mid-level features (e.g., features generated via an earlier layer of feature blocks near the input) are used to align the query (u ref) and key(u sup ) can be calculated by using the action (or motion) information (t sup ) is from the previous time (t-1). ref ) and the key (u sup ) is represented by the motion information (t sup ) to a Multi-Head Self-Attention Block (MHSA) 704. MHSA 704 may be a module for attention mechanisms that allow for independently attending to different parts of a sequence. MHSA 704 may generate a set of features (e.g., high-resolution features t) for the aligned frames at time step t.

[0082] In this example architecture 700, further processing may be performed to provide super-resolution. A fusion block 706 may receive a set of features for the aligned frame at time t. The fusion block 706 may combine the aligned frame at time t and the tensor t sup The fusion block 706 may process the results of the alignment of the fusion block 706 and fuse them into a single tensor. For example, the fusion block 706 may perform channel-wise concatenation and 1×1 convolution to generate boosted features t. The boosted features t may be fed to the super-resolution block 708. The super-resolution block 708 may process the boosted features t through a set of convolution blocks that may further refine the output of the fusion block 706, for example, to compute higher resolution features (e.g., super-resolution features). The higher resolution features may be upsampled via the upsampling block 710, which performs spatial upsampling via sub-pixel convolution to restore the output.

[0083]

[0074] Referring to FIG. 7B, an example architecture 750 provides a multi-frame implementation. In the example architecture 750, a set of windows or frames is provided as input, i.e., frame(tk) to frame(t+k). To up-sample frame t, the windows of the frame and the tensors represented by the features extracted from each window may be warped to be aligned to frame t. A feature block 752 processes each input of the set of windows to generate corresponding features. Features of frames tk to t+k other than frame t are aligned to the features in frame t via MHSA 754. In this example architecture, the supporting information (e.g., features tk to features t+k) may be computed by directly feeding the past and future frames in the temporal window (e.g., three past and future frames) to the network. This may result in a set of 2k-1 aligned features (e.g., aligned features tk to aligned features t+k). A fusion block 756 may then fuse the 2k-1 features with the features at t via channel-wise concatenation and 1×1 convolution to generate boosted features t. The example architecture 750 may then function similarly to the example architecture 700 of FIG. 7A. That is, the boosted features t are fed to a super-resolution block 758. In turn, the super-resolution block 758 may process the boosted features t through a set of convolution blocks that further refine the output of the fusion block 756 to compute higher resolution features (e.g., super-resolution features). The higher resolution features are upsampled via an upsampling block 760, which performs spatial upsampling via sub-pixel convolution to restore the output.

[0084]

[0075] FIG. 8 is a diagram illustrating an example alignment of a frame 800 using a query selection according to aspects of the present disclosure. As described, aspects of the present disclosure may provide increased processing efficiency. Rather than aligning all pixels in a frame, the similarity between sequential frames of a video may be further used to reduce the alignment operations performed to align the sequential frames. That is, because pixels may undergo small displacements between sequential frames, a gating function may be applied to limit the alignment operations to regions with significant changes. The change or difference between frames may be represented, for example, by a residual frame. Upon computing the residual frame, a gating function may be applied to determine whether or not to perform an alignment operation (e.g., similar to the alignment operation of FIG. 6).

[0085]

[0076] Referring to FIG. 8, a binary gate 812 may limit the alignment of frames t (e.g., 802a) and t-1 (e.g., 802b) such that the alignment may be conditional based on the input. This may be beneficial, for example, when there is limited camera motion or a moving object between frames t and t-1. In such cases, many of the pixels may not be involved in the motion between the frames and may therefore already be aligned. Thus, rather than performing the alignment for all pixels in frame t (e.g., 802a), the binary gate 812 may determine a subset of pixels in frame t 802a to be aligned.

[0086] In the example of FIG. 8, low-resolution features of a reference frame 802a at time step t and a support frame 802b at time step t-1 are shown. High-resolution features of an unaligned frame 802c at time t-1 are also shown. A residual frame 810 is a u-pixel image of the support frame 802b. sup tensor and u in reference frame 802a ref The residual frame 810 may be calculated as the difference between the tensors. The residual frame 810 may act as a bias for regions in the frame to be aligned. Regions with high residuals are more likely to be misaligned, while regions with low residuals may already be aligned. In one example, a region may have high residual if it contains a moving object, such as a car moving down a street. In this example, the characteristics of the car and street pixels may be different, such that if a pixel transitions from one frame to another (due to motion), a region containing such a pixel may have high residual. A high residual may indicate a high probability of motion, and therefore alignment may be performed. The relative magnitude of the residual may depend on factors including, for example, the network architecture and the scale of the input image. In some examples, a residual value >1 may be considered a high residual value, while a residual value < .001 may be considered a low residual value.

[0087] The residual frame 810 may be fed to a binary gate 812. The binary gate may be, for example, a one-layer convolution with a single output. In some aspects, a softmax (e.g., Gumbel softmax) operation may be applied along with a straight through estimator to train the binary gate. In the forward pass, an argmax function may be applied with probability p to perform hard attention (z∈{0,1}), but the gradient may be relaxed using a softmax function instead of an argmax function, so that the samples are reduced to the vector

[0088]

number

[0089] can be expressed as:

[0090]

number

[0091] Where:

[0092]

number

[0093] α k is the unnormalized probability, and G k is a sequence of independent and identically distributed random variables (e.g., noise), and τ is the temperature parameter of the softmax function.

[0094]

[0079] Thus, the binary gate 812 may limit the alignment to active pixels (shown as white pixels in the pixel view 816). That is, the binary gate 812 provides a binary value for turning on or off each pixel of the reference frame based on the calculated residual frame 810. If the pixel is active or on, the alignment may be performed. On the other hand, if the pixel is inactive or off, the alignment may not be performed. In some aspects, if the pixel is inactive or off, the pixel value of the support frame 802b corresponding to the inactive pixel of the reference frame 802a may be copied without processing.

[0095]

[0080] Having limited the pixels for alignment, the self-attention process can be performed as described above with reference to FIG. 6 to generate a reconstructed frame (e.g., a high-resolution feature of the frame) at time t.

[0096] Each active pixel 804a, 804b (shown via pixel view 816) of the reference frame 802a may represent a query to the support frame 802b. Each query (e.g., 804a, 804b) has a corresponding window of keys 806a, 806b to the support frame 802b (e.g., the previous frame at time step t-1). Each of the windows of the keys 806a, 806b may define a search space of pixels in the support frame 802b. Thus, each query 804a, 804b may be subjected to a local search within the window of the keys 806a, 806b in the support frame 802b. That is, the active pixels (e.g., 804a, 804b) of the reference frame 802a are compared with each of the pixels in the corresponding window of the keys (806a, 806b) of the support frame 802b. For example, the similarity between a query (e.g., 804a, 804b) in the reference frame 802a and all the keys 806a, 806b in the respective windows in the support frame 802b may be calculated by taking the dot product of the query (e.g., 804a, 804b) and all the keys (e.g., 806a, 806b) in the respective windows. The similarity may be normalized to generate a corresponding set of attention weights 814a, 814b according to an attention function 816. The attention weights 814a, 814b may collectively represent an attention map 814.

[0097] In turn, the attention weights 814a, 814b may be applied to the values ​​(e.g., 808a, 808b) of the unaligned high-resolution frame 802c. The values ​​808a, 808b may be weighted sums of each set of pixels of the unaligned high-resolution frame 802c that correspond to the windows of the keys 806a, 806b of the low-resolution features of the support frame 802b, respectively. That is, each of the weighted sums may be considered as the values ​​(e.g., 808a, 808b) of the unaligned high-resolution frame 802c. A reconstructed video frame 802d at time t may be generated using the improved alignment determined according to the alignment operator 818. The reconstructed video frame 802d may be generated based on the calculated products of the attention weights 814a, 814b and the corresponding values ​​(e.g., 808a, 808b) that are subject to the selection of the query representing the active pixels. The reconstructed video 802d may also have high-resolution features.

[0098]

[0083] In some aspects, the reconstructed video frame 802d may be output and displayed.

[0099] In some aspects, a sparsity loss objective may be applied to further restrict the pixels to be aligned. The sparsity objective may be expressed as follows:

[0100]

number

[0101] Here, β is a hyperparameter to control the tradeoff between efficiency and quality of the reconstruction by further constraining the amount of pixels to be aligned.

[0102]

[0085] Figure 9 is a flow diagram illustrating a method 900 for processing a video according to an aspect of the present disclosure. As shown in Figure 9, at block 902, the method 900 receives a video as an input in an artificial neural network (ANN). The video comprises a sequence of frames. For example, as shown in Figure 7A, the example architecture 700 receives a video including a frame at time t-1 and a frame at time t.

[0103] At block 904, the method 900 extracts a set of features for each of the current frame of the video and a previous frame of the video. The set of features includes a set of supporting features for a set of pixels of the previous frame that are to be aligned with a set of reference features of the current frame. For example, three input tensors, t, which may represent features extracted from a frame of the video (e.g., 500), are sup , u ref , and u sup . The unaligned feature tensor t sup represents the frame or feature to be aligned. The reference feature tensor u ref represents the frame or feature that should be used for the reference coordinate system for alignment. The supporting feature tensor u sup is the reference feature tensor u ref represents spatially-distributed supporting features for use in estimating motion relative to

[0104]

[0087] In block 906, the method 900 may optionally apply a gating function based on a residual comprising a difference between the current frame and the previous frame. For example, as described with reference to FIG. 8, if the residual frame 810 is a u of the support frame 802b, sup tensor and u in reference frame 802a refThe residual frame 810 may be calculated as the difference between the tensors. The residual frame 810 may act as a bias for regions in the frame to be aligned. Regions with high residuals are more likely to be misaligned, while regions with low residuals may already be aligned. In one example, a region may have high residuals if it contains a moving object, such as, for example, a car moving on a street. The residual frame 810 may be provided to a binary gate 812. The binary gate may be, for example, a one-layer convolution with a single output. The binary gate 812 may limit the alignment to active pixels (shown as white pixels in the pixel view 816). That is, the binary gate 812 provides a binary value to turn on or off each pixel of the reference frame based on the calculated residual frame 810. If the pixel is active or on, alignment may be performed. On the other hand, if the pixel is inactive or off, alignment may not be performed. In some aspects, if a pixel is inactive or off, the pixel value of the support frame 802b that corresponds to the inactive pixel of the reference frame 802a may be copied without processing.

[0105] At block 908, the method 900 calculates a similarity between the support feature for each pixel in the set of pixels of the set of support features of the previous frame and the corresponding reference feature of the current frame. As described with reference to FIG. 6, each pixel of the reference frame 602a is compared to each of the pixels in the corresponding window of the keys 606a-606d of the support frame 602b. For example, the similarity between the query of the reference frame 602a and all the keys in the respective windows in the support frame 602b can be calculated by taking the dot product of the query (e.g., 604a-604d) and all the keys (e.g., 606a-606d) in the respective windows.

[0106] At block 908, the method 900 generates an attention map based on the similarity. As described with reference to FIG. 6, the attention weights 608a-608d may be generated using the attention function 614. The attention function 614 may apply a normalization function (e.g., softmax normalization) to the product of the key and the query. Each pixel 604a-604d of the reference frame 602a may represent a query (low-resolution feature at time step t) in frame t. Each query (e.g., 604a-604d) has a corresponding window of the keys 606a-606d to the support frame 602b (e.g., the previous frame at time step t-1). Each of the windows of the keys 606a-606d may be considered as a neighborhood of pixels surrounding the corresponding pixel location of the queries 604a-604d. The windows of the keys 606a-606d may define a search space of pixels in the support frame 602b. Thus, each query 604a-604d is subjected to a local search within the window of keys 606a-606d in the support frame 602b. That is, each pixel (e.g., query) 604a-604d of the reference frame 602a is compared to each of the pixels in the corresponding window of keys 606a-606d in the support frame 602b. For example, the similarity between a query in the reference frame 602a and all the keys in the respective window in the support frame 602b may be calculated by taking the dot product of the query (e.g., 604a-604d) with all the keys (e.g., 606a-606d) in the respective window. As shown in FIG. 6, the similarity may be normalized to generate a set of attention weights 608a-608d corresponding to the keys 606a-606d, respectively. The set of attention weights 608 608a-608d may represent an attention map 608.

[0107]

[0090] At block 910, the method 900 generates an output comprising a reconstruction of the current frame based on the attention map. For example, attention weights 608a-608d, which may represent the attention map 608 as described with reference to FIG. 6, may be applied to respective values ​​610a-610d of the unaligned high-resolution frame 602c. The values ​​610a-610d may represent weighted sums of pixels corresponding to respective windows (e.g., neighborhoods of pixels) of the keys 606a-606d. The unaligned high-resolution frame 602c may have been generated in a previous iteration and may represent features of the neural network at a previous time step. Each of the weighted sums may be considered as a value (e.g., 610a-610d) of the unaligned high-resolution frame 602c. A reconstructed video frame 602d having improved alignment and having high-resolution features at time t may be generated according to an alignment operator 616 based on the calculated products of the values ​​(e.g., 610a-610d) and the respective attention maps 608a-608d.

[0108]

[0091] Implementation examples are provided in the following numbered clauses: 1. A processor-implemented method for processing video, comprising: receiving the video as an input in an artificial neural network (ANN), the video comprising a sequence of frames; extracting a set of features for each of a current frame of the video and a previous frame of the video, the set of features including a set of support features for a set of pixels of the current frame to be aligned with a set of reference features of the current frame; calculating a similarity between a supporting feature for each pixel in the set of pixels of the supporting feature set of the previous frame and a corresponding reference feature of the current frame; generating an attention map based on the similarity; generating an output comprising a reconstruction of the current frame based on the attention map; 23. A processor-implemented method comprising: 2. The processor-implemented method of claim 1, further comprising displaying the output. 3. The processor-implemented method of any one of clauses 1 to 2, wherein the reconstructed current frame includes an adjusted alignment of one or more pixels of the current frame. 4. The processor-implemented method of any one of clauses 1 to 3, further comprising applying a gating function to limit the calculation of the similarity to a subset of the set of pixels of the current frame based on a residual, the residual comprising a difference between the current frame and the previous frame. 5. A processor-implemented method described in any one of clauses 1 to 4, further comprising applying a sparsity goal to limit the number of pixels in the set of pixels of the current frame for which the similarity is calculated. 6. The processor-implemented method of any one of clauses 1-5, further comprising upsampling the reconstructed current frame. 7. An apparatus for processing video, comprising: Memory, at least one processor coupled to the memory; wherein the at least one processor: receiving the video as an input in an artificial neural network (ANN), the video comprising a sequence of frames; extracting a set of features for each of a current frame of the video and a previous frame of the video, the set of features including a set of support features for a set of pixels of the current frame to be aligned with a set of reference features of the current frame; calculating a similarity between a supporting feature for each pixel in the set of pixels of the supporting feature set of the previous frame and a corresponding reference feature of the current frame; generating an attention map based on the similarity; generating an output comprising a reconstruction of the current frame based on the attention map; An apparatus configured to: 8. The apparatus of clause 7, wherein the at least one processor is further configured to display the output. 9. The apparatus of clause 7 or 8, wherein the reconstructed current frame includes an adjusted alignment of one or more pixels of the current frame. 10. The device described in any one of clauses 7 to 9, wherein the at least one processor is further configured to apply a gating function to limit the calculation of the similarity to a subset of the set of pixels of the current frame based on a residual, the residual comprising a difference between the current frame and the previous frame. 11. The apparatus of any one of clauses 7 to 10, wherein the at least one processor is further configured to apply a sparsity goal to limit the number of pixels in the set of pixels of the current frame for which the similarity is calculated. 12. The apparatus of any one of clauses 7 to 11, wherein the at least one processor is further configured to upsample the reconstructed current frame. 13. A non-transitory computer readable medium having program code recorded thereon, the program code being executed by a processor and comprising: program code for receiving the video as input in an artificial neural network (ANN), the video comprising a sequence of frames; program code for extracting a set of features for each of a current frame of the video and a previous frame of the video, the set of features including a set of support features for a set of pixels of the current frame to be aligned with a set of reference features of the current frame; program code for calculating a similarity between a supporting feature for each pixel in the set of pixels of the supporting feature set of the previous frame and a corresponding reference feature of the current frame; a program code for generating an attention map based on the similarity; program code for generating an output comprising a reconstruction of the current frame based on the attention map; 1. A non-transitory computer-readable medium comprising: 14. The non-transitory computer-readable medium of clause 13, wherein the program code further comprises program code for displaying the output. 15. The non-transitory computer-readable medium of clause 13 or 14, wherein the reconstructed current frame includes an adjusted alignment of one or more pixels of the current frame. 16. A non-transitory computer-readable medium as described in any one of clauses 13 to 15, wherein the program code further comprises program code for applying a gating function to limit the calculation of the similarity to a subset of the set of pixels of the current frame based on a residual, the residual comprising a difference between the current frame and the previous frame. 17. The non-transitory computer-readable medium of any one of clauses 13 to 16, wherein the program code further comprises program code for applying a sparsity goal to limit the number of pixels in the set of pixels of the current frame for which the similarity is calculated. 18. The non-transitory computer-readable medium of any one of clauses 13 to 17, wherein the program code further comprises program code for upsampling the reconstructed current frame. 19. An apparatus for processing video, comprising: In an artificial neural network (ANN), means for receiving the video as an input, the video comprising a sequence of frames; means for extracting a set of features for each of a current frame of the video and a previous frame of the video, the set of features including a set of support features for a set of pixels of the current frame to be aligned with a set of reference features of the current frame; means for computing a similarity between a supporting feature for each pixel in the set of pixels of the supporting feature set of the previous frame and a corresponding reference feature of the current frame; means for generating an attention map based on the similarity; means for generating an output comprising a reconstruction of the current frame based on the attention map; An apparatus comprising: 20. The apparatus of clause 19, further comprising means for displaying said output. 21. The apparatus of clause 19 or 20, wherein the reconstructed current frame includes an adjusted alignment of one or more pixels of the current frame. 22. An apparatus described in any one of clauses 19 to 21, further comprising means for applying a gating function to limit the calculation of the similarity to a subset of the set of pixels of the current frame based on a residual, the residual comprising a difference between the current frame and the previous frame. 23. An apparatus described in any one of clauses 19 to 22, further comprising means for applying a sparsity goal to limit the number of pixels in the set of pixels of the current frame for which the similarity is calculated. 24. The apparatus of any one of clauses 19 to 23, further comprising means for upsampling the reconstructed current frame.

[0109] In one aspect, the receiving means, extracting means, computing means, means for generating an attention map, means for generating an output, and / or displaying means may be the CPU 102, the GPU 104, the DSP 106, a program memory associated with the CPU 102, the dedicated memory block 118, the fully connected layer 362, the NPU 428, and / or the routing connection processing unit 216 configured to perform the described functions. In another configuration, the above-described means may be any module or any device configured to perform the described functions by these above-described means.

[0110]

[0093] The various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. These means may include various hardware and / or software component(s) and / or module(s), including, but not limited to, circuits, application specific integrated circuits (ASICs), or processors. Generally, where there are operations illustrated in the figures, these operations may have corresponding counterpart means-plus-function components with similar numbering.

[0111]

[0094] As used, the term "determining" encompasses a wide range of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), determining, and the like. Additionally, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Further, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0112]

[0095] When used, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. By way of example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc.

[0113]

[0096] The various example logic blocks, modules and circuits described in connection with this disclosure may be implemented or performed using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the described functions. A general purpose processor may be a microprocessor, but alternatively, the processor may be any commercially available processor, controller, microcontroller or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0114]

[0097] The steps of the algorithm or method described in connection with the present disclosure may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in any form of storage medium known in the art. Some examples of storage media that may be used include random access memory (RAM), read only memory (ROM), flash memory, erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), registers, hard disk, removable disk, CD-ROM, etc. A software module may comprise a single instruction, or many instructions, and may be distributed across several different code segments, among different programs, and across multiple storage media. The storage medium may be coupled to the processor such that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integral to the processor.

[0115]

[0098] The disclosed methods comprise one or more steps or actions for achieving the described method. The steps and / or actions of the method may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0116]

[0099] The described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, an exemplary hardware configuration may include a processing system in a device. The processing system may be implemented using a bus architecture. The bus may include any number of interconnected buses and bridges, depending on the particular application and overall design constraints of the processing system. The bus may link together various circuits including a processor, a machine-readable medium, and a bus interface. The bus interface may be used, among other things, to connect a network adapter to the processing system via the bus. The network adapter may be used to implement signal processing functions. In certain aspects, a user interface (e.g., a keypad, a display, a mouse, a joystick, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, and the like, which are well known in the art and therefore will not be described further.

[0117]

[0100] The processor may be responsible for managing the bus and general-purpose processing, including executing software stored on the machine-readable medium. The processor may be implemented using one or more general-purpose and / or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuits that may execute software. Software should be broadly construed to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. The machine-readable medium may include, by way of example, a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a magnetic disk, an optical disk, a hard drive, or any other suitable storage medium, or any combination thereof. The machine-readable medium may be embodied in a computer program product. The computer program product may include packaging materials.

[0118]

[0101] In a hardware implementation, the machine-readable medium may be part of a processing system separate from the processor. However, as one skilled in the art would readily appreciate, the machine-readable medium, or any part thereof, may be external to the processing system. By way of example, the machine-readable medium may include a transmission line, a carrier wave modulated by data, and / or a computer product separate from the device, all of which may be accessed by the processor through a bus interface. Alternatively or additionally, the machine-readable medium, or any part thereof, may be integrated into the processor, as may be the case with a cache and / or a general-purpose register file. Although the various components described, such as local components, are described as having specific locations, they may also be configured in various ways, such as when a particular component is configured as part of a distributed computing system.

[0119]

[0102] The processing system may be configured as a general-purpose processing system having one or more microprocessors providing processor functions and external memory providing at least a portion of the machine-readable medium, all linked together with other support circuits through an external bus architecture. Alternatively, the processing system may include one or more neuromorphic processors for implementing the described neuron models and models of neural systems. As another alternative, the processing system may be implemented using an application specific integrated circuit (ASIC) having a processor, a bus interface, a user interface, support circuits, and at least a portion of the machine-readable medium integrated into a single chip, or using one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, or any other suitable circuitry or combination of circuits capable of performing the various functions described throughout this disclosure. Those skilled in the art will recognize how best to implement the functions described for the processing system depending on the particular application and the overall design constraints imposed on the entire system.

[0120]

[0103] The machine-readable medium may comprise a number of software modules. The software modules include instructions that, when executed by a processor, cause the processing system to perform various functions. The software modules may include a transmitting module and a receiving module. Each software module may reside in a single storage device or may be distributed across multiple storage devices. As an example, a software module may be loaded into RAM from a hard drive when a triggering event occurs. During execution of a software module, the processor may load some of the instructions into a cache to increase access speed. One or more cache lines may then be loaded into a general-purpose register file for execution by the processor. When referring to a function of a software module below, it will be understood that such function is implemented by the processor when executing instructions from that software module. Furthermore, it should be understood that aspects of the present disclosure provide improvements to the functionality of a processor, computer, machine, or other system implementing such aspects.

[0121]

[0104] If implemented in software, the functions may be transmitted or stored as one or more instructions or code on a computer-readable medium. Computer-readable media includes both communication media and computer storage media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store or carry desired program code in the form of data structures or instructions and that can be accessed by a computer. In addition, any connection is strictly referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. As used, disk and disc include compact discs (CDs), laser discs, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically while discs reproduce data optically using lasers. Thus, in some aspects, computer readable media may comprise non-transitory computer readable media (e.g., tangible media). Additionally, in other aspects, computer readable media may comprise transitory computer readable media (e.g., signals). Combinations of the above should also be included within the scope of computer readable media.

[0122]

[0105] Accordingly, certain aspects may comprise a computer program product for performing the presented operations. For example, such a computer program product may comprise a computer-readable medium having instructions stored (and / or encoded) thereon, the instructions being executable by one or more processors to perform the described operations. In certain aspects, the computer program product may include packaging materials.

[0123]

[0106] Furthermore, it should be understood that modules and / or other suitable means for performing the described methods and techniques may be downloaded and / or otherwise obtained by a user terminal and / or base station, as appropriate. For example, such devices may be coupled to a server to facilitate the transfer of means for performing the described methods. Alternatively, the various methods described may be provided via a storage means (e.g., RAM, ROM, a physical storage medium such as a compact disk (CD) or floppy disk, etc.), such that a user terminal and / or base station may obtain the various methods upon coupling or providing the storage means to the device. Furthermore, any other suitable technique for providing the described methods and techniques to a device may be utilized.

[0124]

[0107] It should be understood that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. 1. A processor-implemented method for processing video, comprising: receiving the video as input in an artificial neural network (ANN), the video comprising a sequence of frames; extracting a set of features for each of a current frame of the video and a previous frame of the video, the set of features including a set of supporting features for a set of pixels of the previous frame to be aligned with a set of reference features of the current frame; calculating a similarity between a supporting feature for each pixel in the set of pixels of the supporting feature set of the previous frame and a corresponding reference feature of the current frame; applying a gating function to limit the calculation of the similarity to a subset of the set of pixels of the current frame based on a residual, the residual comprising a difference between the current frame and the previous frame; generating an attention map based on the similarity; generating an output comprising a reconstruction of the current frame based on the attention map; 23. A processor-implemented method comprising:

2. The method of claim 1 , further comprising displaying the output.

3. The processor-implemented method of claim 1 , wherein the reconstructed current frame includes an adjusted alignment of one or more pixels of the current frame.

4. 2. The processor-implemented method of claim 1, further comprising applying a sparsity goal to limit a number of pixels in the set of pixels of the current frame for which the similarity measure is calculated.

5. The processor-implemented method of claim 1 , further comprising upsampling the reconstructed current frame.

6. 1. An apparatus for processing video, comprising: Memory, at least one processor coupled to the memory; wherein the at least one processor: receiving the video as input in an artificial neural network (ANN), the video comprising a sequence of frames; extracting a set of features for each of a current frame of the video and a previous frame of the video, the set of features including a set of supporting features for a set of pixels of the previous frame to be aligned with a set of reference features of the current frame; calculating a similarity between a supporting feature for each pixel in the set of pixels of the supporting feature set of the previous frame and a corresponding reference feature of the current frame; applying a gating function to limit the calculation of the similarity to a subset of the set of pixels of the current frame based on a residual, the residual comprising a difference between the current frame and the previous frame; generating an attention map based on the similarity; generating an output comprising a reconstruction of the current frame based on the attention map; An apparatus configured to:

7. The apparatus of claim 6 , wherein the at least one processor is further configured to display the output.

8. The apparatus of claim 6 , wherein the reconstructed current frame includes an adjusted alignment of one or more pixels of the current frame.

9. 7. The apparatus of claim 6, wherein the at least one processor is further configured to apply a sparsity goal to limit a number of pixels in the set of pixels of the current frame for which the similarity measure is calculated.

10. The apparatus of claim 6 , wherein the at least one processor is further configured to upsample the reconstructed current frame.

11. A non-transitory computer readable medium having program code recorded thereon, the program code being executed by a processor to perform the method of any one of claims 1 to 5.

Images