Flow-independent neural video compression

Abstract

A processor-implemented method for video compression using an artificial neural network (ANN) includes receiving video through the ANN. The ANN extracts a first set of features for a current frame of the video and a second set of features for a reference frame of the video. The ANN determines an estimate of a correlation feature between the first set of features for the current frame and the second set of features for the reference frame. The estimate of the correlation feature is coded and transmitted to a receiver.

Description

Cross-reference of related applications

[0001]

[0001] This application claims priority to U.S. Patent Application No. 17 / 975,471, filed on October 27, 2022, and claims the benefit of U.S. Provisional Patent Application No. 63 / 284,569, filed on November 30, 2021, and the entirety of the disclosures thereof are expressly incorporated herein by reference. [Technical Field]

[0002]

[0002] The aspects of this disclosure relate in general to neural networks, and more particularly to video compression using artificial neural networks. [Background technology]

[0003]

[0003] An artificial neural network may comprise an interconnected group of artificial neurons (e.g., neuron models). An artificial neural network may be a computing device or represented as a method to be performed by a computing device. A convolutional neural network is one type of feedforward artificial neural network. A convolutional neural network may comprise a collection of neurons, each having a receptive field and tiling the input space together. Convolutional neural networks (CNNs), such as deep convolutional neural networks (DCNs), have numerous applications. In particular, these neural network architectures are used in a variety of technologies, such as image recognition, pattern recognition, speech recognition, autonomous driving, and other classification tasks.

[0004]

[0004] Neural networks are well applied in multimedia compression, such as video and image compression. Image compression is useful for communicating multimedia, such as images and videos. However, users may have different computer architectures with different computing power and may be operating such devices under different network conditions. Conventional neural video compression methods utilize flow-based warping for motion compensation. However, flow-based warping may not be very efficient for mobile devices that may have limited resources. [Overview of the Initiative]

[0005]

[0005] This disclosure is described in each independent claim. Some aspects of this disclosure are described in the dependent claims.

[0006]

[0006] In one aspect of the present disclosure, a processor implementation for video compression using an artificial neural network (ANN) is provided. The method includes receiving video by the ANN. The method also includes the ANN extracting a first set of features of the current frame of the video and a second set of features of a reference frame of the video. In addition, the method includes the ANN determining an estimate of correlation features between the first set of features of the current frame and the second set of features of the reference frame. Furthermore, the method includes transmitting the estimate of correlation features to a receiver.

[0007]

[0007] In one aspect of the present disclosure, an apparatus for video compression using an artificial neural network (ANN) is provided. The apparatus includes a memory and one or more processors coupled to the memory. The processor(s) is configured to receive a video by the ANN. The processor(s) is also configured to extract, by the ANN, a first set of features of a current frame of the video and a second set of features of a reference frame of the video. In addition, the processor(s) is configured to determine, by the ANN, an estimation of correlation features between the first set of features of the current frame and the second set of features of the reference frame. Further, the processor(s) is configured to transmit the estimation of the correlation features to a receiver.

[0008]

[0008] In one aspect of the present disclosure, a processor-implemented method for video compression using an artificial neural network (ANN) is provided. The method includes receiving, by the ANN, a reference frame of a video and an estimation of correlation features between the current frame of the video and the reference frame of the video. The method also includes extracting, by the ANN, a first set of features of the reference frame of the video. In addition, the method includes decoding, by the ANN, the estimation of the correlation features. Further, the method includes generating, by the ANN, a prediction of the current frame of the video based on the decoded estimation of the correlation features and the first set of features of the reference frame of the video.

[0009]

[0009] In one aspect of the present disclosure, an apparatus for video compression using an artificial neural network (ANN) is provided. The apparatus includes a memory and one or more processors coupled to the memory. The processor(s) is configured to receive, by the ANN, an estimation of a correlation feature between a reference frame of a video and a current frame of the video. The processor(s) is also configured to extract, by the ANN, a first set of features of the reference frame of the video. In addition, the processor(s) is configured to decode, by the ANN, the estimation of the correlation feature. Further, the processor(s) is configured to generate, by the ANN, a prediction of the current frame of the video based on the decoded estimation of the correlation feature and the first set of features of the reference frame of the video.

[0010]

[0010] Additional features and advantages of the present disclosure are described below. It should be understood by those skilled in the art that the present disclosure may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. It should also be recognized by those skilled in the art that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features believed to be characteristic of the present disclosure, both as to its organization and method of operation, together with further objects and advantages, will be better understood from the following description when considered in connection with the accompanying drawings. However, it is to be expressly understood that each of the drawings is provided for the purpose of illustration and description only and is not intended to limit the scope of the present disclosure.

Brief Description of the Drawings

[0011]

[0011] The features, nature, and advantages of the present disclosure will become more apparent from the detailed description set forth below when read in conjunction with the drawings in which like reference characters identify correspondingly throughout. [Figure 1]

[0012] It is a diagram showing an exemplary implementation of a neural network using a system-on-chip (SOC) including a general-purpose processor according to some aspects of the present disclosure. [Figure 2A]

[0013] This figure shows a neural network according to the embodiments of this disclosure. [Figure 2B] This figure shows a neural network according to the embodiments of this disclosure. [Figure 2C] This figure shows a neural network according to the embodiments of this disclosure. [Figure 2D]

[0014] This figure shows an exemplary deep convolutional network (DCN) according to an aspect of the present disclosure. [Figure 3]

[0015] This is a block diagram illustrating an exemplary deep convolutional network (DCN) according to aspects of the present disclosure. [Figure 4]

[0016] Block diagram showing an exemplary software architecture that allows for the modularization of artificial intelligence (AI) functions according to the aspects of this disclosure. [Figure 5]

[0017] This block diagram shows an exemplary architecture for flow-independent video compression according to an aspect of the present disclosure. [Figure 6] This block diagram shows an exemplary architecture for flow-independent video compression according to an aspect of the present disclosure. [Figure 7] This block diagram shows an exemplary architecture for flow-independent video compression according to an aspect of the present disclosure. [Figure 8]

[0018] This is a flowchart illustrating an exemplary process for video compression using an artificial neural network according to aspects of the present disclosure. [Figure 9] This is a flowchart illustrating an exemplary process for video compression using an artificial neural network according to aspects of the present disclosure. [Figure 10] This is a flowchart illustrating an exemplary process for video compression using an artificial neural network according to aspects of the present disclosure. [Modes for carrying out the invention]

[0012]

[0019] The modes for carrying out the invention described below with respect to the attached drawings illustrate various configurations and do not represent only configurations in which the described concepts can be implemented. The “modes for carrying out the invention” include specific details intended to provide a complete understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts can be implemented without these specific details. In some cases, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0013]

[0020] Those skilled in the art will understand, based on the teachings, that the scope of this disclosure is intended to encompass any aspect of the disclosure, whether implemented independently of or in combination with any other aspect of the disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the described aspects. In addition, the scope of this disclosure shall encompass any such apparatus or method practiced using other structures, functions, or structures and functions in addition to or other than the various aspects of the disclosure described. It will be understood that any aspect of the disclosed disclosure may be embodied by one or more elements of the claims.

[0014]

[0021] The word "exemplary" is used to mean "serving as an example, case, or illustration." Any aspect described as "exemplary" should not necessarily be interpreted as being preferable or more favorable than other aspects.

[0015]

[0022] While specific embodiments are described, many variations and substitutions of these embodiments fall within the scope of this disclosure. While some advantages and benefits of preferred embodiments are stated, the scope of this disclosure is not limited to any particular advantage, use, or purpose. Rather, the embodiments of this disclosure are broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated as examples in the figures and the following description of preferred embodiments. The detailed description and drawings are illustrative and not limiting, and the scope of this disclosure is defined by the appended claims and their equivalents.

[0016]

[0023] As explained, neural networks are well-suited for multimedia compression, such as video compression. Traditional neural video compression methods utilize flow-based warping for motion compensation. However, flow-based warping can be hardware-friendly or inefficient for mobile devices that may have limited resources.

[0017]

[0024] To address these and other challenges, aspects of this disclosure relate to flow-independent neural video compression. Instead of using flow-based warping, the warping operator may be replaced with a fully convolutional neural network. In some aspects, the fully convolutional neural network may be configured using one or more autoencoders. The autoencoders may learn to encode unlabeled data and include an encoder that maps inputs to codes and a decoder that maps codes to the reconstruction of inputs.

[0018]

[0025] Figure 1 shows an exemplary implementation of a system-on-a-chip (SOC) 100, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to operate an artificial neural network (e.g., a neural end-to-end network) to provide flow-independent video compression. Variables (e.g., neural signals and synaptic weights), system parameters associated with computing devices (e.g., a weighted neural network), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a memory block 118, or distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from program memory associated with the CPU 102 or from memory block 118.

[0019]

[0026] The SOC100 may also include a GPU104, a DSP106, a connectivity block 110 which may include 5G connectivity, 4G Long-Term Evolution (4G LTE®) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth® connectivity, and additional processing blocks adapted to specific functions, such as a multimedia processor 112 which may detect and recognize gestures. In one implementation, the NPU108 is implemented in the CPU102, DSP106, and / or GPU104. The SOC100 may also include a navigation module 120 which may include a sensor processor 114, image signal processors (ISPs) 116, and / or a global positioning system.

[0020]

[0027] The SOC100 may be based on the ARM instruction set. In aspects of this disclosure, instructions loaded into the general-purpose processor 102 may include code for receiving video by an artificial neural network (ANN). The general-purpose processor 102 may also include code for extracting a first set of features of the current frame of the video and a second set of features of a reference frame of the video via the ANN. In addition, the general-purpose processor 102 may include code for the ANN to determine an estimate of correlation features between the first set of features of the current frame and the second set of features of the reference frame. The general-purpose processor 102 may further include code for transmitting the estimate of correlation features to a receiver.

[0021]

[0028] In other aspects of this disclosure, instructions loaded into the general-purpose processor 102 may include code for an artificial neural network (ANN) to receive a reference frame of video and an estimate of correlation features between the current frame of video and the reference frame of video. The general-purpose processor 102 may also include code for the ANN to extract a first set of features of the reference frame of video. In addition, the general-purpose processor 102 may include code for the ANN to decode the estimate of correlation features. The general-purpose processor 102 may further include code for the ANN to generate a prediction of the current frame of video based on the decoded estimate of correlation features and the first set of features of the reference frame of video.

[0022]

[0029] In a further aspect of this disclosure, instructions loaded into the general-purpose processor 102 may include code for the ANN to receive a history of multiple frames of video. The general-purpose processor 102 may also include code for the ANN to generate a reference frame from the history of multiple frames. In addition, the general-purpose processor 102 may include code for the ANN to generate a prediction of the current frame of video based on the reference frame and the current frame. The general-purpose processor 102 may further include code for generating a reconstruction of the current frame based on the prediction of the current frame of video and a residual calculated based on the difference between the current frame of video and the prediction of the current frame of video.

[0023]

[0030] A deep learning architecture may perform object recognition tasks by learning to represent the input at successively higher levels of abstraction within each layer, thereby constructing a useful feature representation of the input data. In this way, deep learning addresses a major bottleneck in traditional machine learning. Prior to the advent of deep learning, machine learning methods for object recognition problems may have relied heavily on human-designed features, sometimes in combination with shallow classifiers. A shallow classifier may be, for example, a two-class linear classifier that can compare a weighted sum of feature vector components to a threshold to predict which class the input belongs to. Human-designed features may be templates or kernels adapted to a specific problem domain by engineers with domain expertise. In contrast, a deep learning architecture may learn, through training, to represent features that are similar to those that human engineers can design. Furthermore, deep networks may learn to represent and recognize novel types of features that humans may not have considered.

[0024]

[0031] A deep learning architecture may learn a hierarchy of features. If visual data is presented, for example, the first layer may learn to recognize relatively simple features such as edges in the input stream. In another embodiment, if auditory data is presented, the first layer may learn to recognize spectral power at a specific frequency. A second layer, taking the output of the first layer as input, may learn to recognize combinations of features such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, a higher layer may learn to represent complex shapes in visual data or words in auditory data. A further higher layer may learn to recognize common visual objects or spoken phrases.

[0025]

[0032] Deep learning architectures can sometimes work particularly well when applied to problems with natural hierarchical structures. For example, classifying electric vehicles may benefit from initially learning to recognize wheels, windshields, and other features. These features may then be combined in different ways in higher layers to recognize cars, trucks, and airplanes.

[0026]

[0033] Neural networks may be designed using various connectivity patterns. In a feedforward network, each neuron in a given layer communicates with neurons in higher layers to pass information from lower to higher layers. As mentioned above, a hierarchical representation may be constructed within the consecutive layers of a feedforward network. Neural networks may also have recursive or feedback (also called top-down) connectivity. In recursive connectivity, the output from a neuron in a given layer may be transmitted to another neuron in the same layer. Recursive architectures can be useful when recognizing patterns across two or more chunks of input data delivered to the neural network in a sequence. Connectivity from a neuron in a given layer to a neuron in a lower layer is called feedback (or top-down) connectivity. Networks with a large number of feedback connectivitys can be useful when the recognition of a higher-level concept can help distinguish certain lower-level features of the input.

[0027]

[0034] The connections between layers of a neural network may be fully connected or locally connected. Figure 2A shows an example of a fully connected neural network 202. In the fully connected neural network 202, neurons in the first layer may transmit their output to any neuron in the second layer, so that each neuron in the second layer receives input from any neuron in the first layer. Figure 2B shows an example of a locally connected neural network 204. In the locally connected neural network 204, neurons in the first layer may be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured so that each neuron in a given layer has the same or similar connectivity pattern, but with different connection strengths (e.g., 210, 212, 214, and 216). Since higher-layer neurons within a given region can receive inputs that, through training, are tuned to the characteristics of a limited portion of the total inputs to the network, the connectivity patterns of local connections can give rise to spatially distinct receptive fields within the higher layers.

[0028]

[0035] An example of a locally connected neural network is a convolutional neural network. Figure 2C shows one embodiment of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strengths associated with the input for each neuron in the second layer are shared (e.g., 208). Convolutional neural networks may be suitable for problems where the spatial location of the input is meaningful.

[0029]

[0036] One type of convolutional neural network is the deep convolutional network (DCN). Figure 2D shows a detailed example of DCN200, designed to recognize visual features from an image 226 input from an image capture device 230, such as an in-vehicle camera. DCN200 in this example may be trained to identify traffic signs and the numbers written on them. Of course, DCN200 may also be trained for other tasks, such as identifying lane markings or traffic signals.

[0030]

[0037] DCN200 may be trained using supervised learning. During training, DCN200 may be presented with images, such as image 226 of a speed limit sign, and then a forward pass may be computed to produce output 222. DCN200 may include a feature extraction section and a classification section. Upon receiving image 226, the convolutional layer 232 may apply a convolutional kernel (not shown) to image 226 to generate a first set 218 of feature maps. For example, the convolutional kernel for convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this embodiment, four different feature maps are generated in the first set 218 of feature maps, so four different convolutional kernels were applied to image 226 in convolutional layer 232. Convolutional kernels are also sometimes called filters or convolutional filters.

[0031]

[0038] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218; that is, the size of the second set of feature maps 220, such as 14×14, is smaller than the size of the first set of feature maps 218, such as 28×28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0032]

[0039] In the embodiment shown in Figure 2D, a second set of feature maps 220 is convolved to generate a first feature vector 224. Further convolved, the first feature vector 224 generates a second feature vector 228. Each feature in the second feature vector 228 may include a number corresponding to a possible feature of image 226, such as "label", "60", and "100". A softmax function (not shown) may convert the numbers in the second feature vector 228 into probabilities. Therefore, the output 222 of DCN200 may be the probability that image 226 contains one or more features.

[0033]

[0040] In this embodiment, the probabilities in output 222 for “label” and “60” are higher than the probabilities of other outputs 222 such as “30”, “40”, “50”, “70”, “80”, “90”, and “100”. Before training, the outputs 222 generated by DCN200 may be inaccurate. Therefore, an error may be calculated between output 222 and the target output. The target output is the ground truth of image 226 (e.g., “label” and “60”). The weights of DCN200 may then be adjusted so that the output 222 of DCN200 is more closely matched with the target output.

[0034]

[0041] To adjust the weights, the learning algorithm may calculate gradient vectors for the weights. The gradient can indicate the amount by which the error will increase or decrease when the weights are adjusted. In the top layer, the gradient can directly correspond to the weight values ​​connecting the activated neurons in the second-to-last layer to the neurons in the output layer. In lower layers, the gradient can depend on the weight values ​​and the error gradient calculated in the upper layers. The weights can then be adjusted to reduce the error. This method of adjusting weights is sometimes called "backpropagation" because it involves a "backward path" through the neural network.

[0035]

[0042] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate for the entire system stops decreasing or until the error rate reaches a target level. After training, DCN200 may be presented with new images, and a forward pass through DCN200 may yield an output 222 that can be considered DCN200's inference or prediction.

[0036]

[0043] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs may be used to extract a hierarchical representation of a training dataset. DBNs may also be obtained by stacking layers of restricted Boltzmann machines (RBMs). RBMs are a type of artificial neural network that can learn a probability distribution over a set of inputs. Because RBMs can learn a probability distribution without any prior information about the class to which each input should be categorized, RBMs are frequently used in unsupervised learning. Using a hybrid paradigm of unsupervised and supervised learning, the lower RBM of a DBN can be trained unsupervised and can function as a feature extractor, while the upper RBM can be trained supervised (on a joint distribution of inputs from previous layers and the target class) and can function as a classifier.

[0037]

[0044] Deep convolutional networks (DCNs) are networks of convolutional networks composed of additional pooling and normalization layers. DCNs achieve state-of-the-art performance for a wide range of tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for a large number of samples, and the network weights are modified using gradient descent.

[0038]

[0045] A DCN may also be a feedforward network. In addition, as described above, connections from neurons in the first layer of a DCN to groups of neurons in the next higher layer are shared across the neurons in the first layer. The feedforward and shared connections of a DCN can be leveraged for high-speed processing. The computational burden of a DCN may be significantly less than that of a similarly sized neural network, for example, one that includes recursive or feedback connections.

[0039]

[0046] The processing of each layer of a convolutional network may be considered as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, the convolutional network trained on that input may be considered three-dimensional, having two spatial dimensions along the image axes and a third dimension that captures color information. The output of the convolutional connections may be considered as forming a feature map in subsequent layers, where each element of the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map may be further processed using rectification, max(0,x), or other nonlinearities. Values ​​from adjacent neurons may further undergo pooling corresponding to downsampling, which can result in additional local invariance and dimensionality reduction. Normalization corresponding to whitening may also be applied through lateral inhibition between neurons in the feature map.

[0040]

[0047] The performance of deep learning architectures can improve as more labeled data points become available or as computational power increases. Modern deep neural networks are routinely trained using computational resources thousands of times greater than what was available to the typical researcher just 15 years ago. New architectures and training paradigms can further enhance the performance of deep learning. Rectified linear units may mitigate the training problem known as vanishing gradient. New training techniques may reduce overfitting, thus allowing larger models to achieve better generalization. Encapsulation techniques may extract data within a given receptive field, further improving overall performance.

[0041]

[0048] Figure 3 is a block diagram of DCN350. The deep convolutional network 350 may include several different types of layers based on connectivity and weight sharing. As shown in Figure 3, the deep convolutional network 350 includes convolutional blocks 354A and 354B. Each of the convolutional blocks 354A and 354B may consist of a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a maximum pooling layer (MAX POOL) 360. Although only two of the convolutional blocks 354A and 354B are shown, this disclosure is not limited in that way, and instead, any number of convolutional blocks 354A and 354B may be included in the deep convolutional network 350 according to design preferences.

[0042]

[0049] The convolutional layer 356 may include one or more convolutional filters that can be applied to the input data to generate a feature map. The normalization layer 358 may normalize the output of the convolutional filters. For example, the normalization layer 358 may result in whitening or lateral suppression. The maximal pooling layer 360 may result in downsampling aggregation across space for local invariance and dimensionality reduction.

[0043]

[0050] For example, the parallel filter bank of the deep convolutional network may be loaded on the CPU 102 or GPU 104 of the SOC 100 (e.g., Figure 1) to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank may be loaded on the DSP 106 or ISP 116 of the SOC 100. In addition, the deep convolutional network 350 may access other processing blocks that may reside on the SOC 100, such as the sensor processor 114 and navigation module 120, which are dedicated to sensors and navigation, respectively.

[0044]

[0051] The deep convolutional network 350 may also include one or more fully connected layers 362 (FC1 and FC2). The deep convolutional network 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362, 364 of the deep convolutional network 350 are weights (not shown) that will be updated. The output of each layer (e.g., 356, 358, 360, 362, 364) may serve as input to one of the subsequent layers (e.g., 356, 358, 360, 362, 364) in the deep convolutional network 350 to learn a hierarchical feature representation from the initially supplied input data 352 (e.g., image, audio, video, sensor data, and / or other input data) of the convolutional block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 may be a set of probabilities, where each probability is the probability that the input data contains a feature from the set of features.

[0045]

[0052] Figure 4 is a block diagram showing an exemplary software architecture 400 in which artificial intelligence (AI) functions can be modularized. Using this architecture 400, an application can be designed in which various processing blocks of the SOC 420 (which may be similar to the SoC 100 in Figure 1) (e.g., CPU 422, DSP 424, GPU 426, and / or NPU 428) can support adaptive rounding as disclosed for post-training quantization for an AI application 402 according to aspects of this disclosure.

[0046]

[0053] The AI ​​application 402 may be configured to invoke functions defined in user space 404, for example, which can detect and recognize a scene indicating the location in which the device is currently operating. The AI ​​application 402 may configure the microphone and camera differently depending on whether the scene to be recognized is an office, an auditorium, a restaurant, or an outdoor setting such as a lake. The AI ​​application 402 may make requests to compiled program code associated with a library defined in an AI function application programming interface (API), such as the SceneDetect API 406, to provide an estimate of the current scene. This request may ultimately rely on the output of a deep neural network configured to provide an inference response based on video and positioning data, for example. The deep neural network may be a differential neural network configured to provide scene estimation based on video and positioning data, for example.

[0047]

[0054] The runtime engine 408, which may be compiled code of a runtime framework, may be further accessible to the AI ​​application 402. The AI ​​application 402 may request the runtime engine 408 to perform inference, such as scene estimation, triggered, for example, at a specific time interval or by an event detected by the user interface of the application 402. Once the runtime engine 408 has provided an inference response (e.g., for scene estimation), it may then signal an operating system in the operating system (OS) space 410, such as the kernel 412, running on the SOC 420. The operating system may then perform a continuous relaxation of quantization on the CPU 422, DSP 424, GPU 426, NPU 428, or any combination thereof. The CPU 422 may be accessed directly by the operating system, while the other processing blocks may be accessed through drivers, such as drivers 414, 416, or 418 for the DSP 424, GPU 426, or NPU 428, respectively. In an exemplary example, the deep neural network may be configured to operate on a combination of processing blocks such as CPU422, DSP424, and GPU426, or it may operate on NPU428.

[0048]

[0055]

[0049]

[0056] As stated above, aspects of this disclosure relate to flow-independent video compression. According to aspects of this disclosure, motion compensation is provided via a full convolutional operator instead of using a flow warping operator. For example, instead of warping previous frames using optical flow, a convolutional neural network may extract feature representations of video frames. Correlation features between the current frame and previous frames may be determined. Correlation features may be encoded and transmitted to a receiver. The encoded correlation features are decoded to predict the current frame.

[0050]

number

[0051] can be used together with the features of previous frames to determine.

[0052]

[0057] FIG. 5 is a block diagram showing an exemplary architecture 500 for flow-independent video compression according to an aspect of the present disclosure. The architecture 500 may include a motion network 502 and a residual network 504. The motion network 502 and the residual network 504 may be implemented, for example, as autoencoders.

[0053]

[0058] The motion network 502 may provide motion compensation as a feature transformation. In the example of FIG. 5, the motion network 502 may include an encoder 506, a decoder 508, and a prior network 510. The encoder 506 may include a feature extractor g a 514a. The feature extractor g a 514a may be, for example, a convolutional neural network (e.g., 350 shown in FIG. 3). The feature extractor g a 514a may receive a video as input. The video may include a plurality of frames. As shown, the feature extractor g a 514a may receive the current frame x i of the video to be compressed. The feature extractor g a 514a may process the current frame x <� i through one or more convolutional layers, extract a set of features, and generate a feature representation of the current frame x i . In addition, the feature extractor g a [[ID=,33]]514b may receive the previous frame

[0054]

Number

[0055] of the video. FIG. 5 shows a plurality of feature extractors g aWhile this is shown, it is merely an example and is only there to facilitate the illustration. Instead, one or more feature extractors g a The following may be used: Feature extractor g a 514b is the previous frame of the video through one or more convolutional layers.

[0056]

number

[0057] It processes the data, extracts a set of features, and then processes the previous frame.

[0058]

number

[0059] It is possible to generate feature representations of the current and previous frames. The feature representations of the current and previous frames are respectively extracted by the feature extractor g a The output can be obtained via 514a and 514b. The feature representations of the current and previous frames are concatenated via the concatenation layer 516a and the feature correlation estimator g c It can be supplied to 518.

[0060]

[0059] Feature correlation estimator g c 518 could also be a convolutional neural network. Feature correlation estimator g c 518 is the current frame x i and previous frame

[0061]

number

[0062] Correlation characteristics between

[0063]

number

[0064] A set of correlation features can be learned and encoded.

[0065]

number

[0066] The current frame x i and previous frame

[0067]

number

[0068] It can respond to movements between these two points.

[0069]

[0060] Correlation characteristics

[0070]

number

[0071] The data can be entropically encoded via an arithmetic encoder (AE) and transmitted to a receiver. Entropy encoding and decoding can be performed based on the distribution of latent correlation features learned via the prior network 510.

[0072]

[0061] Decoder 508 has correlation features

[0073]

number

[0074] It can receive the entropy-encoded latent representation of the correlation feature.

[0075]

number

[0076] The entropy-encoded latent representation is decoded via an arithmetic decoder (AD) and a code-feature converter g d It can be supplied to 520. Code-feature converter g d 520 is a correlation feature

[0077]

number

[0078] The latent feature representation can be recovered.

[0079]

[0062] The decoder 508 may also receive a reference frame 522. The reference frame 522 is, for example, a previous frame

[0080]

number

[0081] It is possible. Reference frame 522 is a feature extractor g that extracts a set of features and generates a feature representation of reference frame 522. a It is supplied to 514c.

[0082]

[0063] Code-feature converter g d Correlation features output by 520

[0083]

number

[0084] The latent feature representation is referenced via the connected layer 516b to the reference frame 522.

[0085]

number

[0086] The feature representation is linked to the frame decoder g sIt can be supplied to 524. Frame decoder g s 524 is a correlation feature and reference frame 522

[0087]

number

[0088] Process the concatenated feature representation and predict the current frame generated via the motion network 502.

[0089]

number

[0090] It can generate.

[0091]

[0064] The residual network 504 is a residual encoder r a 526, residual decoder r s 528 and the residual leading network 512 may similarly be included. a 526 can receive the residual as input. The residual is the actual current frame x i and current frame prediction

[0092]

number

[0093] It can be given by the difference between [the two values]. Residual encoder r a 526 can extract characteristic representations of residuals.

[0094]

[0065] The residual feature representation is entropy encoded and the receiver (e.g., residual decoder r s It can be transmitted to 528). Entropy coding and decoding can be performed based on the distribution of latent residual features learned via the prior network 512.

[0095]

[0066] Residual Decoder r s528 decodes the feature representation of the residual, and the residual (for example,

[0096]

number

[0097] prediction

[0098]

number

[0099] This can generate residual prediction.

[0100]

number

[0101] The current frame prediction is obtained via the addition block 530.

[0102]

number

[0103] It is added to the current frame x i Restoration

[0104]

number

[0105] It can generate the current frame x i Restoration

[0106]

number

[0107] is the subsequent frame of the video (for example, x i+1It can be transmitted to a receiver (e.g., encoder 506 of motion network 502) for use in motion compensation for )

[0108]

[0067] In some embodiments, the architecture 500 may use a full convolution operator. In doing so, motion compensation can be determined without using conventional motion compensation approaches that use optical flow.

[0109]

[0068] Figure 6 is a block diagram showing an exemplary architecture 600 for flow-independent video compression according to an aspect of the present disclosure. Referring to Figure 6, the input for architecture 500 in Figure 5 may be modified to further utilize motion information from the history of multiple previously decoded frames. In Figure 6, two previously decoded frames

[0110]

number

[0111] While this is shown, the disclosure is not limited in that sense. Alternatively, two or more previously decoded frames may be included in the history of previously decoded frames. Architecture 600 may further include an extrapolator network (E) 602. The extrapolator network 602 may be, for example, a fully convolutional network. The extrapolator network 602 includes multiple previously decoded frames

[0112]

number

[0113] The extrapolator network 602 may receive the previously decoded frame.

[0114]

number

[0115] Processing, reference frame prediction

[0116]

number

[0117] It can generate a reference frame prediction.

[0118]

number

[0119] These are treated as intermediate frames and can be input to the motion network 502 of architecture 500. The accuracy of predicting the current frame can be increased by using multiple previously decoded frames as references instead of a single previously decoded frame (for example, as shown in Figure 5).

[0120]

[0069] Figure 7 is a block diagram showing an exemplary architecture 700 for flow-independent video compression according to an aspect of the present disclosure. Figure 7 is a simplified version of Figure 5, designed for efficiency optimization. Referring to Figure 7, architecture 700 can be simplified by removing the motion network 502. Thus, architecture 700 includes an extrapolator network 602 and a residual network 504. The extrapolator network 602 extracts the history of previous frames (e.g., two or more frames) from the current frame x i prediction

[0121]

number

[0122] This can be determined directly. The residual network 504 predicts

[0123]

number

[0124] and the current frame x i The difference between this and is encoded, and this is the current frame prediction.

[0125]

number

[0126] It is added to the current frame x i Restoration

[0127]

number

[0128] It can generate.

[0129]

[0070] Figure 8 is a flowchart showing an exemplary process 800 for video compression using an artificial neural network (ANN) according to an aspect of the present disclosure. As shown in Figure 8, in block 802, process 800 receives video via the ANN. A feature extractor g, as described with reference to Figure 5, a 514a may receive video as input. Video may contain multiple frames.

[0130]

[0071] In block 804, process 800 extracts a first set of features of the current frame of the video and a second set of features of a reference frame of the video by the ANN. As illustrated with reference to Figure 5, feature extractor g a 514a passes through one or more convolutional layers to the current frame x i It processes the data, extracts a set of features, and then processes the current frame x i A feature representation can be generated. Optionally, in block 806, the feature extractor g a 514b is the previous frame of the video

[0131]

number

[0132] It can receive. Feature extractor g a 514b is the previous frame of the video through one or more convolutional layers.

[0133]

number

[0134] It processes the data, extracts a set of features, and then processes the previous frame.

[0135]

number

[0136] It is possible to generate feature representations.

[0137]

[0072] In block 808, process 800 determines the estimation of correlation features between a first set of features of the current frame and a second set of features of the reference frame by the ANN. As illustrated with reference to Figure 5, the feature representations of the current and previous frames are respectively extracted by feature extractor g a The output can be obtained via 514a and 514b. The feature representations of the current and previous frames are concatenated via the concatenation layer 516a and the feature correlation estimator g c It can be supplied to 518. Feature correlation estimator g c 518 could also be a convolutional neural network. Feature correlation estimator g c 518 is the current frame x i and previous frame

[0138]

number

[0139] Correlation characteristics between

[0140]

number

[0141] A set of correlation features can be learned and encoded.

[0142]

number

[0143] The current frame x i and previous frame

[0144]

number

[0145] It can respond to movements between these two points.

[0146]

[0073] In block 810, process 800 transmits the estimation of correlation features to the receiver. As explained with reference to Figure 5, correlation features

[0147]

number

[0148] This can be entropically encoded via an arithmetic encoder (AE) and transmitted to a receiver (e.g., decoder 508).

[0149]

[0074] Figure 9 is a flowchart showing an exemplary process 900 for video compression using an artificial neural network (ANN) according to an aspect of the present disclosure. As shown in Figure 9, in block 902, process 900 receives a reference frame of video by the ANN. In block 904, process 900 receives an estimate of correlation features between the current frame of video and the reference frame of video by the ANN. As shown in Figure 5, decoder 508 receives the reference frame of video 522 and the current frame of video x i and video reference frame 522

[0150]

number

[0151] Correlation characteristics between

[0152]

number

[0153] Receive the estimate.

[0154]

[0075] In block 906, process 900 extracts a first set of features of the video reference frame by the ANN. For example, as illustrated with reference to Figure 5, decoder 508 may also receive reference frame 522. Reference frame 522 is, for example, a previous frame

[0155]

number

[0156] It is possible. Reference frame 522 is a feature extractor g that extracts a set of features and generates a feature representation of reference frame 522. a It is supplied to 514c.

[0157]

[0076] In block 908, process 900 decodes the estimated correlation features by ANN. As shown in Figure 5, decoder 508 decodes the correlation features

[0158]

number

[0159] It can receive the entropy-encoded latent representation of the correlation feature.

[0160]

number

[0161] The entropy-encoded latent representation is decoded via an arithmetic decoder (AD) and a code-feature converter g d It can be supplied to 520. Code-feature converter g d 520 is a correlation feature

[0162]

number

[0163] The latent feature representation can be recovered.

[0164]

[0077] In block 910, process 900 generates a prediction of the current frame of the video based on the decoded estimate of the correlation features and a first set of features of the reference frame of the video, as illustrated with reference to Figure 5.

[0165]

number

[0166] The latent feature representation is concatenated with the feature representation of reference frame 522 via the concatenation layer 516b, and the frame decoder g s It can be supplied to 524. Frame decoder g s 524 is a correlation feature

[0167]

number

[0168] and process the concatenated feature representation of reference frame 522, and generate the current frame x via motion network 502. i prediction

[0169]

number

[0170] Generates.

[0171]

[0078] Figure 10 is a flowchart showing an exemplary process 1000 for video compression using an artificial neural network (ANN) according to an aspect of the present disclosure. As shown in Figure 10, in block 1002, process 1000 receives a history of multiple frames of video by the ANN. As described with reference to Figure 6, architecture 600 may further include an extrapolator network (E) 602. The extrapolator network 602 may be, for example, a fully convolutional network. The extrapolator network 602 receives a history of multiple previously decoded frames

[0172]

number

[0173] It can receive this as input.

[0174]

[0079] In block 1004, process 1000 generates a reference frame based on the history of multiple frames of the video by the ANN. For example, as illustrated with reference to Figure 6, the extrapolator network 602 uses previously decoded frames

[0175]

number

[0176] Processing, reference frame prediction

[0177]

number

[0178] It can generate.

[0179]

[0080] In block 1006, process 1000 generates a prediction of the current frame of the video based on the reference frame and the current frame of the video using the ANN. Reference frame prediction created by extrapolator network 602 as shown in Figure 6

[0180]

number

[0181] and the current framex i Use this to predict the current frame

[0182]

number

[0183] This is generated.

[0184]

[0081] In block 1008, process 1000 generates a prediction of the current frame of the video by the ANN, and a reconstruction of the current frame based on the residual calculated based on the difference between the current frame of the video and the prediction of the current frame of the video. As shown in Figure 6, the residual network 504 compares the prediction and the current frame x i The difference between this and is encoded, and this is the current frame prediction.

[0185]

number

[0186] It is added to the current frame x i Restoration

[0187]

number

[0188] It can generate.

[0189]

[0082] Implementation examples are described in the following numbered clauses. 1. A processor implementation method for video compression using an artificial neural network (ANN), comprising: receiving a video by the ANN; extracting a first set of features of the current frame of the video and a second set of features of the reference frame of the video by the ANN; determining an estimate of the correlation features between the first set of features of the current frame and the second set of features of the reference frame by the ANN; transmitting the estimate of the correlation features to a receiver; and a processor implementation method. 2. The processor implementation method according to clause 1, further comprising determining an estimate of the correlation features via a fully convolutional operator of the ANN. 3. The processor implementation method according to clause 1 or 2, wherein the reference frame includes the restoration of a previous frame of the video. 4. The processor implementation method according to any one of clauses 1 to 3, wherein the reference frame is predicted based on the restoration of a plurality of preceding frames of the video. 5. A processor implementation method for video compression using an artificial neural network (ANN), comprising: receiving a reference frame of the video by the ANN; receiving an estimate of the correlation features between the current frame of the video and the reference frame of the video by the ANN; extracting a first set of features of the reference frame of the video by the ANN; decoding the estimate of the correlation features by the ANN; generating a prediction of the current frame of the video based on the decoded estimate of the correlation features and the first set of features of the reference frame of the video by the ANN; and a processor implementation method. 6. calculating a residual based on the difference between the current frame of the video and the prediction of the current frame of the video; To generate a reconstruction of the current frame of the video based on the prediction and residual of the current frame, The processor implementation method described in Clause 5, further including the following: 7. The processor implementation method described in Clause 5 or 6, wherein the reference frame is predicted based on the reconstruction of multiple preceding frames of the video. 8. A processor implementation method for video compression using an artificial neural network (ANN), ANN allows receiving the history of multiple frames of video, ANN generates a reference frame from the history of multiple frames of video, ANN generates a prediction of the current frame of the video based on the reference frame and the current frame of the video, The ANN generates a prediction of the current frame of the video, and a reconstruction of the current frame based on the residual calculated based on the difference between the current frame of the video and the prediction of the current frame of the video. Processor implementation method including 9. The processor implementation method described in Clause 8, wherein the history includes multiple previously decoded frames. 10. A processor implementation method according to Clause 8 or 9, further comprising encoding the difference between the current frame of the video and a prediction of the current frame of the video, and adding the encoding to the prediction of the current frame. 11. A device for video compression using an artificial neural network (ANN), Memory and A system comprising at least one processor coupled to memory, wherein at least one processor is ANN receives the video, The ANN extracts a first set of features for the current frame of the video and a second set of features for the reference frame of the video. The ANN determines the estimation of correlation features between a first set of features of the current frame and a second set of features of the reference frame. A device configured to transmit estimated correlation features to a receiver. 12. At least one processor, The apparatus described in Clause 11, further configured to determine the estimation of correlation features via the full convolution operator of the ANN. 13. The apparatus described in Clause 11 or 12, which includes the restoration of a previous frame of video by referencing a frame. 14. The apparatus described in any one of the clauses 11 to 13, wherein at least one processor is further configured to predict a reference frame based on the reconstruction of multiple preceding frames of video. 15. A device for video compression using an artificial neural network (ANN), Memory and A system comprising at least one processor coupled to memory, wherein at least one processor is Receive a reference frame from the video, The estimation of correlation features between the current frame of the video and the reference frame of the video is received. Extract the first set of features from the video's reference frame, Decode the estimated correlation features, A device configured to generate a prediction of the current frame of a video based on a decoded estimation of correlated features and a first set of features of the video's reference frame. 16. At least one processor, The residual is calculated based on the difference between the current frame of the video and the prediction for the current frame of the video. The apparatus described in Clause 15, further configured to generate a reconstruction of the current frame of video based on the prediction and residuals of the current frame. 17. The apparatus according to Clause 15 or 16, wherein at least one processor is further configured to predict a reference frame based on the reconstruction of multiple preceding frames of video. 18. A device for video compression using an artificial neural network (ANN), Memory and A system comprising at least one processor coupled to memory, wherein at least one processor is ANN receives the history of multiple frames of the video, ANN generates a reference frame from the history of multiple frames of video, ANN generates a prediction of the current frame of the video based on the reference frame and the current frame of the video. An ANN-based device configured to generate a prediction of the current frame of a video, and a reconstruction of the current frame based on the residual calculated based on the difference between the current frame of the video and the prediction of the current frame of the video. 19. The device described in Clause 18, whose history includes multiple previously decoded frames. 20. The apparatus according to Clause 18 or 19, wherein at least one processor is further configured to encode the difference between the current frame of video and a prediction of the current frame of video, and to add the encoding to the prediction of the current frame.

[0190]

[0083] In one embodiment, the receiving means, extraction means, determination means, transmission means, decoding means, means for generating predictions by the ANN, means for generating reference frames by the ANN, and / or means for generating restorations by the ANN may be a CPU 102, program memory associated with the CPU 102, a dedicated memory block 118, a fully connected layer 362, an NPU 428 / , and / or a routing connection processing unit 216 configured to perform the functions described above. In another configuration, the means described above may be any module or any device configured to perform the functions enumerated by the means described above.

[0191]

[0084] Various operations of the methods described above may be carried out by any preferred means capable of performing the corresponding functions. The means may include, but are not limited to, various hardware and / or software components and / or modules, including circuits, application-specific integrated circuits (ASICs), or processors. Generally, where there are operations shown in the figures, those operations may have corresponding relative means-plus-function components that are similarly numbered.

[0192]

[0085] As used herein, the term “determining” encompasses a wide variety of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or other data structure), and confirming. In addition, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and so on. Furthermore, “determining” may include resolving, selecting, choosing, and establishing.

[0193]

[0086] As used herein, the phrase “at least one of” the list of items refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” includes a, b, c, ab, ac, bc, and abc.

[0194]

[0087] The various illustrative logical blocks, modules, and circuits described in connection with the present disclosure may be implemented or executed using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gates or transistor logic, discrete hardware components, or any combination thereof designed to perform the described functions. The general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0195]

[0088] The steps of a method or algorithm described in connection with the present disclosure may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may reside in any form of computer-readable medium known in the art. Some examples of computer-readable media that may be used are random access memory (RAM), read only memory (ROM), flash memory, erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM (registered trademark)), registers, hard disk, removable disk, CD-ROM, and the like. The software module may comprise a single instruction or multiple instructions and may be distributed over different programs, across different code segments, or across multiple storage media. The computer-readable medium may be coupled to the processor such that the processor can read information from, and write information to, the computer-readable medium. In the alternative, the computer-readable medium may be integral to the processor.

[0196]

[0089] The methods disclosed herein include one or more steps or actions for achieving the described method. The steps and / or actions of those methods can be replaced with one another without departing from the claims. In other words, unless a particular order of steps or actions is specified, the order and / or use of any particular steps and / or actions can be modified without departing from the claims.

[0197]

[0090] The functions described may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, the exemplary hardware configuration may include a processing system within the device. The processing system may be implemented using a bus architecture. The bus may include any number of interconnection buses and bridges, depending on the specific application of the processing system and the overall design constraints. The bus may link various circuits to each other, including processors, machine-readable media, and bus interfaces. The bus interface may, among other things, be used to connect a network adapter to the processing system via the bus. The network adapter may be used to implement signal processing functions. In some embodiments, a user interface (e.g., a keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits, but these circuits are well known in the art and are therefore not described further.

[0198]

[0091] The processor may be responsible for managing the bus and general processing, including executing software stored on a machine-readable medium. The processor may be implemented using one or more general-purpose processors and / or dedicated processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuits capable of executing software. Software is broadly interpreted to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or by any other name. Machine-readable medium may include, for example, random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. The machine-readable medium may be embodied in a computer program product. The computer program product may include packaging materials.

[0199]

[0092] In hardware implementations, the machine-readable medium may be part of a processing system separate from the processor. However, as will be readily apparent to those skilled in the art, the machine-readable medium or any part thereof may be outside the processing system. For example, the machine-readable medium may include transmission lines, data-modulated carriers, and / or computer products separate from the device, all of which may be accessed by the processor through a bus interface. Alternatively or additionally, the machine-readable medium or any part thereof may be integrated into the processor, such as caches and / or general-purpose register files. The various components discussed may be described as having a specific location, such as local components, but these components may also be configured in various ways, such as several components configured as part of a distributed computing system.

[0200]

[0093] The processing system may be configured as a general-purpose processing system having one or more microprocessors providing processor functions, all of which are linked to each other with other support circuits through an external bus architecture, and external memory providing at least a portion of a machine-readable medium. Alternatively, the processing system may comprise one or more neuromorphological processors for implementing the described neuron model and neural system model. Another alternative is that the processing system may be implemented using an application-specific integrated circuit (ASIC) having a processor, a bus interface, a user interface, support circuits, and at least a portion of a machine-readable medium integrated on a single chip, or using one or more feed-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, individual hardware components, or any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this disclosure. Those skilled in the art will recognize how to best implement the functions described for the processing system depending on the specific application and the overall design constraints imposed on the system as a whole.

[0201]

[0094] The machine-readable medium may comprise several software modules. A software module contains instructions that, when executed by the processor, cause the processing system to perform various functions. A software module may also include a transmit module and a receive module. Each software module may reside in a single storage device or be distributed across multiple storage devices. For example, a software module may be loaded from a hard drive into RAM when a trigger event occurs. While a software module is executing, the processor may load some of the instructions into a cache to increase access speed. One or more cache lines may then be loaded into a general-purpose register file for execution by the processor. When the functions of a software module are referred to below, it will be understood that such functions are implemented by the processor when it executes instructions from that software module. Furthermore, it should be understood that aspects of this disclosure result in improvements to the functionality of processors, computers, machines, or other systems that implement such aspects.

[0202]

[0095] When implemented in software, the functions may be stored on or transmitted via computer-readable media as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any media that facilitates the transfer of computer programs from one place to another. The storage media may be any available media that can be accessed by a computer. Such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be accessed by a computer and that can be used to carry or store desired program code in the form of instructions or data structures. In addition, any connection is appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disk and disc include compact disc (CD), laserdisc (disc), optical disc (disc), digital versatile disc (disc) (DVD), floppy disk (disk), and Blu-ray (disc), where disk typically reproduces data magnetically and disc (disc) reproduces data optically using a laser. Thus, in some embodiments, computer-readable medium may include non-temporary computer-readable medium (e.g., tangible medium). In addition, in other embodiments, the computer-readable medium may include a temporary computer-readable medium (e.g., a signal). Combinations of the above should also be included within the scope of the computer-readable medium.

[0203]

[0096] Accordingly, some embodiments may include a computer program product for performing the operations described herein. For example, such a computer program product may include a computer-readable medium storing (and / or encoding) instructions, the instructions being executable by one or more processors to perform the operations described. In some embodiments, the computer program product may include packaging material.

[0204]

[0097] Furthermore, it should be understood that modules and / or other suitable means for performing the described methods and techniques may be downloaded and / or otherwise obtained by user terminals and / or base stations, where applicable. For example, such devices may be coupled to a server to facilitate the transfer of means for performing the described methods. Alternatively, the various methods described may be provided via storage means so that user terminals and / or base stations can obtain the various methods by coupling or providing storage means (e.g., physical storage media such as RAM, ROM, compact disks (CDs) or floppy disks) to the device. Furthermore, any other suitable techniques for providing the described methods and techniques to the device may be utilized.

[0205]

[0098] It should be understood that the claims are not limited to the exact configuration and components shown above. Various modifications, changes, and variations may be made to the configuration, operation, and details of the method and apparatus described above without departing from the claims. The invention described in the original claims of this application is listed below. [C1] A processor implementation method for video compression using an artificial neural network (ANN), The aforementioned ANN receives video and The ANN extracts a first set of features of the current frame of the video and a second set of features of the reference frame of the video. The ANN determines the estimation of correlation features between the first set of features of the current frame and the second set of features of the reference frame, Transmitting the aforementioned estimation of correlation features to the receiver, A processor implementation method comprising: [C2] The processor implementation according to C1, further comprising determining the estimation of correlation features via the full convolution operator of the ANN. [C3] The processor implementation method according to C1, wherein the reference frame comprises the restoration of a previous frame of the video. [C4] The processor implementation method according to C1, wherein the reference frame is predicted based on the reconstruction of a plurality of preceding frames of the video. [C5] A processor implementation method for video compression using an artificial neural network (ANN), The aforementioned ANN receives a video reference frame, The ANN receives an estimate of the correlation features between the current frame of the video and the reference frame of the video, The ANN extracts a first set of features of the reference frame of the video, The aforementioned ANN is used to decode the aforementioned estimation of correlation features, The ANN generates a prediction of the current frame of the video based on the decoded estimation of the correlation features and the first set of features of the reference frame of the video, A processor implementation method comprising: [C6] Calculate the residual based on the difference between the current frame of the video and the prediction of the current frame of the video, To generate a reconstruction of the current frame of the video based on the prediction and residual of the current frame, The processor implementation method described in C5 further includes the following: [C7] The processor implementation method according to C5, wherein the reference frame is predicted based on the reconstruction of a plurality of preceding frames of the video. [C8] A processor implementation for video compression using an artificial neural network (ANN), The aforementioned ANN receives the history of multiple frames of the video, The ANN generates a reference frame from the history of multiple frames of the video, The ANN generates a prediction of the current frame of the video based on the reference frame and the current frame of the video, The ANN generates a reconstruction of the current frame based on the prediction of the current frame of the video and the residual calculated based on the difference between the current frame of the video and the prediction of the current frame of the video. A processor implementation method comprising: [C9] The processor implementation method according to C8, wherein the history comprises a plurality of previously decoded frames. [C10] The processor implementation method according to C8, further comprising encoding the difference between the current frame of the video and the prediction of the current frame of the video, and adding the encoding to the prediction of the current frame. [C11] A device for video compression using an artificial neural network (ANN), comprising memory and The system comprises at least one processor coupled to the memory, wherein the at least one processor is The aforementioned ANN receives the video, The ANN extracts a first set of features of the current frame of the video and a second set of features of the reference frame of the video. The ANN determines the estimation of correlation features between the first set of features of the current frame and the second set of features of the reference frame. A device configured to transmit the aforementioned estimation of correlation features to a receiver. [C12] The at least one processor, The apparatus according to C11, further configured to determine the estimation of correlation features via the full convolution operator of the ANN. [C13] The apparatus according to C11, wherein the reference frame comprises the restoration of a previous frame of the video. [C14] The apparatus according to C11, wherein the at least one processor is further configured to predict the reference frame based on the reconstruction of a plurality of preceding frames of the video. [C15] A device for video compression using an artificial neural network (ANN), comprising memory and The system comprises at least one processor coupled to the memory, wherein the at least one processor is Receive a reference frame from the video, The estimation of correlation features between the current frame of the video and the reference frame of the video is received. A first set of features of the reference frame of the video is extracted, Decode the aforementioned estimation of correlation features, A device configured to generate a prediction of the current frame of the video based on the decoded estimation of the correlation features and the first set of features of the reference frame of the video. [C16] The at least one processor, The residual is calculated based on the difference between the current frame of the video and the prediction of the current frame of the video. The apparatus according to C15, further configured to generate a reconstruction of the current frame of the video based on the prediction and residual of the current frame. [C17] The apparatus according to C15, wherein the at least one processor is further configured to predict the reference frame based on the reconstruction of a plurality of preceding frames of the video. [C18] A device for video compression using an artificial neural network (ANN), comprising memory and The system comprises at least one processor coupled to the memory, wherein the at least one processor is The aforementioned ANN receives the history of multiple frames of the video, The ANN generates a reference frame from the history of multiple frames of the video, The ANN generates a prediction of the current frame of the video based on the reference frame and the current frame of the video. A device configured to generate a reconstruction of the current frame based on the prediction of the current frame of the video and the residual calculated based on the difference between the current frame of the video and the prediction of the current frame of the video, using the ANN. [C19] The apparatus according to C18, wherein the history comprises a plurality of previously decoded frames. [C20] The apparatus according to C18, wherein the at least one processor is further configured to encode the difference between the current frame of the video and the prediction of the current frame of the video, and to add the encoding to the prediction of the current frame.

Claims

1. A processor implementation for flow-independent neural network-based video compression using an artificial neural network (ANN) that does not use flow-based warping, The aforementioned ANN receives video, The ANN extracts a first set of features of the current frame of the video and a second set of features of a reference frame of the video, wherein the first set of features represents a first feature representation associated with the current frame, and the second set of features represents a second feature representation associated with the reference frame. The ANN determines the estimation of at least one correlated feature between the first set of features of the current frame and the second set of features of the reference frame, Transmitting the estimation of at least one correlation feature to the receiver, A processor implementation method comprising:

2. The processor implementation method according to claim 1, further comprising determining the estimation of the at least one correlation feature via the full convolution operator of the ANN.

3. The processor implementation method according to claim 1, wherein the reference frame comprises the restoration of a previous frame of the video.

4. The processor implementation method according to claim 1, wherein the reference frame is predicted based on the reconstruction of a plurality of preceding frames of the video.

5. A processor implementation for flow-independent neural network-based video compression using an artificial neural network (ANN) that does not use flow-based warping, The aforementioned ANN receives a reference frame of the video, The ANN receives an estimate of at least one correlation feature between the current frame of the video and the reference frame of the video, The ANN extracts a first set of features of the reference frame of the video, wherein the first set of features corresponds to a feature representation of the reference frame. The ANN is used to decode the estimation of the at least one correlation feature, The ANN generates a prediction of the current frame of the video based on the decoded estimation of the at least one correlation feature and the first set of features of the reference frame of the video, A processor implementation method comprising:

6. The residual is calculated based on the difference between the current frame of the video and the prediction of the current frame of the video. To generate a reconstruction of the current frame of the video based on the prediction and residual of the current frame, The processor implementation method according to claim 5, further comprising the above.

7. The processor implementation method according to claim 5, wherein the reference frame is predicted based on the reconstruction of a plurality of preceding frames of the video.

8. An apparatus for flow-independent neural network-based video compression using an artificial neural network (ANN) that does not use flow-based warping, Memory and The system comprises at least one processor coupled to the memory, wherein the at least one processor The ANN receives the video, The ANN extracts a first set of features of the current frame of the video and a second set of features of a reference frame of the video, wherein the first set of features represents a first feature representation associated with the current frame, and the second set of features represents a second feature representation associated with the reference frame. The ANN determines the estimation of at least one correlated feature between the first set of features of the current frame and the second set of features of the reference frame. A device configured to transmit the estimation of at least one correlation feature to a receiver.

9. The aforementioned at least one processor, The apparatus according to claim 8, further configured to determine the estimation of the at least one correlation feature via the full convolution operator of the ANN.

10. The apparatus according to claim 8, wherein the reference frame comprises the restoration of a previous frame of the video.

11. The apparatus according to claim 8, wherein the at least one processor is further configured to predict the reference frame based on the reconstruction of a plurality of preceding frames of the video.

12. An apparatus for flow-independent neural network-based video compression using an artificial neural network (ANN) that does not use flow-based warping, Memory and The system comprises at least one processor coupled to the memory, wherein the at least one processor Receive a reference frame from the video, The estimation of at least one correlation feature between the current frame of the video and the reference frame of the video is received. A first set of features of the reference frame of the video is extracted, wherein the first set of features corresponds to the feature representation of the reference frame. Decode the estimation of the at least one correlation feature, A device configured to generate a prediction of the current frame of the video based on the decoded estimation of the at least one correlation feature and the first set of features of the reference frame of the video.

13. The aforementioned at least one processor, The residual is calculated based on the difference between the current frame of the video and the prediction of the current frame of the video. The apparatus according to claim 12, further configured to generate a reconstruction of the current frame of the video based on the prediction and residual of the current frame.

14. The apparatus according to claim 12, wherein the at least one processor is further configured to predict the reference frame based on the reconstruction of a plurality of preceding frames of the video.

Images