Inter coding transforms for autoencoder

Abstract

The present disclosure provides methods, apparatuses, and systems for video coding. The present implementation more specifically relates to an autoencoder that supports inter-frame coding in a potential domain. A video encoder may transform a video frame from a pixel domain to a potential domain based on a machine learning model. For example, a machine learning model may be trained to transform video frames into potential attribute tensors. In some aspects, a video encoder may combine a generated tensor and a potential attribute tensor associated with a previously encoded video frame and transform the generated tensor to a vector that includes potential attributes from both a current video frame and a previous video frame based on an inter-coding transform. More specifically, the inter coding transform may reduce the dimension of the combined tensor such that the generated vector is less than or more compressible than the original potential attribute tensor.

Description

Technical Field

[0001] The present implementation relates generally to video coding, and in particular to inter-coding transforms for autoencoders. Background Art

[0002] Digital video is a sequence of digital images (or "frames") that can be displayed continuously or otherwise rendered (such as by a video playback device). Each digital image can be represented by an array of pixel values ​​(or multiple arrays of pixel values ​​associated with different channels). Some video playback devices can receive a sequence of images from a source device (such as an image capture device or a video repository) via a communication channel (such as a wired medium or a wireless medium). Due to the bandwidth limitations of the communication channel, digital image data is typically encoded or compressed before being transmitted by the source device. Data compression is a technique for encoding information into smaller data units. As such, data compression can be used to reduce the bandwidth or overhead required to store or transmit video frames over a communication channel.

[0003] Example video compression techniques include "intra-frame" decoding and "inter-frame" decoding. Intra-frame decoding can be used to encode individual video frames as independent images (rather than a temporal sequence of images). In other words, the pixel values ​​of an intra-coded video frame can be decoded independently, or in the absence of information from any other video frame. However, adjacent video frames (such as images displayed successively in time) are likely to be identical or substantially similar in appearance. Thus, inter-frame decoding (also known as "inter-frame decoding" or "differential decoding") can be used to transmit only the differences (also known as "residuals") between consecutive video frames. In other words, the residual video frame must be combined with one or more previous video frames to recover the pixel values ​​to be displayed for the current video frame.

[0004] Some modern video decoding systems (such as autoencoders) use machine learning to achieve higher levels of data compression. However, machine learning models require a considerable amount of processing and memory overhead (especially for encoding residual frames). Thus, new video decoding techniques are needed to exploit machine learning for data compression while reducing overall resource consumption. Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] An innovative aspect of the subject matter of the present disclosure can be implemented in a method of video decoding. The method includes the following steps: receiving an image sequence; encoding a first image in the image sequence into a latent attribute tensor associated with the first image based on a machine learning model; performing an inter-frame decoding operation to generate a latent attribute vector based on the latent attribute tensor associated with the first image and the latent attribute tensor associated with a second image received before the first image in the image sequence; and transmitting the vector to a video decoder through a communication channel.

[0007] Another innovative aspect of the subject matter of the present disclosure can be implemented in a video encoder, the video encoder including a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the video encoder to: receive an image sequence; encode a first image in the image sequence into a latent attribute tensor associated with the first image based on a machine learning model; perform an inter-frame decoding operation to generate a latent attribute vector based on the latent attribute tensor associated with the first image and the latent attribute tensor associated with a second image received before the first image in the image sequence; and transmit the vector to a video decoder through a communication channel. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present implementations are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings.

[0009] Figure 1 An example communication system for encoding and decoding data is shown.

[0010] Figure 2 A block diagram of an example video transmission system is shown in accordance with some implementations.

[0011] Figure 3 A block diagram of an example video encoding system is shown in accordance with some implementations.

[0012] Figure 4 Example inter-coding operations that can be performed in a latent domain are shown according to some implementations.

[0013] Figure 5 Example inverse inter-coding operations that can be performed in a latent domain are shown according to some implementations.

[0014] Figure 6 A block diagram of an example video decoding system is shown in accordance with some implementations.

[0015] Figure 7 A block diagram of an example video encoder is shown in accordance with some implementations.

[0016] Figure 8 An illustrative flow diagram depicting example operations for video coding in accordance with some implementations is shown. DETAILED DESCRIPTION

[0017] In the following description, many specific details, such as examples of specific components, circuits, and processes, are set forth to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected to or connected through one or more intermediate components or circuits. The terms "electronic system" and "electronic device" can be used interchangeably to refer to any system capable of electronically processing information. Similarly, in the following description, and for the purpose of explanation, specific terms are set forth to provide a thorough understanding of aspects of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be required to practice the exemplary embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid making the present disclosure difficult to understand. The subsequent portions of the detailed description are presented based on the procedures, logic blocks, processing, and other symbolic representations of the operations of the data bits in the computer memory.

[0018] These descriptions and representations are the means by which the technical personnel in the field of data processing use to most effectively communicate the essence of their work to other technical personnel in the field. In the present disclosure, procedures, logic blocks, processes, etc. are considered to be the self-consistent (self-consistent) sequence of steps or instructions that lead to the desired result. The steps are those steps that require the physical manipulation of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated in a computer system. However, it should be remembered that all terms in these terms and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to these quantities.

[0019] Unless specifically stated, as will be apparent from the following discussion, it is appreciated that throughout this application, discussions utilizing terms such as "access," "receive," "send," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices.

[0020] In the accompanying drawings, a single block may be described as performing one or more functions; however, in actual practice, one or more functions performed by the block may be performed in a single component or across multiple components, and / or may be performed using hardware, using software, or using a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described below generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Similarly, the example input device may include components other than those shown, including well-known components such as processors, memories, etc.

[0021] Unless the techniques described herein are specifically described as being implemented in a specific manner, the techniques may be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules or components may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory processor-readable storage medium that includes instructions that, when executed, perform one or more of the methods described above. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

[0022] Non-transitory processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, etc. Additionally or alternatively, the technology may be at least partially implemented by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or other processor.

[0023] The various illustrative logical blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein may be executed by one or more processors (or processing systems). The term "processor" as used herein may refer to any general purpose processor, special purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0024] As described above, some modern video decoding systems (such as autoencoders) utilize machine learning to achieve higher levels of data compression. Machine learning is a technique for improving the ability of a computer system or application to perform a specific task. During the training phase, the machine learning system is provided with multiple "answers" and a large amount of raw input data. The machine learning system analyzes the input data to learn a set of rules (also referred to as a "machine learning model") that can be used to map the input data to the answer. During the inference phase, the machine learning system uses the trained machine learning model to infer the answer from new input data.

[0025] Deep learning is a specific form of machine learning in which the inference phase and the training phase are performed on multiple layers. Due to the way information is processed (similar to biological nervous systems), deep learning architectures are often referred to as "artificial neural networks." For example, each layer of an artificial neural network can be composed of one or more "neurons." The neurons of each layer can perform different transformations on the output data from the previous layer so that the final output of the neural network leads to the desired inference. The set of transformations associated with the various layers of the network is called a "neural network model."

[0026] An autoencoder is a type of artificial neural network that is well suited for image compression. For example, an autoencoder can be trained to reproduce at its output the same image received at its input. A bottleneck is imposed between the input and output layers of the neural network, which reduces the dimensionality of the output at the intermediate layers. As a result of this bottleneck, the autoencoder is forced to learn a compressed representation (or "latent attribute") of the input image. Thus, an autoencoder architecture generally includes an encoder trained to convert a frame of image data into a lower dimensional tensor or vector of latent attributes and a decoder trained to reconstruct the original frame of image data from the tensor or vector of latent attributes.

[0027] In a video decoding system, the latent attribute represents compressed image data sent by a video encoder to a video decoder through a communication channel. For example, the video encoder can implement the encoding portion of an autoencoder, and the video decoder can implement the decoding portion of the autoencoder. However, when using an autoencoder to compress a residual frame of image data (as a result of inter-frame decoding), the video encoder must also implement the decoding portion of the autoencoder to recover (from the residual frame of the latent attribute) the previous video frame that will be used to inter-frame decode the next residual frame.

[0028] Performing both the encoding and decoding operations of the autoencoder requires considerable processing and memory overhead, which may be impractical (or infeasible) to implement in some video encoders. However, aspects of the present disclosure recognize that inter-frame decoding can be more efficiently integrated with the autoencoder by moving the inter-frame decoding framework from the pixel domain to the potential domain. As used herein, the term "potential domain" refers to any data associated with the potential attributes of the input image or operations performed on the potential attributes of the input image (such as after the dimensionality of the input image is reduced by the autoencoder). In contrast, the term "pixel domain" refers to any data associated with the pixel values ​​of the input image or operations performed on the pixel values ​​of the input image (such as before the dimensionality of the input image is reduced).

[0029] Various aspects generally relate to video decoding, and more particularly to an automatic encoder that supports inter-frame decoding in a potential domain. A video encoder can convert a video frame from a pixel domain to a potential domain based on a machine learning model. For example, a machine learning model can be trained to transform a video frame into a potential attribute tensor (also referred to as an "encoded tensor"). In some aspects, the video encoder can combine the encoded tensor with a potential attribute tensor associated with a previously encoded video frame (also referred to as a "decoded tensor") and transform the resulting tensor into a vector (also referred to as a "latent vector") including potential attributes from both the current video frame and the previous video frame based on an inter-frame decoding transform. More specifically, the inter-frame decoding transform can reduce the dimensionality of the combined tensor so that the potential vector is smaller or more compressible than the encoded tensor. The video encoder further transmits the potential vector to a video decoder via a communication channel.

[0030] In some aspects, the video decoder may combine the latent vector with the decoded tensor and transform the resulting tensor into a latent attribute tensor associated with the current video frame (also referred to as a "new decoded tensor") based on an inverse inter-coding transform. More specifically, the inverse inter-coding transform may reduce the dimensionality of the combined tensor so that the new decoded tensor can be converted from the latent domain back to the pixel domain by a machine learning model. For example, the machine learning model may be trained to reconstruct the current video frame from the latent attributes included in the new decoded tensor. In some implementations, the video decoder may decode (or decompress) the new decoded tensor into the current video frame based on the machine learning model. In some other implementations, the video encoder may use the new decoded tensor to repeat the inter-coding operation for a new video frame (such as the next video frame in a sequence).

[0031] Specific implementations of the subject matter described in the present disclosure can be implemented to achieve one or more of the following potential advantages. Aspects of the present disclosure combine the benefits of inter-frame decoding with the benefits of an autoencoder in an efficient manner to reduce the size of compressed image data. For example, by performing inter-frame decoding in a latent domain associated with an autoencoder, the video encoder of the present implementation is able to encode latent vectors based on previous video frames without having to transform the latent attributes of each video frame back into the pixel domain. In addition, the inter-frame decoding transform and the inverse inter-frame decoding transform of the present implementation can be substantially smaller than the neural network layer of the autoencoder required to convert the latent attributes from the latent domain to the pixel domain. Thus, compared to existing inter-frame decoding techniques, aspects of the present disclosure can significantly reduce the processing and memory overhead of the video encoder.

[0032] Figure 1 An example communication system 100 for encoding and decoding data is shown. The communication system 100 includes an encoder 110 and a decoder 120. In some implementations, the encoder 110 and the decoder 120 may be provided in a corresponding communication device (such as, for example, a computer, a switch, a router, a hub, a gateway, a camera, a display, or other device capable of transmitting or receiving a communication signal). In some other implementations, the encoder 110 and the decoder 120 may be included in the same device or system.

[0033] The encoder 110 receives input data 102 to be transmitted or stored via a channel 130. For example, the channel 130 may include a wired communication medium or a wireless communication medium that facilitates communication between the encoder 110 and the decoder 120. Alternatively or in addition, the channel 130 may include a data storage medium. In some aspects, the encoder 110 may be configured to compress the size of the input data 102 to accommodate bandwidth, storage, or other resource limitations associated with the channel 130. For example, the encoder 110 may encode each unit of the input data 102 into a corresponding "codeword" (as encoded data 104) that can be transmitted or stored via the channel 130. The decoder 120 is configured to receive the encoded data 104 via the channel 130 and decode the encoded data 104 into output data 106. For example, the decoder 120 may decompress or otherwise reverse the compression performed by the encoder 110 so that the output data 106 is substantially similar (if not identical) to the original input data 102.

[0034] Data compression techniques can generally be categorized as "lossy" or "lossless". Lossy data compression may result in some information loss between the encoding step and the decoding step. As a result, the output data 106 may be different from the input data 102. Example lossy compression techniques include transform coding (such as by applying a spatial frequency transform) and quantization (such as by applying a quantization matrix), among other examples. In contrast, lossless data compression does not result in any information loss between the encoding step and the decoding step as long as the channel 130 does not introduce errors into the encoded data 104. As a result, the output data 106 is identical to the input data 102. Example lossless compression techniques include entropy coding (such as arithmetic coding, Huffman decoding, or Golomb decoding) and run length encoding (RLE), among other examples.

[0035] In some implementations, the encoder 110 and the decoder 120 can utilize machine learning to achieve even greater data compression. As described above, machine learning is a technique for improving the ability of a computer system or application to perform a specific task. During the training phase, a machine learning system is provided with multiple "answers" and a large amount of raw input data. The machine learning system analyzes the input data to learn a set of rules (also referred to as a "machine learning model") that can be used to map the input data to the answer. During the inference phase, the machine learning system uses the trained machine learning model to infer the answer from the new input data.

[0036] Deep learning is a specific form of machine learning in which the inference phase and the training phase are performed on multiple layers. Due to the way information is processed (similar to biological nervous systems), deep learning architectures are often referred to as "artificial neural networks." For example, each layer of an artificial neural network can be composed of one or more "neurons." The neurons of each layer can perform different transformations on the output data from the previous layer so that the final output of the neural network leads to the desired inference. The set of transformations associated with the various layers of the network is called a "neural network model."

[0037] An autoencoder is a type of artificial neural network that is well suited for image compression. For example, an autoencoder can be trained to reproduce at its output the same image received at its input. A bottleneck is imposed between the input and output layers of the neural network, which reduces the dimensionality of the output at the intermediate layers. As a result of this bottleneck, the autoencoder is forced to learn a compressed representation (or "latent attribute") of the input image. Thus, an autoencoder architecture generally includes an encoder trained to convert a frame of image data into a lower dimensional tensor or vector of latent attributes and a decoder trained to reconstruct the original frame of image data from the tensor or vector of latent attributes.

[0038] Figure 22 shows a block diagram of an example video transmission system 200 according to some implementations. The system 200 includes a video encoder 210 and a video decoder 220. In some implementations, the video encoder 210 and the video decoder 220 may be respectively Figure 1 Thus, the video encoder 210 may be transmitted via a channel such as Figure 1 Channel 130) is communicatively coupled to the video decoder 220.

[0039] The video encoder 210 is configured to encode the raw image data 201 into the encoded image data 205 for transmission to the decoder 220. For example, a frame of the raw image data 201 may include an array of pixel values ​​(or multiple arrays of pixel values ​​associated with different color channels) representing a digital image or frame of video captured or collected by an image source (such as a camera or other image output device). In some aspects, the video encoder 210 may transmit a sequence of frames of the encoded image data 205, each representing a corresponding image or frame of the digital video.

[0040] The video decoder 220 is configured to decode the encoded image data 205 into reconstructed image data 208 for display on a display device, such as a television, a computer monitor, a smart phone, or any other device including an electronic display. More specifically, the video decoder 220 may reverse the encoding performed by the video encoder 210 so that the reconstructed image data 208 is substantially similar, if not identical, to the original image data 201. In some aspects, the video decoder 220 may display or render a sequence of frames of the reconstructed image data 208 on the display device.

[0041] The video encoder 210 is shown to include a latent space (LS) conversion component 212, an inter-frame decoding component 214, and a vector encoding component 216. The LS conversion component 212 is configured to convert the original image data 201 from the pixel domain to the latent domain. In some implementations, the LS conversion component 212 can transform the original image data 201 into a tensor 202 of latent attributes based on a machine learning model. For example, the LS conversion component 212 can perform encoding operations associated with an autoencoder trained to reduce the dimension of the original image data 201. As a result, the tensor 202 can be a compressed representation of the original image data 201.

[0042] The inter-coding component 214 is configured to generate a latent vector 203 based on the tensor 202 and the potential information 204 (also referred to as "prior potential information") associated with the image or video frame previously encoded by the encoder 210. In some implementations, the prior potential information 204 may include the potential attributes of the previous image or video frame. Thus, the inter-coding operation can be performed entirely in the latent domain. For example, the inter-coding component 214 can combine the tensor 202 with the prior potential information 204 so that the latent vector 203 includes at least some potential attributes from each of the previous image and the current image. In some implementations, the resulting latent vector 203 can be smaller or more compressible than the tensor 202.

[0043] The vector encoding component 216 is configured to encode the latent vector 203 into one or more codewords of the encoded image data 205 according to one or more coding schemes. In some implementations, the vector encoding component 216 can perform entropy encoding (or other lossless or lossy compression) on the latent vector 203 to further reduce the amount of the encoded image data 205 transmitted to the video decoder 220. In some aspects, the vector encoding component 216 can also recover the a priori latent information 204 for encoding the next image or video frame received by the video encoder 210. In some implementations, the vector encoding component 216 can perform an inverse inter-frame decoding operation on the latent vector 203 to generate the a priori latent information 204.

[0044] The video decoder 220 is shown to include a vector decoding component 222, an inverse inter-coding component 224, and a pixel space (PS) conversion component 226. The vector decoding component 222 is configured to recover the latent vector 206 from the encoded image data 205. In some implementations, the vector decoding component 222 can reverse the encoding performed by the vector encoding component 216. For example, the vector decoding component 222 can decode the encoded image data 205 into the latent vector 206 according to the same decoding scheme implemented by the vector encoding component 216. As a result, the latent vector 206 can be the same as the latent vector 203.

[0045] The inverse inter-coding component 224 is configured to recover a tensor 207 of potential attributes based on the potential vector 206 and a tensor 207' previously recovered by the inverse inter-coding component 224 (also referred to as a "previous tensor"). For example, the previous tensor 207' may include potential attributes of a previous image or video frame decoded by the video decoder 220. In some implementations, the previous tensor 207' may be the same as the a priori potential information 204 used by the video encoder 210 to encode the current image or video frame. The inverse inter-coding component 224 may combine the potential vector 206 with the previous tensor 207' so that the resulting tensor 207 includes at least some potential attributes from each of the previous image and the current image. In some implementations, the resulting tensor 207 may be stored in the buffer 225 as the previous tensor 207' for decoding the next image or video frame.

[0046] The PS conversion component 226 is configured to convert the tensor 207 from the potential domain back to the pixel domain. In some implementations, the PS conversion component 226 can transform the tensor 207 into the reconstructed image data 208 based on the machine learning model. For example, the PS conversion component 226 can perform a decoding operation associated with an autoencoder trained to reconstruct the original image data 201 from the tensor 207. As a result, the reconstructed image data 208 can be substantially similar (if not identical) to the original image data 201. In some implementations, the reconstructed image data 208 can be further displayed or rendered as a digital image on an electronic display (not shown for simplicity).

[0047] In some aspects, the LS conversion component 212, the inter-frame decoding component 214, the inverse inter-frame decoding component 224, and the PS conversion component 226 can be collectively trained as an autoencoder that reproduces the same image received at its input at its output. However, unlike existing autoencoders, the autoencoder of the present implementation is trained to reproduce each output image based on the potential attributes of the current input image and the potential attributes of the previous input image. Due to the inter-frame decoding operations performed in the potential domain (such as by the inter-frame decoding component 214 and the inverse inter-frame decoding component 224), the output layer of the LS conversion component 212 is different from the input layer of the PS conversion component 224. Accordingly, the tensor 207 can be different from the tensor 202 of the same image.

[0048] Figure 3 A block diagram of an example video encoding system 300 is shown according to some implementations. In some implementations, the video encoding system 300 may be Figure 2More specifically, the video encoding system 300 is configured to encode an input image 301 into a vector 304 of latent attributes (also referred to as a "latent vector") based at least in part on latent attributes of input images previously encoded by the video encoding system 300 (also referred to as "previous input images"). Figure 2 , input image 301 may be an example of original image data 201 , and latent vector 304 may be an example of latent vector 203 .

[0049] The video encoding system 300 includes an analysis transform 310, an inter-coding transform 320, and an inverse inter-coding transform 330. The analysis transform 310 transforms an input image 301 into a tensor 302 (also referred to as an "encoding tensor") of potential attributes associated with the input image 301 based on a neural network model. In other words, the analysis transform 310 converts the input image 301 from a pixel domain to a potential domain. Figure 2 , the analysis transformation 310 can be an example of the LS conversion component 212.

[0050] In some implementations, the analysis transformation 310 may include multiple layers of a convolutional neural network (CNN) trained to reduce the dimensionality of the input image 301. For example, depending on the image format, the input image 301 may be composed of a plurality of layers having a certain height (h i )、Width(w i ) and depth (d i ). As a result of processing the input image 301 through various convolutional layers, pooling layers, or any combination thereof that reduce the dimensionality of the input image 301, the CNN produces an encoded tensor 302 at its output. Thus, the encoded tensor 302 at the output of the CNN is a compressed representation of the input image 301. In some implementations, the encoded tensor 302 may be a 3D array of pixel values ​​having a height h, a width w, and a depth d a The three-dimensional tensor (h,w,d a ), where h <h i , w <w i , and d a >d i .

[0051] The video encoding system 300 combines the encoded tensor 302 and a tensor 306 of latent attributes associated with a previous input image (also referred to as a "decoded tensor") to produce a reconstructed tensor 303. Figure 2 , the decoded tensor 306 may be an example of the prior latent information 204. In some implementations, the decoded tensor 306 may be a three-dimensional tensor (h, w, d s ), which has the same height h and width w as the encoded tensor 302, where the depth ds With the depth d of the encoded tensor 302 a The same or different, and the reconstructed tensor 303 can be a concatenation of the encoding tensor 302 and the decoding tensor 306. Thus, the reconstructed tensor 303 can also be a three-dimensional tensor (h, w, d r1 ), which has the same height h and width w as each of tensors 302 and 306 and a depth d equal to the combined depth of tensors 302 and 306 r1 (d r1 =d a +d s ).

[0052] The inter-coding transform 320 transforms the reconstructed tensor 303 into the latent vector 304 by reducing the dimensionality of the reconstructed tensor 303. In some aspects, the inter-coding transform 320 may be a transform that reduces the depth d of the reconstructed tensor 303. r1 , while maintaining its height h and width w. As a result, the latent vector 304 may have the same height h and width w as the reconstructed tensor 303 and a depth d smaller than the reconstructed tensor 303. r1 The depth d l The three-dimensional tensor (h,w,d l ). In some implementations, the inter-coding transform 320 may include one or more layers of a CNN trained to transform the reconstructed tensor 303 into a latent vector 304 that is smaller or more compressible than the encoded tensor 302. Figure 2 As described, the latent vector 304 may be further entropy encoded and transmitted to a video decoder (not shown for simplicity) via a communication channel.

[0053] In some aspects, the video encoding system 300 can use the latent vector 304 in a recursive feedback loop to encode the next input image in the image sequence. For example, the video encoding system 300 can combine the latent vector 304 with the decoded tensor 306 to produce another reconstructed tensor 305. In some implementations, the reconstructed tensor 305 can be a concatenation of the latent vector 304 and the decoded tensor 306. As such, the reconstructed tensor 305 can also be a tensor having the same height h and width w as the latent vector 304 and the decoded tensor 306 and a depth d equal to the combined depth of the latent vector 304 and the decoded tensor 306. r2 (d r2 =d l +d s ) of a three-dimensional tensor (h,w,d r2 ).

[0054] The inverse inter-coding transform 330 transforms the reconstructed tensor 305 into a new decoded tensor 306' by reducing the dimensionality of the reconstructed tensor 305, the new decoded tensor 306' including the latent attributes of the current input image 301. In some aspects, the inverse inter-coding transform 330 may be a transform that reduces the depth d of the reconstructed tensor 305. r2 Any linear or nonlinear transformation while maintaining its height h and width w. As a result, the depth d of the decoded tensor 306' s is less than the depth d of the reconstructed tensor 305 r2 In some implementations, the inverse inter-coding transform 330 may include one or more layers of a CNN trained to transform the reconstructed tensor 305 into a decoded tensor 306' that can be converted back to the pixel domain (as the original input image 301). The new decoded tensor 306' may be stored in the tensor buffer 340 as the decoded tensor 306 for encoding the next input image.

[0055] Figure 4 4 shows an example inter-frame decoding operation 400 that can be performed in a latent domain according to some implementations. The example operation 400 can be performed by a video encoder such as Figure 2 The video encoder 210 or Figure 3 The video encoding system 300 of the embodiment of the present invention is executed to generate a latent vector 408 based on the encoded tensor 402 and the decoded tensor 404. In some implementations, the encoded tensor 402, the decoded tensor 404, and the latent vector 408 may be respectively Figure 3 An example of an encoded tensor 302, a decoded tensor 306, and a latent vector 304 of FIG.

[0056] exist Figure 4 In the example of FIG. 4 , the encoded tensor 402 is depicted as having a height h=4, a width w=4, and a depth d a =6 three-dimensional tensor (h,w,d a ), and the decoded tensor 404 is depicted as having a height h=4, a width w=4, and a depth d s =6 three-dimensional tensor (h,w,d s ). For simplicity, the depth d of the encoding tensor 402 is a is shown as the depth d of the decoded tensor 404 s Same (d a =d s ). However, in actual implementations, tensor 402 and tensor 404 may have different depths. Encoded tensor 402 is combined with decoded tensor 404 to produce reconstructed tensor 406. In some implementations, reconstructed tensor 406 may be Figure 3 An example of a reconstructed tensor 303 of . Figure 4As shown in FIG. , tensors 402 and 404 are separated by a depth d a and depth d s The reconstructed tensor 406 is a tensor with a height of h=4, a width of w=4, and a depth of d r1 =12(d r1 =d a +d s ) of a three-dimensional tensor (h,w,d r1 ).

[0057] The inter-coding transform 410 converts the reconstructed tensor 406 into a latent vector 408. In some implementations, the inter-coding transform 410 may be Figure 3 An example of an inter-coding transform 320 of FIG. For example, the inter-coding transform 410 may include one or more layers of a CNN trained to reduce the dimensionality of the reconstructed tensor 406 to produce a latent vector 408 that is smaller or more compressible than the encoded tensor 402. Figure 4 In the example of FIG. 4 , the latent vector 408 is depicted as having a height h=4, a width w=4, and a depth d l =4 three-dimensional tensor (h,w,d l ). However, in actual implementations, the latent vector 408 may have any depth d less than the depth of the reconstructed tensor 406. l (d l <d r1 ). In some implementations, the latent vector 408 can have the same depth as the encoded tensor 402 or the decoded tensor 404.

[0058] Figure 5 An example inverse inter-frame decoding operation 500 that can be performed in a latent domain according to some implementations is shown. The example operation 500 may be performed by a video encoder such as Figure 2 The video encoder 210 or Figure 3 ) or a video decoder such as Figure 2 The video decoder 220) executes based on Figure 4 The latent vector 408 and the decoded tensor 404 are used to recover the decoded tensor 504. In some implementations, the decoded tensor 504 may be Figure 3 An example of a decoded tensor 306 ′.

[0059] As reference Figure 4 As described, the decoded tensor 404 has a height h=4, a width w=4, and a depth d s =6 three-dimensional tensor (h,w,d s ), and the latent vector 408 has a height h=4, a width w=4, and a depth dl =4 three-dimensional tensor (h,w,d l ). The latent vector 408 is combined with the decoded tensor 404 to produce a reconstructed tensor 502. In some implementations, the reconstructed tensor 502 may be Figure 3 An example of a reconstructed tensor 305 of . Figure 5 As shown in FIG. , the latent vector 408 and the decoded tensor 404 are transformed into a vector at a depth d l and depth d s The reconstructed tensor 502 is a tensor with a height of h=4, a width of w=4, and a depth of d r2 =10(d r2 =d l +d s ) of a three-dimensional tensor (h,w,d r2 ).

[0060] The inverse inter-coding transform 510 converts the reconstructed tensor 502 into a decoded tensor 504. In some implementations, the inverse inter-coding transform 510 may be Figure 3 For example, the inverse inter-coding transform 510 may include one or more layers of a CNN trained to reduce the dimensionality of the reconstructed tensor 502 to produce a decoded tensor 504 that can be converted back to the pixel domain (as the original input image). Figure 5 As shown in , decoded tensor 504 has the same size as decoded tensor 404. In other words, decoded tensor 504 is a tensor with height h=4, width w=4, and depth d s =6 three-dimensional tensor (h,w,d s ).

[0061] Figure 6 A block diagram of an example video decoding system 600 is shown according to some implementations. In some implementations, the video decoding system 600 may be Figure 2 More specifically, the video decoding system 600 is configured to decode the latent vector 602 into an output image 608 based at least in part on latent properties of an output image previously decoded by the video decoding system 600 (also referred to as a “previous output image”). Figure 2 , latent vector 602 may be an example of latent vector 206 , and output image 608 may be an example of reconstructed image data 208 .

[0062] In some implementations, latent vector 602 may be a vector having height h, width w, and depth d l The three-dimensional tensor (h,w,d l). The video decoding system 600 combines the latent vector 602 with a tensor 606 of latent attributes associated with a previous output image (also referred to as a "decoded tensor") to produce a reconstructed tensor 604. Figure 2 , the decoded tensor 606 may be an example of the tensor 207'. In some implementations, the decoded tensor 606 may be a three-dimensional tensor (h, w, d s ), which has the same height h and width w as the latent vector 602, where the depth d s and the depth d of the latent vector 602 l The same or different, and the reconstructed tensor 604 can be a cascade of the latent vector 602 and the decoded tensor 606 (such as reference Figure 5 Thus, the reconstructed tensor 604 may also be of the same height h and width w as the latent vector 602 and the decoded tensor 606, and a depth d equal to the combined depth of the latent vector 602 and the decoded tensor 606. r (d r =d l +d s ) of a three-dimensional tensor (h,w,d r ).

[0063] The video decoding system 600 includes an inverse inter-coding transform 610 and a composite transform 620. The inverse inter-coding transform 610 transforms the reconstructed tensor 604 into a new decoded tensor 606' by reducing the dimensionality of the reconstructed tensor 604. The new decoded tensor 606' includes potential attributes of the current output image 608 (such as reference Figure 5 In some implementations, the inverse inter-frame decoding transform 610 can be Figure 3 Thus, the inverse inter-coding transform 610 may be the same as the inverse inter-coding transform 330 of the reconstructed tensor 604. r Any linear or nonlinear transformation while maintaining its height h and width w. As a result, the depth d of the decoded tensor 606' s is less than the depth d of the reconstructed tensor 604 r The new decoded tensor 606' may be stored as decoded tensor 606 in tensor buffer 630 for use in decoding the next latent vector.

[0064] The synthesis transformation 620 transforms the decoded tensor 606' into the output image 608 based on the neural network model. In other words, the synthesis transformation 620 converts the decoded tensor 606' from the potential domain to the pixel domain. Figure 2, the synthesis transform 620 can be an example of the PS conversion component 226. In some implementations, the synthesis transform 620 may include multiple layers of a CNN trained to upsample the decoded tensor 606'. For example, as a result of processing the decoded tensor 606' through various convolutional layers, pooling layers, or any combination thereof that increase the dimension of the decoded tensor 606', the CNN may reconstruct an output image 608. Thus, the output image 608 is a decompressed representation of the decoded tensor 606'. In some implementations, the output image 608 may be composed of a plurality of pixels having a particular height (h o )、Width(w o ) and depth (d o ) is a three-dimensional array representation of the pixel values, where h o >h,w o >w, and d o <d s .

[0065] Figure 7 FIG. 7 is a block diagram of a video encoder 700 according to some implementations. In some implementations, the video encoder 700 may be Figure 2 The video encoder 210 or Figure 3 More specifically, the video encoder 700 may be configured to encode a sequence of images or video frames for transmission over a communication channel.

[0066] In some implementations, the video encoder 700 may include a data interface 710, a processing system 720, and a memory 730. The data interface 710 is configured to receive image data from an image source and output a vector of potential attributes associated with the image data through a communication channel. In some aspects, the data interface 710 may include an image source interface (I / F) 712 for communicating with an image source and a channel interface 714 for communicating via a communication channel. In some implementations, the image source interface 712 may receive an image sequence from an image source, and the channel interface 714 may transmit the potential vector to a video decoder through a communication channel.

[0067] The memory 730 may include an image data buffer 732 to store any image data or intermediate data associated with the encoding operation. The memory 730 may also include a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, hard drive, etc.) that may store at least the following software (SW) modules: A latent space (LS) transformation SW module 734 for encoding a first image in the image sequence into a latent attribute tensor associated with the first image based on a machine learning model; as well as • An inter-coding SW module 736 to perform an inter-coding operation to generate a latent attribute vector based on a latent attribute tensor associated with a first image and a latent attribute tensor associated with a second image received before the first image in a sequence of images. Each software module includes instructions that, when executed by the processing system 720 , cause the video encoder 700 to perform a corresponding function.

[0068] The processing system 720 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the encoder 700, such as in the memory 730. For example, the processing system 720 may execute the LS conversion SW module 734 to encode a first image in the image sequence into a latent attribute tensor associated with the first image based on the machine learning model. The processing system 720 may further execute the inter-frame decoding SW module 736 to perform an inter-frame decoding operation to generate a latent attribute vector based on the latent attribute tensor associated with the first image and the latent attribute tensor associated with a second image received before the first image in the image sequence.

[0069] Figure 8 An illustrative flow chart depicting example operations 800 for video decoding according to some implementations is shown. In some implementations, the example operations 800 may be performed by a video encoder such as Figure 2 The video encoder 210, Figure 3 The video encoding system 300 or Figure 7 The video encoder 700) is executed.

[0070] A video encoder receives a sequence of images (810). The video encoder encodes a first image in the sequence of images into a first latent attribute tensor associated with the first image based on a machine learning model (820). The video encoder performs a first inter-frame decoding operation (830) to generate a first latent attribute vector based on the first tensor and a latent attribute tensor associated with a second image received before the first image in the sequence of images. In some implementations, the execution of the first inter-frame decoding operation may include generating a first reconstructed tensor based on the first tensor and the latent attribute tensor associated with the second image; and encoding the first reconstructed tensor into a first vector based on an inter-frame decoding transform that reduces the dimension of the first reconstructed tensor. The video encoder further transmits the first vector to a video decoder (840) via a communication channel.

[0071] In some implementations, the first tensor, the potential attribute tensor associated with the second image, the first reconstructed tensor, and the first vector may each have a corresponding height, width, and depth. In some implementations, the first vector may have the same height and the same width as each of the first reconstructed tensor, the first tensor, and the potential attribute tensor associated with the second image. In some implementations, the depth of the first vector may be less than the depth of the first reconstructed tensor. In some implementations, the generation of the first reconstructed tensor may include concatenating the first tensor and the potential attribute tensor associated with the second image so that the depth of the first reconstructed tensor is equal to the sum of the depths of the first tensor and the potential attribute tensor associated with the second image.

[0072] In some aspects, the video encoder may further perform an inverse inter-frame decoding operation to generate a second latent attribute tensor associated with the first image based on the first vector and the latent attribute tensor associated with the second image; encode a third image in the image sequence as a latent attribute tensor associated with the third image based on the machine learning model; perform a second inter-frame decoding operation to generate a second latent attribute vector based on the second tensor and the latent attribute tensor associated with the third image; and transmit the second vector to the video decoder through a communication channel. In some implementations, the execution of the second inter-frame decoding operation may include: generating a second reconstructed tensor based on the second tensor and the latent attribute tensor associated with the third image; and encoding the second reconstructed tensor into a second vector based on an inter-frame decoding transform.

[0073] In some aspects, the performing of the inverse inter-coding operation may include: generating a second reconstructed tensor based on the first vector and a latent attribute tensor associated with the second image; and decoding the second reconstructed tensor into a second tensor based on an inverse inter-coding transform that reduces the dimension of the second reconstructed tensor. In some implementations, the generating of the second reconstructed tensor may include concatenating the first vector and the latent attribute tensor associated with the second image so that the depth of the second reconstructed tensor is equal to the sum of the depths of the first vector and the latent attribute tensor associated with the second image.

[0074] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0075] In addition, those skilled in the art will appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician can implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0076] The methods, sequences or algorithms described in conjunction with the aspects disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or in any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated with the processor.

[0077] In the foregoing description, embodiments have been described with reference to specific examples thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader scope of the present disclosure as set forth in the appended claims. Accordingly, the description and drawings are to be viewed in an illustrative rather than a restrictive sense.

Claims

1. A video decoding method, comprising: receiving an image sequence; encoding a first image in the sequence of images into a first latent attribute tensor associated with the first image based on a machine learning model; performing a first inter-coding operation to generate a first latent attribute vector based on the first tensor and a latent attribute tensor associated with a second image in the sequence of images received before the first image; The first vector is transmitted to a video decoder via a communication channel.

2. The method according to claim 1, wherein: The execution of the first inter-frame decoding operation includes: generating a first reconstructed tensor based on the first tensor and the latent attribute tensor associated with the second image; and The first reconstructed tensor is encoded into the first vector based on an inter-coding transform that reduces the dimension of the first reconstructed tensor.

3. The method according to claim 2, wherein: The first tensor, the latent attribute tensor associated with the second image, the first reconstructed tensor, and the first vector each have a corresponding height, width, and depth.

4. The method according to claim 3, wherein: The first vector has the same height and the same width as each of the first reconstructed tensor, the first tensor, and the latent attribute tensor associated with the second image.

5. The method according to claim 4, wherein: The depth of the first vector is less than the depth of the first reconstructed tensor.

6. The method according to claim 4, wherein: The generating of the first reconstructed tensor comprises: The first tensor and the latent attribute tensor associated with the second image are concatenated such that a depth of the first reconstructed tensor is equal to a sum of the depths of the first tensor and the latent attribute tensor associated with the second image.

7. The method according to claim 4, further comprising: performing an inverse inter-coding operation to generate a second latent attribute tensor associated with the first image based on the first vector and the latent attribute tensor associated with the second image; encoding a third image in the sequence of images into a latent attribute tensor associated with the third image based on the machine learning model; performing a second inter-coding operation to generate a second latent attribute vector based on the second tensor and the latent attribute tensor associated with the third image; The second vector is transmitted to the video decoder through the communication channel.

8. The method according to claim 7, wherein: The execution of the inverse inter-frame decoding operation includes: generating a second reconstructed tensor based on the first vector and the latent attribute tensor associated with the second image; and The second reconstructed tensor is decoded into the second tensor based on an inverse inter-coding transform that reduces the dimension of the second reconstructed tensor.

9. The method according to claim 8, wherein: The generating of the second reconstructed tensor comprises: The first vector and the latent attribute tensor associated with the second image are concatenated such that a depth of the second reconstructed tensor is equal to a sum of the first vector and a depth of the latent attribute tensor associated with the second image.

10. The method according to claim 7, wherein: The execution of the second inter-frame decoding operation includes: generating a second reconstructed tensor based on the second tensor and the latent attribute tensor associated with the third image; and The second reconstructed tensor is encoded into the second vector based on the inter-coding transform.

11. A video encoder, comprising: Processing systems; as well as a memory storing instructions that, when executed by the processing system, cause the video encoder to: receiving an image sequence; encoding a first image in the sequence of images into a first latent attribute tensor associated with the first image based on a machine learning model; performing a first inter-coding operation to generate a first latent attribute vector based on the first tensor and a latent attribute tensor associated with a second image in the sequence of images received before the first image; The first vector is transmitted to a video decoder via a communication channel.

12. The video encoder according to claim 11, wherein: The execution of the first inter-frame decoding operation includes: generating a first reconstructed tensor based on the first tensor and the latent attribute tensor associated with the second image; and The first reconstructed tensor is encoded into the first vector based on an inter-coding transform that reduces the dimension of the first reconstructed tensor.

13. The video encoder according to claim 12, wherein: The first tensor, the latent attribute tensor associated with the second image, the first reconstructed tensor, and the first vector each have a corresponding height, width, and depth.

14. The video encoder according to claim 13, wherein: The first vector has the same height and the same width as each of the first reconstructed tensor, the first tensor, and the latent attribute tensor associated with the second image.

15. The video encoder according to claim 14, wherein: The depth of the first vector is less than the depth of the first reconstructed tensor.

16. The video encoder of claim 14, wherein: The generating of the first reconstructed tensor comprises: The first tensor and the latent attribute tensor associated with the second image are concatenated such that a depth of the first reconstructed tensor is equal to a sum of the depths of the first tensor and the latent attribute tensor associated with the second image.

17. The video encoder of claim 14, further comprising: performing an inverse inter-coding operation to generate a second latent attribute tensor associated with the first image based on the first vector and the latent attribute tensor associated with the second image; encoding a third image in the sequence of images into a latent attribute tensor associated with the third image based on the machine learning model; performing a second inter-coding operation to generate a second latent attribute vector based on the second tensor and the latent attribute tensor associated with the third image; The second vector is transmitted to the video decoder through the communication channel.

18. The video encoder according to claim 17, wherein: The execution of the inverse inter-frame decoding operation includes: generating a second reconstructed tensor based on the first vector and the latent attribute tensor associated with the second image; and The second reconstructed tensor is decoded into the second tensor based on an inverse inter-coding transform that reduces the dimension of the second reconstructed tensor.

19. The video encoder of claim 18, wherein: The generating of the second reconstructed tensor comprises: The first vector and the latent attribute tensor associated with the second image are concatenated such that a depth of the second reconstructed tensor is equal to a sum of the first vector and a depth of the latent attribute tensor associated with the second image.

20. The video encoder of claim 17, wherein: The execution of the second inter-frame decoding operation includes: generating a second reconstructed tensor based on the second tensor and the latent attribute tensor associated with the third image; and The second reconstructed tensor is encoded into the second vector based on the inter-coding transform.

Images