Method for multi-granularity temporal trajectory characterization for generative video compression

Abstract

A video decoding method includes decoding an image bitstream, the image bitstream being associated with a video sequence, where the decoding of the image bitstream reconstructs a key reference frame; decomposing the reconstructed key reference frame into key frame potential features and a first set of compact motion vectors associated with the reconstructed key reference frame; decoding a feature bitstream, the feature bitstream associated with the video sequence to obtain a second set of compact motion vectors associated with an inter frame; transforming the key frame potential features into a first fine-grained motion field for the reconstructed key reference frame and a second fine-grained motion field for the inter frame based on the first set of compact motion vectors and the second set of compact motion vectors; predicting dense motion based on the first fine-grained motion field and the second fine-grained motion field; and generating the inter frame based on the dense motion and the reconstructed key reference frame.

Description

Cross Reference to Related Applications

[0001] This disclosure claims priority benefit of U.S. provisional application 63 / 705,035, filed October 9, 2024, which is incorporated herein by reference in its entirety. This disclosure also claims priority benefit of U.S. patent application 19 / 319,319, filed September 4, 2025. TECHNICAL FIELD

[0002] The present disclosure relates generally to video processing, and more particularly, to a method for generative video compression based on multi-granularity temporal trajectory representation. BACKGROUND

[0003] A video is composed of a set of still images (or “frames”) that capture visual information. To reduce storage memory and transmission bandwidth, a video can be compressed before being stored or transmitted, and decompressed before being displayed. The compression process is commonly referred to as encoding, while the decompression process is commonly referred to as decoding. There are many video coding formats that use standardized video coding techniques, the most common of which are based on prediction, transform, quantization, entropy coding, and in-loop filtering. Standardization organizations develop video coding standards that specify certain video coding formats, such as the High Efficiency Video Coding (HEVC / H.265) standard, the Versatile Video Coding (VVC / H.266) standard, and the AVS standard. As more and more advanced video coding techniques are incorporated into video standards, the coding efficiency of new video coding standards is also increasing. SUMMARY

[0004] Embodiments of the present disclosure provide a video decoding method. The video decoding method comprises: decoding an image bitstream, the image bitstream being associated with a video sequence, wherein the decoding of the image bitstream reconstructs a key reference frame; decomposing the reconstructed key reference frame into key frame latent features associated with the reconstructed key reference frame and a first set of compact motion vectors; decoding a feature bitstream, the feature bitstream being associated with the video sequence, to obtain a second set of compact motion vectors associated with an inter frame; transforming the key frame latent features into a first fine-grained motion field for the reconstructed key reference frame and a second fine-grained motion field for the inter frame based on the first set of compact motion vectors and the second set of compact motion vectors; predicting dense motion based on the first fine-grained motion field and the second fine-grained motion field; and generating the inter frame based on the dense motion and the reconstructed key reference frame.

[0005] Embodiments of the present disclosure provide an encoding method. The encoding method comprises: encoding an image bitstream, the image bitstream comprising encoded information of key reference frames of a video sequence, wherein the encoded information of the image bitstream is decomposed into key frame latent features associated with reconstructed key reference frames and a first set of compact motion vectors; and encoding a feature bitstream, the feature bitstream comprising encoded information of inter frames of the video sequence, wherein the encoded information of the feature bitstream comprises a second set of compact motion vectors associated with the inter frames; wherein, based on the first set of compact motion vectors and the second set of compact motion vectors, a first fine-grained motion field for the reconstructed key reference frames and a second fine-grained motion field for the inter frames are generated by transforming the key frame latent features; wherein the first fine-grained motion field and the second fine-grained motion field are used for predicting dense motion; and wherein the dense motion is used to generate the inter frames.

[0006] Embodiments of the present disclosure provide a method for transmitting a bitstream. The method comprises: receiving a video sequence; encoding the video sequence by: encoding an image bitstream, the image bitstream comprising encoded information of key reference frames of a video sequence, wherein the encoded information of the image bitstream is decomposed into key frame latent features associated with reconstructed key reference frames and a first set of compact motion vectors; and encoding a feature bitstream, the feature bitstream comprising encoded information of inter frames of the video sequence, wherein the encoded information of the feature bitstream comprises a second set of compact motion vectors associated with the inter frames; wherein, based on the first set of compact motion vectors and the second set of compact motion vectors, a first fine-grained motion field for the reconstructed key reference frames and a second fine-grained motion field for the inter frames are generated by transforming the key frame latent features; wherein the first fine-grained motion field and the second fine-grained motion field are used for predicting dense motion; and wherein the dense motion is used to generate the inter frames; and transmitting the image bitstream and the feature bitstream generated based on the encoding. BRIEF DESCRIPTION OF DRAWINGS

[0007] Embodiments of the present disclosure and various aspects thereof are illustrated in the following detailed description and in the accompanying drawings. The various features shown in the figures are not drawn to scale.

[0008] Figure 1 is a schematic diagram illustrating an exemplary system for encoding image data, in accordance with some embodiments of the present disclosure.

[0009] Figure 2 is a schematic diagram illustrating an architecture of a block-based video compression framework, in accordance with some embodiments of the present disclosure.

[0010] Figure 3 is a schematic diagram illustrating the structure of an example video sequence according to some embodiments of the disclosure.

[0011] Figure 4A is a schematic diagram illustrating an example block-based encoding process according to some embodiments of the disclosure.

[0012] Figure 4B is a schematic diagram illustrating another example block-based encoding process according to some embodiments of the disclosure.

[0013] Figure 5A is a schematic diagram illustrating an example block-based decoding process according to some embodiments of the disclosure.

[0014] Figure 5B is a schematic diagram illustrating another example block-based decoding process according to some embodiments of the disclosure.

[0015] Figure 6 is a schematic diagram illustrating an example architecture of an end-to-end deep learning based video compression framework according to some embodiments of the disclosure.

[0016] Figure 7 is a schematic diagram illustrating an example architecture of a deep learning based video generative compression framework according to some embodiments of the disclosure.

[0017] Figure 8 is a schematic diagram illustrating an example encoder-decoder encoding framework for talking face videos with compact feature size of 1x4x4 according to some embodiments of the disclosure.

[0018] Figure 9 is a schematic diagram illustrating a general flowchart of a generative face video compression (GFVC) system according to some embodiments of the disclosure.

[0019] Figure 10 illustrates an example structure of multi-granularity temporal trajectory factorization according to some embodiments of the disclosure.

[0020] Figure 11 is a flowchart of an example method for multi-granularity temporal trajectory factorization (MTTF) according to some embodiments of the disclosure.

[0021] Figure 12 is a schematic diagram illustrating a generative video encoding framework according to some disclosed embodiments.

[0022] Figure 13 is a flowchart of an example video decoding method using multi-granularity temporal trajectory factorization (MTTF) according to some embodiments of the disclosure.

[0023] Figure 14 is a flowchart of an exemplary video encoding method using multi-granular temporal trajectory factorization (MTTF) according to some embodiments of the present disclosure.

[0024] Figure 15 is a block diagram of an exemplary apparatus for encoding image data according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] Reference will now be made in detail embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers represent the same or similar elements between the several figures. The implementations set forth in the following description of exemplary embodiments do not represent all of the implementations consistent with the application. Instead, they are merely examples of apparatuses and methods consistent with aspects related to the described application as recited in the attached claims. Specific aspects of the present disclosure are described in further detail below. To the extent not inconsistent with the text of this specification, the terminology used herein merges with that used in the documents incorporated by reference.

[0026] The Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (ITU-T VCEG) and the ISO / IEC Moving Picture Experts Group (ISO / IEC MPEG) is currently developing a Versatile Video Coding (VVC / H.266) standard. The VVC standard aims to double the compression efficiency of its predecessor, the High Efficiency Video Coding (HEVC / H.265) standard. In other words, the goal of VVC is to achieve the same subjective quality using half the bandwidth of HEVC / H.265.

[0027] To achieve the same subjective quality using half the bandwidth of HEVC / H.265, the JVET has been developing techniques beyond HEVC using the Joint Exploration Model (JEM) reference software. As coding techniques are incorporated into JEM, JEM achieves much higher coding performance than HEVC.

[0028] The VVC standard has recently been finalized and continues to incorporate more coding techniques to provide better compression performance. VVC follows the hybrid video coding system employed by modern video compression standards such as HEVC, H.264 / AVC, MPEG2, H.263, etc.

[0029] A video is a set of still images (or "frames") arranged in time order to store visual information. A video capture device (e.g., a camera) can be used to capture and store those images in time order, and a video playback device (e.g., a television, a computer, a smartphone, a tablet computer, a video player, or any end-user terminal with display function) can be used to display those images in time order. Moreover, in some applications, a video capture device can transmit the captured video in real time to the video playback device (e.g., a computer with a display), for example, for video surveillance, conference, or live broadcast.

[0030] To reduce the storage space and transmission bandwidth required for these applications, the video can be compressed before storing or transmitting, and decompressed before displaying. The compression and decompression can be implemented by software executed by a processor (e.g., a processor of a general-purpose computer) or special-purpose hardware. A module for compression is often referred to as an "encoder", while a module for decompression is often referred to as a "decoder". The encoder and decoder can be collectively referred to as a "codec". The encoder and decoder can be implemented as any of various suitable hardware, software, or combination thereof. For example, the hardware implementation of the encoder and decoder can include circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, or any combination thereof. The software implementation of the encoder and decoder can include program code, computer-executable instructions, firmware, or any suitable computer-implemented algorithm or process fixed in a computer-readable medium. Video compression and decompression can be implemented by various algorithms or standards, such as MPEG-1, MPEG-2, MPEG-4, H.26x series, etc. In some applications, the codec can decompress the video according to a first encoding standard, and re-compress the decompressed video using a second encoding standard, in which case the codec can be referred to as a "transcoder".

[0031] A video encoding process can identify and retain useful information that can be used to reconstruct the images, and ignore information that is not important for reconstruction. If the ignored, unimportant information cannot be completely reconstructed, such an encoding process can be referred to as "lossy". Otherwise, it can be referred to as "lossless". Most encoding processes are lossy, which is a trade-off for reducing the required storage space and transmission bandwidth.

[0032] Useful information of a certain picture being coded (referred to as the "current picture") includes changes relative to a reference picture (e.g., a previously coded and reconstructed picture). Such changes can include changes in position, changes in intensity, or changes in color of pixels, among which, changes in position are of most interest. Changes in position of a group of pixels representing a certain object can reflect a motion of the object between the reference picture and the current picture.

[0033] A picture that is coded without reference to another picture (i.e., whose reference picture is itself) is referred to as an "I picture." A picture is referred to as a "P picture" if some or all blocks (e.g., blocks generally refer to portions of video pictures) in the picture are predicted using intra-prediction or inter-prediction with one reference picture (e.g., uni-prediction). A picture is referred to as a "B picture" if at least one block in the picture is predicted with two reference pictures (e.g., bi-prediction).

[0034] Figure 1 FIG. 1 is a block diagram illustrating a system 100 for coding image data according to an embodiment of the present disclosure. The image data can include an image (also referred to as a "picture" or a "frame"), a plurality of images, or a video. An image is a still image. The plurality of images can be spatially or temporally related or unrelated. A video is a set of images arranged in a temporal order.

[0035] As Figure 1 shown in FIG. 1, the system 100 includes a source device 120 that provides encoded video data to be decoded at a later time by a destination device 140. Consistent with embodiments disclosed herein, the source device 120 and the destination device 140 each can comprise any of a variety of devices, including desktop computers, notebook (e.g., laptop) computers, servers, tablet computers, set-top boxes, smartphones, vehicles, cameras, image sensors, robots, televisions, wearable devices (e.g., smart-watches or wearable cameras), display devices, digital media players, video gaming consoles, video streaming devices, and the like. The source device 120 and the destination device 140 can be equipped for wireless or wired communication.

[0036] Referring to Figure 1Source device 120 can include an image / video preprocessor 122, an image / video encoder 124, and an output interface 126. Destination device 140 can include an input interface 142, an image / video decoder 144, and a machine vision application 146. Image / video encoder 124 encodes the input bitstream and outputs the encoded bitstream 162 via output interface 126. Encoded bitstream 162 is transmitted through a communication medium 160 and received by input interface 142. Image / video decoder 144 then decodes encoded bitstream 162 to generate decoded data.

[0037] More specifically, source device 120 can also include various devices (not shown) for providing source image data to be processed by image / video encoder 124. Devices for providing source image data can include an image / video capture device such as a camera, an image / video archive or storage device containing previously captured images / videos, or an image / video feed interface for receiving images / videos from an image / video content provider.

[0038] Image / video encoder 124 and image / video decoder 144 each can be implemented as any of a variety of suitable encoder or decoder circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic circuitry, software, hardware, firmware, or any combinations thereof. When the coding is implemented partially in software, image / video encoder 124 or image / video decoder 144 can store

[0039] Image / video encoder 124 and image / video decoder 144 can operate according to any video coding standards, such as advanced video coding (AVC), high efficiency video coding (HEVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), Joint Photographic Experts Group (JPEG), Moving Picture Experts Group (MPEG), etc. Alternatively, image / video encoder 124 and image / video decoder 144 can be custom devices that do not conform to a standard. Although not specifically shown, source device 120 and destination device 140 can include additional features that are common in such devices, such as input / output (I / O) devices, a power supply, and a communication interface. For example, source device 120 can include a communication interface that enables source device 120 to communicate with other devices. Figure 1Although not shown, in some embodiments, image / video encoder 124 and image / video decoder 144 can each be integrated with an audio encoder and decoder and can include appropriate MUX-DEMUX units, or other hardware and software, to handle encoding of both audio and video, including multiplexing the audio and video into a composite data stream or demultiplexing the composite data stream into audio and video.

[0040] Output interface 126 can include any type of medium or device capable of transmitting encoded bitstream 162 from source device 120 to destination device 140. For example, output interface 126 can include a transmitter or a transceiver configured to transmit encoded bitstream 162 from source device 120 directly to destination device 140 in real-time. Encoded bitstream 162 can be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 140.

[0041] Communication medium 160 can include a transitory medium, such as a wireless broadcast or wired network transmission. For example, communication medium 160 can include radio frequency (RF) spectrum or one or more physical transmission lines such as a cable. Communication medium 160 can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet. In some embodiments, communication medium 160 can include routers, switches, base stations, or any other equipment that can be useful to facilitate communication from source device 120 to destination device 140. For example, a network server (not shown) can receive encoded bitstream 162 from source device 120 and provide encoded bitstream 162 to destination device 140, such as via a network transmission.

[0042] Communication medium 160 can also be in the form of a storage medium, such as a non-transitory storage medium, for example, a hard disk, flash drive, compact disk, digital video disk, Blu-ray disk, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded picture data. In some embodiments, a computing device of a medium production facility, such as a compact disk pressing facility, can receive encoded picture data from source device 120 and produce a compact disk including the encoded video data.

[0043] Input interface 142 can include any type of medium or device capable of receiving information from communication medium 160. Received information includes encoded bitstream 162. For example, input interface 142 can include a receiver or a transceiver configured to receive encoded bitstream 162 in real-time.

[0044] System 100 can be configured to perform video encoding and decoding in accordance with block-based video compression techniques, deep learning-based video compression techniques, talking face video compression techniques, or the like.

[0045] The block-based video compression techniques utilize spatial redundancy, temporal redundancy, and information entropy redundancy in a video using a block-based hybrid video coding framework. This hybrid video coding framework includes motion compensation (e.g., intra / inter prediction), transform (e.g., discrete cosine transform), quantization, and entropy coding. The block-based video compression techniques can comply with various image / video coding standards, such as JPEG, JPEG2000, H.264 / MPEG4 Part 10, Audio Video Coding Standard (AVS), H.265 / HEVC standard, Versatile Video Coding (VVC) standard, and the like.

[0046] Figure 2 is a schematic diagram showing a block-based video compression framework 200 according to some embodiments of the present disclosure. The block-based video compression framework 200 can include an encoder configured to generate a bitstream based on input video frames, and a decoder configured to reconstruct video frames based on the bitstream. For simplicity, Figure 2 Only the encoder end of the block-based video compression framework 200 is shown. It is contemplated that the decoder end of the block-based video compression framework 200 is the inverse operation of the encoder end.

[0047] In particular, as Figure 2 shown, the input frame x t is partitioned into a set of blocks of the same size (e.g., 8x8), such as square regions. The block-based video compression framework 200 includes the following steps.

[0048] The block-based video compression framework 200 performs motion estimation by using a block-based motion estimation module 201. The motion estimation module 201 can estimate the motion between the current frame x t and the previously reconstructed frame Obtain the corresponding motion vector v t for each block.

[0049] The block-based video compression framework 200 performs motion compensation by using a motion compensation module 202. Based on the motion vector v t determined by the motion estimation module 201, the predicted frame is obtained by copying the corresponding pixels in the previously reconstructed frame to the current frame Then, the residual r t between the original frame x t and the predicted frame

[0050] The block-based video compression framework 200 performs transform and quantization by using a transform module 203 and a Q module 204, respectively. The residual rt Quantified as Before quantization, the transform module 203 uses a linear transform (e.g., DCT) to obtain better compression performance.

[0051] The block-based video compression framework 200 performs an inverse transform using an inverse transform module 205. The inverse transform uses the quantization result. To obtain the reconstructed residual

[0052] The block-based video compression framework 200 performs entropy coding using an entropy coding module 206. The motion vector v is encoded using an entropy coding method. t and the quantification results All are encoded into one or more bit streams and the one or more bit streams are sent to the decoder.

[0053] The block-based video compression framework 200 performs frame reconstruction using the reconstruction module 207. and The reconstructed frame is obtained by adding the two parts. Right now, The reconstructed frame is used for motion estimation in the (t+1)th frame.

[0054] The bitstream generated by the entropy coding module 206 can be processed at the decoder end. Figure 2 Decoding (not shown). Motion compensation, inverse quantization, and frame reconstruction can be performed to obtain the reconstructed frame.

[0055] Combination Figure 3 , 4A Sections 4B, 5A, and 5B further describe the details of the block-based video compression framework 200. Specifically, Figure 3 The structure of an example video sequence 300 according to some embodiments of the present disclosure is shown. The video sequence 300 may be live video or video that has already been captured and archived. The video sequence 300 may be real-life video, computer-generated video (e.g., computer game video), or a combination thereof (e.g., real-life video with augmented reality effects). The video sequence 300 may originate from a video capture device (e.g., a camera), a video archive containing previously captured video (e.g., a video file stored on a storage device), or a video feed interface (e.g., a video broadcast transceiver) that receives video from a video content provider.

[0056] like Figure 3 As shown, video sequence 300 may include a series of images arranged chronologically along a timeline, including images 302, 304, 306, and 308. Images 302-306 are consecutive, and there are more images between images 306 and 308.Figure 3 In particular, image 302 is an I-image, whose reference image is image 302 itself. Image 304 is a P-image, whose reference image is image 302, as indicated by the arrow. Image 306 is a B-image, whose reference images are images 304 and 308, as indicated by the arrows. In some embodiments, the reference image of an image (e.g., image 304) can not be immediately before or after the image. For example, the reference image of image 304 can be a picture before image 302. It should be noted that the reference images of images 302-306 are merely examples, and the present disclosure does not limit the embodiments of the reference images to Figure 3 the example shown.

[0057] Generally, due to the computational complexity of such tasks, video codecs do not encode or decode an entire picture at once. Instead, they can partition the picture into multiple basic segments and encode or decode the picture segment by segment. Such a basic segment is referred to as a basic processing unit (“BPU”) in the present disclosure. For example, Figure 3 Structure 310 in FIG. 3 illustrates an example structure of a picture (e.g., any of pictures 302-308) of video sequence 300. In structure 310, the picture is partitioned into 4x4 basic processing units, the boundaries of which are shown in dashed lines. In some embodiments, the basic processing units can be referred to as “macroblocks” in some video coding standards (e.g., MPEG series, H.261, H.263, or H.264 / AVC), or “coding tree units” (“CTUs”) in some other video coding standards (e.g., H.265 / HEVC, H.266 / VVC, or AVS). The basic processing units can have variable sizes in the picture, such as 128x128, 64x64, 32x32, 16x16, 4x8, 16x32, or any arbitrary shape and size of pixels. The size and shape of the basic processing units can be selected for a picture based on a balance of coding efficiency and level of detail to be preserved in the basic processing units.

[0058] The basic processing unit can be a logical unit that can include a set of different types of video data stored in computer memory (e.g., in a video frame buffer). For example, a basic processing unit of a color image can include a luma component (Y) that characterizes non-color information, one or more chroma components (e.g., Cb and Cr) that characterize color information, and associated syntax elements, where the luma and chroma components can be the same size as the basic processing unit. In some video coding standards (e.g., H.265 / HEVC, H.266 / VVC, or AVS), the luma and chroma components can be referred to as “coding tree blocks” (“CTBs”). Any operation performed on a basic processing unit can be repeated on the luma and chroma components of the basic processing unit, respectively.

[0059] Video coding includes multiple stages of operations, examples of which are shown in Figures 4A-4B and Figures 5A-5B For each stage, the basic processing unit can still be too large to process and can be further divided into segments, referred to in this disclosure as “basic processing subunits.” In some embodiments, the basic processing subunits can be referred to as “blocks” in some video coding standards (e.g., the MPEG series, H.261, H.263, H.264 / AVC, or AVS), or as “coding units” (“CUs”) in some other video coding standards (e.g., H.265 / HEVC, H.266 / VVC, or AVS). A basic processing subunit can have the same or smaller size than the basic processing unit. Similar to the basic processing unit, a basic processing subunit is also a logical unit that can include a set of different types of video data (e.g., Y, Cb, Cr, and associated syntax elements) stored in computer memory (e.g., in a video frame buffer). Any operation performed on a basic processing subunit can be repeated on each of the luma and chroma components of the basic processing subunit. It should be noted that such division can be performed to a deeper level depending on processing needs. It should also be noted that different stages can use different schemes to divide the basic processing unit.

[0060] For example, at the mode decision stage (examples of which are shown in Figure 4B ), the encoder can decide what prediction mode (e.g., intra-picture prediction or inter-picture prediction) to use for a basic processing unit that can be too large to make such a decision. The encoder can split the basic processing unit into multiple basic processing subunits (e.g., as CUs in H.265 / HEVC, H.266 / VVC, or AVS) and decide the prediction type for each individual basic processing subunit.

[0061] For another example, at the prediction stage (examples of which are shown inFigures 4A-4B (As shown), the encoder can perform prediction operations at the level of a basic processing subunit (e.g., CU). However, in some cases, the basic processing subunit may still be too large to process. The encoder can further divide the basic processing subunit into smaller segments (e.g., referred to as "prediction blocks" or "PBs" in H.265 / HEVC or H.266 / VVC), at which the prediction operations can be performed.

[0062] For another example, in the transformation phase (the example of which is in...) Figures 4A-4B (As shown), the encoder can perform transform operations on residual basic processing subunits (e.g., CUs). However, in some cases, the basic processing subunits may still be too large to process. The encoder can further divide the basic processing subunits into smaller segments (e.g., referred to as "transform blocks" or "TBs" in H.265 / HEVC, H.266 / VVC, or AVS), and the transform operation can be performed at the level of these smaller segments. It should be noted that the partitioning scheme of the same basic processing subunit can differ between the prediction and transform phases. For example, in H.265 / HEVC, H.266 / VVC, or AVS, the prediction blocks and transform blocks of the same CU can have different sizes and numbers.

[0063] exist Figure 3 In structure 310, the basic processing unit 312 is further divided into 3×3 basic processing sub-units, the boundaries of which are shown as dashed lines. Different schemes can be used to divide different basic processing units of the same image into basic processing sub-units.

[0064] In some implementations, to provide parallel processing and fault tolerance for video encoding and decoding, an image can be divided into multiple regions for processing, such that for a particular region of the image, the encoding or decoding process can be independent of information from any other region of the image. In other words, each region of the image can be processed independently. By doing so, the codec can process different regions of an image in parallel, thereby improving encoding efficiency. Furthermore, when data in one region is corrupted during processing or lost during network transmission, the codec can correctly encode or decode other regions of the same image without relying on the corrupted or lost data, thus providing fault tolerance. In some video coding standards, images can be divided into different types of regions. For example, H.265 / HEVC, H.266 / VVC, and AVS provide two types of regions: "slices" and "tiles." It should also be noted that different images in the video sequence 300 can have different segmentation schemes for dividing an image into multiple regions.

[0065] For example, in Figure 3 structure 310 is divided into three regions 314, 316, and 318, the boundaries of which are shown in solid lines within structure 310. Region 314 includes four basic processing units. Each of regions 316 and 318 includes six basic processing units. It should be noted that Figure 3 The basic processing units, basic processing sub-units, and regions of structure 310 in

[0066] Figure 4A A diagram illustrating an example encoding process 400A consistent with embodiments of the present disclosure is shown. For example, the encoding process 400A can be performed by an encoder. As Figure 4A indicated, the encoder can encode a video sequence 402 into a video bitstream 428 according to process 400A. Similar to Figure 3 video sequence 300 in Figure 3 structure 310 in In some embodiments, the encoder can perform process 400A on a hierarchy of basic processing units for each original picture of video sequence 402. For example, the encoder can perform process 400A in an iterative manner, where the encoder can encode a basic processing unit in one iteration of process 400A. In some embodiments, the encoder can perform process 400A in parallel for multiple regions (e.g., regions 314-318) of each original picture of video sequence 402.

[0067] Figure 4AIn particular embodiments, the encoder can feed a basic processing unit (referred to as an “original BPU”) of an original picture of video sequence 402 to prediction stage 404 to generate prediction data 406 and a predicted BPU 408. The encoder can subtract predicted BPU 408 from original BPU to generate a residual BPU 410. The encoder can feed residual BPU 410 to transform stage 412 and quantization stage 414 to generate quantized transform coefficients 416. The encoder can feed prediction data 406 and quantized transform coefficients 416 to binary encoding stage 426 to generate video bitstream 428. Components 402, 404, 406, 408, 410, 412, 414, 416, 426, and 428 can be referred to as a “forward path.” During process 400A, after quantization stage 414, the encoder can feed quantized transform coefficients 416 to inverse quantization stage 418 and inverse transform stage 420 to generate a reconstructed residual BPU 422. The encoder can add reconstructed residual BPU 422 to predicted BPU 408 to generate a prediction reference 424, which is used in prediction stage 404 for the next iteration of process 400A. Components 418, 420, 422, and 424 of process 400A can be referred to as a “reconstruction path.” The reconstruction path can be used to ensure that the encoder and the decoder both use the same reference data for prediction.

[0068] The encoder can iteratively perform process 400A to encode each original BPU of the original picture (in the forward path) and generate prediction reference 424 for use in encoding the next original BPU of the original picture (in the reconstruction path). After encoding all original BPUs of the original picture, the encoder can proceed to encode the next picture in video sequence 402.

[0069] With reference to process 400A, the encoder can receive video sequence 402 generated by a video capture device (e.g., a camera). As used herein, the term “receive” can refer to any action that gets, obtains, retrieves, acquires, accesses, or inputs data in any manner.

[0070] At prediction stage 404, at a current iteration, the encoder can receive an original BPU and prediction reference 424 and perform a prediction operation to generate prediction data 406 and predicted BPU 408. Prediction reference 424 can be generated from the reconstruction path of a previous iteration of process 400A. The purpose of prediction stage 404 is to reduce information redundancy by extracting prediction data 406 from prediction data 406 and prediction reference 424 that can be used to reconstruct the original BPU into predicted BPU 408.

[0071] Ideally, the predicted BPU 408 can be identical to the original BPU. However, due to non-ideal prediction and reconstruction operations, the predicted BPU 408 is typically slightly different from the original BPU. To account for such differences, after generating the predicted BPU 408, the encoder can subtract the predicted BPU 408 from the original BPU to generate a residual BPU 410. For example, the encoder can subtract the values (e.g., grayscale or RGB values) of the pixels of the predicted BPU 408 from the values of the corresponding pixels of the original BPU. Each pixel of the residual BPU 410 can have a residual value, which is generated by such subtraction between the corresponding pixels of the original BPU and the predicted BPU 408. Compared to the original BPU, the prediction data 406 and the residual BPU 410 can have fewer bits, but they can be used to reconstruct the original BPU without apparent quality degradation. Thus, the original BPU is compressed.

[0072] To further compress the residual BPU 410, at a transform stage 412, the encoder can reduce the spatial redundancy of the residual BPU 410 by decomposing the residual BPU 410 into a set of two-dimensional “basis patterns,” each basis pattern being associated with a “transform coefficient.” The basis patterns can have the same size (e.g., the size of the residual BPU 410). Each basis pattern can characterize a component of a variation frequency (e.g., a frequency of luminance variation) of the residual BPU 410. Any basis pattern cannot be reproduced by any combination (e.g., linear combination) of any other basis pattern. In other words, the decomposition can decompose the variations of the residual BPU 410 into the frequency domain. This decomposition is similar to a discrete Fourier transform of a function, where the basis patterns are similar to the basis functions (e.g., trigonometric functions) of the discrete Fourier transform, and the transform coefficients are similar to the coefficients associated with the basis functions.

[0073] Different transform algorithms can use different basis patterns. Various transform algorithms can be used at transform stage 412, such as discrete cosine transform, discrete sine transform, etc. The transform at transform stage 412 is invertible. That is, the encoder can recover residual BPUs 410 by the inverse operation of the transform, called "inverse transform." For example, to recover a certain pixel of residual BPUs 410, the inverse transform can be multiplying the value of the corresponding pixel of the basis pattern by the respective correlation coefficient and adding the products to produce a weighted sum. For video coding standards, the encoder and decoder can use the same transform algorithm (and thus the same basis pattern). Thus, the encoder can record only the transform coefficients, and the decoder can reconstruct residual BPUs 410 from the transform coefficients without receiving the basis pattern from the encoder. The transform coefficients can have fewer bits than residual BPUs 410, but they can be used to reconstruct residual BPUs 410 without noticeable quality degradation. Thus, residual BPUs 410 are further compressed.

[0074] The encoder can further compress the transform coefficients at quantization stage 414. During the transform process, different basis patterns can represent different frequencies of variation (e.g., frequencies of luminance variation). Because the human eye is generally better at recognizing low-frequency variations, the encoder can ignore information of high-frequency variations without causing noticeable quality degradation in decoding. For example, at quantization stage 414, the encoder can generate quantized transform coefficients 416 by dividing each transform coefficient by an integer value called "quantization scale factor" and performing a round-off to the nearest integer on the quotient. After this operation, some transform coefficients of high-frequency basis patterns can be converted to zero, and the transform coefficients of low-frequency basis patterns can be converted to smaller integers. The encoder can ignore quantized transform coefficients 416 of zero value, whereby transform coefficients are further compressed. The quantization process is also invertible, in which quantized transform coefficients 416 can be reconstructed to the transform coefficients in the inverse operation of quantization, called "inverse quantization."

[0075] Because the encoder ignores the remainder of this division in the round-off operation, quantization stage 414 can be lossy. Generally, quantization stage 414 can constitute the most significant source of information loss in process 400A. The greater the information loss, the fewer bits quantized transform coefficients 416 can require. To obtain different degrees of information loss, the encoder can use different values of the quantization parameter or any other parameter of the quantization process.

[0076] At the binarization stage 426, the encoder can use binarization techniques to encode the prediction data 406 and the quantized transform coefficients 416, such as, for example, entropy coding, variable length coding, arithmetic coding, Huffman coding, context adaptive binary arithmetic coding, or any other lossless or lossy compression algorithm. In some embodiments, in addition to the prediction data 406 and the quantized transform coefficients 416, the encoder can also encode other information at the binarization stage 426, such as, for example, the prediction modes used at the prediction stage 404, parameters of the prediction operations, the transform types of the transform stage 412, parameters of the quantization process (e.g., quantization parameters), encoder control parameters (e.g., bitrate control parameters), and the like. The encoder can use the output data of the binarization stage 426 to generate a video bitstream 428. In some embodiments, the video bitstream 428 can be further packetized for network transmission.

[0077] Referring to the reconstruction path of the process 400A, at the inverse quantization stage 418, the encoder can perform inverse quantization on the quantized transform coefficients 416 to generate reconstructed transform coefficients. At the inverse transform stage 420, the encoder can generate a reconstructed residual BPU 422 based on the reconstructed transform coefficients. The encoder can add the reconstructed residual BPU 422 to the predicted BPU 408 to generate a prediction reference 424 to be used in the next iteration of the process 400A.

[0078] It should be noted that other variants of the process 400A can be used to encode the video sequence 402. In some embodiments, the stages of the process 400A can be performed in a different order by the encoder. In some embodiments, one or more stages of the process 400A can be combined into a single stage. In some embodiments, a single stage of the process 400A can be split into multiple stages. For example, the transform stage 412 and the quantization stage 414 can be combined into a single stage. In some embodiments, the process 400A can include additional stages. In some embodiments, the process 400A can omit one or more stages of Figure 4A

[0079] Figure 4B A schematic diagram of another example encoding process 400B consistent with embodiments of the present disclosure is shown. The process 400B can be modified from the process 400A. For example, the process 400B can be used by an encoder conforming to a hybrid video coding standard (e.g., the H.26x family). Compared to the process 400A, the forward path of the process 400B additionally includes a mode decision stage 430, and divides the prediction stage 404 into a spatial prediction stage 4042 and a temporal prediction stage 4044. The reconstruction path of the process 400B additionally includes a loop filtering stage 432 and a buffer 434.​

[0080] Generally, prediction techniques can be categorized into two types: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-image prediction or "intra-frame prediction") uses pixels from one or more encoded neighboring BPUs within the same image to predict the current BPU. That is, the prediction reference 424 in spatial prediction can include the neighboring BPUs. Spatial prediction can reduce the inherent spatial redundancy of the image. Temporal prediction (e.g., inter-image prediction or "inter-frame prediction") uses regions from one or more encoded images to predict the current BPU. That is, the prediction reference 424 in temporal prediction can include the encoded image. Temporal prediction can reduce the inherent temporal redundancy of the image.

[0081] In reference process 400B, during the forward path, the encoder performs prediction operations in spatial prediction phase 4042 and temporal prediction phase 4044. For example, in spatial prediction phase 4042, the encoder may perform intra-frame prediction. For a given original BPU of an image being encoded, prediction reference 424 may include one or more adjacent BPUs that have already been encoded (in the forward path) and reconstructed (in the reconstruction path) in the same image. The encoder can generate a predicted BPU 408 by extrapolating adjacent BPUs. The extrapolation technique may include, for example, linear extrapolation or interpolation, polynomial extrapolation or interpolation, etc. In some embodiments, the encoder may perform the extrapolation at the pixel level, for example, extrapolating the corresponding pixel value for each pixel of the predicted BPU 408. The adjacent BPU used for extrapolation can be located from various directions relative to the original BPU, such as in the vertical direction (e.g., at the top of the original BPU), the horizontal direction (e.g., to the left of the original BPU), the diagonal direction (e.g., at the lower left, lower right, upper left, or upper right of the original BPU), or any direction defined in the video coding standard used. For intra-frame prediction, prediction data 406 may include, for example, the location (e.g., coordinates) of the adjacent BPU used, the size of the adjacent BPU used, the extrapolation parameters, the orientation of the adjacent BPU used relative to the original BPU, etc.

[0082] In another example, during the temporal prediction phase 4044, the encoder may perform the inter-frame prediction. For a given original BPU of the current image, the prediction reference 424 may include one or more images (referred to as "reference images") that have been encoded (in the forward path) and reconstructed (in the reconstruction path). In some embodiments, the reference images may be encoded and reconstructed on a BPU-by-BPU basis. For example, the encoder may add a reconstructed residual BPU 422 to the prediction BPU 408 to generate a reconstructed BPU. After all reconstructed BPUs of the same image have been generated, the encoder may generate a reconstructed image as the reference image. The encoder may perform a "motion estimation" operation to search for a matching region within a range (referred to as a "search window") of the reference image. The location of the search window in the reference image may be determined based on the location of the original BPU in the current image. For example, the search window may be centered at a location in the reference image that has the same coordinates as the original BPU in the current image and may extend outward by a predetermined distance. When the encoder identifies a region similar to the original BPU in the search window (e.g., using a pixel recursive algorithm, block matching algorithm, etc.), the encoder can determine such a region as a matching region. The matching region may have different specifications than the original BPU (e.g., less than, equal to, greater than, or with a different shape). Because the reference image and the current image are temporally separated in the timeline (e.g., as...), Figure 3 As shown), the matching region can be considered to have "moved" to the original BPU location over time. The encoder can record the direction and distance of this movement as a "motion vector." When using multiple reference images (e.g., Figure 3 When the encoder is used to search for a matching region for each reference image (306), it can determine the associated motion vector of the matching region. In some embodiments, the encoder can assign weights to the pixel values ​​of the matching region for each matching reference image.

[0083] The motion estimation can be used to identify various types of motion, such as translation, rotation, scaling, etc. For inter-frame prediction, the prediction data 406 may include, for example, the location (e.g., coordinates) of the matching region, the motion vector associated with the matching region, the number of reference images, the weights associated with the reference images, etc.

[0084] To generate the predicted BPU 408, the encoder can perform a "motion compensation" operation. The motion compensation can be used to reconstruct the predicted BPU 408 based on the prediction data 406 (e.g., motion vectors) and the prediction reference 424. For example, the encoder can move the matching region of the reference image according to the motion vectors, where the encoder can predict the original BPU of the current image. When multiple reference images (e.g., images 306 in FIG. 3) are used, the encoder can move the matching region of the reference images according to the motion vectors and average pixel values of the matching regions, respectively. In some embodiments, if the encoder has assigned weights to the pixel values of the matching regions of the respective matching reference images, the encoder can perform a weighted sum of the pixel values of the moved matching regions. Figure 3

[0085] In some embodiments, the inter-frame prediction can be uni-directional or bi-directional. Uni-directional inter-frame prediction can use one or more reference images in the same temporal direction relative to the current image. For example, image 304 in FIG. 3 is a uni-directional inter-frame predicted image, where the reference image (e.g., image 302) is before image 304. Figure 3 Bi-directional inter-frame prediction can use one or more reference images in both temporal directions relative to the current image. For example, image 306 in FIG. 3 is a bi-directional inter-frame predicted image, where the reference images (e.g., images 304 and 308) are in both temporal directions relative to image 304. Figure 3

[0086] Still referring to the forward path of process 400B, after the spatial prediction stage 4042 and the temporal prediction stage 4044, at a mode decision stage 430, the encoder can select a prediction mode (e.g., one of intra-prediction or inter-prediction) for the current iteration of process 400B. For example, the encoder can perform a rate-distortion optimization technique, where the encoder can select a prediction mode according to the bit rate of the candidate prediction mode and the distortion of the reconstructed reference image under the candidate prediction mode to minimize the value of a cost function. According to the selected prediction mode, the encoder can generate the corresponding predicted BPU 408 and prediction data 406.

[0087] ​​In the reconstruction path of process 400B, if an intra prediction mode has been selected in the forward path, after generating a prediction reference 424 (e.g., a current BPU that has been encoded and reconstructed in the current picture), the encoder can feed the prediction reference 424 directly to a spatial prediction stage 4042 for subsequent use (e.g., for extrapolation of a next BPU of the current picture). The encoder can feed the prediction reference 424 to a loop filtering stage 432, where the encoder can apply loop filtering to the prediction reference 424 to reduce or eliminate distortion (e.g., blocking artifacts) introduced during encoding of the prediction reference 424. The encoder can apply various loop filtering techniques at the loop filtering stage 432, such as deblocking, sample adaptive offset, adaptive loop filtering, etc. The loop-filtered reference picture can be stored in a buffer 434 (or “decoded picture buffer”) for subsequent use (e.g., as an inter prediction reference picture for future pictures of the video sequence 402). The encoder can store one or more reference pictures in the buffer 434 for use at the temporal prediction stage 4044. In some embodiments, the encoder can encode parameters of the loop filtering (e.g., loop filtering strength) along with the quantized transform coefficients 416, prediction data 406, and other information at the binary encoding stage 426.

[0088] Figure 5A A schematic diagram illustrating an example decoding process 500A consistent with embodiments of the present disclosure is shown. The process 500A can be a decompression process corresponding to the compression process 400A in Figure 4A some embodiments, the process 500A can be similar to the reconstruction path of the process 400A. A decoder can decode the video bitstream 428 into a video stream 504 according to the process 500A. The video stream 504 can be very similar to the video sequence 402. However, due to information loss in the compression and decompression processes (e.g., the quantization stage 414 in the process 400A), in general, the video stream 504 is not identical to the video sequence 402. Similar to the process 400A and 400B in Figures 4A-4B some embodiments, the process 500A can be similar to the reconstruction path of the process 400A. A decoder can decode the video bitstream 428 into a video stream 504 according to the process 500A. The video stream 504 can be very similar to the video sequence 402. However, due to information loss in the compression and decompression processes (e.g., the quantization stage 414 in the process 400A), in general, the video stream 504 is not identical to the video sequence 402. Similar to the process 400A and 400B in Figures 4A-4B some embodiments, the process 500A can be similar to the reconstruction path of the process 400A. A decoder can decode the video bitstream 428 into a video stream 504 according to the process 500A. The video stream 504 can be very similar to the video sequence 402. However, due to information loss in the compression and decompression processes (e.g., the quantization stage 414 in the process 400A), in general, the video stream 504 is not identical to the video sequence 402. Similar to the process 400A and 400B in

[0089] In some embodiments, the process 500A can be similar to the reconstruction path of the process 400A. A decoder can decode the video bitstream 428 into a video stream 504 according to the process 500A. The video stream 504 can be very similar to the video sequence 402. However, due to information loss in the compression and decompression processes (e.g., the quantization stage 414 in the process 400A), in general, the video stream 504 is not identical to the video sequence 402. Similar to the process 400A and 400B in Figure 5AIn some embodiments, the decoder can feed a portion of the video bitstream 428 associated with a basic processing unit (referred to as a "coded BPU") of a coded picture to a binarization stage 502. In the binarization stage 502, the decoder can binarize the portion into prediction data 406 and quantized transform coefficients 416. The decoder can feed the quantized transform coefficients 416 to an inverse quantization stage 418 and an inverse transform stage 420 to generate a reconstructed residual BPU 422. The decoder can feed the prediction data 406 to a prediction stage 404 to generate a predicted BPU 408. The decoder can add the reconstructed residual BPU 422 to the predicted BPU 408 to generate a prediction reference 424. In some embodiments, the prediction reference 424 can be stored in a buffer (e.g., a decoded picture buffer in computer memory). The decoder can feed the prediction reference 424 to the prediction stage 404 for performing a prediction operation in a next iteration of the process 500A.

[0090] The decoder can iteratively perform the process 500A to decode each coded BPU of a coded picture and generate a prediction reference 424 for encoding a next coded BPU of the coded picture. After decoding all coded BPUs of the coded picture, the decoder can output the picture to a video stream 504 for display and continue decoding a next coded picture in the video bitstream 428.

[0091] In the binarization stage 502, the decoder can perform an inverse operation of a binarization technique used by the encoder (e.g., entropy encoding, variable length encoding, arithmetic encoding, Huffman encoding, context adaptive binary arithmetic encoding, or any other lossless compression algorithm). In some embodiments, in addition to the prediction data 406 and the quantized transform coefficients 416, the decoder can also decode other information in the binarization stage 502, such as, for example, a prediction mode, parameters of a prediction operation, a transform type, parameters of a quantization process (e.g., a quantization parameter), encoder control parameters (e.g., bit rate control parameters), and the like. In some embodiments, if the video bitstream 428 is transmitted over a network in the form of data packets, the decoder can unpack the video bitstream 428 before feeding the video bitstream 428 to the binarization stage 502.

[0092] Figure 5BA diagram illustrating another example decoding process 500B consistent with embodiments of the present disclosure is shown. Process 500B can be modified from process 500A. For example, process 500B can be used by a decoder that conforms to a hybrid video coding standard (e.g., the H.26x family). In comparison to process 500A, process 500B additionally divides prediction stage 404 into a spatial prediction stage 4042 and a temporal prediction stage 4044, and additionally includes a loop filtering stage 432 and a buffer 434.

[0093] In process 500B, for a coded base processing unit (referred to as a "current BPU") of a coded picture (referred to as a "current picture") being decoded by the decoder, prediction data 406 decoded from binary decoding stage 502 by the decoder can include various types of data depending on what prediction mode was used by the encoder to code the current BPU. For example, if the current BPU was coded by the encoder using intra prediction, prediction data 406 can include a prediction mode indicator (e.g., a flag value) indicating intra prediction, parameters of the intra prediction operation, etc. The parameters of the intra prediction operation can include, for example, locations (e.g., coordinates) of one or more neighboring BPUs used as references, sizes of the neighboring BPUs, parameters of extrapolation, directions of the neighboring BPUs relative to the original BPU, etc. For another example, if the current BPU was coded by the encoder using inter prediction, prediction data 406 can include a prediction mode indicator (e.g., a flag value) indicating inter prediction, parameters of the inter prediction operation, etc. The parameters of the inter prediction operation can include, for example, a number of reference pictures associated with the current BPU, weights respectively associated with the reference pictures, locations (e.g., coordinates) of one or more matching regions in the respective reference pictures, one or more motion vectors respectively associated with the matching regions, etc.

[0094] Based on the prediction mode indicator, the decoder can determine whether to perform spatial prediction (e.g., intra prediction) at spatial prediction stage 4042 or temporal prediction (e.g., inter prediction) at temporal prediction stage 4044. Figure 4B Details of performing such spatial prediction or temporal prediction are described in the Background section and will not be repeated here. After performing such spatial prediction or temporal prediction, the decoder can generate a predicted BPU 408. The decoder can add predicted BPU 408 and reconstructed residual BPU 422 to generate a predicted reference 424, as Figure 5A described.

[0095] In process 500B, the decoder can feed the prediction reference 424 to the spatial prediction stage 4042 or the temporal prediction stage 4044 for performing a prediction operation in the next iteration of process 500B. For example, if the current BPU is decoded using the intra prediction at the spatial prediction stage 4042, after the prediction reference 424 (e.g., the decoded current BPU) is generated, the decoder can feed the prediction reference 424 directly to the spatial prediction stage 4042 for subsequent use (e.g., for extrapolation of the next BPU of the current picture). If the current BPU is decoded using the inter prediction at the temporal prediction stage 4044, after the prediction reference 424 (e.g., a reference picture in which all BPU have been decoded) is generated, the decoder can feed the prediction reference 424 to the loop filtering stage 432 to reduce or eliminate distortion (e.g., blocking artifacts). The decoder can apply loop filtering to the prediction reference 424 in the manner as described in Figure 4B

[0096] In addition to the block-based video compression techniques, deep learning can be used in video compression to achieve competitive performance compared to traditional compression schemes. For example, end-to-end image compression algorithms show better rate-distortion (RD) performance than JPEG, JPEG2000, even HEVC due to end-to-end training and non-linear transformation. Furthermore, the video compression algorithms based on deep neural networks (DNNs), such as deep video compression model (DVC), can achieve satisfactory RD performance. These schemes can be performed without prior knowledge of the video content. With respect to the application of video conferencing / telephony, deep generative models such as first-order motion model (FOMM) and face video-to-video synthesis (Face vid2vid) can achieve more potential performance at ultra-low bitrates. In particular, these models exploit the fact that the changes in these videos often exist in the human motion information, providing strong prior information that can be used for frame synthesis. These features are described by the changes in human structure such as landmarks or keypoints, and are further transmitted to animate the reference frames and generate human motion videos.​

[0097] Deep learning based algorithms can be used to replace or augment some operations or functions of the block-based video coding tools, including intra / inter prediction, entropy coding, in-loop filtering, etc. End-to-end image / video compression algorithms can be used for joint optimization of the whole image / video compression framework instead of designing only one specific module. For example, an end-to-end video coding scheme, DVC scheme, can be used to jointly optimize all components of video compression. Furthermore, to address the content adaptation and error propagation awareness issues, an online encoder update scheme can be used to improve the video compression performance. In addition, a flexible feature coding (FVC) can be used, which develops all major modules of the end-to-end compression framework in the feature space. Recursive learning for video compression (RLVC) and HLVC can be used to exploit the temporal correlation between video frames based on recursive probabilistic models and weighted recurrent quality enhancement networks. Four effective modules in multi-frame prediction for learning-based video compression (M-LVC) can be used. However, like the traditional video coding tools, these learning-based video compression methods are targeted at general natural scenes without specific consideration of human content, such as faces, body, or other parts.

[0098] Figure 6 is a schematic diagram illustrating an exemplary architecture of an end-to-end deep learning based video compression framework 600 according to some embodiments of the present disclosure. The framework 600 uses various deep learning models that jointly optimize the components of video compression, such as motion estimation, motion compression, and residual compression. Specifically, learning-based optical flow estimation is utilized to obtain the motion information and reconstruct the current frame. Then, two autoencoder-like neural networks are employed to compress the corresponding motion and residual information. The modules in the framework 600 are jointly learned using a single loss function, where they collaborate with each other by trading off the reduction of the number of compressed bits against the improvement of the quality of the decoded video. Figure 2 The block-based video compression framework 200 shown is in one-to-one correspondence with Figure 6 The end-to-end deep learning based video compression framework 600 shown. The relationship between them is introduced below, and a brief overview of the differences between the two is given. The end-to-end deep learning based video compression framework 600 can include an encoder configured to generate a bitstream based on input video frames, and a decoder configured to reconstruct video frames based on the bitstream. For simplicity, Figure 6 Only the encoder side of the end-to-end deep learning based video compression framework 600 is shown.

[0099] As Figure 6As shown, framework 600 can perform motion estimation and compression. In the optical flow network module 601, a CNN (Convolutional Neural Network) model can be used to estimate the optical flow, which is treated as motion information v. t Instead of directly encoding the raw optical flow value, an MV encoder-decoder network is used to compress and decode the optical flow value. First, the MV encoder network module 602 can be used to compress and decode the motion information v. t Encode the motion information v. t The encoded motion is characterized as m t The encoded motion representation can be further quantized by the Q module 603. Then, the corresponding reconstructed motion information can be decoded using the MV decoder network module 604.

[0100] Frame 600 can also perform motion compensation. A motion compensation network, provided by a motion compensation network module 605, is designed to obtain the predicted frame based on the acquired optical flow. Then, obtain the original frame x. t With the predicted frame The residual r between t ,for

[0101] The Frame 600 can also perform transform, quantization, and inverse transform. This is achieved by using a highly nonlinear residual encoder-decoder network, such as... Figure 6 The residual encoder network module 606 shown replaces the linear transformation and converts the residual r t Nonlinear mapping to representation y t Then, the Q module 607 will transfer y t Quantified as To construct an end-to-end training scheme, quantization methods are used. The quantized representation... The data is fed into a residual decoder network provided by a residual decoder network module 608 to obtain the reconstructed residual.

[0102] Framework 600 can also perform entropy coding. During the testing phase, the quantized motion representation is performed by the bit rate estimation network module 609. and the residual characterization The data is encoded into bits and sent to the decoder. During the training phase, the CNN is used to estimate the required bit count overhead. and The probability distribution of each symbol in the array.

[0103] Further, the loss of the framework 600 can be determined according to the original frame, the reconstructed frame and the coded frame. The loss determined here can also be used to optimize the network connection within the framework 600 to achieve better performance.

[0104] The framework 600 can also perform frame reconstruction (not shown in the middle), in the same way as the frame reconstruction described in connection with the framework 200. Figure 6

[0105] The end-to-end deep learning-based video compression framework 600 can be used for face video compression, for example, for talking face generative video coding. For example, the end-to-end deep learning-based talking face generative video coding can use generative models, such as variational auto-encoding (VAE) and generative adversarial networks (GAN). The face video compression can achieve satisfactory performance improvement. For example, X2Face can be used to control face generation through image, audio and pose coding. In addition, realistic neural talking head models can be used through few-shot adversarial learning. For the video-to-video synthesis task, Face-vid2vid can be used. In addition, a scheme that drives generative models to render the target frame using compact 3D keypoint representation can also be used. In addition, a motion-compatible video chat system based on FOMM (First Order Motion Model) can be used. VSBNet that reconstructs the original frame from the landmark points using adversarial learning can also be used. In addition, an end-to-end talking head video compression framework based on CFTE (Compact Feature Transfer and Encoding) can be used, which is designed for efficient talking face video compression for ultra-low bandwidth scenarios. The CFTE scheme uses compact feature representation to compensate for temporal evolution, and reconstructs the target face video frame in an end-to-end manner. In addition, under the supervision of the rate-distortion target, the CFTE scheme can be integrated into the video coding framework. Although these algorithms achieve frame reconstruction using a small number of face parameters through the powerful rendering capability of deep generative models, certain head pose motion and facial expression motion still cannot be accurately rendered compared with the original motion video.

[0106] Figure 7 ​is a schematic diagram illustrating an exemplary deep learning based video generative compression framework 700 according to some embodiments of the present disclosure. The framework 700 is suitable for compressing and generating talking face videos. For example, the framework 700 can be based on the first order motion model (FOMM). The FOMM deforms a reference source frame to follow the motion in a driving video. While this approach is suitable for various types of videos (e.g., motion pictures, cartoons), this approach can also be used for face animation applications. The FOMM follows an encoder-decoder architecture, whose motion transfer component includes the following steps.

[0107] First, a keypoint extractor (also referred to as a motion module) is learned using a constant variation loss without explicit labels. Using this keypoint extractor, two sets of ten learned keypoints are calculated for the source frame and the driving frame. The learned keypoints are transformed by a Gaussian mapping function from a feature map of channel dimension x 64 x 64 size, so that each corresponding keypoint can represent the feature information of different channels. It should be noted that each keypoint is a point represented by coordinates (x, y), which can represent the most important information of the feature map.

[0108] Second, a dense motion network uses the landmark points and the source frame to generate a dense motion field and an occlusion map.

[0109] Then, the encoder 710 encodes the source frame by a conventional image / video compression method such as HEVC / VVC or JPEG / BPG. Here, the VVC is used to compress the source frame.

[0110] In a subsequent stage, the generated feature map is warped using the dense motion field (using a differentiable grid sampling operation) and then multiplied by the occlusion map.

[0111] Finally, the decoder 720 generates an image from the warped map.

[0112] Figure 8 is a schematic diagram illustrating an exemplary encoder-decoder encoding framework 600 for talking face videos with compact feature size of 1 x 4 x 4 according to some embodiments of the present disclosure. Figure 8 Another basic framework of the deep video generative compression scheme based on compact feature representation, i.e., CFTE, is provided. The basic framework follows an encoder-decoder architecture that applies a context-based encoding scheme.

[0113] At the encoder 810 side, the compression framework includes three modules: an encoder for compressing the key frames (also referred to as VVC encoding module), a feature extractor for extracting compact human features for the inter frames, and a feature encoding module for compressing the inter-frame prediction residuals of the compact human features. First, the key frames representing human texture are compressed with the VVC encoder. Each of the subsequent inter frames is represented with a compact feature matrix of size 1 x 4 x 4 by the compact feature extractor. It should be noted that the size of the compact feature matrix is not fixed, and the number of feature parameters can also be increased or decreased according to the specific requirements of bit consumption. Then, the extracted features are inter-predicted and quantized, and the residuals are entropy encoded into the final bitstream.

[0114] At the decoder 820 side, this compression framework also contains three main modules, including decoding for reconstructing the key frames, reconstructing the compact features through entropy decoding and compensation, and generating the final video by utilizing the reconstructed features and decoded key frames. More specifically, in the process of generating the final video, the decoded key frames from the VVC bitstream can be further represented in the form of features by compact feature extraction. Subsequently, given the features from the key frames and inter frames, the relevant sparse motion fields are calculated, facilitating the generation of pixel-level dense motion maps and occlusion maps. Finally, based on the deep generative model, the decoded key frames, pixel-level dense motion maps, and occlusion maps with implicit motion field features are used to produce the final video with accurate appearance, pose, and expression.

[0115] Inspired by the recent advances in deep generative models, especially generative adversarial networks (GANs), the low-quality face reconstruction of early MBC techniques can be well remedied and improved. In particular, learning-based face reproduction or animation models bring great prospects for generative face video compression (GFVC). Figure 9 is a schematic diagram showing a general flowchart of a generative face video compression (GFVC) system according to some embodiments of the present disclosure. As Figure 9 shown, the key reference frame 901 is encoded and decoded by a conventional image / video codec 910 to obtain a decoded key reference frame 903. The subsequent inter frames 902 are processed by a model-based codec 920. Specifically, at the encoder end, the subsequent inter frames 902 are represented with compact transmission symbols by an analysis model 921, and encoded into an encoded bitstream by a parameter encoding 922. At the decoder end, the decoded key reference frame 903 and the face representation parameters decoded by a parameter decoding 923 are jointly fed into a synthesis model 924 to output a reconstructed inter frame 904. In this way, video communication with ultra-low bit rate and high-quality reconstruction can be achieved.

[0116] While the above generative video compression techniques can significantly improve the rate-distortion (RD) performance, they still have some drawbacks and challenges that limit further performance improvement and practical applications. For example, the above generative video codecs mainly use explicit features with actual physical manifestations for representation, thus leading to unnecessary compression redundancy. Meanwhile, such representation lacks the required expressiveness and generalizability to handle more complex scenarios, such as human body movements.

[0117] Embodiments of the present disclosure provide a multi-granularity temporal trajectory factorization (MTTF) for generative video compression.

[0118] The MTTF is designed to improve the performance of generative video coding by enhancing the generalizability and stability of the codec. For example, temporal trajectory can be implicitly factorized into fine-grained features to achieve input-adaptive motion representation, while the transmitted features can still remain coarse-grained compact vectors to achieve ultra-low bitrate compression. With this solution, the MTTF can leverage the compactness and expressiveness of feature representation to achieve both high compressibility and high-quality reconstruction, and can be generalized to various video contents.

[0119] Figure 10 An exemplary structure of a multi-granularity temporal trajectory factorization 1000 is shown in accordance with some embodiments of the present disclosure. Figure 11 is a flowchart of an exemplary method for multi-granularity temporal trajectory factorization (MTTF) 1100 in accordance with some embodiments of the present disclosure. The method 1100 can be performed by a GFVC system (e.g., by a GFVC system 900 of Figure 9 in accordance with some embodiments of the present disclosure. The method 1100 can be performed by one or more software or hardware components of an apparatus. In some embodiments, the method 1100 can be implemented by a computer program product including computer-executable instructions, such as program code, that are executed by a computer. Referring to Figure 10 and Figure 11 , the method 1100 can include the following steps 1102-1110.

[0120] At step 1102, a reconstructed key reference frame is decomposed into key frame latent features and a first set of compact motion vectors. Latent features are hidden or inferred features that cannot be directly observed in the original data, but can be learned by a model to help explain patterns or structures in the data. The latent features can include implicit vectors for controlling aspects of the generator output (e.g., pose, expression, or background of a generated face).

[0121] The reconstructed key reference frame It is obtained by decoding and reconstructing a key reference frame I of the video sequence from the image bitstream. The image bitstream is encoded and decoded through an image / video codec process, for example, by... Figure 9 The image / video codec 910, along with processes 400A, 400B, 500A, and 500B as shown in Figures 4 and 5, performs encoding and decoding. The reconstructed key reference frame is represented as... Where H represents the height of a frame in the video sequence, and W represents the width of a frame in the video sequence. Then, the reconstructed key reference frame... Downsampling is performed at a ratio s, and the data is fed into the feature extractor E. F In order to obtain the latent features of the keyframe. As shown in equation (1): Where D represents the downsampling operation, represents the potential features of the keyframe, and s represents the downsampling ratio.

[0122] Then, the keyframe latent features Feed to the key weight predictor E W (not shown) and key bias predictor E B In (not shown), the first set of compact motion vectors is obtained, as shown in equations (2)-(3): in, and These are the reconstruction keyframes. The weight vector and bias vector. In some embodiments, the weight predictor or bias predictor is a module or algorithm that can transform two-dimensional spatial features into one-dimensional spatial vectors.

[0123] In step 1104, the inter-frame is decomposed into a second set of compact motion vectors. The inter-frame is represented as... The inter-frame P is downsampled at a ratio s and fed into the feature extractor E. F In order to obtain inter-frame latent features L P As shown in equation (4): L P =E F (D(P,s)) (4) Where D represents the downsampling operation, L P The term represents the inter-frame latent features, and s represents the downsampling ratio. In some embodiments, the reconstructed keyframe... The downsampling ratio is the same as the downsampling ratio of the inter-frame P. Used for the reconstruction keyframe. The feature extractor of the inter-frame P is the same as the feature extractor of the key-frame. In some embodiments, the key-frame latent feature and the inter-frame latent feature L P share the dimension N wherein, is the dimension of the latent feature of a single frame, N F is the number of channels of each latent feature (including the key-frame latent feature and the inter-frame latent feature). In some embodiments, one video sequence includes one key reference frame and multiple inter-frames, thus the number of latent features is equal to the number of inter-frames plus one.

[0124] The inter-frame latent feature L P is then fed into the inter-frame weight predictor E W 1013and the inter-frame bias predictor E B 1014to obtain a second set of compact motion vectors as shown in equations (5)-(6): w P = E W (L P ) (5) b P = E B (L P ) (6) wherein w P and b P are the weight vector and bias vector of the inter-frame, respectively. In some embodiments, w P , b P share the same dimension N F x 1.

[0125] In some embodiments, the second set of compact vectors are encoded and decoded by the model-based codec 920 and transmitted through feature bitstream. Figure 9

[0126] At step 1106, based on the first set of compact motion vectors and the second set of compact motion vectors, the key-frame latent feature is transformed into a first fine-grained motion field for the reconstructed key-frame and a second fine-grained motion field for the inter-frames.

[0127] In some embodiments, the multi-granularity motion transformation is implemented by modulating the key-frame latent feature with weight and bias after spatial feature transformation as shown in equations (7) and (8): wherein,​ is a fine-grained motion field for the reconstructed key frame, F P is a fine-grained motion field for the inter-frame. In some embodiments, the key fine-grained motion field for the reconstructed key reference frame and an inter-frame fine-grained motion field F P share dimensions and • denotes a channel-wise multiplication operation. The spatial feature transform can be performed by conventional methods, which are not elaborated here.

[0128] Reference Figure 10 , the multi-granularity feature decomposition 1011 includes a down-sampling unit for performing a down-sampling operation D, a feature extractor E F , a weight predictor E W and a bias predictor E B The structure of the multi-granularity feature decomposition 1021 in the decoder 1020 can be the same as that of the multi-granularity feature decomposition 1011 in the encoder 1010, Figure 10 The structure of the multi-granularity feature decomposition 1021 is not shown in detail.

[0129] It can be understood that in some embodiments, steps 1102 and 1106 are performed by the decoder 1020, and step 1104 is performed by the encoder 1010.

[0130] The proposed multi-granularity temporal trajectory decomposition (MTTF) enables the input frame to be decomposed not only into higher-dimensional space (e.g., with the number of latent features), but also into more diversified representations (e.g., using the predictors described above).

[0131] Figure 12 is a schematic diagram showing a generative video coding framework 1200 according to some disclosed embodiments. As Figure 12As shown, in the generative video coding framework 1200, an encoder 1210 compresses a key reference frame (e.g., the first frame of a video sequence) by using a video encoder 1211 (e.g., a VVC codec) and transmits the compressed key reference frame as an image bitstream. A feature decomposition module 1212 decomposes a plurality of sets of compact motion vectors associated with a plurality of inter frames from the plurality of inter frames and transmits as a feature bitstream. For a decoder 1220, the key reference frame is reconstructed by a video decoder 1221 (e.g., a VVC codec) to obtain a reconstructed key reference frame. Then, the reconstructed key reference frame is decomposed by a feature decomposition module 1222 into a key frame latent feature (e.g., a spatial latent feature) and a set of compact motion vectors. With the compact motion vectors associated with the reconstructed key reference frame and the plurality of inter frames, the key frame latent feature is transformed by the compact motion vectors to form two fine-grained motion fields by a multi-granularity motion transformation 1223. For example, each set of compact vectors from the key frame or inter frames is used as a modulation weight and bias to perform a spatial feature transformation on the key frame latent feature. Thus, the decoder 1220 explores the intrinsic correlation between two frames (e.g., the reconstructed key frame and the inter frame) by utilizing a spatial feature transformation to implicitly decompose the temporal trajectory information in the two frames into multi-granularity representations, i.e., compact motion vectors and fine-grained motion fields. Then, the fine-grained motion fields are fed into a motion predictor 1224 to predict dense motion 1225. Finally, the dense motion and the reconstructed key reference frame are utilized by a generator 1226 to animate the reconstructed inter frame.

[0132] Figure 13 is a flowchart of an exemplary video decoding method 1300 using multi-granularity temporal trajectory factorization (MTTF) according to some embodiments of the present disclosure. The method 1300 can be performed by a GFVC system (e.g., by a GFVC system 1200 of Figure 12 ), or by one or more software or hardware components of a device. In some embodiments, the method 1300 can be implemented by a computer program product comprising computer-executable instructions, e.g., a program code, which is executed by a computer and comprises the method. Referring to Figure 12 and Figure 13 , the method 1300 can comprise the following steps 1302 to 1312.

[0133] At step 1302, a picture bitstream associated with the video sequence is decoded, and a reconstructed key reference frame is obtained by decoding a key reference frame and reconstructing the decoded key reference frame. The key reference frame is encoded using a regular codec 1211 of the encoder 1210 and transmitted in the picture bitstream. The key reference frame is then decoded from the picture bitstream by a regular codec 1221 of the decoder 1220 to obtain a reconstructed key reference frame. The reconstructed key reference frame can be used to generate reconstructed inter frames.

[0134] At step 1304, the reconstructed key reference frame is decomposed into key frame latent features associated with the reconstructed key reference frame and a first set of compact motion vectors. The reconstructed key reference frame is decomposed by a feature decomposition module 1222 to obtain the first set of compact motion vectors (e.g., and ) and key frame latent features.

[0135] In some embodiments, step 1304 further comprises the steps of: downsampling the reconstructed key reference frame; feeding the downsampled reconstructed key reference frame to a feature extractor to obtain the key frame latent features; and feeding the key frame latent features into a weight predictor and a bias predictor, respectively, to obtain the first set of compact motion vectors, wherein the first set of compact motion vectors comprises a key weight vector and a key bias vector.

[0136] At step 1306, a feature bitstream associated with the video sequence is decoded to obtain a second set of compact motion vectors associated with inter frames. In some embodiments, the second set of compact motion vectors comprises an inter weight vector and an inter bias vector. A plurality of inter frames is decomposed by a feature decomposition module 1212 to obtain a plurality of second sets of compact motion vectors (e.g., P and P ). Each second set of compact motion vectors is associated with each inter frame. All second sets of compact motion vectors are encoded by the encoder 1210 and transmitted in the feature bitstream. The second sets of compact motion vectors are then decoded from the feature bitstream by the decoder 1220.

[0137] At step 1308, the key frame latent features are transformed into a first fine-grained motion field for the reconstructed key frame and a second fine-grained motion field for the inter frames based on the first set of compact motion vectors and the second set of compact motion vectors. The key frame latent features, the first set of compact motion vectors, and the second set of compact motion vectors are transformed by a multi-grained motion transformation 1223 to generate a key fine-grained motion field and an inter fine-grained motion field.

[0138] In some embodiments, step 1308 further includes modulating the key frame latent features with the key weight vector and the key bias vector to generate the first fine-grained motion field for the reconstructed key reference frame; and modulating the key frame latent features with the inter-frame weight vector and the inter-frame bias vector to generate the second fine-grained motion field for the inter-frame.

[0139] For details regarding steps 1302 to 1308, please refer to the descriptions of Figure 10 and Figure 11 , which are not repeated here.

[0140] At step 1310, dense motion is predicted based on the first fine-grained motion field and the second fine-grained motion field. The dense motion 1225 is predicted by a motion predictor 1224 based on the key fine-grained motion field and the inter-frame fine-grained motion field.

[0141] At step 1312, the inter-frame is generated based on the dense motion and the reconstructed key reference frame. The inter-frame is generated by a generator 1226 based on the dense motion 1225 and the reconstructed key reference frame.

[0142] In some embodiments, the video sequence includes a plurality of inter-frames, and dimensions of the first fine-grained motion field and the second fine-grained motion field are based on a number of the inter-frames.

[0143] In some embodiments, dimensions of the first set of compact motion vectors and the second set of compact motion vectors are based on the number of the inter-frames.

[0144] Figure 14 is a flowchart of an exemplary video encoding method 1400 using multi-granularity temporal trajectory factorization (MTTF) according to some embodiments of the present disclosure. The method 1400 can be performed by a GFVC system (e.g., by the GFVC system 1200 of Figure 12 ), or by one or more software or hardware components of a device. In some embodiments, the method 1400 can be implemented by a computer program product included in a computer readable medium, the computer program product including computer executable instructions, such as program code, executed by a computer. Referring to Figure 12 and Figure 14 , the method 1400 can include the following steps 1402 to 1410.

[0145] At step 1402, an image bitstream is encoded, and the image bitstream includes encoded information for a key reference frame of a video sequence. The encoded information of the image bitstream is factorized into key frame latent features associated with a reconstructed key reference frame and a first set of compact motion vectors.

[0146] At step 1404, a feature bitstream is encoded, and the feature bitstream includes encoded information for inter frames of the video sequence. The encoded information of the feature bitstream includes a second set of compact motion vectors associated with the inter frames. Based on the first set of compact motion vectors and the second set of compact motion vectors, a first fine-grained motion field for the reconstructed key reference frame and a second fine-grained motion field for the inter frames are generated by transforming the key frame latent features. The first fine-grained motion field and the second fine-grained motion field are used for predicting dense motion, and the dense motion is used to generate the inter frames.

[0147] In some embodiments, step 1404 further includes down-sampling the inter frames; feeding the down-sampled inter frames to a feature extractor to obtain inter frame latent features; feeding the inter frame latent features into a weight predictor and a bias predictor, respectively, to obtain the second set of compact motion vectors, wherein the second set of compact motion vectors includes an inter frame weight vector and an inter frame bias vector; and encoding the second set of compact motion vectors.

[0148] For details of steps 1402 and 1404, please refer to the descriptions of Figure 10 and Figure 11 , which are not repeated here.

[0149] In some embodiments, the video sequence includes a plurality of inter frames, and dimensions of the first fine-grained motion field and the second fine-grained motion field are based on a number of the inter frames.

[0150] In some embodiments, dimensions of the first set of compact motion vectors and the second set of compact motion vectors are based on the number of the inter frames.

[0151] Figure 15 is a block diagram of an exemplary apparatus 1500 for encoding image data according to some embodiments of the present disclosure. The apparatus 1500 can be used to perform the video compression method described above. As Figure 15As shown, the apparatus 1500 can include a processor 1502. When the processor 1502 executes instructions as described herein, the apparatus 1500 can become a special purpose machine for video encoding or decoding. The processor 1502 can be any type of circuitry capable of manipulating or processing information. For example, the processor 1502 can include any combination of central processing units (or “CPUs”), graphics processing units (or “GPUs”), neural processing units (“NPUs”), microcontroller units (“MCUs”), optical processors, programmable logic controllers, microcontrollers, microprocessors, digital signal processors, intellectual property (IP) cores, programmable logic arrays (PLAs), programmable array logic (PAL), generic array logic (GAL), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), systems on a chip (SoCs), application-specific integrated circuits (ASICs), and the like. In some embodiments, the processor 1502 can also be a group of processors grouped as a single logical component. For example, as Figure 15 shown, the processor 1502 can include multiple processors, including a processor 1502a, a processor 1502b, and a processor 1502n.

[0152] The apparatus 1500 can also include a memory 1504 configured to store data (e.g., instruction sets, computer code, intermediate data, and the like). For example, as Figure 15 shown, the stored data can include program instructions (e.g., program instructions for implementing the methods described in this disclosure). The processor 1502 can access the program instructions and data for processing (e.g., via a bus 1510) and execute the program instructions to perform operations or manipulations on the data for processing. The memory 1504 can include a high-speed random access memory or a non-volatile storage device. In some embodiments, the memory 1504 can include any combination of any number of random access memories (RAMs), read only memories (ROMs), optical disks, magnetic disks, hard drives, solid-state drives, flash drives, secure digital (SD) cards, memory sticks, compact flash (CF) cards, and the like. The memory 1504 can also be a group of memories grouped as a single logical component (not shown in FIG. 15). Figure 15

[0153] The bus 1510 can be a communication device that transfers data between components within the apparatus 1500, such as an internal bus (e.g., a CPU-memory bus), an external bus (e.g., a universal serial bus port, a peripheral component interconnect express port), and the like.

[0154] ​For ease of explanation and without causing ambiguity, the processor 1502 and other data processing circuitry are collectively referred to as “data processing circuitry” in this disclosure. The data processing circuitry can be implemented entirely as hardware, or as a combination of software, hardware, or firmware. Moreover, the data processing circuitry can be a single standalone module, or can be combined in whole or in part into any other component of the apparatus 1500.

[0155] The apparatus 1500 can also include a network interface 1506 to provide wired or wireless communication with a network (e.g., the Internet, an intranet, a local area network, a mobile communication network, etc.). In some embodiments, the network interface 1506 can include any combination of any number of network interface controllers (NICs), radio frequency (RF) modules, transponders, transceivers, modems, routers, gateways, wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (“NFC”) adapters, cellular network chips, etc.

[0156] In some embodiments, the apparatus 1500 can also include a peripheral interface 1508 to provide connectivity to one or more peripheral devices. As shown, the peripheral devices can include, but are not limited to, a cursor control device (e.g., a mouse, touchpad, or touchscreen), a keyboard, a display (e.g., a cathode ray tube display, a liquid crystal display, or a light-emitting diode display), a video input device (e.g., a camera or an input interface coupled to a video archive), etc. Figure 15

[0157] It should be noted that a video codec consistent with this disclosure can be implemented as any combination of any number of software or hardware modules in the apparatus 1500. For example, some or all stages of the disclosed methods can be implemented as one or more software modules of the apparatus 1500, e.g., program instructions that can be loaded into the memory 1504. For another example, some or all stages of the disclosed methods can be implemented as one or more hardware modules of the apparatus 1500, e.g., a special-purpose data processing circuit (e.g., an FPGA, an ASIC, an NPU, etc.).

[0158] In some embodiments, a method of transmitting a bitstream is provided. The method includes receiving a video sequence, encoding the video sequence by the above-described method (e.g., the method 1400( Figure 14 ), and transmitting an image bitstream and a feature bitstream generated based on the encoding.

[0159] In some embodiments, a non-transitory computer-readable storage medium storing a bitstream is also provided. The bitstream can be encoded and decoded according to the above-described method using multi-granularity temporal trajectory factorization (MTTF). For example, the bitstream can include an image bitstream and a feature bitstream generated based on the above-described method (e.g., the method 1400​Figure 14 )) encoded image bitstream and feature bitstream.

[0160] In some embodiments, a non-transitory computer-readable storage medium comprising instructions is also provided, and the instructions can be executed by a device (e.g., the disclosed encoder and decoder) for performing the above-mentioned methods. Common forms of non-transitory media include, for example, a floppy disk, a flexible disk, a hard disk, a solid-state drive, a magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM or any other flash memory, NVRAM, a cache, a register, any other memory chip or cartridge, and a networked version of any of the foregoing. The device can include one or more processors (CPUs), input / output interfaces, network interfaces, and / or memory.

[0161] It should be noted that the embodiments described in the present disclosure can be used freely in combination or individually.

[0162] In summary, the above-mentioned method and framework using multi-granularity temporal trajectory factorization (MTTF) have the following technical features.

[0163] In contrast to latent animators (LIA) that rely on a set of motion vectors shared among all inputs for motion decomposition, using the MTTF, the input images are decomposed into a multi-channel trajectory representation, enabling each dimension to implicitly encode different motion patterns. This decomposition is learnable, spatially-aware, and self-adapting to the inputs.

[0164] In contrast to the compact features used in CFTE, which employ a highly compressed 4x4 matrix per frame, the proposed MTTF leverages the keyframe latent features as a basis to construct fine-grained motion fields. This approach captures richer appearance information for motion representation, provides greater spatial expressiveness, and enables zero-bit extraction from VVC-decoded keyframes.

[0165] With the proposed MTTF, only the compact motion vectors associated with the interframe are needed to be encoded and transmitted, which are the transform coefficients for multi-granularity motion transformation. Therefore, the encoded part is still compact enough to meet the requirement of ultra-low bitrate encoding.

[0166] Embodiments can be further described using the following clauses: 1. A video decoding method comprising: decoding an image bitstream, the image bitstream being associated with a video sequence, wherein the decoding reconstructs a key reference frame of the image bitstream; decomposing the reconstructed key reference frame into key frame latent features associated with the reconstructed key reference frame and a first set of compact motion vectors; decoding a feature bitstream associated with the video sequence to obtain a second set of compact motion vectors associated with an inter frame; transforming the key frame latent features into a first fine-grained motion field for the reconstructed key reference frame and a second fine-grained motion field for the inter frame based on the first set of compact motion vectors and the second set of compact motion vectors; predicting dense motion based on the first fine-grained motion field and the second fine-grained motion field; and generating the inter frame based on the dense motion and the reconstructed key reference frame. 2. The method of clause 1, wherein decomposing the reconstructed key reference frame into the key frame latent features and the first set of compact motion vectors further comprises: downsampling the reconstructed key reference frame; feeding the downsampled reconstructed key reference frame to a feature extractor to obtain the key frame latent features; and feeding the key frame latent features into a weight predictor and a bias predictor, respectively, to obtain the first set of compact motion vectors, wherein the first set of compact motion vectors comprises a key weight vector and a key bias vector. 3. The method of clause 2, wherein the second set of compact motion vectors comprises an inter weight vector and an inter bias vector. 4. The method of clause 3, wherein transforming the key frame latent features into the first fine-grained motion field for the reconstructed key reference frame and the second fine-grained motion field for the inter frame based on the first set of compact motion vectors and the second set of compact motion vectors further comprises: modulating the key frame latent features with the key weight vector and the key bias vector to generate the first fine-grained motion field for the reconstructed key reference frame; and modulating the key frame latent features with the inter weight vector and the inter bias vector to generate the second fine-grained motion field for the inter frame. 5. The method of clause 1, wherein the video sequence comprises a plurality of inter frames, and dimensions of the first fine-grained motion field and the second fine-grained motion field are based on a number of the inter frames. 6. The method of clause 5, wherein dimensions of the first set of compact motion vectors and the second set of compact motion vectors are based on the number of the inter frames. 7. A method of video encoding, comprising: an encoded image bitstream comprising encoded information for key reference frames of a video sequence, wherein the encoded information of the image bitstream is decomposable into key frame latent features associated with reconstructed key reference frames and a first set of compact motion vectors; and an encoded feature bitstream comprising encoded information for inter frames of the video sequence, wherein the encoded information of the feature bitstream comprises a second set of compact motion vectors associated with the inter frames; wherein, based on the first set of compact motion vectors and the second set of compact motion vectors, a first fine-grained motion field for the reconstructed key reference frames and a second fine-grained motion field for the inter frames are generated by transforming the key frame latent features; wherein the first fine-grained motion field and the second fine-grained motion field are used for predicting dense motion; and wherein the dense motion is used to generate the inter frames. 8. The method of clause 7, wherein the encoded feature bitstream comprising encoded information for the inter frames further comprises: down-sampling the inter frames; feeding the down-sampled inter frames to a feature extractor to obtain inter frame latent features; feeding the inter frame latent features into a weight predictor and a bias predictor, respectively, to obtain the second set of compact motion vectors, wherein the second set of compact motion vectors comprises an inter frame weight vector and an inter frame bias vector; and encoding the second set of compact motion vectors. 9. The method of clause 7, wherein the video sequence comprises a plurality of inter frames, and dimensions of the first fine-grained motion field and the second fine-grained motion field are based on a number of the inter frames. 10. The method of clause 9, wherein dimensions of the first set of compact motion vectors and the second set of compact motion vectors are based on the number of the inter frames. 11. A method for transmitting a bitstream, the method comprising: receiving a video sequence; encoding the video sequence by: encoding an image bitstream comprising encoded information for key reference frames of a video sequence, wherein the encoded information of the image bitstream is decomposable into key frame latent features associated with reconstructed key reference frames and a first set of compact motion vectors; and encoding a feature bitstream, the feature bitstream comprising encoded information for an inter-frame of the video sequence, wherein the encoded information of the feature bitstream comprises a second set of compact motion vectors associated with the inter-frame; wherein, based on the first set of compact motion vectors and the second set of compact motion vectors, a first fine-grained motion field for the reconstructed key reference frame and a second fine-grained motion field for the inter-frame are generated by transforming the key-frame latent features; wherein, the first fine-grained motion field and the second fine-grained motion field are used to predict dense motion; and wherein, the dense motion is used to generate the inter-frame; and transmitting the image bitstream and the feature bitstream generated based on the encoding. 12. The method of clause 11, wherein encoding the feature bitstream, the feature bitstream comprising encoded information for the inter-frame further comprises: down-sampling the inter-frame; feeding the down-sampled inter-frame to a feature extractor to obtain inter-frame latent features; and feeding the inter-frame latent features into a weight predictor and a bias predictor, respectively, to obtain the second set of compact motion vectors, wherein the second set of compact motion vectors comprises an inter-frame weight vector and an inter-frame bias vector. 13. The method of clause 11, wherein the video sequence comprises a plurality of inter-frames, and dimensions of the first fine-grained motion field and the second fine-grained motion field are based on a number of the inter-frames. 14. The method of clause 13, wherein dimensions of the first set of compact motion vectors and the second set of compact motion vectors are based on the number of the inter-frames. 15. A method for multi-granularity temporal trajectory factorization (MTTF) of a video sequence, comprising: factorizing a reconstructed key reference frame into key-frame latent features and a first set of compact motion vectors; factorizing an inter-frame into a second set of compact motion vectors; and transforming, based on the first set of compact motion vectors and the second set of compact motion vectors, the key-frame latent features into a first fine-grained motion field for the reconstructed key reference frame and a second fine-grained motion field for the inter-frame. 16. The method of clause 15, wherein factorizing the reconstructed key reference frame into the key-frame latent features and the first set of compact motion vectors further comprises: down-sampling the reconstructed key reference frame; feeding the down-sampled reconstructed key reference frame to a first feature extractor to obtain the key frame latent features; and feeding the key frame latent features into a key weight predictor and a key bias predictor, respectively, to obtain the first set of compact motion vectors, wherein the first set of compact motion vectors comprises a key weight vector and a key bias vector. 17. The method of clause 16, wherein decomposing the inter-frame into the second set of compact motion vectors comprises: down-sampling the inter-frame; feeding the down-sampled inter-frame to a second feature extractor to obtain inter-frame latent features; and feeding the inter-frame latent features into an inter-frame weight predictor and an inter-frame bias predictor, respectively, to obtain the second set of compact motion vectors, wherein the second set of compact motion vectors comprises an inter-frame weight vector and an inter-frame bias vector. 18. The method of clause 17, wherein transforming the key frame latent features into the first fine-grained motion field for the reconstructed key reference frame and the second fine-grained motion field for the inter-coded frame based on the first set of compact motion vectors and the second set of compact motion vectors further comprises: modulating the key frame latent features with the key weight vector and the key bias vector to generate the first fine-grained motion field for the reconstructed key reference frame; and modulating the key frame latent features with the inter-frame weight vector and the inter-frame bias vector to generate the second fine-grained motion field for the inter-frame. 19. The method of clause 17, wherein dimensions of the first fine-grained motion field and the second fine-grained motion field are based on a number of inter-frame latent features. 20. The method of clause 19, wherein dimensions of the first set of compact motion vectors and the second set of compact motion vectors are based on the number of inter-frame latent features. 21. The method of clause 17, wherein the first feature extractor and the second feature extractor are the same. 22. The method of clause 17, wherein the reconstructed key reference frame and the inter-frame are down-sampled at a same rate.

[0167] It should be noted that the relational terms herein, such as first, second, and the like, are used solely to distinguish one from another entity or action without necessarily implying any actual relationship or order between such entities or actions. Moreover, the words "comprising," "having," "containing," and "including," and other similar forms, are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, nor is it meant to be limited to only the listed item or items.

[0168] As used herein, unless specifically stated otherwise, the term "or" encompasses all possible combinations. For example, if a database is stated as including A or B, then unless specifically stated otherwise, a database can include A; or B; or A and B. As a second example, if a database is stated as including A, B, or C, then unless specifically stated otherwise, a database can include A; or B; or C; or A and B; or A and C; or B and C; or A and B and C.

[0169] It should be understood that the above-described embodiments can be implemented by hardware, or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above-described computer-readable medium. The software, when executed by a processor, can perform the disclosed methods. The computing units and other functional units described in the disclosure can be implemented by hardware, or software, or a combination of hardware and software. Those of ordinary skill in the art should also understand that a plurality of the above-described modules / units can be combined into one module / unit, and each of the above-described modules / units can be further divided into a plurality of sub-modules / sub-units.

[0170] In the above specification, the embodiments have been described with reference to numerous specific details that can be varied in accordance with implementation. Certain adjustments and modifications can be made to the described embodiments. Other embodiments will be apparent to those of ordinary skill in the art from a consideration of the specification and practice of the invention disclosed herein. The specification and examples are to be considered exemplary only, with the true scope and spirit of the invention indicated by the appended claims. The order of steps shown in the figures is also considered to be merely illustrative, and not intended to be limiting to any particular order of steps. Thus, those of ordinary skill in the art will appreciate that the steps can be performed in different orders while implementing the same method.

[0171] In the drawings and specification, there have been disclosed exemplary embodiments. However, many variations and modifications can be made to these embodiments. Consequently, it is intended that the scope of the application be limited only by the broadest interpretation of the appended claims to be accorded under 35 U.S.C. § 112.

Claims

1. A video decoding method, comprising: Decode an image bitstream, the image bitstream being associated with a video sequence, wherein the decoding of the image bitstream is used to reconstruct key reference frames; The reconstructed key reference frame is decomposed into key frame latent features associated with the reconstructed key reference frame and a first set of compact motion vectors; Decode the feature bitstream, which is associated with the video sequence, to obtain a second set of compact motion vectors associated with inter-frame frames; Based on the first set of compact motion vectors and the second set of compact motion vectors, the latent features of the key frame are transformed into a first fine-grained motion field for the reconstructed key reference frame and a second fine-grained motion field for the inter-frame frame. Predicting dense motion based on the first and second fine-grained motion fields; and The inter-frames are generated based on the dense motion and the reconstructed key reference frames.

2. The method according to claim 1, wherein, Decomposing the reconstructed key reference frame into the key frame latent features and the first set of compact motion vectors further includes: The key reference frames for reconstruction are downsampled; The downsampled reconstructed key reference frame is fed into a feature extractor to obtain the latent features of the key frame; and The keyframe latent features are fed into a weight predictor and a bias predictor, respectively, to obtain the first set of compact motion vectors, wherein the first set of compact motion vectors includes a key weight vector and a key bias vector.

3. The method according to claim 2, wherein, The second set of compact motion vectors includes inter-frame weight vectors and inter-frame bias vectors.

4. The method according to claim 3, wherein, Based on the first set of compact motion vectors and the second set of compact motion vectors, transforming the latent features of the keyframes into a first fine-grained motion field for reconstructing the key reference frame and a second fine-grained motion field for the inter-frames further includes: The keyframe latent features are modulated using the key weight vector and the key bias vector to generate the first fine-grained motion field for the reconstructed key reference frame; and The keyframe latent features are modulated using the inter-frame weight vector and the inter-frame bias vector to generate the second fine-grained motion field for the inter-frame frames.

5. The method according to claim 1, wherein, The video sequence includes multiple inter-frame frames, and the dimensions of the first fine-grained motion field and the second fine-grained motion field are based on the number of inter-frame frames.

6. The method according to claim 5, wherein, The dimensions of the first set of compact motion vectors and the second set of compact motion vectors are based on the number of inter-frame frames.

7. A video encoding method, comprising: An encoded image bitstream, the image bitstream including encoded information for key reference frames of a video sequence, wherein the encoded information of the image bitstream is decomposed into key frame latent features associated with the reconstructed key reference frames and a first set of compact motion vectors; and Encode a feature bitstream, the feature bitstream including encoded information for inter-frames of the video sequence, wherein the encoded information of the feature bitstream includes a second set of compact motion vectors associated with the inter-frames; Specifically, based on the first set of compact motion vectors and the second set of compact motion vectors, a first fine-grained motion field for the reconstructed key reference frame and a second fine-grained motion field for the inter-frame frame are generated by transforming the latent features of the key frame. The first fine-grained motion field and the second fine-grained motion field are used to predict dense motion; and The dense motion is used to generate the inter-frames.

8. The method according to claim 7, wherein, An encoded feature bitstream, the feature bitstream including encoded information for the inter-frames, and further including: The inter-frames are downsampled; The downsampled inter-frames are fed into a feature extractor to obtain the potential features of the inter-frames. The inter-frame latent features are fed into a weight predictor and a bias predictor, respectively, to obtain the second set of compact motion vectors, wherein the second set of compact motion vectors includes inter-frame weight vectors and inter-frame bias vectors; and The second set of compact motion vectors is encoded.

9. The method according to claim 7, wherein, The video sequence includes multiple inter-frame frames, and the dimensions of the first fine-grained motion field and the second fine-grained motion field are based on the number of inter-frame frames.

10. The method according to claim 9, wherein, The dimensions of the first set of compact motion vectors and the second set of compact motion vectors are based on the number of inter-frame frames.

11. A method for transmitting a bit stream, the method comprising: Receive video sequences; The video sequence is encoded in the following manner: An encoded image bitstream, the image bitstream including encoded information for key reference frames of a video sequence, wherein the encoded information of the image bitstream is decomposed into key frame latent features associated with the reconstructed key reference frames and a first set of compact motion vectors; and Encode a feature bitstream, the feature bitstream including encoded information for inter-frames of the video sequence, wherein the encoded information of the feature bitstream includes a second set of compact motion vectors associated with the inter-frames; Specifically, based on the first set of compact motion vectors and the second set of compact motion vectors, a first fine-grained motion field for the reconstructed key reference frame and a second fine-grained motion field for the inter-frame frame are generated by transforming the latent features of the key frame. The first fine-grained motion field and the second fine-grained motion field are used to predict dense motion; and Wherein, the dense motion is used to generate the inter-frame frames; and The image bitstream and the feature bitstream generated based on the encoding are transmitted.

12. The method according to claim 11, wherein, The encoded feature bitstream, which includes encoded information for the inter-frames, further includes: The inter-frames are downsampled; The downsampled inter-frames are fed into the feature extractor to obtain latent features of the inter-frames; and The inter-frame latent features are fed into the weight predictor and the bias predictor respectively to obtain the second set of compact motion vectors, wherein the second set of compact motion vectors includes inter-frame weight vectors and inter-frame bias vectors.

13. The method according to claim 11, wherein, The video sequence includes multiple inter-frame frames, and the dimensions of the first fine-grained motion field and the second fine-grained motion field are based on the number of inter-frame frames.

14. The method according to claim 13, wherein, The dimensions of the first set of compact motion vectors and the second set of compact motion vectors are based on the number of inter-frame frames.

15. A non-transitory computer-readable storage medium storing an instruction set, an image bitstream, and a feature bitstream thereon, the instruction set being executable by one or more processors in a method to decode the image bitstream and the feature bitstream, the method comprising: Decode an image bitstream, the image bitstream being associated with a video sequence, wherein the decoding of the image bitstream is used to reconstruct key reference frames; The reconstructed key reference frame is decomposed into key frame latent features associated with the reconstructed key reference frame and a first set of compact motion vectors; Decode the feature bitstream, which is associated with the video sequence, to obtain a second set of compact motion vectors associated with inter-frame frames; Based on the first set of compact motion vectors and the second set of compact motion vectors, the latent features of the key frame are transformed into a first fine-grained motion field for the reconstructed key reference frame and a second fine-grained motion field for the inter-frame frame. Predicting dense motion based on the first and second fine-grained motion fields; and The inter-frames are generated based on the dense motion and the reconstructed key reference frames.

Images