Tokenized video compression with text-guided referenced video generation

Abstract

A method performed by an encoder that processes consecutive groups of pictures (GoPs). One or more reference frames are selected from each GoP, and remaining frames are designated as non-reference frames. The reference frames are patchified and encoded to generate discrete reference latent features. A subset of discrete reference tokens is selected from a discrete video token codebook, and a reference token mask is generated. The latent features are mapped to the selected tokens to produce discrete reference token indices, which are partially masked. Token index differences between consecutive GoPs are determined. Masked token indices are predicted using a token prediction network. A text description associated with the GoPs is encoded into tokenized textual latent indices. Reconstructed non-reference frames are generated based on predicted discrete tokens and textual latents. Residuals between original and reconstructed non-reference frames are encoded together with token information for transmission toward a decoder.

Description

Atty. Docket: 4502-86501 (6000765PCT02)Tokenized Video Compression with Text-Guided Referenced Video GenerationCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 845,126 filed on July 16, 2025, which is hereby incorporated by reference.TECHNICAL FIELD

[0002] The present disclosure describes techniques for learned video compression. More specifically, this disclosure is related to video compression by tokenized video generation.BACKGROUND

[0003] Artificial intelligence generated content (AIGC) uses a wide range of image generative models, including generative adversarial networks (GAN) and diffusion models. The goal is to enable fast and accessible high-quality content creation. Various methods have been developed to allow for efficient manipulation of the generated content using different types of inputs, such as using text descriptions and / or spatial / spatiotemporal compositions, like sketches or segmentations.

[0004] Large-scale pretrained vision-language models (VLM) have reached a milestone in text-to-image generation for AIGC. By training a very large model using very large datasets of captioned images from the internet, a multi-modal language-image pre-training representation, like contrastive language-image pre-training (CLIP) can be successfully learned through self-supervised contrastive learning. The joint embedding space of text and image is robust to image distribution shift.SUMMARY

[0005] The disclosed embodiments provide a general framework that jointly uses discrete and continuous video tokenizers, text tokenizers, data-adaptive token selection, masked token prediction, as well as text-guided video generation, for video compression. Text-guide video generation is performed inside a Group of Picture (GoP) where tokenized discrete (and optionally continuous) video representations of selected reference frames are used to generate non-reference frames to reduce transmission overhead, and non-reference residuals are leveraged to balance compression rate and reconstruction quality. In addition, text prompts are optionally adjusted inAtty. Docket: 4502-86501 (6000765PCT02)the encoder to further improve the reconstruction quality according to individual input GoPs, without increasing transmitted bits.

[0006] A first aspect relates to a method implemented by an encoder, comprising selecting one or more reference frames from a first group of pictures (GoP) and a second GoP of a video sequence, wherein remaining frames from the first GoP and the second GoP comprise first nonreference frames; patchifying the one or more reference frames to generate patchified reference frames; encoding the patchified reference frames using a discrete video tokenizer to generate discrete reference latent features; selecting, based on the discrete reference latent features, a subset of discrete reference tokens from a pre-trained discrete video token codebook and generating a reference token mask indicating the subset of discrete reference tokens; mapping the discrete reference latent features to the subset of discrete reference tokens to generate discrete reference token indices for the first GoP and the second GoP; applying one or more masks to the discrete reference token indices to generate masked discrete reference token indices; determining token index differences between the first GoP and the second GoP based on the masked discrete reference token indices; and encoding the masked discrete reference token indices, the reference token mask, and the token index differences for transmission toward a decoder.

[0007] Optionally, in any of the preceding aspects, another implementation of the aspect provides encoding the patchified reference frames to generate continuous reference latent features; determining a reference feature mask based on the continuous reference latent features; determining tokenized reference latent representations using a pre-trained continuous video tokenizer based on the continuous reference latent features and the reference feature mask; and encoding the tokenized reference latent representations for transmission toward the decoder.

[0008] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the pre-trained continuous video tokenizer comprises a continuous-valued autoencoder that compresses the one or more reference frames into a continuous latent space.

[0009] Optionally, in any of the preceding aspects, another implementation of the aspect provides generating predicted discrete reference token indices from the masked discrete reference token indices using a token prediction network; receiving a text description of the first GoP and the second GoP and encoding the text description into tokenized textual latent indices; generating reconstructed non-reference frames using the predicted discrete reference token indices and the tokenized textual latent indices; determining non-reference residuals based on differences betweenAtty. Docket: 4502-86501 (6000765PCT02)the first non-reference frames and the reconstructed non-reference frames; and encoding the masked discrete reference token indices, the reference token mask, the token index differences, the tokenized textual latent indices, and the non-reference residuals for transmission toward a decoder, wherein the token index differences represent token index variations between the first GoP and the second GoP.

[0010] Optionally, in any of the preceding aspects, another implementation of the aspect provides encoding the masked discrete reference token indices into a discrete reference indices string; encoding the token index differences into a difference indices string; encoding the tokenized textual latent indices into a textual string; and encoding the non-reference residuals into a residual string.

[0011] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the second GoP is temporally subsequent to the first GoP in the video sequence.

[0012] Optionally, in any of the preceding aspects, another implementation of the aspect provides adjusting the text description comprises processing the text description to compute a text latent feature; updating the text latent feature using a gradient of the loss function; and generating an updated text description from the updated text latent feature using a diffusion-based language model.

[0013] Optionally, in any of the preceding aspects, another implementation of the aspect provides encoding the patchified reference frames comprises using a neural network encoder trained to extract spatial and temporal features from the one or more reference frames.

[0014] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the discrete reference latent features comprise a plurality of latent feature vectors, and wherein the mapping comprises selecting a token closest to each latent feature vector according to a distance metric.

[0015] Optionally, in any of the preceding aspects, another implementation of the aspect provides applying the masks to the discrete reference token indices comprises masking out elements in the discrete reference token indices according to a predefined masking ratio, and wherein masked-out elements are set to a symbol and are not transmitted.

[0016] Optionally, in any of the preceding aspects, another implementation of the aspect provides generating the discrete reference token indices comprises processing the masked discrete reference token indices through a transformer encoder to generate contextual embeddings; andAtty. Docket: 4502-86501 (6000765PCT02)decoding the contextual embeddings through a transformer decoder to predict values of the masked-out elements.

[0017] Optionally, in any of the preceding aspects, another implementation of the aspect provides selecting the subset of discrete reference tokens comprises using a pre-trained neural network model that predicts most relevant tokens to represent video content based on the discrete reference latent features.

[0018] Optionally, in any of the preceding aspects, another implementation of the aspect provides generating the reconstructed non-reference frames comprises: inputting the predicted discrete reference token indices and the tokenized textual latent indices into a pre-trained video generation model; and synthesizing the reconstructed non-reference frames conditioned on both the predicted discrete reference token indices and the tokenized textual latent indices

[0019] A second aspect relates to an encoder comprising a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the encoder to perform the method in any of the disclosed embodiments.

[0020] A third aspect relates to a method comprising receiving a discrete reference indices string, a reference token mask, and a difference indices string; decoding, using the reference token mask, the discrete reference indices string to compute first masked discrete reference token indices for a first group of pictures (GoP); decoding the difference indices string to compute token index differences; determining second masked discrete reference token indices for a second GoP based on the first masked discrete reference token indices and the token index differences; and outputting the frames of the first GoP and the second GoP based on the first masked discrete reference token indices and the second masked discrete reference token indices.

[0021] Optionally, in any of the preceding aspects, another implementation of the aspect provides decoding the discrete reference indices string comprises applying lossless integer entropy decoding to recover the first masked discrete reference token indices.

[0022] Optionally, in any of the preceding aspects, another implementation of the aspect provides generating predicted discrete reference token indices from the first masked discrete reference token indices and the second masked discrete reference token indices using a token prediction network, wherein generating the discrete reference token indices comprises: processing the first masked discrete reference token indices and the second masked discrete reference tokenAtty. Docket: 4502-86501 (6000765PCT02)indices using a transformer encoder to generate contextual embeddings; and decoding the contextual embeddings using a transformer decoder to predict values for masked-out elements in the first masked discrete reference token indices and the second masked discrete reference token indices; decoding the textual string to recover tokenized textual latent indices; generating reconstructed non-reference frames based on the predicted discrete reference token indices and the tokenized textual latent indices; decoding the residual string to recover non-reference residuals; reconstructing final non-reference frames by combining the reconstructed non-reference frames with the non-reference residuals; and outputting the final non-reference frames

[0023] Optionally, in any of the preceding aspects, another implementation of the aspect provides generating the reconstructed non-reference frames comprises using a pre-trained video generation model based on the predicted discrete reference token indices and the tokenized textual latent indices.

[0024] Optionally, in any of the preceding aspects, another implementation of the aspect provides wherein recovering the non-reference residuals comprises applying entropy decoding to the residual string to obtain quantized residuals; and dequantizing the quantized residuals to recover the non-reference residuals.

[0025] Optionally, in any of the preceding aspects, another implementation of the aspect provides wherein reconstructing the final non-reference frames comprises adding the non-reference residuals to the reconstructed non-reference frames to produce intermediate non-reference frames; and processing the intermediate non-reference frames using a post-processing neural network to compute the final non-reference frames.

[0026] Optionally, in any of the preceding aspects, another implementation of the aspect provides depatchifying the reconstructed non-reference frames by reordering pixel blocks according to a reverse patchification process.

[0027] Optionally, in any of the preceding aspects, another implementation of the aspect provides receiving and decoding a continuous latent string to recover tokenized reference latent representations; and generating the reconstructed non-reference frames further based on the recovered tokenized reference latent representations.

[0028] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the tokenized textual latent indices represent natural language text describing visual content, actions, objects, or scenes present in the first GoP and the second GoP.Atty. Docket: 4502-86501 (6000765PCT02)

[0029] A fourth aspect relates to a decoder comprising a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the decoder to perform the method in any of the disclosed embodiments.

[0030] A fifth aspect relates to a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computerexecutable instructions when executed by one or more processors of an encoder, cause the apparatus to perform the method in any of the disclosed embodiments.

[0031] A sixth aspect relates to a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computerexecutable instructions when executed by one or more processors of a decoder, cause the apparatus to perform the method in any of the disclosed embodiments.

[0032] A seventh aspect relates to a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause an encoder to perform operations in any of the disclosed embodiments.

[0033] An eighth aspect relates to a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause a decoder to perform operations in any of the disclosed embodiments.

[0034] For the purpose of clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create a new embodiment within the scope of the present disclosure.

[0035] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0036] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0037] FIG. 1 is a diagram illustrating a general framework for tokenized visual representation (TVR) according to an embodiment of the present disclosure.Atty. Docket: 4502-86501 (6000765PCT02)

[0038] FIGS. 2A-2B illustrate an encoding framework according to an embodiment of the present disclosure.

[0039] FIGS. 3A-3B illustrate an encoding framework according to an embodiment of the present disclosure.

[0040] FIG. 4 illustrates a decoding framework according to an embodiment of the present disclosure.

[0041] FIG. 5 illustrates a decoding framework according to an embodiment of the present disclosure.

[0042] FIG. 6 illustrates a prompt adjustment process according to an embodiment of the present disclosure.

[0043] FIG. 7 illustrates a flowchart according to an embodiment of the present disclosure.

[0044] FIG. 8 illustrates a flowchart according to an embodiment of the present disclosure.

[0045] FIG. 9 is a diagram illustrating an apparatus according to an embodiment of the present disclosure.DETAILED DESCRIPTION

[0046] It should be understood at the outset that although an illustrative implementation of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0047] The disclosed embodiments provide a general framework that jointly uses discrete and continuous video tokenizers, text tokenizers, data-adaptive token selection, masked token prediction, as well as text-guided video generation, for video compression. Text-guide video generation is performed inside a Group of Picture (GoP) where tokenized discrete (and optionally continuous) video representations of selected reference frames are used to generate non-reference frames to reduce transmission overhead, and non-reference residuals are leveraged to balance compression rate and reconstruction quality. In addition, text prompts are optionally adjusted inAtty. Docket: 4502-86501 (6000765PCT02)the encoder to further improve the reconstruction quality according to individual input GoPs, without increasing transmitted bits.

[0048] The disclosed framework is applicable to a wide range of practical video compression and transmission scenarios where high visual quality and reduced bandwidth are critical. The technology may be used in video streaming platforms, cloud-based video storage systems, video conferencing services, remote collaboration tools, and virtual or augmented reality applications. By generating non-reference frames at the decoder using discrete tokenized representations and text-guided video generation, the disclosed framework significantly reduces transmission overhead while maintaining perceptual quality. The residual coding mechanism enables flexible rate control, allowing adaptation to varying network conditions. The optional continuous latent stream enhances reconstruction fidelity for professional content creation, post-production workflows, and archival storage. Additionally, the adaptive text adjustment mechanism permits offline optimization for storage applications without increasing decoding complexity, making the approach suitable for large-scale content distribution systems where compressed video is encoded once and decoded multiple times across heterogeneous devices.

[0049] A general framework for AIGC framework can be described as follows. A prompt input y is passed through a prompt encoder. The prompt input y may be a text input that provides a description of content to be generated using Al. In some embodiments, the prompt input y may also include images related to the to-be-generated Al content. The prompt encoder is a component configured to encode the input prompts y into a format that a multi-modal embedding network can understand and process. In an embodiment, the prompt encoder is configured to generate a prompt embedding feature zyfrom the prompt input y, which represents the prompt input y encoded into an input format for the multi-modal embedding network. The prompt embedding feature zycaptures the semantic meaning and contextual information of the prompt input y, enabling the multi-modal embedding network to understand and process the input effectively. The multi-modal embedding network is a type of neural network architecture designed to merge information from multiple modalities, such as text, images, audio, or other types of data. In an embodiment, the multi-modal embedding network is configured to compute an image embedding feature zxthat models the prior of P(zy|zx). In an embodiment, the image embedding feature zxis a numerical representation of an image in a high-dimensional vector space. The image embedding feature zxis passed to a decoder. The decoder is a decoding neural network configured to compute an outputAtty. Docket: 4502-86501 (6000765PCT02)image x based on the image embedding feature zxand the prompt embedding feature zy. The target is to achieve high visual perceptual quality (e.g., natural and photo realistic, low level of visible artifacts) of the generated image x, and the semantic alignment of x to the requirement described by the prompt input y.

[0050] A general framework for LIC can be described as follows. LIC is a modern approach to image compression that utilizes deep learning techniques to learn efficient representations of images. Traditional image compression techniques rely on handcrafted algorithms that transform the image data into a compressed format. However, LIC aims to improve compression efficiency by training neural networks to automatically learn the most effective compression strategies directly from the data. LIC based on neural networks (NN) has been largely studied in recent years and has shown superior performance over traditional coding methods like Joint Photographic Experts Group (JPEG), Versatile Video Coding (VVC), and High Efficiency Video Coding (HEVC). In an embodiment, on a sender side, an input image x is passed through an input encoder to generate an image embedding feature zx, which is a representation of the input image x in a numerical format. In an embodiment, the input encoder is a neural network configured to convert the raw pixel values of the input image x into a compressed and semantically meaningful numerical representation in a high-dimensional vector space. In some embodiments, the image embedding feature zxis further compressed through quantization and arithmetic coding into a data string that is efficient for storage and transmission from the sender to a receiver.

[0051] In an embodiment, on the receiver side, a decoded image embedding feature zxis recovered from the received data string sent by the sender using arithmetic decoding and dequantization. The decoded image embedding feature zxis used as input for a decoder. The decoder is configured to reconstruct an output image x based on the decoded image embedding feature zx. The target is to minimize the restoration loss between the reconstructed output x and the original input x, and to minimize the bits to represent the image embedding feature zxfor storage and transmission.

[0052] FIG. 1 is a diagram 100 illustrating a general framework for tokenized visual representation (TVR) according to an embodiment of the present disclosure. The core component of image / video generation is the TVR, where an input tokenizer transforms input visual signals into discretized sequences of latent codes (e.g., using a vector-quantized generative adversarial network (VQGAN)) or into a continuous latent space (e.g., using a variational autoencoder (VAE)),Atty. Docket: 4502-86501 (6000765PCT02)by modelling the probability distributions of the visual space. As shown in FIG. 1, a patchify module 102 configured to divide an input image or video x into a sequence of overlapping or nonoverlapping two-dimensional (2D) or three-dimensional (3D) patches xp. The input tokenizer module 104 configured to transform each patch into a latent code or a latent feature. The resulting sequence of latent codes or latent features is concatenated along a sequence dimension to form an embedding zx(when transformed to discrete latent codes) or zx(when transformed to continuous latent features). A decoder 106 is configured to transform the embedding back to the pixel space to reconstruct the output x. The input tokenizer module 104, the latent space, and the decoder 106 are optimized end-to-end to balance the efficiency of representation and the reconstruction quality. Various tokenizers may be designed for either images (e.g., VQGAN) or videos (like masked generative video transformer (Mag Vi t).

[0053] Pretrained Large Language Models (LLM) has shown strong capability in natural language understanding and natural language generation. The LLM is trained over large-scale text data like spreadsheets, fictional novels, etc., and can perform a variety of linguistic tasks. By recasting visual synthesis as a sequence prediction problem, LLM-based methods generate images / videos by predicting tokens using Autoregressive (AR) models like OpenAI’s DALL-E or Non-Autoregressive (Non-AR) models like Magvit.

[0054] Learned Video compression (LVC) based on NN has been actively studied in recent years and has shown superior performance over other coding methods such as HEVC and VVC. Given a video segment consisting of n frames X = x1, ..., xn, in a general LVC, on the sender side, a set of embedding features Y = y1, ..., yn, are generated. These embedding features are further compressed through quantization and entropy coding into a data string that is efficient for storage and transmission. On the receiver side, a set of decoded embedding features Y = y1, ..., ynis recovered from the received data string through entropy decoding and dequantization. Based on the decoded embedding features, a corresponding set of reconstructed frames X = x1, ..., xnis generated.

[0055] Previous LVC methods follow the traditional video coding pipeline. On the sender side, for a GoP consisting of n frames X = x1, ..., xn, a number of m (1 < m < n) I-frames XI= x1, ..., xmare selected, which are compressed by an NN-based LIC Encoder using to generate a set of compressed I-frame features Y1= y[}, —,y’m- Y1are transmitted to the receiver side, basedAtty. Docket: 4502-86501 (6000765PCT02)on which the decoder computes the reconstructed I-framesX7= x-, ■ ■■,xilmby a corresponding NN-based LIC Decoder module. The remaining predictive (P) frames and bidirectional predictive (B) frames are then compressed based on x, —,x- In general, to compress a remaining frame xt, a set of reference frames Rtare selected consisting of the reconstructed 1-framesxf, —,xmand a set of k previously reconstructed frames x^,..., xk, where x can be before or after t. Then a motion vector vtis computed based on the reference frames Rtand xtby an NN-based Motion Estimation module. Then a residual rtis computed by an NN-based Motion Compensation module using the motion vector vt, the reference frames Rt, and xt. The motion vector vtand residual rtare transferred to the receiver side, usually with additional entropy encoding and decoding process to further reduce transmission overhead. Finally, based on the reference frames Rt, the residual rtand the motion vector vt, an NN-based Decoder reconstructs the output xt.

[0056] On the receiver side, based on the received 1-frame features Y1= y,..., y the decoder computes the reconstructed I-frames X1= x{, —,x- by the NN-based LIC Decoder module. Then for each remaining frame xt, the same set of reference frames Rtare selected as on the sender side. Then based on the received motion vector vtand residual rt, as well as the reference frames Rt, the NN-based Decoder reconstructs the output xt.

[0057] LIC with image tokenizers

[0058] Both discrete image tokenizers such as VQGAN and continuous tokenizers such as VAE has been used for LIC. For discrete tokens, the sender side encodes an input image into a latent space spanned by the discrete visual tokens, which are mapped into a sequence of token indices z1,...,zK. The token indices are integers, which can be effectively transferred to the receiver side. The receiver then uses the same discrete visual tokens to retrieve a decoded feature y, based on which a decoder reconstructs the output image x. Transferring integer token indices is very robust to heterogeneous computing platforms. By encoding token indices instead of latent features, token-based LIC avoids the sensitivity to computation mismatches between the sender and receiver. Moreover, transferring token indices gives the freedom of expanding latent feature dimension (often associated with better representation power for better reconstruction) without increasing bitrate, in comparison to transferring latent features or residues. In addition, discrete-token-based LIC increases robustness to input degradations. Realistic and rich textures can be generated using high-quality (HQ) visual tokens even for low-quality (LQ) inputs. For continuousAtty. Docket: 4502-86501 (6000765PCT02)tokens, the sender side encodes the input image into a latent feature using a continuous tokenizer (e g., a VAE). On the receiver side, the transmitted bitstream is entropy decoded and dequantized to obtain a decoded latent feature, which is subsequently used by a diffusion model to reconstruct the output image x.

[0059] Despite the advantages of transferring token indices for the compression application, such as robustness to heterogeneous platforms, the freedom to expand latent feature dimensions without increasing bitrates, robustness of HQ visual tokens against LQ inputs, it is non-trivial to extend discrete-token-based LIC to LVC. In the case of LVC, sequence of individual frames, which leads to temporal inconsistency (often manifested as temporal jittering artifacts) and inefficiency, as they fail to exploit the substantial temporal redundancy inherent in video data that could otherwise be used to reduce bitrates. Conversely, video tokenizers often employ a very large number of spatiotemporal tokens — on the order of tens or hundreds of millions — to accurately represent complex spatiotemporal visual content. This results in a dramatic increase in the number of bits required to represent the token indices, thereby defeating the primary purpose of discrete-token-based LVC. Similarly, for LVC with continuous tokenizers, the dimensionality of the continuous latent feature is usually very large to reliably encode the complex spatial-temporal visual content. Consequently, the resulting bitrates after quantization and entropy encoding are excessively high, making efficient transmission for practical LVC applications difficult.

[0060] The present disclosure proposes a general framework for LVC that leverages a powerful TVR for LVC. Video tokenizers capture rich and complex spatial -temporal information and enable high-quality video reconstruction. To resolve the dilemma of using a very large number of visual tokens for reliable video reconstruction and at the same time maintaining low bitrates, the proposed framework utilizes seven mechanisms as follows. First, the dual stream framework where both the discrete TVR (D-TVR) via the discrete video tokenizer and the continuous TVR (C-TVR) via the continuous video tokenizer are both utilized to extract complementary information for effective video reconstruction. Second, an adjustive token selection mechanism is used to select only the most relevant tokens so that the required bits to represent each used token can be reduced. Third, reference frames are selected in each GoP processing unit which can be effectively reconstructed using a small number of tokens capturing mainly spatial visual content, and then through spatial-temporal referenced attention, the remaining frames can be efficiently reconstructed using a small number of tokens capturing mainly the temporal movements of the spatial visual content from theAtty. Docket: 4502-86501 (6000765PCT02)reference frames. This reference frame-based tokenization through spatial -temporal referenced attention decomposes the vast highly complicated spatial-temporal visual space along spatial and temporal dimensions, largely reducing the difficulty of learning many spatial-temporal tokens sufficient to represent the complex entangled spatial-temporal visual space. For instance, it requires a lot of tokens to describe a moving train and a moving car, which not only need to capture various spatial appearance of the train, and the car coupled with different types of movements, but also need to differentiate the moving train and the moving car. In contrast, it is relatively easier to describe and separate a train and a car with tokens focusing on spatial visual appearance and then describe their movements with tokens focusing on temporal motion conditioned on the spatial visual appearance. Fourth, an effective spatial-temporal masked token prediction mechanism is leveraged to further improve the token efficiency and reduce bitrates for LVC. Fifth, the text-guided video generation mechanism is utilized to perform the intra-GoP non-reference frame generation. By generating the non-reference frames in decoder based on reference frames instead of transmitting them, transmission overhead can be further reduced. Sixth, the non-reference residuals are leveraged to balance compression rate and reconstruction quality and provide flexible rate control. Seventh, text guidance can be adaptively adjusted in the sender to further improve the reconstruction quality without increasing transmission overhead or decoder complexity.

[0061] FIGS. 2A-2B illustrates an encoding framework 200 (a.k.a., system) according to an embodiment of the present disclosure. An encoder may be implemented by the encoding device including one or more processors and a memory storing instructions executable by the one or more processors. The encoder may include a plurality of functional modules configured to process video segments. The modules are configured to select one or more reference frames from a GoP, designate remaining frames as non-reference frames, and patchify the reference frames into spatial blocks. The patchified reference frames are encoded to generate discrete reference latent features. Based on the discrete reference latent features, a subset of discrete reference tokens is selected from a pre-trained discrete video token codebook, and a reference token mask is generated. The discrete reference latent features are mapped to the selected tokens to generate discrete reference token indices. The encoder further applies masking to the discrete reference token indices, encodes the masked token indices into a bitstream, and predicts masked token indices using a token prediction network. The encoder also encodes a text description associated with the GoP and computes non-reference residuals based on differences between original and reconstructed nonAtty. Docket: 4502-86501 (6000765PCT02)reference frames for transmission. Although illustrated and described as separate modules for clarity, the modules may be combined, separated, or share functionality. Each module may be implemented in hardware, software, firmware, or any combination thereof. An encoder can be interpreted as any system that performs encoding, regardless of implementation.

[0062] On the sender side, the system 200 is provided with a GoP xtof length T > 1. xtis a general 4D tensor of shape T X w x h X c, where w, h, c are the width, height, and number of channels. For example, c = 3 for color videos, c = 1 for spectral videos, or c = 4 for RGB-D (color and depth) videos. The system 200 further provides a first reference frame selection module 202 configured to select one or multiple reference framesfrom xt, and the remaining nonreference frames are denoted as xr. In an embodiment, the reference frames can be selected by uniform sampling, e.g., one reference frame for every 4 or 8 frames, or by predefined rules, e.g., the first or middle frame in xtis the reference frame, or by representation saliency, e.g., the reference frames are the most representative frames in xt. The system 200 further provides a first patchify module 204 configured to compute a patchified xrpand xrpbased on x and xpr, respectively. In an embodiment, the patchify module 204 processes x frame by frame. That is, for each frame in x, it is partitioned along the width and height dimensions into nwand nhblocks, each block having size w / nwx h / nhx c. Note that the original frame can be first padded to ensure integer block size. The patchify module 204 usually processes xrdifferently from x. For example, frames in xrare processed together as a whole unit, which is partitioned along the width, height, and temporal dimensions into nw, nh, and nTblocks, each block having size Tnr / nTx w / nwx h / nhx c, where Tnris the length of xr.

[0063] The system 200 further provides a first discrete reference embedding module EMBrd206 configured to encode the patchified reference xtr pinto an embedded discrete reference latent feature e^d. In an embodiment, for each block in the patchified xrp, a discrete latent feature tensor with cddimensions is computed by the first discrete reference embedding module EMBrd206. In an embodiment, all the discrete latent feature tensors of all the blocks in the patchified xtr pform the embedded discrete reference latent featureVarious neural networks can be used as the discrete reference embedding module EMBrd. In a preferred embodiment, a Visual Transformer (ViT) is used. In an embodiment, the neural network structure of EMBrdand EMBrldoes not have temporal sequential processing. For example, for each frame, a two-dimensional (2D) embeddingAtty. Docket: 4502-86501 (6000765PCT02)neural network computes a feature tensor from each w / nwx h / nhx c block, which is serialized and processed by transformer encoders to compute a cd-dim latent feature tensor. In an embodiment, the neural network structure of EMBrddoes not have temporal sequential processing.

[0064] The system 200 is also provided with another GoP xt+1of length T > 1, where xt+1and xthave the same shape. In an embodiment, xt+1is the next consecutive video segment of xtalong the timeline. The system 200 further provides a second reference frame selection module 208 configured to select, same as xt, the one or multiple reference frames x^+1from xt+1, and the remaining non-reference frames are denoted as x^. The system 200 further provides a second patchify module 210 configured to compute a patchified x^ and x"^ based on x+1and x^, respectively, in the same way patchified xtr pand xrpare computed based on x^ and xr. That is, the patchified x^ and x^phave the same shape, and the patchified x^pand xrphave the same shape. The system 200 further provides a second discrete reference embedding module EMBrd212 configured to encode the patchified reference x^tinto an embedded discrete reference latent feature t+ 1.

[0065] The system 200 further provides a discrete token selection module 214 configured to select, based on the embedded discrete reference latent featuresand a subset of discrete reference tokensfrom a list of reference video tokens Tknrdin a pre-trained discrete video tokenizer (such as Nvidia’s cosmos discrete video tokenizer), and compute a reference token mask mJdX(+ito mask out the unselected tokens in Tknrd.

[0066] The system 200 further provides a reference discrete tokenization module 216 configured to compute a discrete reference token indices zxdbased on exdand mxdXt+i, and compute a discrete reference token indices zd+ibased on exd+iand mxdxtIn an embodiment, the pre-learned reference video tokens Tknrdcomprisesof tokens, each is a latent feature vector of cddimension. Each latent feature vector in exdor exd+1corresponds to each block in xtr por x^15which is mapped to a token tkn^ E TknxdXt+ior tknxd+i iE TknxdXt+1, i = 1,...,nwx nhx nT, where tknxdi or tknrdis closest to the latent feature vector of the block in exdorext+1measured by a distance metric (such as LI or L2 norm). That is, the entire zxdor zxd+ihas nwx nhx nTintegers correspond to the indices of the mapped selected tokens. Since the number of selected reference tokens is much less than the number of original pre-learned reference tokens, the bit consumption of encoding indices of the selected reference tokens is largely reduced. ThereAtty. Docket: 4502-86501 (6000765PCT02)are many ways the discrete token selection module 214 selects the subset of discrete tokens. For example, a pre-trained neural network model can be used, which predicts the most relevant tokens in Tknrdto represent the video content of xtrpand x^ based on exdand e£d+i.

[0067] The system 200 further provides a reference masking module 218 configured to apply masks to the discrete reference token indices z£dto compute a masked discrete reference token indicesz^d’maskanc[ apply masks to the discrete reference token indices z£d+ito compute a masked discrete reference token indices zx^ask. Usually, a pre-defined masking ratio is given a user, and elements in zxdor z£d+iare masked out according to the ratio. Masked-out elements are set to a special symbol agreed between sender and receiver to avoid transmission. The reference masking module 218 can randomly mask out elements in zxdor z£d+ior use some selection mechanism to mask out less frequent indices in zxdor zd+i.

[0068] The system 200 further provides a compute reference discrete token difference module 220 configured to compute a different token indices 5dt+1based on the masked discrete reference token indices Ud'm“-S'kanc[ rd, maskas as referencetoken masksd. The target is.. cd x-r xi • rd, mask < rd, mask,<. < x • to compute oXf+ito utilize the redundancy in zXfand zXt^iso that less transmission is required.

[0069] In an embodiment, the masked discrete reference token indicesZx^’maskand the reference token maskifed into an indices encoding module 222 to compute a discrete reference indices string sXt. Also, SXt+1is fed into a difference encoding module 224 to compute a difference indices string sdt+i. The discrete reference indices string sXt, the difference indices string §Xand the reference token mask mxdXt+iare all transmitted to the receiver side. The indices encoding module 222 and the difference encoding module 224 usually employ lossless integer entropy coding methods to further reduce the bit consumption of representing the token indices zx^’maskand the different token indices SXt+i. Due to the spatial -temporal redundancy in the visual content of video sequences, the token indices between consecutive GoPs are highly predictable and redundant, and therefore sXtand sXt+1usually consume much less bits than a string directly encoded from the original non-masked token indices.Atty. Docket: 4502-86501 (6000765PCT02)

[0070] Still on the sender side, the masked discrete reference token indicesz^'maskanj zx^askare fed into a reference discrete token prediction module 226 to compute a predicted discrete reference token indices zx^ andwhere the masked-out elements inzxr^maskailcjzxt+iarerefilled by token prediction.

[0071] In addition, the system 200 is provided with a text descriptionYXfwhich describes the content of xtand xt+1, such as the semantics, objects, object relations, motions, etc. YXtXt+1can be provided by user inputs, or by a VLM (such as CLIP) or other video captioning methods based on xtand xt+1. The text description YXt Xf+1is further fed into a text tokenizer 228 to compute a tokenized textual latent• The text tokenizer usually uses the text encoder of an LLM (such as OpenAI’s DALL-E) or a VLM (such as CLIP) to compute the tokenized textual latent ztX, which usually consists of a sequence of integer indices corresponding to codeword indices of a pre-trained textual codebook. The pre-trained textual codebook can be a semantic dictionary where each codeword is a natural semantic word in the dictionary or can be a prelearned codebook where each codeword comprises a combination of semantic words. Then, a text token encoding module 230 computes a bit-efficient textual string sXf xfrom the tokenized textual latent zXtXThe textual string Sxt> Xf. is transmitted to the receiver side. The text token encoding module 230 usually uses lossless integer entropy encoding methods to reduce the bit consumption of the tokenized textual latent Zxt> Xt.

[0072] In the embodiments, the system 200 further provides a first continuous reference embedding module EMBrl232 also encodes the patchified xtr pinto an embedded continuous reference latent feature extl. For each block in the patchifieda continuous latent feature tensor with cldimensions is computed by the first continuous reference embedding module EMBrl232. All the continuous feature tensors of all the blocks in the patchified xrpform the embedded continuous reference latent feature extl. Various neural networks can be used as the first continuous reference embedding module EMBrl232. In the preferred embodiment, a Visual Transformer (ViT) like is also used as EMBrl. Typically, the neural network structure of EMBrldoes not have temporal sequential processing. For example, for each frame, a 2D embedding neural network computes a feature tensor from each w / nwx h / nhx c block, which is serialized and processed by transformer encoders to compute a cJ-dim latent feature tensor. Similarly, a second continuousAtty. Docket: 4502-86501 (6000765PCT02)reference embedding module EMBrl234 encodes the patchified x^p±into an embedded continuous reference latent feature e£*.

[0073] In an embodiment, the embedded continuous reference latent featuresand e^+ iare fed into a continuous token selection module 236 to compute a reference feature maskwhich is used by a reference continuous tokenization module 240 to compute a tokenized reference latent zxtlbased on the continuous reference latent features extland to compute a tokenized reference latent zx\,, based on the continuous reference latent features e£*. The reference continuous tokenization module 238 uses a pre-trained continuous video tokenizer like Nvidia’s Cosmos continuous video tokenizer to compute zx\ and z%*+1, and the feature maskcan mask out a set of feature channels that are unimportant to represent the visual content of the current extland e^+1so that the corresponding channels in the tokenized reference latent z * and z£+1are masked out.

[0074] In an embodiment, the tokenized reference latent z * is processed by a latent encoding module 240 to generate a continuous latent string sxwhich is sent to the receiver. Optionally, the tokenized reference latent z£+iis also processed by the latent encoding module 240 and put into the continuous latent strings^. The latent encoding module 240 usually comprises quantization and entropy encoding processes so that the generated continuous latent stringconsumes less bits than the tokenized reference latent z^ (and optionally zxtl+i). Then, in the sender, sXltis further processed by a latent decoding module 242 to compute a recovered tokenized reference latent zxtl(and optionally a recovered tokenized latent zxl). The latent decoding module 242 usually comprises corresponding entropy decoding and dequantization processes which is used also in the receiver to recover zxtl(and optionally z£* ) from sXt.

[0075] In an embodiment, the predicted discrete reference token indices z^ and z^+i, the tokenized textual latent zXtiX, and the recovered zxtl(and optionally Z^+1) are fed into a tokenized video generation module 244, which computes a generated non-reference xrpand x^p.

[0076] In an embodiment, a compute non-reference difference module 246 computes a nonreference residualbased on the original xrpand x^pand the generated non-reference xrpand x^p. A residual encoding module 248 further encodes the non-reference residual fx^Xt+iAtty. Docket: 4502-86501 (6000765PCT02)into a residual string swhich is sent to the receiver side. The residual encoding module 248 usually comprises quantization and entropy encoding processes so that the generated residual string s^Xt+iconsumes less bits than the non-reference residualfor transmission.

[0077] FIGS. 3A-3B illustrate an encoding framework 300 according to an embodiment of the present disclosure corresponding to a discrete-token-only embodiment of the proposed video compression framework. Unless otherwise stated, the functional modules shown in FIGS. 3A-3B may correspond to and operate in the same manner as the identically named modules described with respect to FIGS. 2A-2B. For brevity, detailed descriptions of such modules are not repeated. In the embodiments of FIGS. 3A-3B, the continuous reference embedding stream and associated continuous token selection, tokenization, latent encoding, and latent decoding modules described in FIGS. 2A-2B are omitted. The encoding process relies solely on the discrete tokenized visual representation (D-TVR) stream. Accordingly, FIGS. 3A-3B represents a simplified embodiment that excludes the continuous latent stream while preserving the discrete token selection, masked token prediction, text-guided generation, and residual coding mechanisms previously described. As shown in FIGS. 3A-3B, the predicted discrete reference token indicesand zXtd+1, and the tokenized textual latent zX(.,Xt+1arebed '111:0the tokenized video generation module 244 to compute the generated non-reference xrpand x^p.

[0078] In an embodiment, the input text description KXt,Xt+1canbe optionally adaptively adjusted on the sender side during inference to further improve the compression performance of the current target GoPs in an online learning fashion, and the process are marked as dotted lines in FIGS. 2A-2B and 3A-3B respectively. This online adjustment does not change the receiver side processing pipeline and does not cause transmission overhead. The online adjustment process is usually iterative. During an iteration, in both FIGS. 2A-2B and 3 A-3B, a residual decoding module 250 same as the receiver side computes the recovered non-reference residualbased on the non-reference residual stringd. Also, a reconstruction module 252 same as the receiver side computes the non-reference output xrpand x^pbased on the generated non-reference xrpand x^p, and the recovered non-reference residualAfter that, a compute loss module 254 computes a loss function TXt> Xf. based on the estimated non-reference output xrpand x^p, the original non-reference xrpand xP, and the non-reference residual rxnXt+1, and the gradient ofAtty. Docket: 4502-86501 (6000765PCT02)the loss function is used by a prompt adjustment module 256 to update the text description YXtiX. Then, the system goes into the next iteration, where the updated text description YXt Xt+1is used instead of the original text description to compute the non-reference output xrpand x^pand the recovered non-reference residualthrough the aforementioned processing modules on the sender side. Finally, after several iterations (the iteration number can be preset or automatically determined to balance computation time and compression performance), the non-reference residual string sx^Xfcomputed based on the final updated text description YXf >xis sent to the receiver side, together with all other aforementioned encoded strings that are computed based on the final updated text descriptionYXf >xand need to be sent to the receiver side.

[0079] In an embodiment, the prompt adjustment module 256 can take various strategies to update the adjusted text description YXbXwhich usually depends on the model structure of the text tokenizer.

[0080] FIG. 4 illustrates a decoding framework 400 (a.k.a., system) according to an embodiment of the present disclosure. A decoder may be implemented by a decoding device including one or more processors and a memory storing instructions executable by the one or more processors. The decoder may comprise a plurality of functional modules configured to reconstruct video frames from data received from the encoder. The modules may be configured to receive and decode a discrete reference indices bitstream, a textual bitstream, a residual bitstream, and optionally a difference indices bitstream. The decoder may reconstruct masked discrete reference token indices from the discrete reference indices bitstream and, when present, compute token index differences to reconstruct masked discrete reference token indices for a subsequent group of pictures (GoP). A token prediction module may predict masked discrete reference token indices to generate predicted discrete reference token indices. A video generation module may generate reconstructed non-reference frames conditioned on the predicted discrete reference token indices and decoded textual latent indices. A residual decoding module may recover non-reference residuals, which are combined with the reconstructed non-reference frames to produce final non-reference frames. Although illustrated and described as separate modules for clarity, the modules may be combined, separated, or share functionality. Each module may be implemented in hardware, software, firmware, or any combination thereof. A decoder can be interpreted as any system that performs decoding, regardless of implementation.Atty. Docket: 4502-86501 (6000765PCT02)

[0081] As shown in FIG. 4, on the receiver side, using the received difference indices string a difference decoding module 402 configured to compute the different token indices 8x. In an embodiment, using the received discrete reference indices string sxthe reference token mask m t*» t+,,l,’ and the different token indices 8Xt,+, „l,’ an indices decoding ° module 404 computes c rd, mask rd, maskthe masked discrete reterence token indices zXtand zXt^. The indices decoding module 404 and the difference decoding module 402 usually employ lossless integer entropy decoding methods corresponding to the lossless integer entropy encoding methods in the indices encoding module and the difference encoding module, respectively, so thatzxr^inaskand zx^askare fully recovered. The masked discretez ’, zr' are fed into a reference discrete token prediction module 406 to compute the predicted discrete reference token indices zxandIn an embodiment, at the same time, using the received textual string sXt, a text token decoding module 408 recovers the tokenized textual latent zx. The text token decoding module 408 usually employs lossless integer entropy decoding methods corresponding to the integer entropy encoding methods in the text token encoding module to fully recover z^.* In an embodiment, using the received non-reference residual string s^xt+1, a residual decoding module 410 computes a recovered non-reference residualThe residual decoding module 410 usually comprises entropy decoding and dequantization processes corresponding to the residual encoding module in the sender. In the embodiment, using the received continuous latent string sxla latent decoding module 412 (same as the one in sender) computes the recovered tokenized reference latent z^ (and optionally zxtl+i). In the embodiment, the predicted discrete reference token indices zxand z£^+1, the tokenized textual latent zXt> Xt, and the recovered tokenized reference latent z^ (and optionally zxtl+i) are fed into a tokenized video generation module 414 (same as the sender side in FIGS. 2A-2B) to compute the generated non-reference xrpand x^p.

[0082] FIG. 5 illustrates a decoding framework 500 (a.k.a., system) according to an embodiment of the present disclosure corresponding to the discrete-token-based encoding embodiment, and further depicting an optional continuous latent decoding path. Unless otherwise stated, modules shown in FIG. 5 may correspond to and operate in the same manner as identically named modules described with respect to FIG. 4, and detailed descriptions are not repeated for brevity. In an embodiment shown in FIG. 5, in addition to the operations described with referenceAtty. Docket: 4502-86501 (6000765PCT02)to FIG. 4, on the receiver side of the embodiment 2, the predicted discrete reference token indicesz£rand and the tokenized textual latent Zxt,xt+1are'nto the tokenized video generation module 414. Tokenized video generation module 414 may correspond to the tokenized video generation module described with respect to the sender side in FIGS. 3A-3B. Based on the predicted discrete reference token indices and the tokenized textual latent, tokenized video generation module 414 generates non-reference frames xrpand x^p, which are subsequently processed by residual decoding and reconstruction modules to obtain final reconstructed nonreference frames.

[0083] After that for both embodiments of FIG. 4 and FIG. 5, the generated non-reference x'irpand x!?, as well as the recovered non-reference residual iAA,, are fed into a reconstruction module 416 to compute the non-reference xrpand x^p, which are further processed by a depatchify module 418 to obtain the final non-reference output xrand x^. In an embodiment, the system goes into the next encoding / decoding cycle to process GoPs xt+1and xt+2. The Reconstruction module will be described in detail later. The depatchify process is the reverse process of the patchify process, where the pixel blocks are reordered to obtain the final non-reference frames.

[0084] FIG. 6 illustrates a prompt adjustment process 600 according to an embodiment of the present disclosure. In an embodiment, a text tokenizer 602 comprises two major modules, where the input text descriptionYXfis processed by a text embedding module 604 to compute a text latent feature eXt, which is further processed by a text tokenization module 606 to generate the tokenized textual latent zXf X■ The gradient of loss functionis used to update the text latent feature eXt x. In an embodiment, the updated text latent featureeXfis fed into a generative diffusion module 608, where based on a latent text diffusion model the adjusted text description YXf Xt+ican be computed. The loss function LyXtXtusually consists of distortion between the non-reference output xrpand x”^ and the generated non-reference xrpand x^*, e.g., LI or L2 norm, perceptual difference between the non-reference output x”rpand x^pand the generated non-reference xrpand x^p, e.g., LPIPS, SSIM, and the bitrate of the non- reference residual• This disclosure does not put restrictions on loss functions used for online text description adjustments.Atty. Docket: 4502-86501 (6000765PCT02)

[0085] Individual Modules

[0086] This section describes the details of the reference discrete token prediction module, the compute reference discrete token difference module, the tokenized video generation module, and the reconstruction module.

[0087] The reference discrete token prediction module usually has a transformer encoderdecoder structure. In one embodiment, the masked discrete reference token indicesZx^’maskare first fed into a transformer encoder-decoder network to compute the predicted discrete reference token indices zx, where mainly spatial attention is used. Then, zx* is further fed into another transformer encoder-decoder network together with zx^askto compute the predicted discrete reference token indices zd+i, with both spatial and temporal attention mechanisms. In another embodiment,zXtand zXt^ are directly serialized into a sequence and fed into a transformer encoder-decoder network to compute the predicted zxrdand zxrd. This disclosure does not put restrictions on the order how ẑxrdand ẑxrdare computed, or the network structures used for such prediction.

[0088] The compute reference discrete token difference module computes the different token indices 8Xto reduce the bits of transferring the masked discrete reference token indices zx^’maskand ẑxrd,mask. In the preferred embodiment, δxdconsists of the difference between ẑxrd,maskandz,,. If an elementin z,, — z ' is zero (the same index), it does not need to be transmitted and is marked as a special symbol known by both sender and receiver in δxd. The indices of non-zero elements are kept unchanged in δxd, whose representation are determined by the reference token maskxt>xt+,l.

[0089] The tokenized video generation module usually has the architecture of an autoregressive transformer model, comprising of interleaved self-attention and feedforward layers as its main building blocks. For example, the LLM-style GPT model structure can be used, where inputs to the model are tokenized visual latent. The tokenized textual latent z^can be added to the main transformer blocks through cross-attention layers. In addition, post-refinement modules such as denoising diffusion modules can be attached after the autoregressive transformer model to further enhance the performance.Atty. Docket: 4502-86501 (6000765PCT02)

[0090] The reconstruction module first adds back the recovered non-reference residual„ to the generated non-reference x™rpand x^ to recover an intermediate non-reference xrp' and x^p', which are then used as inputs to a neural network model to compute the final non-reference x^rpand xp. This neural network model aims to reduce temporal inconsistencies between xrpand xpand / or further enhance their visual qualities. For example, a convolutional UNet or a denoising diffusion model can be used.

[0091] This disclosure does not put any restrictions on the model structures of or how to obtain the pre-trained models for the previously mentioned modules like LLM, discrete video tokenizers, continuous video tokenizer, text tokenizer, etc. This invention also does not put any restrictions on the model structure of or how to obtain the neural network models for any model-based modules in the framework such as the reference discrete token prediction, tokenized video generation, reconstruction, etc. The specific methods used by other non-model related modules such as the compute reference discrete difference module and compute non-reference difference module and the specific format of the computed different token indices δxdand residual rx,xnrare not restricted either. Furthermore, this disclosure does not put any restrictions on the method and order to train different network modules in the framework, which modules are pre-trained which modules are finetuned or trained from scratch, the exact loss functions to train these modules, etc.

[0092] FIG. 7 is a method 700 implemented by an encoding device according to an embodiment of the disclosure. In an embodiment, the encoding device is a computer, a smart phone, a smart tablet, or other device configured to process and display video content. The encoding device may execute one or more software modules, hardware accelerators, or combinations thereof to perform the tokenization, token selection, token prediction, and encoding operations described herein. In certain embodiments, the encoding framework operates in real time during gameplay or video rendering to compress visual data for transmission or storage.

[0093] In block 702, the method comprises selecting one or more reference frames from a first group of pictures (GoP) and a second GoP of a video sequence, wherein remaining frames from the first GoP and the second GoP comprise first non-reference frames.

[0094] In block 704, the method further comprises patchifying the one or more reference frames to generate patchified reference frames.Atty. Docket: 4502-86501 (6000765PCT02)

[0095] In block 706, the method further comprises encoding the patchified reference frames using a discrete video tokenizer to generate discrete reference latent features.

[0096] In block 708, the method further comprises selecting, based on the discrete reference latent features, a subset of discrete reference tokens from a pre-trained discrete video token codebook and generating a reference token mask indicating the subset of discrete reference tokens.

[0097] In block 710, the method further comprises mapping the discrete reference latent features to the subset of discrete reference tokens to generate discrete reference token indices for the first GoP and the second GoP.

[0098] In block 712, the method further comprises applying one or more masks to the discrete reference token indices to generate masked discrete reference token indices.

[0099] In block 714, the method further comprises determining token index differences between the first GoP and the second GoP based on the masked discrete reference token indices.

[0100] In block 716, the method further comprises encoding the masked discrete reference token indices, the reference token mask, and the token index differences for transmission toward a decoder.

[0101] FIG. 8 is a method 800 implemented by a decoding device according to an embodiment of the disclosure. In an embodiment, the decoding method may be implemented by the decoding device configured to reconstruct and display video content. The decoding device may comprise one or more processors, memory, and communication interfaces for receiving encoded video bitstreams and performing the decoding operations described herein. Examples of such decoding devices include, but are not limited to, a computer, a smart phone, a smart tablet, a smart television, a gaming console, or any other electronic device capable of decoding and displaying video content.

[0102] In block 802, the method comprises receiving a discrete reference indices string, a reference token mask, and a difference indices string.

[0103] In block 804, the method further comprises decoding, using the reference token mask, the discrete reference indices string to compute first masked discrete reference token indices for a first group of pictures (GoP).

[0104] In block 806, the method further comprises decoding the difference indices string to compute token index differences.Atty. Docket: 4502-86501 (6000765PCT02)

[0105] In block 808, the method further comprises determining second masked discrete reference token indices for a second GoP based on the first masked discrete reference token indices and the token index differences.

[0106] In block 810, the method further outputting the frames of the first GoP and the second GoP based on the first masked discrete reference token indices and the second masked discrete reference token indices.

[0107] FIG. 9 is a schematic diagram of a computing device 900 (e.g., a personal computer, smart phone, smart tablet, handheld gaming device, etc.) according to an embodiment of the disclosure. The computing device 900 is suitable for implementing the disclosed embodiments as described herein. The computing device 900 comprises ingress ports / ingress means 910 (a.k.a., upstream ports) and receiver units (Rx) / receiving means 920 for receiving data; a processor, logic unit, or central processing unit (CPU) / processing means 930 to process the data; transmitter units (Tx) / transmitting means 940 and egress ports / egress means 950 (a.k.a., downstream ports) for transmitting the data; and a memory / memory means 960 for storing the data. In an embodiment, the receiver units (Rx) / receiving means 920 comprise a discrete circuit, integrated circuit, chip set, package, hardware module, electronic device, or other structure capable of receiving signals. In an embodiment, the transmitter units (Tx) / transmitting means 940 comprise a discrete circuit, integrated circuit, chip set, package, hardware module, electronic device, or other structure capable of transmitting signals. The computing device 900 may also comprise optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress ports / ingress means 910, the receiver units / receiving means 920, the transmitter units / transmitting means 940, and the egress ports / egress means 950 for egress or ingress of optical or electrical signals.

[0108] The processor / processing means 930 is implemented by hardware and software. The processor / processing means 930 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor / processing means 930 is in communication with the ingress ports / ingress means 910, receiver units / receiving means 920, transmitter units / transmitting means 940, egress ports / egress means 950, and memory / memory means 960. The processor / processing means 930 comprises a tokenized video generation module 970. The tokenized video generation module 970 is able to implement the methods disclosed herein. The inclusion of the hybrid-style video generation module 970 therefore provides a substantialAtty. Docket: 4502-86501 (6000765PCT02)improvement to the functionality of the computing device 900 and effects a transformation of the computing device 900 to a different state. Alternatively, the tokenized video generation 970 is implemented as instructions stored in the memory / memory means 960 and executed by the processor / processing means 930.

[0109] The computing device 900 may also include input and / or output (I / O) devices or I / O means 980 for communicating data to and from a user. The I / O devices or I / O means 980 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices or I / O means 980 may also include input devices, such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices. The memory / memory means 960 comprises one or more disks, tape drives, and solid-state drives and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory / memory means 960 may be volatile and / or non-volatile and may be read-only memory (ROM), random-access memory (RAM), ternary content-addressable memory (TCAM), and / or static RAM (SRAM).

[0110] Embodiments of the present disclosure provide at least the following technical advantages.

[0111] This disclosed framework offers high compression rate with high-quality video generation at the same time by exploiting the power of both discrete video tokenizers and continuous video tokenizers. The video tokenizers have more representation power in describing the spatial-temporal visual content than image tokenizers, giving high-quality to the reconstructed videos. The token selection and token prediction mechanism reduce the required bits to represent the tokens and the required tokens to be encoded for transmission, giving effective bitrate reduction.

[0112] The disclosed framework improves rate-distortion performance by the reference-frame-based tokenization mechanism. Reference frames that can be effectively reconstructed using a small number of spatial-content-focusing tokens are first selected. Then through spatial -temporal referenced attention the remaining non-reference frames are efficiently reconstructed using a small number of temporal-movement-focusing tokens.

[0113] The discrete video tokenizer employs a pre-learned spatial-temporal visual codebook that is robust to input degradations and optimized for perceptual reconstruction quality. The continuous video tokenizer extracts fine-grained detail information from the input video to ensureAtty. Docket: 4502-86501 (6000765PCT02)high-fidelity reconstruction. By combining discrete and continuous tokenizers, the system achieves a balanced tradeoff between perceptual quality and fidelity.

[0114] The framework further supports flexible, task-oriented quality control. Depending on application requirements, the system can dynamically adjust the contribution of continuous tokenized latents relative to discrete tokens to emphasize either perceptual quality or fidelity during reconstruction.

[0115] The text-guided video generation mechanism performs intra-GoP non-reference frame generation on the decoder side to largely reduce transmission overhead of non-reference frames, which leverages the power of pre-trained VLM models that captured the joint distribution of the video tokenizers and the corresponding text descriptions. Trained over a large scale of image-text and / or video-text data pairs, the VLM enables more abundant features from both visual and text domains to better describe the input video than using visual domain alone.

[0116] The residuals of generated non-reference frames can be used to balance compression rate and reconstruction quality and provide flexible rate control. Highly compressed residual gives less accuracy of recovered non-reference frames but consumes less bits.

[0117] The text guidance can be adaptively adjusted in the sender during inference stage in an online fashion to further improve the reconstruction quality without increasing transmission overhead or decoder complexity. It can be used to tradeoff computation time and reconstruction quality in certain non-time-critical applications. For example, a large video can be compressed once for storage in the sender, and efficiently transmitted to multiple users, and the online adjustment does not affect the decoding time in receiver.

[0118] While several embodiments have been provided in the present disclosure, it may be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.

[0119] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure.Atty. Docket: 4502-86501 (6000765PCT02)Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

Atty. Docket: 4502-86501 (6000765PCT02)CLAIMSWhat is claimed is:

1. A method implemented by an encoder, comprising:selecting one or more reference frames from a first group of pictures (GoP) and a second GoP of a video sequence, wherein remaining frames from the first GoP and the second GoP comprise first non-reference frames;patchifying the one or more reference frames to generate patchified reference frames; encoding the patchified reference frames using a discrete video tokenizer to generate discrete reference latent features;selecting, based on the discrete reference latent features, a subset of discrete reference tokens from a pre-trained discrete video token codebook and generating a reference token mask indicating the subset of discrete reference tokens;mapping the discrete reference latent features to the subset of discrete reference tokens to generate discrete reference token indices for the first GoP and the second GoP;applying one or more masks to the discrete reference token indices to generate masked discrete reference token indices;determining token index differences between the first GoP and the second GoP based on the masked discrete reference token indices; andencoding the masked discrete reference token indices, the reference token mask, the token index differences, the tokenized textual latent indices, and the non-reference residuals for transmission toward a decoder.

2. The method of claim 1, further comprising:encoding the patchified reference frames to generate continuous reference latent features; determining a reference feature mask based on the continuous reference latent features; determining tokenized reference latent representations using a pre-trained continuous video tokenizer based on the continuous reference latent features and the reference feature mask; and encoding the tokenized reference latent representations for transmission toward the decoder.Atty. Docket: 4502-86501 (6000765PCT02)3. The method of any of claims 1-2, wherein the pre-trained continuous video tokenizer comprises a continuous-valued autoencoder that compresses the one or more reference frames into a continuous latent space.

4. The method of any of claims 1-3, further comprising:generating predicted discrete reference token indices from the masked discrete reference token indices using a token prediction network;receiving a text description of the first GoP and the second GoP and encoding the text description into tokenized textual latent indices;generating reconstructed non-reference frames using the predicted discrete reference token indices and the tokenized textual latent indices;determining non-reference residuals based on differences between the first non-reference frames and the reconstructed non-reference frames; andencoding the masked discrete reference token indices, the reference token mask, the token index differences, the tokenized textual latent indices, and the non-reference residuals for transmission toward the decoder, wherein the token index differences represent token index variations between the first GoP and the second GoP.

5. The method of any of claims 1-4, wherein the encoding comprises:encoding the masked discrete reference token indices into a discrete reference indices string;encoding the token index differences into a difference indices string;encoding the tokenized textual latent indices into a textual string; andencoding the non-reference residuals into a residual string.

6. The method of any of claims 1-5, wherein the second GoP is temporally subsequent to the first GoP in the video sequence.

7. The method of any of claims 1-6, further comprising iteratively adjusting the text description during encoding based on a loss function that measures distortion between the firstAtty. Docket: 4502-86501 (6000765PCT02)non-reference frames and the reconstructed non-reference frames and bitrate of the non-reference residuals.

8. The method of any of claims 1-7, wherein adjusting the text description comprises:processing the text description to compute a text latent feature;updating the text latent feature using a gradient of the loss function; andgenerating an updated text description from the updated text latent feature using a diffusion-based language model.

9. The method of any of claims 1-8, wherein encoding the patchified reference frames comprises using a neural network encoder trained to extract spatial and temporal features from the one or more reference frames.

10. The method of any of claims 1-9, wherein the discrete reference latent features comprise a plurality of latent feature vectors, and wherein the mapping comprises selecting a token closest to each latent feature vector according to a distance metric.

11. The method of any of claims 1-10, wherein applying the masks to the discrete reference token indices comprises masking out elements in the discrete reference token indices according to a predefined masking ratio, and wherein masked-out elements are set to a symbol and are not transmitted.

12. The method of any of claims 1-11, wherein generating the discrete reference token indices comprises:processing the masked discrete reference token indices through a transformer encoder to generate contextual embeddings; anddecoding the contextual embeddings through a transformer decoder to predict values of the masked-out elements.Atty. Docket: 4502-86501 (6000765PCT02)13. The method of any of claims 1-12, wherein selecting the subset of discrete reference tokens comprises using a pre-trained neural network model that predicts most relevant tokens to represent video content based on the discrete reference latent features.

14. The method of any of claims 1-13, wherein generating the reconstructed non-reference frames comprises:inputting the predicted discrete reference token indices and the tokenized textual latent indices into a pre-trained video generation model; andsynthesizing the reconstructed non-reference frames conditioned on both the predicted discrete reference token indices and the tokenized textual latent indices.

15. An encoder, comprising:a memory configured to store instructions; andone or more processors coupled to the memory and configured to execute the instructions to cause the encoder to perform a method according to any of claims 1-14.

16. A method implemented by a decoder, comprising:receiving a discrete reference indices string, a reference token mask, and a difference indices string;decoding, using the reference token mask, the discrete reference indices string to compute first masked discrete reference token indices for a first group of pictures (GoP);decoding the difference indices string to compute token index differences; determining second masked discrete reference token indices for a second GoP based on the first masked discrete reference token indices and the token index differences; andoutputting frames of the first GoP and the second GoP based on the first masked discrete reference token indices and the second masked discrete reference token indices.

17. The method of claim 16, wherein decoding the discrete reference indices string comprises applying lossless integer entropy decoding to recover the first masked discrete reference token indices.Atty. Docket: 4502-86501 (6000765PCT02)18. The method of any of claims 16-17, further comprising:receiving a textual string and a residual string;generating predicted discrete reference token indices from the first masked discrete reference token indices and the second masked discrete reference token indices using a token prediction network, wherein generating the predicted discrete reference token indices comprises:processing the first masked discrete reference token indices and the second masked discrete reference token indices using a transformer encoder to generate contextual embeddings; anddecoding the contextual embeddings using a transformer decoder to predict values for masked-out elements in the first masked discrete reference token indices and the second masked discrete reference token indices;decoding the textual string to recover tokenized textual latent indices;generating reconstructed non-reference frames based on the predicted discrete reference token indices and the tokenized textual latent indices;decoding the residual string to recover non-reference residuals;reconstructing final non-reference frames by combining the reconstructed non-reference frames with the non-reference residuals; andoutputting the final non-reference frames.

19. The method of any of claims 16-18, wherein generating the reconstructed non-reference frames comprises using a pre-trained video generation model based on the predicted discrete reference token indices and the tokenized textual latent indices.

20. The method of any of claims 16-19, wherein recovering the non-reference residuals comprises:applying entropy decoding to the residual string to obtain quantized residuals; and dequantizing the quantized residuals to recover the non-reference residuals.

21. The method of any of claims 16-20, wherein reconstructing the final non-reference frames comprises:Atty. Docket: 4502-86501 (6000765PCT02)adding the non-reference residuals to the reconstructed non-reference frames to produce intermediate non-reference frames; andprocessing the intermediate non-reference frames using a post-processing neural network to compute the final non-reference frames.

22. The method of any of claims 16-21, further comprising depatchifying the reconstructed non-reference frames by reordering pixel blocks according to a reverse patchification process.

23. The method of any of claims 16-22, further comprising:receiving and decoding a continuous latent string to recover tokenized reference latent representations; andgenerating the reconstructed non-reference frames further based on recovered tokenized reference latent representations.

24. The method of any of claims 16-23, wherein the tokenized textual latent indices represent natural language text describing visual content, actions, objects, or scenes present in the first GoP and the second GoP.

25. A decoder, comprising:a memory configured to store instructions; andone or more processors coupled to the memory and configured to execute the instructions to cause the decoder to perform a method according to any of claims 16-24.

26. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions when executed by one or more processor of an encoder, cause the encoder to perform a method according to any of claims 1-14.

27. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions whenAtty. Docket: 4502-86501 (6000765PCT02)executed by one or more processor of a decoder, cause the decoder to perform a method according to any of claims 16-24.

28. An encoder, comprising:a storage means configured to store instructions; andone or more processing means coupled to the storage means and configured to execute the instructions to cause the encoder to perform a method according to any of claims 1-14.

29. A decoder, comprising:a storage means configured to store instructions; andone or more processing means coupled to the storage means and configured to execute the instructions to cause the decoder to perform a method according to any of claims 16-24.

30. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause an encoder to perform operations according to any of claims 1-14.

31. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause an encoder to perform operations according to any of claims 16-24.

Images