Generative video coding and decoding method based on multi-modal large model

Abstract

The invention relates to the technical field of video coding and decoding, and discloses a multi-mode large model-based generative video coding and decoding, which comprises a key frame selection module for determining a key frame by analyzing the semantic and motion characteristics of a video frame; the multi-modal semantic description generation module is used for generating semantic description according to the key frame and the video clip; the key frame compression module is used for realizing efficient compression through latent variable modeling and entropy coding; the key frame reconstruction module reconstructs a key frame by using a conditional diffusion model in combination with the compressed data and the semantic description information; and the video generation module generates a non-key frame by using the semantic description and the key frame, and reconstructs a complete video. Through key frame screening combining semantic and motion information, key frame compression and reconstruction based on a conditional latent variable diffusion model, and frame supplementation and frame insertion generation based on semantic description, efficient compression and high-quality video reconstruction can be realized under a low code rate, and the video storage efficiency and the visual quality are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video coding and decoding, and in particular to a generative video coding and decoding based on a multimodal large model. Background Art

[0002] Video compression technology plays a crucial role in multimedia processing, primarily reducing the storage and transmission requirements of video data. Uncompressed video files are typically large, placing significant strain on storage devices and network bandwidth. The goal of video compression technology is to remove redundant information, simplify data representation, and maintain high visual quality. Traditional video compression methods typically rely on standard coding techniques, such as H.264 and HEVC, which primarily improve compression efficiency through inter- and intra-frame redundancy removal, transform coding, and entropy coding.

[0003] While these traditional methods perform well in some application scenarios, they often fail to maintain good reconstruction quality when processing complex video content, especially in low bitrate environments. As video content becomes more complex and bitrate requirements decrease, the limitations of traditional video coding techniques are becoming increasingly apparent. Compressed images can lose important details and even exhibit noticeable blurring and compression artifacts.

[0004] In recent years, the application of deep learning technology in video compression has brought new breakthroughs. Deep learning-based video codec methods, through end-to-end training, can adaptively learn video features, thereby providing better compression and reconstruction results. These methods can automatically adjust encoding strategies based on the content of different videos, improving compression efficiency and enhancing reconstruction quality. However, despite the significant improvements in compression efficiency achieved by deep learning codecs, they still face several challenges. In particular, existing methods struggle to achieve both high compression rates and high-quality visual quality at extremely low bitrates.

[0005] Currently, deep learning-based video encoding and decoding methods can be roughly divided into two categories: general-purpose neural codecs and domain-specific neural codecs. General-purpose neural codecs are designed to adapt to a variety of video content. Their core technologies include spatiotemporal context modeling and generative models (such as generative adversarial networks (GANs). Although these codecs are highly versatile and can handle a variety of video scenarios, they typically require higher bitrates at low bitrates and, due to the complexity of their internal mechanisms, often exhibit black-box characteristics, lacking sufficient interpretability and control capabilities.

[0006] On the other hand, domain-specific neural codecs are typically optimized for specific video types, such as facial or human videos. These methods achieve compression through symbolic feature representations (such as 2D keypoints or 2D landmarks), enabling efficient compression at low bitrates and strong interpretability. However, their limited scope of optimization limits their applicability and prevents their widespread application to a wide range of video content.

[0007] Currently, neural video codecs that are both versatile and controllable are still relatively rare, especially those that achieve efficient compression at very low bitrates. Consequently, existing technologies have limitations in low-bitrate video compression and high-quality video reconstruction, failing to meet modern video compression requirements, especially in bandwidth-constrained or storage-limited scenarios. Summary of the Invention

[0008] In response to the shortcomings of the existing technology, the present invention provides a generative video codec based on a multimodal large model, which solves the problem that the existing video compression technology cannot achieve both high compression efficiency and high reconstruction quality under ultra-low bit rate conditions.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: a generative video codec based on a multimodal large model, comprising:

[0010] A key frame selection module is used to determine key frames based on the semantic features and motion features of the input video frame;

[0011] a multimodal semantic description generation module, configured to generate semantic description information based on the key frames and corresponding video clips;

[0012] The key frame compression module is used to perform latent variable modeling and entropy coding on key frames to generate compressed data;

[0013] A key frame reconstruction module, configured to reconstruct the key frame using a conditional diffusion model based on the compressed data and semantic description information;

[0014] The video generation module is used to supplement or insert non-key frames based on the semantic description information and key frames to reconstruct the complete video.

[0015] Preferably, the key frame selection module includes:

[0016] A semantic feature extraction unit, used to extract semantic feature vectors of video frames based on a pre-trained graph-text neural network model;

[0017] A motion feature extraction unit, configured to extract the motion amplitude between video frames based on an optical flow estimation algorithm;

[0018] The inter-frame difference judgment unit is used to perform key frame selection judgment based on the semantic features and motion features.

[0019] Preferably, the inter-frame difference judgment unit performs judgment based on the following key frame decision function:

[0020] D(F n ,F l )=λ1(1-f Clip (F n ,F l ))+λ2f Raft (F n ,F l )-D th ;

[0021] Among them, F n is the current frame to be judged, F l is the last selected keyframe; f Clip (F n ,F l ) is the semantic similarity between the current frame and the previous key frame, and the cosine similarity is calculated after extracting the feature vector based on the joint image-text model; f Raft (F n ,F l ) is the average motion change between the current frame and the previous key frame, which is calculated based on the optical flow network; λ1 and λ2 are the weighted coefficients of the semantic difference term and the motion change term respectively; D th is the set difference judgment threshold; when D(F n ,F l )>0, frame F n is identified as a key frame.

[0022] Preferably, the multimodal semantic description generation module includes:

[0023] Semantic extraction unit, which uses a multimodal large model to generate frame-level semantic descriptions of key frames, including background information, object descriptions, and local differences;

[0024] A motion information generation unit, configured to further generate a video-level description of a segment corresponding to the key frame based on the key frame description, reflecting the motion trajectory and timing changes within the segment;

[0025] The compression encoding unit is used to perform lossless compression encoding on the frame-level description and the segment description.

[0026] Preferably, the key frame compression module includes:

[0027] Latent variable encoding unit, used to map keyframes to latent space variables;

[0028] A hyper-prior modeling unit for generating feature distribution representations before quantization;

[0029] The vector quantization submodule is used to perform vector matching according to the codebook and generate an index matrix;

[0030] An entropy encoder is used to perform compression encoding on the index matrix.

[0031] Preferably, the key frame reconstruction module includes:

[0032] Conditional diffusion reconstruction network, which inputs quantized latent variable indices and frame-level semantic descriptions and recovers the latent variables through the diffusion modeling process;

[0033] The VAE decoder is used to decode the recovered latent variables and output the restored image frame.

[0034] Preferably, the conditional diffusion reconstruction network performs latent variable reconstruction on the key frames based on the following denoising diffusion process:

[0035]

[0036] in, represents the random sampling of latent variables from the standard normal distribution at the beginning of the diffusion process; z d,t represents the latent variable vector generated in the t-th step of the diffusion process; z d,t-1 represents the latent variable of the diffusion process at step t-1; α t represents the noise attenuation coefficient of the diffusion process at step t; represents the cumulative noise product from step 1 to step t; σ t represents the standard deviation of the sampling noise at step t in the diffusion process; z i Represents the key frame latent variables obtained at the encoding end; Represents the semantic description text or vector representation of the corresponding key frame; t represents the current diffusion time step; ∈ θ (·) represents the denoising prediction model parameterized by a neural network, which is used to estimate the noise residual under the current conditions; z~N(0,I) represents the noise vector sampled from the standard Gaussian distribution;

[0037] The denoising diffusion process is performed by multi-step iteration. d,T Backtrack to z d,0 , and finally use the VAE decoder to decode z d,0 Decode and obtain the corresponding key frame image.

[0038] Preferably, the video generation module includes:

[0039] A prediction generation unit, configured to predict and generate subsequent frames based on the current key frame and its segment description information;

[0040] An interpolation generation unit, configured to interpolate and generate an intermediate frame between two adjacent key frames based on their semantic descriptions;

[0041] Parameter control unit, used to support dynamic adjustment of frame rate and resolution.

[0042] The present invention provides a generative video codec based on a multimodal large model. It has the following beneficial effects:

[0043] 1. The present invention introduces a key frame screening strategy that combines semantic and motion information, which can dynamically select representative and high-information key frames. Unlike traditional key frame selection methods based on simple motion changes or fixed intervals, the present invention effectively reduces the storage and processing requirements of redundant frames while retaining the core information of the video content. This method allows video compression to significantly improve storage and transmission efficiency while ensuring quality.

[0044] 2. The present invention proposes a key frame compression strategy based on the conditional latent variable diffusion model. It performs preliminary dimensionality reduction through a variational autoencoder and combines it with a diffusion model to effectively compress key frame information, significantly reducing the space requirements for key frame storage. Compared with traditional video encoding methods, this method can still provide high-quality video reconstruction at extremely low bit rates, breaking through the bottleneck of low-bit-rate video compression and ensuring that details are retained and quality is not lost during the compression process.

[0045] 3. The semantically guided video reconstruction method of the present invention improves the accuracy and consistency of inter-frame interpolation when supplementing frames by introducing text descriptions or additional contextual information. This method solves the common artifact problems in traditional frame supplementation technologies based on motion compensation or interpolation, and effectively improves the detail consistency of the generated frames. Especially in long-term video generation tasks, it can more naturally restore motion and scene changes, ensuring video smoothness and visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a framework diagram of the video encoding and decoding of the present invention;

[0047] Figure 2 This is a general diagram of the video compression framework of the present invention;

[0048] Figure 3 A schematic diagram of a multimodal semantic description generation module of the present invention;

[0049] Figure 4 Schematic diagram of an efficient key frame compression module of the present invention;

[0050] Figure 5 Schematic diagram comparing the RD performance of the present invention with that of various classic frameworks. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] Please see the attached Figure 1 -Attached Figure 2 The embodiment of the present invention provides a generative video codec based on a multimodal large model, including:

[0053] A key frame selection module is used to determine key frames based on the semantic features and motion features of the input video frame;

[0054] First, the keyframe selection module receives a series of input video frames and performs feature extraction and comparative analysis on each frame to determine which frames should be selected as keyframes. Keyframes not only represent important changes in the video content but also effectively reduce redundant data, thereby improving the efficiency of video encoding and compression.

[0055] Specifically, in this embodiment, the key frame selection module includes three core units: a semantic feature extraction unit, a motion feature extraction unit, and an inter-frame difference judgment unit.

[0056] The semantic feature extraction unit is responsible for extracting semantic information from each frame of video. This unit uses a pre-trained graph-text neural network model (such as CLIP) to extract the semantic features of the video frame. The image encoder converts each frame of image F n Converted to a d-dimensional semantic feature vector Represents the visual content of the frame.

[0057] The motion feature extraction unit extracts the motion amplitude between consecutive frames through the optical flow estimation algorithm. Specifically, the optical flow estimation algorithm (such as RAFT) is used to calculate the F n and F l The pixel-level motion information between the two frames is averaged and the motion field is averaged to obtain the motion amplitude f of each frame. Raft (F n ,F l ).

[0058] The inter-frame difference judgment unit combines semantic features and motion features to determine whether the current frame is a key frame. Specifically, it calculates the difference D(F) between the current frame and the previous key frame using the following formula: n ,F l ):

[0059] D(Fn ,F l )=λ1(1-f Clip (F n ,F l ))+λ2f Raft (F n ,F l )-D th ;

[0060] Among them, F n is the current frame to be judged, F l is the last selected keyframe; f Clip (F n ,F l ) is the semantic similarity between the current frame and the previous key frame, and the cosine similarity calculated after extracting the feature vector based on the joint image-text model is calculated as their feature vector v n and v l The cosine similarity between:

[0061]

[0062] f Raft (F n ,F l ) is the motion amplitude calculated by the optical flow network, representing the frame F n With F l The intensity of exercise between

[0063] λ1 and λ2 are weighting coefficients of semantic difference term and motion change term, which are used to adjust the relative importance of the two. Both are positive real numbers;

[0064] D th It is the preset difference judgment threshold used to determine whether the current frame is a key frame, usually set to 0.

[0065] When D(F n ,F l )>0, indicating the current frame F n The semantic change or motion change between the previous key frame and the current frame F is large, which determines the n as a new keyframe.

[0066] The semantic feature extraction unit and the motion feature extraction unit work in parallel to extract semantic and motion features for each frame. The inter-frame difference determination unit then receives these features and calculates a keyframe selection decision value. Each frame is compared with the previous keyframe. If the difference exceeds a threshold, the current keyframe is updated.

[0067] In this embodiment, the inter-frame difference judgment unit not only considers the comprehensive influence of semantic features and motion features, but also can adjust the parameters λ1, λ2 and Dth To optimize the model to adapt to different video content or application requirements.

[0068] In the specific implementation, the threshold D th It can be dynamically adjusted according to the characteristics of the data set, or an automatic adjustment mechanism can be used to improve accuracy.

[0069] In one possible implementation, this module can be deployed on an edge computing platform for real-time video analysis, improving computational efficiency through hardware acceleration. It can also work in conjunction with other video processing modules to form a complete video encoding and decoding system.

[0070] Please see the attached Figure 3 ,Multimodal semantic description generation module, used to generate semantic description information based on key frames and corresponding video clips;

[0071] In this embodiment, the image and video description module extracts visual information from the video using a large multimodal model (LMM) and combines it with spatiotemporal information to generate semantic descriptions of keyframes and video segments. This module aims to generate more accurate semantic descriptions based on selected keyframes and their corresponding segments, further analyzing the motion information in the video, and ultimately improving the quality and efficiency of the video encoding and reconstruction processes.

[0072] First, assume that the key frames selected by the key frame selection module are F1, F2, ..., F n , these key frames divide the video into n n video clips {C1,C2,…,C n}, each video segment C i corresponds to a set of consecutive frames, surrounding the key frame F i Perform time division. In this way, each video segment C i Contains the surrounding F i and the motion information and semantic information of the frames before and after it.

[0073] For each key frame F i and its corresponding video clip C i We use a pre-trained multimodal large model (such as the Qwen-VL-Instruct-7B model) to extract visual information. This model maps the key frame image into a feature vector v in the semantic space through image-text joint representation. i ∈R d , and further generate text descriptions related to each key frame

[0074] The description Contains the following elements:

[0075] Object background information: describes the objects or background that appear in the scene, such as people, objects, environment, etc.

[0076] Difference information: the semantic difference between the current key frame and the previous and next frames.

[0077] Detailed information: Supplementary description of the details of an object, scene or action.

[0078] Overall structure: Describe the key objects in the scene and the relationships between them.

[0079] Therefore, the key frame D i Description It can be expressed as:

[0080] To better describe the video clip C i We use the key frames generated above to describe the timing information and motion changes in Combined with the spatiotemporal information, the motion information in the video clip is further analyzed. Specifically, the motion amplitude f is calculated based on the optical flow algorithm (such as RAFT). flow (F i-1 ,F i ) and the timing changes dynamically to generate video clip C i Motion information description

[0081] Fragment Description Combined with keyframe description and motion information, forming a complete semantic and temporal description of the clip. The formula is expressed as:

[0082]

[0083] Among them, f flow (F i-1 ,F i ) and f flow (F i ,F i+1 ) represents the optical flow information between consecutive frames, reflecting the motion amplitude and direction.

[0084] In order to improve the transmission efficiency, we describe the generated key frames and video clip description The LZW (Lempel-Ziv-Welch) algorithm is used for lossless compression. The LZW algorithm significantly reduces the storage space of data by encoding the redundant parts of the text description.

[0085] Specifically, the description information and Compressed into:

[0086]

[0087] This process ensures the efficiency of storage and transmission of the generated description information while maintaining the integrity and reconstructibility of the description.

[0088] During implementation, the semantic extraction unit and the motion information generation unit work together. First, the semantic extraction unit extracts visual information from key frames and generates preliminary frame-level descriptions. Then, the motion information generation unit generates video segment-level descriptions based on the frame-level descriptions and motion features. Finally, the compression encoding unit compresses these descriptions to ensure efficient storage and transmission.

[0089] For example, in a sports video, a keyframe might correspond to the moment a player shoots a basketball, describing the Contains semantic information about the shooting action, the player, and the background. The motion information generation unit further combines the player's motion trajectory and the flight path of the basketball to generate a description of the video clip. Finally, all descriptions are encoded using the LZW compression algorithm to ensure efficient storage and transmission at low bit rates.

[0090] Through the above implementation, the image and video description module of the present invention can effectively enhance the semantic modeling capabilities of video content, leveraging multimodal information to generate more accurate keyframe and video segment descriptions. Furthermore, the use of LZW lossless compression technology improves the efficiency of video encoding, transmission, and storage. This module provides semantic consistency and efficient description support for video compression and reconstruction, and is widely applicable in scenarios such as streaming media, video surveillance, and virtual reality.

[0091] Please see the attached Figure 4 ,Keyframe compression module, used to perform latent variable modeling and entropy coding on key frames to generate compressed data;

[0092] The keyframe compression module proposed in this paper aims to efficiently compress keyframes in a video, generating compressed data through latent variable modeling and entropy coding techniques. In this module, keyframes are first mapped to a latent space using latent variable coding. Then, hyper-prior modeling is used to generate a pre-quantized feature distribution. Subsequently, a vector quantization submodule uses a codebook for vector matching, and finally, an entropy encoder is used for compression encoding. This process not only improves compression efficiency but also ensures high-quality reconstruction during decoding.

[0093] The latent variable encoding unit is responsible for converting the input key frame F i This process is done by a variational autoencoder (VAE) or similar latent variable model. Specifically, the key frame F i Projected to the latent space variables And generate the latent variable distribution q(z i ). The formula of the mapping process is expressed as:

[0094] z i =Encoder(F i ), where z i represents the encoding variable of the latent space, F i is the input key frame image.

[0095] The role of the super prior modeling unit is to generate the feature distribution representation before quantization. This unit is constructed by i Modeling, generating its hyper-prior distribution parameters This modeling process takes into account the structural characteristics of the latent space and generates its distribution model for the subsequent quantization process. The formula is expressed as:

[0096] h i =Hyperprior(z i ), where h i represents the hyper-prior features of the latent space, z i is the latent space variable obtained from the latent variable encoder.

[0097] The task of the vector quantization submodule is to match the latent variables according to the codebook and generate an index matrix. This process converts the latent variables h i Mapped to a discrete dictionary index K i , thus performing data compression. The quantization operation selects the closest codeword based on the minimum Euclidean distance:

[0098]

[0099] Among them, h i (u,v) represents an element of the super prior feature, v k is the kth dictionary element in the codebook, d is the Euclidean distance, k * (u,v) represents the quantized index matrix.

[0100] Finally, the quantization result K i It will be used as input for subsequent entropy coding.

[0101] The entropy encoder is responsible for the index matrix K i Entropy coding is a lossless data compression method that effectively reduces the amount of data by performing short coding on the values that appear more frequently in the data and long coding on the values that appear less frequently. i , the entropy encoder generates the compressed code stream Enc(K i ):

[0102] Enc(K i )=Entropy Encode(K i );

[0103] The encoding process may employ common entropy coding techniques, such as Huffman coding or arithmetic coding, to further improve compression efficiency.

[0104] In implementation, each unit is connected via a data bus. The latent variable encoding unit maps the input keyframes into a latent space and generates a distributed representation via the hyper-prior modeling unit. Subsequently, the vector quantization submodule performs quantization based on a dictionary, and finally, an entropy encoder performs lossless compression on the quantized index matrix. The design of this entire module ensures the efficiency and reversibility of the keyframe compression process, enabling high-quality reconstruction during decoding.

[0105] Assume that the input key frame F i For a basketball shooting scene in a video, the latent variable encoding unit maps the frame to the latent space z i The super-prior modeling unit generates a super-prior feature h according to the latent space variable i , which is then quantized into an index matrix K by the vector quantization submodule i Finally, the entropy encoder performs K i Compression is performed to save bandwidth and space during video transmission or storage.

[0106] The keyframe compression module in this embodiment effectively improves the compression efficiency of video keyframes through techniques such as latent variable modeling, hyper-prior modeling, vector quantization, and entropy coding. Latent variable coding ensures accurate representation of image information, hyper-prior modeling provides additional distribution information, vector quantization further reduces data storage requirements, and entropy coding ensures lossless compression. This module achieves high-quality keyframe compression at low bitrates, making it suitable for a variety of application scenarios such as streaming media transmission and video surveillance.

[0107] The key frame reconstruction module is used to reconstruct the key frame based on the compressed data and semantic description information using the conditional diffusion model;

[0108] The keyframe reconstruction module in this embodiment of the present invention is used to restore keyframe images with high quality at the decoding end based on compressed data and frame-level semantic description information. This module primarily consists of a conditional diffusion reconstruction network and a VAE decoder, which collaboratively perform latent variable recovery and image reconstruction.

[0109] The key frame reconstruction module includes a diffusion modeling unit and an image decoding unit. The diffusion modeling unit connects the latent variables after entropy decoding with the semantic description information and outputs the restored latent variables; the image decoding unit is connected to the diffusion modeling unit and outputs the final image frame.

[0110] The index matrix k obtained after entropy decoding * After the inverse quantization mapping, the initial input h of the latent variable is obtained i , and its reconstruction method is:

[0111] in, is the kth * vectors; is the initial latent variable after decoding.

[0112] The reconstruction process then uses a conditional diffusion modeling mechanism to complete the fitting and recovery of the latent variables at the encoding end. The diffusion process iterates in a time-reverse manner, moving from random noise to the original latent variables.

[0113] The initial state of the diffusion process is sampled from a standard normal distribution:

[0114] During the diffusion process, the current latent variable state z d,t According to the conditional denoising network ∈ θ Calculate the next moment latent variable z with the previous state d,t-1 The denoising process is as follows:

[0115]

[0116] in, represents the latent variable at step t; z i Represents the latent variable at the coding end (used to provide conditions); Represents key frame F i Semantic description (text or its vectorized representation); t represents the diffusion time step; α t represents the noise attenuation coefficient at step t; represents the noise cumulative product; σ t represents the standard deviation of the noise corresponding to the diffusion time step; ∈ θ (·) represents the parameterized denoising prediction model of the conditional diffusion network; represents Gaussian sampled noise.

[0117] The above diffusion process is iterated z d,T After steps, the final restored latent variable z is obtained d,0 .

[0118] The final latent variable is input to the decoder for restoration:

[0119] in, is the restored key frame image, and Decoder represents the decoder network structure, corresponding to the VAE decoder during training.

[0120] The denoising network structure ∈ θ It combines temporal embedding and multimodal attention mechanism to support the simultaneous processing of latent variable states and semantic description vectors, improving semantic consistency.

[0121] Semantic Description The multi-round dialogue information extracted from the image and video description module using Qwen-VL-Instruct-7B is first converted into a vector by the language encoder and then used as the denoising condition input.

[0122] In actual deployment, the module implementation process is as follows: First, the index matrix of the entropy coding at the encoder is decoded and mapped to the latent variable h i ; Then initialize the latent variable to random noise z d,T , combined with z i and Reconstruct the latent variable z through T diffusion steps d,0 ;Finally, call the VAE decoder to generate the key frame image

[0123] For example, if a keyframe is a street view image, the semantic description This includes information such as pedestrians, vehicles, buildings, and sunlight direction. The diffusion model uses this semantic information to guide the direction of latent variable recovery, thereby ensuring the accuracy and structural integrity of the street scene elements in the restored image.

[0124] The key frame reconstruction module and the compression module form an encoding-decoding closed loop, ensuring the fidelity of semantic information during the compression and reconstruction process, and significantly improving the reconstruction quality under ultra-low bit rate conditions.

[0125] The video generation module is used to supplement or insert non-key frames based on semantic description information and key frames to reconstruct the complete video.

[0126] The video generation module is used at the decoding end to reconstruct non-keyframes between or after keyframes based on the semantic descriptions of keyframe images and their fragments, thereby restoring the complete video. The module comprises a prediction generation unit, an interpolation generation unit, and a parameter control unit, which work together to construct the non-keyframe generation path.

[0127] The prediction generation unit is used to generate consecutive frames after the current key frame, and the input is the current key frame image F i The corresponding fragment description information Output is a sequence of subsequent frame images

[0128] The interpolation generation unit is used to insert intermediate frames between adjacent key frames. The input is two adjacent key frames F i ,F i+1 and its corresponding semantic description Output is a sequence of intermediate frame images

[0129] The parameter control unit is connected to the above two generation units, and is used to schedule the generation path and dynamically set the frame rate and image resolution. The input is the target output frame timestamp sequence and resolution configuration, and the output is the scheduling instruction and sampled image frame.

[0130] In the prediction generation unit, the video latent variable generation model is first established, and the input is the key frame semantic description With the key frame image F i , construct the initial state of conditional diffusion:

[0131] in, For the predicted frame The initial state of the diffusion latent variable is

[0132] The conditional diffusion inversion process is then used to gradually predict the latent variables:

[0133] in, The latent variable for the predicted frame at step t; z i is the key frame latent variable, used to provide context prior; ∈ θ (·) is the conditional denoising sub-model in the video diffusion network (SEINE); α t 、 σ t They are diffusion scheduling coefficient, accumulation coefficient and noise coefficient respectively.

[0134] Finally, the latent variable is input into the image decoder to restore the generated frame image:

[0135] The Decoder is an image reconstruction network, which is consistent with the decoder structure used in the key frame compression module.

[0136] In the interpolation generation unit, let the two key frames be F i 、F i+1 , m-1 frames need to be inserted in between, the interpolation frame index is k∈{1,…,m-1}, and the interpolation factor is defined as:

[0137] Construct the initial latent variable of the interpolation frame as the linear fusion of the key frame features:

[0138] Where E(·) is the shared encoder network structure.

[0139] The interpolation frame generation process reuses the conditional diffusion process and introduces semantic description and interpolation weights:

[0140]

[0141] The final intermediate frame image is as follows:

[0142]

[0143] The parameter control unit reads the target output timestamp sequence Determine whether each time point falls within the prediction path or the interpolation path, and call the corresponding generation process; when the output image size is inconsistent with the target resolution (h', w'), call the super-resolution module for reconstruction:

[0144] SRNet is a pre-trained super-resolution reconstruction network used to improve the ability to restore image details.

[0145] The module's structure is as follows: the prediction generation unit and interpolation generation unit are connected to the keyframe reconstruction module and semantic description module, respectively, with their outputs connected to the frame buffer. The parameter control unit is located before the generation path, reading user or system parameters and then calling the generation module to perform inference.

[0146] In one embodiment, the input clip is a sports game scene, and the key frame is the player's jump frame F i With floor frame F i+1 , semantic description This includes "fast movement during the run-up phase, smooth movements, and a backdrop of spectator stands." Based on the set frame rate, the system interpolates between two keyframes to generate images of each stage of the jump process, achieving a natural and continuous presentation of the movements and outputting a complete, continuous clip.

[0147] Through the above structure and process, the video generation module can combine semantic information to supplement non-key frame content under ultra-low bit rate conditions, improve the coherence and detail fidelity of the restored video, and provide support for overall video reconstruction.

[0148] Experimental Design: Comparative Experiment on Generative Video Encoding and Decoding Performance Based on Multimodal Large Model

[0149] Purpose of the experiment:

[0150] This experiment aims to validate the performance of our proposed video codec framework (hereafter referred to as "this framework") based on a large multimodal model (LMM) and a conditional diffusion model (CDM) in video compression and reconstruction, particularly at low bitrates. The experiment will focus on comparing this framework with classic video codecs (such as VVC) and state-of-the-art learning codecs (such as DCVC) in terms of RD (Rate-Distortion) performance, specifically focusing on compression efficiency and visual quality in ultra-low bitrate scenarios.

[0151] Experimental setup:

[0152] This experiment uses a standard video compression test dataset, which includes video clips from various scenes, including faces, natural scenes, sports, and virtual reality videos. We adjust the bitrate, especially at very low bitrates (such as 0.010bpp and 0.014bpp), to compare the RD performance of the following encoders:

[0153] VVC (the latest standard video codec);

[0154] DCVC (deep learning based convolutional video encoder family);

[0155] M3-CVC (our proposed video encoding and decoding framework based on multimodal large models).

[0156] Experimental process:

[0157] Data preprocessing and video segmentation: First, the video segment is divided into multiple sub-segments, and important frames are selected using a keyframe extraction algorithm. For each video segment, its keyframes and corresponding semantic description information are extracted, and LMM is used for feature extraction and description generation.

[0158] Encoding process: Video clips are encoded using various video codecs. In this framework, keyframes are first compressed using a VQVAE encoder to generate latent variables. These are then decoded using a conditional diffusion model to reconstruct the keyframes. Non-keyframes are then generated using prediction and interpolation modes based on the keyframes and semantic descriptions.

[0159] Decoding and reconstruction: Each encoded video stream is decoded and reconstructed using the generative video reconstruction method of the M3-CVC framework (including prediction generation and interpolation generation), and then the reconstruction quality and semantic consistency of the video are evaluated.

[0160] Performance evaluation indicators:

[0161] The performance of each codec was compared using the following criteria:

[0162] BD-rate (bitrate-distortion ratio): used to evaluate the relationship between the distortion (such as PSNR or SSIM) of the reconstructed video and the bit rate at different bit rates.

[0163] PSNR (Peak Signal-to-Noise Ratio): Measures the image quality between the reconstructed video and the original video. A higher value indicates better image quality.

[0164] SSIM (Structural Similarity Index): measures the structural and visual consistency of an image. Values closer to 1 indicate better video quality.

[0165] Visual perception quality: Through manual evaluation of the naturalness, clarity, and detail retention of video reconstruction, the visual fidelity of video reconstruction at low bit rates is emphasized.

[0166] Experimental results: According to the experimental settings, the experimental results show (see Appendix Figure 5 ), the M3-CVC framework shows significant advantages over VVC and DCVC in ultra-low bitrate conditions in multiple scenarios. The specific results are as follows:

[0167] Bitrate and distortion comparison: Compared to VVC, the M3-CVC framework reduces BD-rate by 20%-40% in extremely low bitrate scenarios between 0.010bpp (bits per pixel) and 0.014bpp. This means that at the same distortion level, this framework saves more bitrate than VVC, demonstrating its significant advantage in compression efficiency.

[0168] Video Quality Evaluation: In PSNR and SSIM evaluations, the M3-CVC framework maintains high image quality at low bitrates, particularly excelling in detail preservation and blur reduction. Compared to VVC, at the same compression rate, this framework reconstructs clearer, more detailed, and significantly reduces blur.

[0169] Competitiveness with DCVC: Compared to the DCVC family, this framework not only maintains competitive RD performance but also improves semantic consistency and perceptual quality. In particular, in motion videos and scenes rich in details, M3-CVC can better preserve the structure and dynamic changes of video content, improving the visual quality of video reconstruction.

[0170] Experimental results demonstrate that the generative video codec framework (M3-CVC), based on a large multimodal model and a conditional diffusion model, offers significant advantages over VVC and DCVC in ultra-low bitrate scenarios, particularly in terms of compression efficiency and video quality. This framework significantly reduces bitrate and improves the quality of video reconstruction while maintaining high semantic consistency and perceptual fidelity, demonstrating broad application prospects, particularly in bandwidth-constrained scenarios such as streaming media, remote conferencing, and virtual reality.

[0171] How it works: The video encoding and decoding process begins with video input. During the video input phase, each frame of the original video sequence is decomposed into multiple information streams, including image content, motion information, and semantic features. These information streams are then passed to the keyframe selection module. This module's task is to select representative keyframes based on the image content and motion information of the input video frames. These keyframes play a central role in the subsequent video generation and reconstruction process. Keyframe selection criteria include the amount of image content change, motion characteristics, and their contribution to the overall semantics of the video.

[0172] Next, the selected keyframes and their corresponding semantic descriptions are fed into the multimodal semantic description generation module. This module generates corresponding semantic descriptions based on image content and motion features. These descriptions encompass not only the image's content hierarchy but also the temporal changes between frames, the dynamics of objects, and the scene's background information. These generated semantic descriptions serve as conditional input for subsequent video generation, providing semantic guidance for the model when generating non-keyframes.

[0173] The video generation module plays a crucial role in the video generation process. This module includes multiple sub-units, among which the prediction generation unit uses the current key frame and its semantic description information to predict subsequent non-key frames, thereby maintaining video continuity. Through the conditional generation of key frames and their semantic descriptions, subsequent frame images are generated, thereby maintaining the smoothness and consistency of the video content. At the same time, the interpolation generation unit interpolates two adjacent key frames and their semantic description information, inserting an intermediate frame between the two key frames to increase the coherence and richness of detail in the video. This process controls the weight of frame generation through interpolation factors, allowing the generated frames to transition naturally and supplement the lost details in the video.

[0174] During the video generation process, the parameter control unit is responsible for dynamically adjusting the frame rate and resolution. This unit plays a key scheduling role in the video generation process, ensuring that the output video quality meets the specified requirements by adaptively adjusting the frame rate and resolution. For example, at high frame rates, the system can increase the number of interpolated frames to refine the video content; at low frame rates, the number of generated frames is reduced to maintain efficiency and conserve computing resources.

[0175] After being processed by these units, the generated non-keyframes still require further encoding to reduce the data size. This is where the keyframe compression module comes in. This module performs latent variable modeling on the keyframes and compresses the image information of the keyframes using methods such as entropy coding to reduce the data size while preserving as much image detail as possible. During this process, the keyframe compression method not only considers image content but also optimizes semantic information, ensuring that the compressed video still reflects the semantic characteristics of the original video.

[0176] The compressed keyframe data and corresponding semantic description information are input into the keyframe reconstruction module. This module uses the conditional diffusion model to reconstruct the keyframes, combining the compressed data with the semantic description to generate high-quality keyframe images. Through this reconstruction process, the system can recover missing or lost image content in the original video while maintaining a visual representation consistent with the semantic description.

[0177] In the final stage of video reconstruction, the system stitches all the generated segments together to create a complete video sequence. By properly arranging keyframes and interpolated frames, the system seamlessly connects the various video sections to produce a complete and smooth video output.

[0178] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A generative video codec based on a multimodal large model, characterized in that: include: A key frame selection module is used to determine key frames based on the semantic features and motion features of the input video frame; a multimodal semantic description generation module, configured to generate semantic description information based on the key frames and corresponding video clips; The key frame compression module is used to perform latent variable modeling and entropy coding on key frames to generate compressed data; A key frame reconstruction module, configured to reconstruct the key frame using a conditional diffusion model based on the compressed data and semantic description information; The video generation module is used to supplement or insert non-key frames based on the semantic description information and key frames to reconstruct the complete video.

2. The generative video codec based on a multimodal large model according to claim 1, characterized in that: The key frame selection module includes: A semantic feature extraction unit, used to extract semantic feature vectors of video frames based on a pre-trained graph-text neural network model; A motion feature extraction unit, configured to extract the motion amplitude between video frames based on an optical flow estimation algorithm; The inter-frame difference judgment unit is used to perform key frame selection judgment based on the semantic features and motion features.

3. The generative video codec based on a multimodal large model according to claim 2, characterized in that: The inter-frame difference judgment unit makes judgment based on the following key frame decision function: D(F n ,F l )=λ1(1-f Clip (F n ,F l ))+λ2f Raft (F n ,F l )-D th ; Among them, F n is the current frame to be judged, F l is the last selected keyframe; f Clip (F n ,F l ) is the semantic similarity between the current frame and the previous key frame, and the cosine similarity is calculated after extracting the feature vector based on the joint image-text model; f Raft (F n ,F l ) is the average motion change between the current frame and the previous key frame, which is calculated based on the optical flow network; λ1 and λ2 are the weighted coefficients of the semantic difference term and the motion change term respectively; D th is the set difference judgment threshold; when D(F n ,F l )>0, frame F n is identified as a key frame.

4. The generative video codec based on a multimodal large model according to claim 1, characterized in that: The multimodal semantic description generation module includes: Semantic extraction unit, which uses a multimodal large model to generate frame-level semantic descriptions of key frames, including background information, object descriptions, and local differences; A motion information generation unit, configured to further generate a video-level description of a segment corresponding to the key frame based on the key frame description, reflecting the motion trajectory and timing changes within the segment; The compression encoding unit is used to perform lossless compression encoding on the frame-level description and the segment description.

5. The generative video codec based on a multimodal large model according to claim 1, characterized in that: The key frame compression module includes: Latent variable encoding unit, used to map keyframes to latent space variables; A hyper-prior modeling unit for generating feature distribution representations before quantization; The vector quantization submodule is used to perform vector matching according to the codebook and generate an index matrix; An entropy encoder is used to perform compression encoding on the index matrix.

6. The generative video codec based on a multimodal large model according to claim 1, characterized in that: The key frame reconstruction module includes: Conditional diffusion reconstruction network, which inputs quantized latent variable indices and frame-level semantic descriptions and recovers the latent variables through the diffusion modeling process; The VAE decoder is used to decode the recovered latent variables and output the restored image frame.

7. The generative video codec based on a multimodal large model according to claim 6, characterized in that: The conditional diffusion reconstruction network reconstructs the latent variables of the key frames based on the following denoising diffusion process: in, represents the random sampling of latent variables from the standard normal distribution at the beginning of the diffusion process; z d,t represents the latent variable vector generated in the t-th step of the diffusion process; z d,t-1 represents the latent variable of the diffusion process at step t-1; α t represents the noise attenuation coefficient of the diffusion process at step t; represents the cumulative noise product from step 1 to step t; σ t represents the standard deviation of the sampling noise at step t in the diffusion process; z i Represents the key frame latent variables obtained at the encoding end; Represents the semantic description text or vector representation of the corresponding key frame; t represents the current diffusion time step; ∈ θ (·) represents the denoising prediction model parameterized by a neural network, which is used to estimate the noise residual under the current conditions; z~N(0,I) represents the noise vector sampled from the standard Gaussian distribution; The denoising diffusion process is performed by multi-step iteration. d,T Backtrack to z d,0 , and finally use the VAE decoder to decode z d,0 Decode and obtain the corresponding key frame image.

8. The generative video codec based on a multimodal large model according to claim 1, characterized in that: The video generation module includes: A prediction generation unit, configured to predict and generate subsequent frames based on the current key frame and its segment description information; An interpolation generation unit, configured to interpolate and generate an intermediate frame between two adjacent key frames based on their semantic descriptions; Parameter control unit, used to support dynamic adjustment of frame rate and resolution.

Images