A video compression transmission method and related device

Abstract

The application discloses a video compression transmission method and related equipment, the method comprises the following steps: obtaining a video sequence to be compressed, taking the first frame and the last frame as conditional frames; inputting the video sequence into the encoder of a pre-trained variational autoencoder to obtain a target latent space representation; processing the target latent space representation through the downsampling module of the compressor to generate an extreme compression representation; transmitting the conditional frames and the extreme compression representation to the receiving end to enable the receiving end to reconstruct the video through the upsampling module of the compressor, the generation model and the decoder of the variational autoencoder to obtain a reconstructed video sequence; the application provides time sequence boundary information through the conditional frames, and in combination with the reconstruction capability of the generation model, can effectively reduce the block effect, blur and high-frequency detail loss commonly seen in traditional methods; the conditional frames and the extreme compression representation significantly reduce the transmission code rate and bandwidth demand, can significantly improve the rate distortion performance, and can be widely applied to the technical field of video compression.

Description

Technical Field

[0001] This invention relates to the field of video compression technology, and in particular to a video compression and transmission method and related equipment. Background Technology

[0002] In the field of video processing and compression coding, video compression technology is mainly divided into three categories: traditional coding, neural compression based on deep learning, and methods based on diffusion models. However, each of these methods has obvious limitations in practical applications.

[0003] Current mainstream video compression standards (such as H.264, H.265, and H.266) employ a hybrid coding strategy, combining motion estimation, transform coding, and entropy modeling. However, at extremely low bitrates, they exhibit severe blockiness, blurring, and loss of high-frequency details. In recent years, deep learning-based neural video compression methods (such as the DCVC series) have improved rate-distortion performance, but at extremely low bitrates (e.g., 0.01 bpp), the perceptual quality of reconstructed videos deteriorates sharply, resulting in severe blurring and distortion. Summary of the Invention

[0004] The main objective of this invention is to provide a video compression and transmission method, apparatus, electronic device, storage medium, and program product, aiming to solve at least one problem of the prior art.

[0005] To achieve the above objectives, one aspect of the present invention provides a video compression and transmission method, the method comprising:

[0006] Obtain the video sequence to be compressed, and use the first and last frames as conditional frames;

[0007] The video sequence is input into the encoder of the pre-trained variational autoencoder to obtain the target latent space representation. The target latent space representation is then processed by the downsampling module of the compressor to generate an extremely compressed representation.

[0008] The conditional frame and the extreme compression representation are transmitted to the receiving end so that the receiving end can reconstruct the video through the upsampling module of the compressor, the generation model and the decoder of the variational autoencoder to obtain the reconstructed video sequence.

[0009] The compressor and the generative model are pre-trained using compression loss based on conditional prior distribution and posterior distribution, and reconstruction loss based on latent space representation, compression representation, and velocity field. The conditional prior distribution is constructed based on the encoded information of the masked video after removing the first and last frames, and the posterior distribution is constructed based on the Gaussian distribution parameters recovered by the upsampling module. The latent space representation represents the encoder output, the compression representation represents the Gaussian distribution parameters recovered by the upsampling module, and the velocity field represents the training output of the generative model.

[0010] In some embodiments, video reconstruction is performed using the upsampling module of the compressor, the generative model, and the decoder of the variational autoencoder to obtain a reconstructed video sequence, including the following steps:

[0011] The extreme compression representation is processed by the upsampling module of the compressor, and the mean value of the output Gaussian distribution parameters is used as the target compression representation.

[0012] The conditional frame and the target compressed representation are input into the generative model, and the intermediate frame potential representation is obtained by the generative model through step-by-step reconstruction based on a preset equidistant time step.

[0013] The intermediate video frame is obtained by converting the latent representation of the intermediate frame back to the pixel space through the decoder of the variational autoencoder.

[0014] The intermediate video frames and conditional frames are stitched together to obtain the reconstructed video sequence.

[0015] In some embodiments, transmitting a condition frame to the receiving end includes the following steps:

[0016] The conditional frame is compressed using image compression methods to obtain compressed information;

[0017] The compressed information is transmitted to the receiving end so that the receiving end can decode the compressed information and recover the condition frame.

[0018] In some embodiments, the method further includes the following steps:

[0019] Obtain the training video sequence, set the first and last frames of the training video sequence to zero, and obtain the masked video;

[0020] The training video sequence is input into the encoder and processed to obtain the latent space representation;

[0021] The masked video is input into the encoder, processed to obtain the prior parameters of the Gaussian distribution, and a conditional prior distribution is constructed based on the prior parameters.

[0022] The latent space representation is progressively compressed using a downsampling module to generate compressed features.

[0023] The compressed features are sampled and recovered by the upsampling module, and the output Gaussian distribution parameters are reparameterized to obtain a compressed representation.

[0024] The posterior distribution is constructed based on the Gaussian distribution parameters output by the upsampling module;

[0025] Training input samples for the generative model are constructed based on compressed representation;

[0026] Based on the conditional frames of the training input samples and training video sequences, and combined with preset equidistant time steps, the velocity field of the training input samples to the latent space representation is predicted by the generative model.

[0027] The compression loss is constructed based on the conditional prior and posterior distributions, the reconstruction loss is constructed based on the latent space representation, the compression representation and the velocity field, and the total loss is constructed based on the compression loss and the reconstruction loss.

[0028] Using the total loss, the parameters of the compressor and the parameters of the generative model are jointly optimized through gradient backpropagation.

[0029] In some embodiments, obtaining the compressed representation through reparameterization may include the following steps:

[0030] The standard deviation of the Gaussian distribution parameters output by the upsampling module is weighted by random parameters that follow a standard normal distribution.

[0031] The result of the weighted operation is summed with the mean value of the Gaussian distribution parameters output by the upsampling module to obtain the compressed representation.

[0032] In some embodiments, constructing training input samples for a generative model based on compressed representation includes the following steps:

[0033] Based on compressed representation, a straight path connecting the latent space representation is constructed;

[0034] Randomly select a point along the straight path as the training input sample;

[0035] The expression for the training input sample is:

[0036] ;

[0037] In the formula, Let represent the training input sample at time step t, where t represents the corresponding time step on the straight path. Representing the latent space representation, This indicates compressed representation.

[0038] In some embodiments, a compression loss is constructed based on the conditional prior distribution and the posterior distribution, a reconstruction loss is constructed based on the latent space representation, the compression representation, and the velocity field, and a total loss is constructed based on the compression loss and the reconstruction loss, including the following steps:

[0039] The compression loss is constructed based on the KL divergence between the posterior distribution and the conditional prior distribution;

[0040] The velocity field is used as the generated value, and the difference between the compressed representation and the latent space representation is used as the original value. The reconstruction loss is constructed based on the difference index between the generated value and the original value.

[0041] The compression loss is weighted using preset hyperparameters, and then combined with the reconstruction loss to construct the total loss.

[0042] To achieve the above objectives, another aspect of the present invention provides a video compression and transmission apparatus, the apparatus comprising:

[0043] The first module is used to obtain the video sequence to be compressed, and the first and last frames are used as conditional frames.

[0044] The second module is used to input the video sequence into the encoder of the pre-trained variational autoencoder, process it to obtain the target latent space representation, and process the target latent space representation through the downsampling module of the compressor to generate an extremely compressed representation;

[0045] The third module is used to transmit the conditional frame and the extreme compression representation to the receiving end, so that the receiving end can reconstruct the video through the upsampling module of the compressor, the generation model and the decoder of the variational autoencoder to obtain the reconstructed video sequence.

[0046] The compressor and the generative model are pre-trained using compression loss based on conditional prior distribution and posterior distribution, and reconstruction loss based on latent space representation, compression representation, and velocity field. The conditional prior distribution is constructed based on the encoded information of the masked video after removing the first and last frames, and the posterior distribution is constructed based on the Gaussian distribution parameters recovered by the upsampling module. The latent space representation represents the encoder output, the compression representation represents the Gaussian distribution parameters recovered by the upsampling module, and the velocity field represents the training output of the generative model.

[0047] In some embodiments, the apparatus further includes a fourth module for performing the following operations:

[0048] Obtain the training video sequence, set the first and last frames of the training video sequence to zero, and obtain the masked video;

[0049] The training video sequence is input into the encoder and processed to obtain the latent space representation;

[0050] The masked video is input into the encoder, processed to obtain the prior parameters of the Gaussian distribution, and a conditional prior distribution is constructed based on the prior parameters.

[0051] The latent space representation is progressively compressed using a downsampling module to generate compressed features.

[0052] The compressed features are sampled and recovered by the upsampling module, and the output Gaussian distribution parameters are reparameterized to obtain a compressed representation.

[0053] The posterior distribution is constructed based on the Gaussian distribution parameters output by the upsampling module;

[0054] Training input samples for the generative model are constructed based on compressed representation;

[0055] Based on the conditional frames of the training input samples and training video sequences, and combined with preset equidistant time steps, the velocity field of the training input samples to the latent space representation is predicted by the generative model.

[0056] The compression loss is constructed based on the conditional prior and posterior distributions, the reconstruction loss is constructed based on the latent space representation, the compression representation and the velocity field, and the total loss is constructed based on the compression loss and the reconstruction loss.

[0057] Using the total loss, the parameters of the compressor and the parameters of the generative model are jointly optimized through gradient backpropagation.

[0058] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method.

[0059] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0060] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0061] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a video compression and transmission method, apparatus, electronic device, storage medium, and program product. This scheme acquires a video sequence to be compressed, using the first and last frames as conditional frames; inputs the video sequence into the encoder of a pre-trained variational autoencoder, processes it to obtain a target latent space representation, processes the target latent space representation through the downsampling module of the compressor to generate an extreme compressed representation; transmits the conditional frames and the extreme compressed representation to the receiving end, so that the receiving end can perform video reconstruction through the upsampling module of the compressor, the generative model, and the decoder of the variational autoencoder to obtain a reconstructed video sequence; wherein, the compressor and the generative model are pre-trained using compression losses based on conditional prior distributions and posterior distributions, and reconstruction losses based on latent space representation, compressed representation, and velocity field; the conditional prior distribution is constructed based on the encoding information of the masked video after removing the first and last frames, the posterior distribution is constructed based on the Gaussian distribution parameters recovered by the upsampling module, the latent space representation represents the output of the encoder, the compressed representation represents the Gaussian distribution parameters recovered by the upsampling module, and the velocity field represents the training output of the generative model. This invention uses the first and last frames of a video sequence as conditional frames and utilizes a pre-trained variational autoencoder and compressor to generate an extremely compressed representation. This allows the invention to maintain high reconstructed video quality even at extremely low bitrates. The conditional frames provide temporal boundary information, which, combined with the reconstruction capabilities of the generative model, effectively reduces block artifacts, blurring, and loss of high-frequency details common in traditional methods. Furthermore, the latent space representation of the variational autoencoder and the downsampling module of the compressor achieve extreme compression of video data, significantly reducing the transmission bitrate. At the same time, by transmitting only the conditional frames and the extremely compressed representation, bandwidth requirements are reduced, making it suitable for scenarios with limited network bandwidth. Specifically, the compressor and generative model are pre-trained using compression losses based on conditional prior and posterior distributions, as well as reconstruction losses based on latent space representation, compressed representation, and velocity field. This allows the entire system to achieve a balance between compression efficiency and reconstruction quality, significantly improving rate-distortion performance. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of an implementation environment for the video compression and transmission method provided in this embodiment of the invention;

[0063] Figure 2 This is a flowchart illustrating a video compression and transmission method provided in an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of the unfolding process of transmitting a conditional frame to the receiving end according to an embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of the video reconstruction unfolding process provided in an embodiment of the present invention;

[0066] Figure 5 This is a schematic diagram of an extended training process provided in an embodiment of the present invention;

[0067] Figure 6 This is a schematic diagram of the expansion process for obtaining a compressed representation through reparameterization provided in an embodiment of the present invention;

[0068] Figure 7 This is a schematic diagram of the structure of the video compression and transmission system provided in an embodiment of the present invention;

[0069] Figure 8 This is a schematic diagram of the video compression and transmission service process provided in an embodiment of the present invention;

[0070] Figure 9 This is a schematic diagram of the structure of a video compression and transmission device provided in an embodiment of the present invention;

[0071] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0073] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0074] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0076] To facilitate understanding of the technical solution of this invention, the technical terms that may be applied in the technical solution of this invention will first be explained:

[0077] Flow Matching: A method for controlling the generation and training of diffusion models, which generates data by learning a velocity field from the noise distribution to the data distribution. This invention reforms it into a direct path learning from compressed features to the target data.

[0078] I-frames: Keyframes in a video sequence, containing complete image information. In this invention, they specifically refer to the first and last frames of the video, transmitted as conditional information.

[0079] Reparameterization Trick: A training technique that allows parameters to be sampled from a random distribution and optimized through backpropagation for training compressor networks.

[0080] VAE: Variational Autoencoder. It consists of two parts: an encoder and a decoder. The encoder encodes the original image into latent variables, and the decoder decodes the latent variables to generate the image. The latent variables are the compressed information of the image and are the key information and core indicators of the generated image.

[0081] Among related technologies, the current mainstream video compression standards (such as H.264, H.265, and H.266) adopt a hybrid coding strategy, combining motion estimation, transform coding, and entropy modeling. However, at extremely low bitrates, they exhibit severe block artifacts, blurring, and loss of high-frequency details. While deep learning-based neural video compression methods (such as the DCVC series) have improved rate-distortion performance, the perceptual quality of their reconstructed videos drops sharply at extremely low bitrates (e.g., 0.01 bpp), resulting in severe blurring and distortion.

[0082] In view of this, this invention provides a video compression and transmission method and related equipment. The method involves acquiring a video sequence to be compressed, using the first and last frames as conditional frames; inputting the video sequence into the encoder of a pre-trained variational autoencoder to obtain a target latent space representation; processing the target latent space representation through the downsampling module of the compressor to generate an extremely compressed representation; and transmitting the conditional frames and the extremely compressed representation to the receiving end, enabling the receiving end to reconstruct the video sequence through the upsampling module of the compressor, the generative model, and the decoder of the variational autoencoder. The compressor and generative model are pre-trained using compression losses based on conditional prior and posterior distributions, and reconstruction losses based on the latent space representation, the compressed representation, and the velocity field. The conditional prior distribution is constructed based on the encoding information of a masked video after removing the first and last frames, and the posterior distribution is constructed based on the Gaussian distribution parameters recovered by the upsampling module. The latent space representation represents the encoder output, the compressed representation represents the Gaussian distribution parameters recovered by the upsampling module, and the velocity field represents the training output of the generative model. This invention uses the first and last frames of a video sequence as conditional frames and utilizes a pre-trained variational autoencoder and compressor to generate an extremely compressed representation. This allows the invention to maintain high reconstructed video quality even at extremely low bitrates. The conditional frames provide temporal boundary information, which, combined with the reconstruction capabilities of the generative model, effectively reduces block artifacts, blurring, and loss of high-frequency details common in traditional methods. Furthermore, the latent space representation of the variational autoencoder and the downsampling module of the compressor achieve extreme compression of video data, significantly reducing the transmission bitrate. At the same time, by transmitting only the conditional frames and the extremely compressed representation, bandwidth requirements are reduced, making it suitable for scenarios with limited network bandwidth. Specifically, the compressor and generative model are pre-trained using compression losses based on conditional prior and posterior distributions, as well as reconstruction losses based on latent space representation, compressed representation, and velocity field. This allows the entire system to achieve a balance between compression efficiency and reconstruction quality, significantly improving rate-distortion performance.

[0083] It is understood that the video compression and transmission method provided by this invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet computer, laptop computer, or desktop computer, but it is not limited to these.

[0084] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0085] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0086] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0087] Terminal 102 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0088] For example, based on Figure 1 The implementation environment shown in this embodiment of the invention provides a video compression and transmission method. The following description uses the application of this video compression and transmission method in server 101 as an example. It can be understood that this video compression and transmission method can also be applied in terminal 102.

[0089] Reference Figure 2 , Figure 2 This is an optional flowchart of the video compression and transmission method provided in the embodiments of the present invention. The execution subject of the video compression and transmission method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 The method may include, but is not limited to, steps S100 to S300.

[0090] Step S100: Obtain the video sequence to be compressed, and use the first and last frames as conditional frames;

[0091] For example, in some specific implementations, the input video sequence X to be compressed is used, and its first and last frames are taken as condition frames C. Specifically, the first and last frames can be compressed independently as I-frames.

[0092] Step S200: Input the video sequence into the encoder of the pre-trained variational autoencoder to obtain the target latent space representation. Then, process the target latent space representation through the downsampling module of the compressor to generate an extremely compressed representation.

[0093] For example, in some specific implementations, the complete video X is input into a pre-trained VAE encoder to obtain a latent space representation. The latent space is represented using the compressor's downsampling module Q_down. The data is processed to generate an extremely compressed representation h for network transmission.

[0094] Step S300: The conditional frame and the extreme compression representation are transmitted to the receiving end so that the receiving end can perform video reconstruction through the upsampling module of the compressor, the generation model and the decoder of the variational autoencoder to obtain the reconstructed video sequence.

[0095] It should be noted that in some embodiments, such as Figure 3 As shown, transmitting a condition frame to the receiving end may include the following steps: S310, compressing the condition frame using an image compression method to obtain compressed information; S320, transmitting the compressed information to the receiving end so that the receiving end can decode the compressed information and recover the condition frame.

[0096] For example, in some specific implementations, the condition frame C and the extreme compression representation h are sent to the receiving end through the channel. Specifically, the condition frame can be compressed using a state-of-the-art image compression method. Then, the receiving end receives the compressed condition frame and the extreme compression representation h, and decodes the condition frame to recover the first and last frames C.

[0097] Specifically, this embodiment of the invention reduces the amount of data transmitted for conditional frames by compressing them using an image compression method before transmitting the compressed information, thereby lowering the overall bitrate requirement for video transmission. Furthermore, by employing a standard image compression method, it is easy to implement and integrate with existing video transmission systems, reducing deployment costs.

[0098] It should be noted that in some embodiments, such as Figure 4As shown, video reconstruction is performed through the upsampling module of the compressor, the generative model, and the decoder of the variational autoencoder to obtain a reconstructed video sequence. This process may include the following steps: S330, the extreme compression representation is processed by the upsampling module of the compressor, and the mean value of the output Gaussian distribution parameters is used as the target compression representation; S340, the conditional frame and the target compression representation are input into the generative model, and the generative model gradually reconstructs the intermediate frame latent representation based on a preset equidistant time step; S350, the intermediate frame latent representation is converted back to pixel space by the decoder of the variational autoencoder to obtain the intermediate video frame; S360, the intermediate video frame and the conditional frame are concatenated to obtain the reconstructed video sequence.

[0099] For example, in some specific implementations, the compressor's upsampling module Q_up is used to process h to recover the mean parameter of the Gaussian distribution. During the inference phase, for deterministic output, the mean is directly used as the compressed representation: Z = The recovered conditional frame C and the compressed representation Z are input together into the generative model. The generative model performs sampling reconstruction based on improved stream matching: 10 time points are equally spaced from t=0 to t=1. Starting from Z, the latent representation of the intermediate frames is gradually reconstructed by iterating from t=0 to t=1 using the velocity field predicted by the generative model, according to the equally spaced time steps. After outputting the reconstructed latent representation of the intermediate frames, the combined latent representation is converted back to pixel space by the VAE decoder and concatenated with the first and last frames to output a complete high-quality reconstructed video sequence.

[0100] Specifically, this embodiment of the invention recovers the target compressed representation through an upsampling module and uses a generative model to gradually reconstruct the latent representation of intermediate frames based on preset equidistant time steps. This generates smoothly transitioning intermediate video frames, avoiding frame jumps or distortion, and effectively improving the temporal coherence and visual smoothness of the video. Furthermore, this embodiment of the invention uses a decoder of a variational autoencoder to convert the latent representation of intermediate frames back to pixel space and concatenate it with conditional frames to form a complete reconstructed video sequence. This process ensures the structural integrity and detail preservation of the reconstructed video, making it particularly suitable for video compression in dynamic scenes.

[0101] The compressor and the generative model are pre-trained using compression loss based on conditional prior distribution and posterior distribution, and reconstruction loss based on latent space representation, compressed representation, and velocity field. The conditional prior distribution is constructed based on the encoding information of the masked video after removing the first and last frames, and the posterior distribution is constructed based on the Gaussian distribution parameters recovered by the upsampling module. The latent space representation represents the output of the encoder, the compressed representation represents the Gaussian distribution parameters recovered by the upsampling module, and the velocity field represents the training output of the generative model.

[0102] It should be noted that in some embodiments, such as Figure 5 As shown, the method may further include the following steps: T100, acquiring a training video sequence, setting the first and last frames of the training video sequence to zero to obtain a masked video; T200, inputting the training video sequence into an encoder, processing it to obtain a latent space representation; T300, inputting the masked video into an encoder, processing it to obtain the prior parameters of the Gaussian distribution, and constructing a conditional prior distribution based on the prior parameters; T400, progressively compressing the latent space representation through a downsampling module to generate compressed features; T500, sampling and recovering the compressed features through an upsampling module, and obtaining a compressed representation from the output Gaussian distribution parameters through reparameterization; T600, based on... The posterior distribution is constructed from the Gaussian distribution parameters output by the upsampling module; T700, the training input samples of the generative model are constructed based on the compressed representation; T800, the velocity field from the training input samples to the latent space representation is predicted by the generative model based on the training input samples and the conditional frames of the training video sequence, combined with preset equidistant time steps; T900, the compression loss is constructed based on the conditional prior distribution and the posterior distribution, the reconstruction loss is constructed based on the latent space representation, the compressed representation and the velocity field, and the total loss is constructed based on the compression loss and the reconstruction loss; T1000, the parameters of the compressor and the parameters of the generative model are jointly optimized using the total loss through gradient backpropagation.

[0103] Specifically, this embodiment of the invention acquires training video sequences and constructs masked videos, thereby simultaneously optimizing the parameters of the compressor and the generation model during training. This ensures the collaborative work of the compression and reconstruction modules, improving overall performance. Furthermore, this embodiment constructs a compression loss based on the KL divergence of the conditional prior and posterior distributions, making the latent space representation more consistent with prior assumptions and reducing information loss during compression. Simultaneously, the reconstruction loss based on the velocity field constrains the output of the generation model, improving the accuracy of intermediate frame reconstruction. Specifically, the total loss is used to jointly optimize parameters through gradient backpropagation, enabling the model to converge quickly and adapt to the compression requirements of different video content.

[0104] It should be noted that in some embodiments, such as Figure 6 As shown, obtaining a compressed representation through reparameterization can include the following steps: T510, weighting the standard deviation of the Gaussian distribution parameters output by the upsampling module using random parameters that follow a standard normal distribution; T520, summing the result of the weighting operation with the mean of the Gaussian distribution parameters output by the upsampling module to obtain the compressed representation.

[0105] For example, in some specific implementations, a compressed representation Z= is obtained by sampling from a Gaussian distribution using a reparameterization technique. , The mean, Standard deviation These are random parameters that follow a standard normal distribution.

[0106] Specifically, this embodiment of the invention uses a reparameterization technique to generate a target compressed representation by weighting and summing Gaussian distributed parameters with random parameters following a standard normal distribution. This method introduces randomness into training, preventing model overfitting and improving the adaptability of the compressed representation to different video content. Furthermore, the Gaussian distribution-based parameterization gives the latent space representation and compressed representation probabilistic characteristics, enhancing the model's ability to model uncertainty and thus maintaining stable reconstruction quality even at extremely low bitrates.

[0107] It should be noted that, in some embodiments, constructing the training input samples for the generative model based on the compressed representation may include the following steps: constructing a straight path connecting the latent space representation based on the compressed representation; randomly selecting a point from the straight path as a training input sample; wherein, the expression of the training input sample is:

[0108] ;

[0109] In the formula, Let represent the training input sample at time step t, where t represents the corresponding time step on the straight path. Representing the latent space representation, This indicates compressed representation.

[0110] Specifically, this embodiment of the invention provides training sample diversity by constructing a straight path connecting the latent space representation and the compressed representation, and randomly selecting points as training input samples, enabling the generative model to learn a smooth mapping from the compressed representation to the latent space representation; wherein, the expression of the training input samples ensures the continuity and rationality of the sampling points, which can avoid instability in training, thereby improving the reliability of the model in the reconstruction process.

[0111] It should be noted that in some embodiments, the compression loss is constructed based on the conditional prior distribution and the posterior distribution, and the reconstruction loss is constructed based on the latent space representation, the compression representation, and the velocity field. The total loss is constructed based on the compression loss and the reconstruction loss, which may include the following steps: constructing the compression loss based on the KL divergence between the posterior distribution and the conditional prior distribution; using the velocity field as the generated value and the difference between the compression representation and the latent space representation as the original value, constructing the reconstruction loss based on the difference index between the generated value and the original value; weighting the compression loss using preset hyperparameters, and then combining it with the reconstruction loss to construct the total loss.

[0112] Specifically, this embodiment of the invention constructs a compression loss based on the KL divergence between the posterior distribution and the conditional prior distribution, effectively reducing redundancy in the latent space representation and improving compression efficiency. Simultaneously, a reconstruction loss is constructed based on the difference between the velocity field and the original value (the difference between the compressed representation and the latent space representation), directly constraining the output of the generative model and ensuring the accuracy of the reconstructed video. Specifically, this embodiment of the invention weights the compression loss using preset hyperparameters and combines it with the reconstruction loss to construct the total loss. This allows for adjusting the trade-off between compression rate and distortion according to specific application scenarios (such as different bitrate requirements), thereby adapting to diverse video compression needs.

[0113] Exemplary, in some specific embodiments, the present invention proposes a generative video compression framework based on latent spatial feature extraction and stream matching. Let the original video sequence be... Where T is the number of frames and H×W is the spatial resolution. The core idea of ​​this framework is to transmit only the first and last frames as conditional information C, while simultaneously learning a highly compressed representation Z, enabling the generative model to reconstruct the entire video sequence X with high quality based on C and Z. The training and testing process is as follows: Figure 7 The training process can be implemented as follows:

[0114] 1. The original video sequence is input into a pre-trained VAE encoder, and the latent space representation is obtained by sampling. .

[0115] Apply a time mask to the original video X, setting the first and last frames to zero to obtain the masked video. .Will Input the pre-trained VAE encoder to obtain the prior parameters of the Gaussian distribution. ( This represents the mean. (The standard deviation is represented by the subscript p, indicating the Gaussian distribution obtained by the VAE encoder.) Establish the conditional prior distribution. This prior is used to guide the compressor to focus on learning dynamic information not included in the conditional frame C.

[0116] 2. Latent space representation This will be used as input to compressor Q. Compressor Q is a UNet-structured network; see [link to relevant documentation]. Figure 8(Attention Layer represents the attention layer, Down represents downsampling, Up represents upsampling, SelfAttention represents applying self-attention, and FFN represents a feedforward neural network.) It consists of a downsampling module Q_down and an upsampling module Q_up. The Q_down module is responsible for generating an extremely compressed representation h for network transmission during training and inference. The Q_up module mainly works in conjunction with Q_down during training to upsample intermediate features to output distribution parameters, thus restoring the feature size required for model inference.

[0117] (1) The downsampling module is responsible for progressively compressing the latent space representation of the input. It adopts a hierarchical coding structure: Stage 1: 3D convolutional downsampling with a stride of 1×2×2 is used to halve the spatial resolution (width and height) while keeping the temporal dimension unchanged; Stage 2: 3D convolutional downsampling with a stride of 2×2×2 is used to compress both the temporal and spatial dimensions, halving the resolution of each dimension; Stage 3: 3D convolutional downsampling with a stride of 2×2×2 is used again to further compress the spatiotemporal dimensions. Each downsampling stage is followed by an attention layer, which captures long-range spatiotemporal dependencies in the video sequence through a self-attention mechanism. The output of this module is a highly compressed intermediate feature h.

[0118] (2) The upsampling module employs a decoding structure symmetrical to the downsampling module. It is responsible for recovering the target distribution parameters from the compressed features and sampling to obtain the compressed representation Z of the dimension required for the generated model. Its structure uses 3D transposed convolution for progressive upsampling, precisely reversing the resolution changes during the downsampling process. The upsampling module ultimately outputs Gaussian distributed parameters. (The subscript q represents the Gaussian distribution obtained by the upsampling module), and the posterior distribution of the compressed representation is established based on these parameters. And by using reparameterization techniques, a compressed representation Z= is obtained from this distribution. .

[0119] 3. Generative Model The core of this training lies in its innovative flow matching training paradigm. Unlike traditional diffusion models that start from random noise, this invention directly learns from the compressed representation Z to the latent space representation. The linear path is specifically implemented as follows: 1) During the training process, this invention constructs a path connecting the compressed representation Z and the latent space representation. Find a straight path and randomly select a point on the path. 2) As input to the generative model, predict from arrive Velocity field: .

[0120] 4. Loss Calculation and End-to-End Optimization:

[0121] (1) Reconstruction losses: , The measurement functions representing the differences include mean squared error (MSE), mean absolute error (MAE), perceptual loss, and structural similarity loss.

[0122] (2) Compression loss: That is, the KL divergence between the posterior distribution of the compressor Q output and the conditional prior distribution. The function representing the measure of KL divergence.

[0123] (3) Total loss: hyperparameters To balance reconstruction quality and compression ratio, it is set in this invention as follows: .

[0124] (4) The compressor parameters and the generative model parameters are jointly optimized through gradient backpropagation. For the generative model, this invention uses Low-Rank Adaptation (LoRA) technology for efficient fine-tuning.

[0125] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0126] First, it's important to note that generative models have shown potential in video synthesis and reconstruction, and have therefore been used for video compression tasks. Existing methods rely on handcrafted features (such as sketches, optical flow, or text descriptions) to guide reconstruction. While this achieves extremely low bitrates, these features struggle to capture the dynamic semantics of the video. Furthermore, the separation of feature extraction and generative model optimization leads to severely insufficient fidelity in the reconstructed video, particularly in maintaining visual coherence and realism in dynamic and complex scenes. In addition, traditional generative models involve numerous sampling steps and have low inference efficiency.

[0127] Therefore, the purpose of this invention is to provide a video compression framework based on a generative model that can be trained end-to-end, to address the fidelity challenges faced by existing technologies at extremely low bitrates. Specifically, this invention aims to overcome the following shortcomings of existing technologies and achieve the corresponding technical objectives:

[0128] 1. Overcome the problems of traditional methods at extremely low bit rates:

[0129] Traditional video codecs (such as H.266 / VVC) and early neural codecs experience a sharp decline in performance as the bit rate continues to decrease, manifesting as severe block artifacts, blurring, and loss of detail, rendering the video content visually and semantically unusable. The purpose of this invention is to overcome this bottleneck and generate visually natural and semantically complete video reconstructions even at extremely low bit rates (e.g., 0.01 bpp).

[0130] 2. Address the dependence on and limitations of existing generative compression methods regarding handcrafted features:

[0131] Current generative model-based video compression methods typically rely on manually defined low-level features (such as edges, optical flow, and text descriptions) to guide reconstruction. These features cannot fully capture the rich high-level semantics and complex dynamics in videos, resulting in poor fidelity in the reconstructed videos. The purpose of this invention is to eliminate the reliance on manually defined features and, through end-to-end learning, enable the model to automatically extract and transmit the most essential and effective information for reconstruction from the data.

[0132] 3. Achieve end-to-end joint optimization of the compression and reconstruction processes:

[0133] In existing methods, the feature extractor (sender) and the generative model (receiver) are often designed separately and trained in stages, leading to suboptimal system-level performance. The extracted features may not be the information most needed by the generative model. The purpose of this invention is to construct a fully differentiable, end-to-end training framework that places the compressor (encoder) and generator (decoder) under the same optimization objective, enabling the compressor to explicitly learn and prioritize the retention of information most valuable to the generative model, thereby achieving overall performance optimization.

[0134] 4. Improve the inference efficiency of generative compression:

[0135] Traditional diffusion models typically require dozens or even hundreds of iterative denoising steps to generate high-quality samples, resulting in slow inference speeds that are difficult to meet the demands of real-time applications. The purpose of this invention is to innovate the generation process by reforming the flow matching objective, simplifying the complex "noise-to-data" generation path into an efficient "compressed feature-to-data" linear path. This allows for high-fidelity reconstruction in very few sampling steps (e.g., 10 steps or less), significantly improving inference speed.

[0136] This invention proposes a generative video compression framework based on latent spatial feature extraction and stream matching. Let the original video sequence be... Where T is the number of frames and H×W is the spatial resolution. The core idea of ​​this framework is to transmit only the first and last frames as conditional information C, while simultaneously learning a highly compressed representation Z, enabling the generative model to reconstruct the entire video sequence X with high quality based on C and Z. The training and testing process is as follows: Figure 7 As shown.

[0137] Among them, such as Figure 7 The training process for the left-hand flow architecture (i.e., the part corresponding to label (a)) is as follows:

[0138] 1. The original video sequence is input into a pre-trained VAE encoder, and the latent space representation is obtained by sampling. .

[0139] Apply a time mask to the original video X, setting the first and last frames to zero to obtain the masked video. .Will Input the pre-trained VAE encoder to obtain the prior parameters of the Gaussian distribution. Establish conditional prior distributions. This prior is used to guide the compressor to focus on learning dynamic information not included in the conditional frame C.

[0140] 2. Latent space representation This will be used as input to compressor Q. Compressor Q is a UNet-structured network; see [link to relevant documentation]. Figure 8 It consists of a downsampling module Q_down and an upsampling module Q_up. The Q_down module is responsible for generating an extremely compressed representation h for network transmission during training and inference. The Q_up module mainly works in conjunction with Q_down during training to upsample intermediate features to output distribution parameters, thus restoring the feature size required for model inference.

[0141] (1) The downsampling module is responsible for progressively compressing the latent space representation of the input. It adopts a hierarchical coding structure: Stage 1: 3D convolutional downsampling with a stride of 1×2×2 is used to halve the spatial resolution (width and height) while keeping the temporal dimension unchanged; Stage 2: 3D convolutional downsampling with a stride of 2×2×2 is used to compress both the temporal and spatial dimensions, halving the resolution of each dimension; Stage 3: 3D convolutional downsampling with a stride of 2×2×2 is used again to further compress the spatiotemporal dimensions. Each downsampling stage is followed by an attention layer, which captures long-range spatiotemporal dependencies in the video sequence through a self-attention mechanism. The output of this module is a highly compressed intermediate feature h.

[0142] (2) The upsampling module employs a decoding structure symmetrical to the downsampling module. It is responsible for recovering the target distribution parameters from the compressed features and sampling to obtain the compressed representation Z of the dimension required for the generated model. Its structure uses 3D transposed convolution for progressive upsampling, precisely reversing the resolution changes during the downsampling process. The upsampling module ultimately outputs Gaussian distributed parameters. Based on these parameters, a compressed representation of the posterior distribution is established. And by using reparameterization techniques, a compressed representation Z= is obtained from this distribution. .

[0143] 3. Generative Model The core of this training lies in its innovative flow matching training paradigm. Unlike traditional diffusion models that start from random noise, this invention directly learns from the compressed representation Z to the latent space representation. The linear path is specifically implemented as follows: 1) During the training process, this invention constructs a path connecting the compressed representation Z and the latent space representation. Find a straight path and randomly select a point on the path. 2) As input to the generative model, predict from arrive Velocity field: .

[0144] 4. Loss calculation and end-to-end optimization.

[0145] (1) Reconstruction losses: , The measurement functions representing the differences include mean squared error (MSE), mean absolute error (MAE), perceptual loss, and structural similarity loss.

[0146] (2) Compression loss: That is, the KL divergence between the posterior distribution of the compressor Q output and the conditional prior distribution. The function representing the measure of KL divergence.

[0147] (3) Total loss: hyperparameters To balance reconstruction quality and compression ratio, it is set in this invention as follows: .

[0148] (4) The compressor parameters and the generative model parameters are jointly optimized through gradient backpropagation. For the generative model, this invention uses Low-Rank Adaptation (LoRA) technology for efficient fine-tuning.

[0149] Among them, such as Figure 7The right-hand flow architecture (i.e., the part corresponding to marker (b)) and the testing process (the actual application process logic is the same) are as follows:

[0150] Sender:

[0151] 1. Input the video sequence X to be compressed.

[0152] 2. Take its first and last frames as conditional frames C, and compress the first and last frames as I-frames independently.

[0153] 3. Input the complete video X into the pre-trained VAE encoder to obtain the latent space representation. .

[0154] 4. Represent the latent space using the compressor's downsampling module Q_down. The data is processed to generate an extremely compressed representation h for network transmission. The compressed conditional frame C and the extremely compressed representation h are then transmitted to the receiving end via the channel.

[0155] Receiver:

[0156] 1. Receive the compressed conditional frame and the extreme compression representation h, and decode the conditional frame to recover the first and last frames C.

[0157] 2. Use the compressor's upsampling module Q_up to process h and recover the mean parameters of the Gaussian distribution. During the inference phase, for deterministic output, the mean is directly used as the compressed representation: Z = .

[0158] 3. Input the recovered conditional frame C and the compressed representation Z into the generative model. .

[0159] 4. The generative model is based on improved flow matching for sampling and reconstruction: 10 time points are divided at equal intervals from t=0 to t=1. Starting from Z, the potential representation of the intermediate frame is gradually reconstructed by iterating from t=0 to t=1 according to the equally divided time steps through the velocity field predicted by the generative model.

[0160] After outputting the reconstructed intermediate frame latent representation, the combined latent representation is converted back to pixel space by the VAE decoder and spliced ​​with the first and last frames to output a complete high-quality reconstructed video sequence.

[0161] In some specific application scenarios, the video compression transmission of this invention can be achieved as follows:

[0162] 1. Each video segment is adjusted to 480p resolution (i.e., 720x480), and then divided into 45-frame segments. The condition information C is the first and last frames of each video segment. The condition frames are compressed using the state-of-the-art image compression method.

[0163] 2. Using a pre-trained WAN2.1 (WAN2.1) VAE, its latent space representation... Compared to the original video X, the downsampling ratio is 4*8*8. The extreme compression representation h output by the compressor Q_down module will be further compressed by 2*8*8, reaching a maximum compression ratio of 32,768 times. h is represented using bfloat16 and uses a 16-channel feature dimension, so the total bandwidth of transmission h and conditional frames (bpp) is approximately 0.01. When using LORA to fine-tune the generation model parameters, the rank r=16 and alpha=16.

[0164] In summary, this invention proposes the first video compression framework based on the collaboration of latent spatial feature extraction and a controlled diffusion model. It overcomes the efficiency-quality contradiction caused by traditional coding relying on manual modules, as well as the limitations of neural compression techniques such as detail loss or mode collapse, achieving a better balance between perceptual quality and fidelity at high compression ratios. Furthermore, by combining the diffusion model with network conditional extraction and a temporal mask prior mechanism, this invention establishes a precise correlation between latent spatial feature condensation and diffusion generation: the conditional extraction module retains core semantics and dynamic features, while the random mask forces the model to focus on key information. Simultaneously, thanks to the high-fidelity generation capability of the controlled diffusion model and the conditional driving strategy, this invention maintains high fidelity in video structure and dynamics even at high compression ratios, achieving stable perceptual quality maintenance. This overcomes the bottleneck of "high compression inevitably accompanied by a sharp drop in quality" in traditional rate-distortion trade-offs, adapting to high-quality compression requirements such as high-definition video transmission.

[0165] Compared with the prior art, the present invention has at least the following beneficial effects:

[0166] End-to-End Video Compression Framework: The core of this invention lies in constructing an end-to-end video compression framework based on latent spatial feature extraction. This is the first time that end-to-end training has been systematically applied to generative video compression, and a differentiable loss function has been proposed. At an ultra-high compression ratio of 32,768 times, the reconstructed video is visually natural and coherent, and semantically complete and usable, breaking through the "performance cliff" of traditional encoders. The specific implementation of this framework includes: 1. Differentiable optimization objective: The function explicitly includes reconstruction and compression terms, which are balanced by the hyperparameter β; 2. Deep coupling between the sender and receiver: The compressor (sender) and the generative model (receiver) are no longer independent modules, but are jointly trained under the guidance of the same information theory objective. This allows the compressor to learn to "sense" the needs of the generative model and actively extract the information most valuable for its reconstruction.

[0167] Reforming the Flow Matching Generation Model: To improve reconstruction efficiency and quality, this invention makes a key reform to the traditional flow matching generation model. The learning objective of the generation model is no longer to predict noise, but to directly regress the velocity field from compressed features to the original data. This not only simplifies the learning task and achieves better convergence performance, but also significantly accelerates the inference process. The reformed flow matching model only requires 10 sampling steps to achieve the reconstruction quality of more than 50 steps of the traditional diffusion model.

[0168] Temporal masking prior design: A masked video is generated by masking (zeroing out) the conditional frames (first and last frames) in the temporal dimension. The Gaussian distribution obtained by passing the masked video through a pre-trained VAE is used as the learning prior. This operation forces the model to focus on the dynamically changing intermediate content that is not present in the conditional frames.

[0169] like Figure 9 As shown, this embodiment of the invention also provides a video compression and transmission device 900, which can implement the above-described method. This device may include:

[0170] The first module 910 is used to obtain the video sequence to be compressed, and to use the first frame and the last frame as conditional frames.

[0171] The second module 920 is used to input the video sequence into the encoder of the pre-trained variational autoencoder, process it to obtain the target latent space representation, and process the target latent space representation through the downsampling module of the compressor to generate an extremely compressed representation;

[0172] The third module 930 is used to transmit the conditional frame and the extreme compression representation to the receiving end, so that the receiving end can perform video reconstruction through the upsampling module of the compressor, the generation model and the decoder of the variational autoencoder to obtain the reconstructed video sequence.

[0173] The compressor and the generative model are pre-trained using compression loss based on conditional prior distribution and posterior distribution, and reconstruction loss based on latent space representation, compression representation, and velocity field. The conditional prior distribution is constructed based on the encoded information of the masked video after removing the first and last frames, and the posterior distribution is constructed based on the Gaussian distribution parameters recovered by the upsampling module. The latent space representation represents the encoder output, the compression representation represents the Gaussian distribution parameters recovered by the upsampling module, and the velocity field represents the training output of the generative model.

[0174] In some embodiments, the apparatus may further include a fourth module for performing the following operations:

[0175] Obtain the training video sequence, set the first and last frames of the training video sequence to zero, and obtain the masked video;

[0176] The training video sequence is input into the encoder and processed to obtain the latent space representation;

[0177] The masked video is input into the encoder, processed to obtain the prior parameters of the Gaussian distribution, and a conditional prior distribution is constructed based on the prior parameters.

[0178] The latent space representation is progressively compressed using a downsampling module to generate compressed features.

[0179] The compressed features are sampled and recovered by the upsampling module, and the output Gaussian distribution parameters are reparameterized to obtain a compressed representation.

[0180] The posterior distribution is constructed based on the Gaussian distribution parameters output by the upsampling module;

[0181] Training input samples for the generative model are constructed based on compressed representation;

[0182] Based on the conditional frames of the training input samples and training video sequences, and combined with preset equidistant time steps, the velocity field of the training input samples to the latent space representation is predicted by the generative model.

[0183] The compression loss is constructed based on the conditional prior and posterior distributions, the reconstruction loss is constructed based on the latent space representation, the compression representation and the velocity field, and the total loss is constructed based on the compression loss and the reconstruction loss.

[0184] Using the total loss, the parameters of the compressor and the parameters of the generative model are jointly optimized through gradient backpropagation.

[0185] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0186] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0187] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0188] like Figure 10 As shown, Figure 10 The hardware structure of an electronic device 1000 according to another embodiment is illustrated. The electronic device 1000 includes:

[0189] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (aSIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0190] The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RaM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001.

[0191] Input / output interface 1003 is used to implement information input and output;

[0192] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0193] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0194] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0195] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0197] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0198] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0199] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0200] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0201] The video compression and transmission method, apparatus, electronic device, storage medium, and program product provided in this invention acquire a video sequence to be compressed, using the first and last frames as conditional frames; input the video sequence into the encoder of a pre-trained variational autoencoder, processing it to obtain a target latent space representation; process the target latent space representation through the downsampling module of the compressor to generate an extreme compressed representation; transmit the conditional frames and the extreme compressed representation to the receiving end, so that the receiving end can perform video reconstruction through the upsampling module of the compressor, the generative model, and the decoder of the variational autoencoder to obtain a reconstructed video sequence; wherein, the compressor and the generative model are pre-trained using compression losses based on conditional prior distributions and posterior distributions, and reconstruction losses based on latent space representation, compressed representation, and velocity field; the conditional prior distribution is constructed based on the encoding information of the masked video after removing the first and last frames, the posterior distribution is constructed based on the Gaussian distribution parameters recovered by the upsampling module, the latent space representation represents the encoder output, the compressed representation represents the Gaussian distribution parameters recovered by the upsampling module, and the velocity field represents the training output of the generative model. This invention uses the first and last frames of a video sequence as conditional frames and utilizes a pre-trained variational autoencoder and compressor to generate an extremely compressed representation. This allows the invention to maintain high reconstructed video quality even at extremely low bitrates. The conditional frames provide temporal boundary information, which, combined with the reconstruction capabilities of the generative model, effectively reduces block artifacts, blurring, and loss of high-frequency details common in traditional methods. Furthermore, the latent space representation of the variational autoencoder and the downsampling module of the compressor achieve extreme compression of video data, significantly reducing the transmission bitrate. At the same time, by transmitting only the conditional frames and the extremely compressed representation, bandwidth requirements are reduced, making it suitable for scenarios with limited network bandwidth. Specifically, the compressor and generative model are pre-trained using compression losses based on conditional prior and posterior distributions, as well as reconstruction losses based on latent space representation, compressed representation, and velocity field. This allows the entire system to achieve a balance between compression efficiency and reconstruction quality, significantly improving rate-distortion performance.

[0202] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0203] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0204] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0205] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0206] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A video compression and transmission method, characterized in that, The method includes the following steps: Obtain the video sequence to be compressed, and use the first and last frames as conditional frames; The video sequence is input into the encoder of a pre-trained variational autoencoder to obtain the target latent space representation. The target latent space representation is then processed by the downsampling module of the compressor to generate an extremely compressed representation. The conditional frame and the extreme compression representation are transmitted to the receiving end so that the receiving end can perform video reconstruction through the upsampling module of the compressor, the generation model and the decoder of the variational autoencoder to obtain the reconstructed video sequence; The compressor and the generative model are pre-trained using compression loss based on conditional prior distribution and posterior distribution, and reconstruction loss based on latent space representation, compressed representation, and velocity field. The conditional prior distribution is constructed based on the encoding information of the masked video after removing the first and last frames, and the posterior distribution is constructed based on the Gaussian distribution parameters recovered by the upsampling module. The latent space representation represents the output of the encoder, the compressed representation represents the Gaussian distribution parameters recovered by the upsampling module, and the velocity field represents the training output of the generative model.

2. The method according to claim 1, characterized in that, The process of reconstructing the video sequence by using the upsampling module of the compressor, the generation model, and the decoder of the variational autoencoder includes the following steps: The extreme compression representation is processed by the upsampling module of the compressor, and the mean value of the output Gaussian distribution parameters is used as the target compression representation. The conditional frame and the target compressed representation are input into the generation model, and the intermediate frame potential representation is gradually reconstructed by the generation model based on a preset equidistant time step. The intermediate frame latent representation is converted back to pixel space by the decoder of the variational autoencoder to obtain the intermediate video frame; The intermediate video frames and the conditional frames are spliced ​​together to obtain the reconstructed video sequence.

3. The method according to claim 1, characterized in that, Transmitting the condition frame to the receiving end includes the following steps: The conditional frame is compressed using an image compression method to obtain compressed information; The compressed information is transmitted to the receiving end so that the receiving end can decode the compressed information and recover the condition frame.

4. The method according to claim 1, characterized in that, The method further includes the following steps: Obtain the training video sequence, set the first and last frames of the training video sequence to zero, and obtain the masked video; The training video sequence is input into the encoder and processed to obtain the latent space representation; The masked video is input into the encoder, processed to obtain the prior parameters of the Gaussian distribution, and the conditional prior distribution is constructed based on the prior parameters. The latent space representation is progressively compressed using the downsampling module to generate compressed features. The compressed features are sampled and recovered by the upsampling module, and the output Gaussian distribution parameters are reparameterized to obtain the compressed representation. The posterior distribution is constructed based on the Gaussian distribution parameters output by the upsampling module. The training input samples for the generative model are constructed based on the compressed representation; Based on the training input samples and the conditional frames of the training video sequence, the velocity field represented by the latent space is predicted by the generative model using a preset equidistant time step; wherein, the conditional frames of the training video sequence are the first frame and the last frame of the training video sequence. A compression loss is constructed based on the conditional prior distribution and the posterior distribution; a reconstruction loss is constructed based on the latent space representation, the compression representation, and the velocity field; and a total loss is constructed based on the compression loss and the reconstruction loss. Using the total loss, the parameters of the compressor and the parameters of the generative model are jointly optimized through gradient backpropagation.

5. The method according to claim 4, characterized in that, The process of obtaining the compressed representation through reparameterization includes the following steps: The standard deviation of the Gaussian distribution parameters output by the upsampling module is weighted by random parameters that follow a standard normal distribution. The result of the weighted operation is summed with the mean value of the Gaussian distribution parameters output by the upsampling module to obtain the compressed representation.

6. The method according to claim 4, characterized in that, The process of constructing the training input samples for the generative model based on the compressed representation includes the following steps: Based on the compressed representation, a straight path connecting the latent space representation is constructed; A point is randomly selected from the straight path as the training input sample; The expression for the training input sample is: ； In the formula, Let represent the training input sample at time step t, where t represents the corresponding time step on the straight path. Representing the latent space representation, This indicates compressed representation.

7. The method according to claim 4, characterized in that, The process of constructing a compression loss based on the conditional prior distribution and the posterior distribution, constructing a reconstruction loss based on the latent space representation, the compression representation, and the velocity field, and constructing a total loss based on the compression loss and the reconstruction loss includes the following steps: The compression loss is constructed based on the KL divergence between the posterior distribution and the conditional prior distribution; The velocity field is used as the generated value, and the difference between the compressed representation and the latent space representation is used as the original value. The reconstruction loss is constructed based on the difference index between the generated value and the original value. The compression loss is weighted using preset hyperparameters, and then combined with the reconstruction loss to construct the total loss.

8. A video compression and transmission device, characterized in that, The device includes: The first module is used to obtain the video sequence to be compressed, and the first and last frames are used as conditional frames. The second module is used to input the video sequence into the encoder of the pre-trained variational autoencoder, process it to obtain the target latent space representation, and process the target latent space representation through the downsampling module of the compressor to generate an extremely compressed representation; The third module is used to transmit the conditional frame and the extreme compression representation to the receiving end, so that the receiving end can perform video reconstruction through the upsampling module of the compressor, the generation model and the decoder of the variational autoencoder to obtain the reconstructed video sequence. The compressor and the generative model are pre-trained using compression loss based on conditional prior distribution and posterior distribution, and reconstruction loss based on latent space representation, compressed representation, and velocity field. The conditional prior distribution is constructed based on the encoding information of the masked video after removing the first and last frames, and the posterior distribution is constructed based on the Gaussian distribution parameters recovered by the upsampling module. The latent space representation represents the output of the encoder, the compressed representation represents the Gaussian distribution parameters recovered by the upsampling module, and the velocity field represents the training output of the generative model.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images