Video compression method and system based on variational auto-encoder improved entropy model
Through the entropy model based on the variational autoencoder, combined with the space-time pyramid structure and continuous coding algorithm, video compression technology is improved, and the problems of low encoding efficiency and high computational complexity in the existing technology are solved, achieving high efficiency and real-time video compression and reconstruction.
Patent Information
- Application Number
- CN202411481911.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-10-23
AI Technical Summary
The existing video compression technology is inefficient in coding when processing complex statistical video content and has high computational complexity, which cannot meet the needs of real-time video compression, especially when processing high-resolution videos.
Using an improved entropy model based on variational autoencoder, multi-scale temporal context features are extracted through space-time pyramid structure and continuous coding algorithm, combined with preset visual features, and using improved hierarchical conditional variational autoencoder and algebraic geometry methods to generate potential Variables are adaptive quantization and dynamic entropy coding optimization based on compression perception to realize perceptually guided decoding reconstruction.
It improves video compression efficiency, reduces calculation complexity, realizes real-time video compression that maintains high video quality under high compression rates, adapts to different video content characteristics, and enhances the system's processing ability of complex video sequences.
Smart Images

Figure CN119011851A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of video compression, and in particular to a video compression method and system based on a variational autoencoder improved entropy model. Background Art
[0002] Video compression aims to reduce the bit rate of video data for easy storage and transmission while maintaining video quality as much as possible. Traditional video compression standards, such as H.264 and H.265, rely on a hybrid framework that includes steps such as prediction, transformation, quantization, entropy coding, and loop filtering. With the development of deep learning technology, neural network-based video compression methods (NVC) have begun to attract attention, which compress by learning the intrinsic representation of data.
[0003] In the prior art, a common method is to use a variational autoencoder (VAE) to improve the entropy model. The advantage of a variational autoencoder is that it can learn the probability distribution of data and perform more efficient encoding based on it. In existing video compression, entropy models usually use a fixed probability model to generate latent variables to optimize the quantization step size, which may lead to low coding efficiency and poor results, especially when processing video content with complex statistical characteristics, and cannot adapt to the diversity of video content; at the same time, the entropy model encoding step in the traditional method may involve complex calculations, especially when processing high-resolution videos; in resource-constrained environments, existing technologies cannot meet the needs of real-time video compression. Summary of the invention
[0004] The purpose of the invention is to provide a video compression method and system based on a variational autoencoder with an improved entropy model to solve the above-mentioned problems existing in the prior art.
[0005] The technical solution is a video compression method based on a variational autoencoder to improve the entropy model, comprising the following steps:
[0006] S1. Receive the current frame data in the video stream, and construct a spatiotemporal pyramid structure based on the current frame data; apply the continuous homology algorithm to the spatiotemporal pyramid structure to generate a multi-scale persistence map; fuse the multi-scale persistence map with the preset traditional visual features to obtain enhanced temporal context features; extract super prior data and potential prior data based on the current frame data, and splice the temporal context features, super prior data and potential prior data to form an input feature set;
[0007] S2. Based on the input feature set, an improved hierarchical conditional variational autoencoder is used to generate multi-level latent variables; based on the multi-level latent variables, an algebraic geometry method is used to estimate the probability distribution parameters; based on the probability distribution parameters, the probability value of each latent variable under predetermined conditions is calculated to generate probability distribution data;
[0008] S3, based on multi-level latent variables and probability distribution data, context-aware adaptive quantization based on compressed sensing theory is performed to obtain quantized data;
[0009] S4, based on the quantized data, performing dynamic entropy coding optimization to obtain a compressed data packet; based on the final compressed data packet, performing rate-distortion optimization based on information geometry to obtain an optimized compressed data packet;
[0010] S5. Based on the optimized compressed data packet, a perception-guided decoding reconstruction is performed to obtain a reconstructed video frame with optimal perceptual quality.
[0011] The video compression system based on the variational autoencoder improved entropy model includes:
[0012] at least one processor; and,
[0013] a memory communicatively connected to at least one of the processors; wherein,
[0014] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the video compression method based on the variational autoencoder improved entropy model.
[0015] Beneficial effect: The present invention can intelligently adjust the quantization step size according to the different characteristics of the video content through an adaptive quantization mechanism, effectively filter noise and retain useful information, thereby improving compression efficiency; by optimizing the network structure, the computational complexity is reduced, making it easier for the present invention to achieve real-time video compression while maintaining high compression efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 It is a flow chart of the present invention.
[0017] Figure 2 This is a flow chart of step S1 of the present invention.
[0018] Figure 3 This is a flow chart of step S2 of the present invention.
[0019] Figure 4 This is a flow chart of step S3 of the present invention.
[0020] Figure 5 This is a flow chart of step S4 of the present invention.
[0021] Figure 6 This is a flow chart of step S5 of the present invention.
[0022] Figure 7 This is an overall structural diagram of a video compression autoencoder with an improved entropy model according to an embodiment of the present invention. DETAILED DESCRIPTION
[0023] like Figure 1 As shown, the present application proposes a video compression method based on a variational autoencoder improved entropy model, comprising the following steps:
[0024] S1. Receive the current frame data in the video stream, and construct a spatiotemporal pyramid structure based on the current frame data; apply the continuous homology algorithm to the spatiotemporal pyramid structure to generate a multi-scale persistence map; fuse the multi-scale persistence map with the preset traditional visual features to obtain enhanced temporal context features; extract super prior data and potential prior data based on the current frame data, and splice the temporal context features, super prior data and potential prior data to form an input feature set;
[0025] S2. Based on the input feature set, an improved hierarchical conditional variational autoencoder is used to generate multi-level latent variables; based on the multi-level latent variables, an algebraic geometry method is used to estimate the probability distribution parameters; based on the probability distribution parameters, the probability value of each latent variable under predetermined conditions is calculated to generate probability distribution data;
[0026] S3, based on multi-level latent variables and probability distribution data, context-aware adaptive quantization based on compressed sensing theory is performed to obtain quantized data;
[0027] S4, based on the quantized data, performing dynamic entropy coding optimization to obtain a compressed data packet; based on the final compressed data packet, performing rate-distortion optimization based on information geometry to obtain an optimized compressed data packet;
[0028] S5. Based on the optimized compressed data packet, a perception-guided decoding reconstruction is performed to obtain a reconstructed video frame with optimal perceptual quality.
[0029] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:
[0030] S11, reading the original video data stream from the video input device or storage medium, extracting the video frame data at the current moment, and obtaining the current frame data; temporarily storing the current frame data in a buffer, and recording its timestamp information; preprocessing the current frame data in the buffer, including denoising, color space conversion, and resolution adjustment, to obtain the preprocessed current frame data;
[0031] S12, based on the preprocessed current frame data, construct a spatiotemporal pyramid structure, each level of the spatiotemporal pyramid structure contains data of different time scales and spatial resolutions; apply a persistent homology algorithm to the data of each level to calculate the topological features; summarize the topological features to generate a multi-scale persistence map;
[0032] S13, based on the preprocessed current frame data, using the pre-trained convolutional neural network, extracting traditional visual features; using a weighted fusion method, combining the multi-scale persistence map and the traditional visual features to obtain enhanced temporal context features;
[0033] S14. Based on the preprocessed current frame data, extract super prior data from the pre-trained neural network model; based on the encoding results of the pre-stored historical frames, extract potential prior data; standardize the super prior data and potential prior data to obtain standardized super prior data and potential prior data; splice the standardized super prior data and potential prior data with the enhanced temporal context features to form a complete input feature set.
[0034] In this embodiment, the super prior data includes high-level semantic information of the video content; the latent prior data reflects the temporal correlation of the video sequence.
[0035] In one embodiment of the present application, the OpenCV library is used to read the video stream, one frame at a time; the frame is converted into a numpy array format, denoted as F t , where t represents the current time step. Apply Gaussian filtering to denoise: F t_denoised =cv2.GaussianBlur(F t , (5,5), 0); Convert RGB color space to YUV color space: F t_yuv =cv2.cvtColor(F t_denoised , cv2.COLOR BGR2YUV ); Adjust the resolution to a standard size (such as 640x480): F t_resized = cv2.resize(F t_yuv , (640, 480)). Among them, F t Represents the original image data of the tth frame; F t_denoised Represents the denoised image data; F t_yuv Represents image data in YUV color space; F t_resized Represents the image data after adjusting the resolution, cv2.GaussianBlur represents the Gaussian blur function, which is used to perform Gaussian filtering on the image to remove noise; cv2.cvtColor represents the color space conversion function, which is used to convert the image from one color space to another color space; cv2.COLOR BGR2YUV Represents the color conversion code; cv2.resize represents the image scaling function.
[0036] Construct a spatiotemporal pyramid structure: define the number of layers L (e.g., 3 layers), and for each layer l (l = 0, 1, ..., L-1), the time window size is Wt =2 l +1; spatial scale is S l =F t_resized.shape / / 2 l ;Store data for each layer: P l =[downsample(F t_resized , S l ) for t in range(tW t / / 2, t+W t / / 2+1)]. Apply the continuous coherence algorithm and use the continuous coherence calculation toolkit (ripser) library to calculate the continuous coherence of each layer: PH l =ripser.ripser(P l , maxdim=1)['dgms'], generate multi-scale persistence diagram: MPD=[PH l , l∈L]; use the pre-trained residual network 50 layers (ResNet50) to extract traditional visual features: VF=ResNet50(weights='imagenet')(F t_resized ) ; fused features: fused feature =concatenate([flatten(MPD), VF]). Where L represents the number of pyramid layers, W t Indicates the time window size; S l represents the spatial scale of the lth layer; P l Represents the pyramid data of the lth layer; PH l represents the continuous homology result of the lth layer; MPD represents the multi-scale persistence map; VF represents the traditional visual feature; fused feature represents the fused features, F t_resized.shape Represents the shape of the resized image; downsample() represents the downsampling function; for t inrange represents a loop statement; ripser.ripser() represents a function in the Ripser library, which is used to calculate continuous homology; maxdim represents the maximum dimension for calculating continuous homology; dgms represents the result of continuous homology; weights='imagenet' represents the use of weights pre-trained on the dataset; concatenate() represents the connection function; flatten() represents a function that flattens a multidimensional array into a one-dimensional array.
[0037] Use the pre-trained Visual Geometry Group Network 16-layer (VGG16) network to extract hyper-prior data: z t =VGG16(weights='imagenet')(F t_resized); Use the pre-trained recurrent neural network (RNN) model to extract potential prior data: y t_1 = RNN(previous frames ); standardize: z t_norm = (z t - mean(z t )) / std(z t );y t_1_norm = (y t_1 -mean(y t_1 )) / std(y t_1 ); perform feature splicing: input features = concatenate([fused feature , z t_norm , y t_1_norm ]). Where z t represents the hyper-prior data; y t_1 represents potential prior data; z t_norm represents the normalized hyper-prior data; y t_1_norm Represents the standardized potential prior data; input features represents the final input feature set, previous frames Represents the previous frame data; mean() represents the mean function, and std() represents the standard deviation function.
[0038] In another embodiment of the present application, the current frame data is read from the video stream and stored as a three-dimensional array F t , where t represents the current time step. Apply Gaussian filtering to denoise: Create a 5x5 Gaussian kernel G' with a standard deviation of σ=1; t Perform a two-dimensional convolution: F t_denoised = F t * G'. Convert RGB color space to YUV color space: convert each pixel (r, g, b) as follows: Y = 0.299R + 0.587G + 0.114B; U = -0.147R - 0.289G +0.436B; V = 0.615R - 0.515G - 0.100B; resample the image using a bilinear interpolation algorithm and adjust the resolution to a standard size (such as 640x480). Where F t Represents the original image data of the tth frame; F t_denoisedRepresents the denoised image data; Y, U, V represent the components of the YUV color space, r, g, b represent the lowercase variables of the red, green, and blue components in the RGB color space, respectively; R, G, B represent the uppercase variables of the red, green, and blue components in the RGB color space, respectively.
[0039] Construct a spatiotemporal pyramid structure: define the number of layers L (e.g., 3 layers), and for each layer l (l = 0, 1, ..., L-1), the time window size is W t =2 l +1; spatial scale is S l =F t_resized.shape / / 2 l ; Use average pooling with a step size of 2 l Downsample each layer. Apply the continuous homology algorithm: construct the distance matrix D for each layer of data, calculate the simplicial complex under different thresholds ε, and calculate the continuous homology group H k (ε), k = 0, 1; Generate multi-scale persistence graph: record the birth and death time of each homology class and store the results in MPD. Extract traditional visual features: implement a simplified convolutional neural network, including 3 convolutional layers, each followed by ReLU activation and maximum pooling; 2 fully connected layers. Concatenate MPD and traditional visual features into a vector.
[0040] Build a simplified VGG network: 5 convolutional blocks, each containing 2-3 convolutional layers and a max pooling layer, using pre-trained weights. Extract potential prior data: implement a simple RNN, an LSTM layer and a fully connected layer, using the features of the previous N frames as input to predict the features of the current frame. Calculate the mean μ and standard deviation σ, and normalize the data: x norm = (x -μ) / σ; concatenate all eigenvectors into a large vector, where x norm Represents the normalized data.
[0041] This embodiment improves the extraction efficiency and quality of temporal context features by introducing a spatiotemporal pyramid structure and a continuous homology algorithm. The spatiotemporal pyramid structure can capture video features of different time scales and spatial resolutions, so that the subsequent encoding process can better adapt to the spatiotemporal changes of the video content. The application of the continuous homology algorithm enables the system to extract topological features of video data, which are highly robust to the motion patterns and scene structure changes in the video. By fusing topological features with traditional visual features, a more comprehensive and stable temporal context representation is formed. This fusion feature can not only more accurately reflect the temporal continuity of the video content, but also maintain stability when the video scene changes drastically. The introduction of super-prior and potential prior provides additional prior knowledge for the subsequent encoding process, which helps to improve the encoding efficiency. Especially when processing complex video sequences, this prior information can help the system better predict and encode the correlation between video frames. In addition, by standardizing and splicing features from different sources, a unified and information-rich input feature set is formed, which provides high-quality input data for the subsequent variational autoencoder. This fusion of multi-source information not only improves the expressiveness of features, but also enhances the adaptability of the system to different types of video content. This embodiment lays a solid foundation for the video compression process through multi-scale and multi-modal feature extraction and fusion, and effectively improves the efficiency and quality of subsequent encoding.
[0042] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:
[0043] S21, inputting the input feature set into the encoder network to generate initial latent variables; obtaining conditional information of the current frame data, combining the conditional information with the initial latent variables, and constructing a multi-level latent variable generation network model; wherein the conditional information includes scene type and motion complexity;
[0044] S22. Based on the conditional information and the initial latent variables, an adaptive prior network is used to dynamically adjust the prior distribution parameters of each layer in the multi-level latent variable generation network model to obtain an adjusted multi-level latent variable generation network model; using the adjusted multi-level latent variable generation network model, latent variables of different abstraction levels are generated layer by layer; and latent variables of all levels are combined to form a multi-level latent variable;
[0045] S23, mapping multi-level latent variables to predefined algebraic clusters, constructing a geometric structure representation of probability distribution; obtaining the encoding results of the most recent N frames from the pre-stored data cache as observation data, and using algebraic statistical methods based on the observation data and the geometric structure representation to estimate the parameters on the algebraic cluster; using semi-algebraic set theory, using pre-set constraints to constrain the space of parameters on the algebraic cluster, and obtaining the constrained parameters; wherein N is a natural number greater than 0;
[0046] S24. Based on the constrained parameters, a conditional probability model of the latent variables is constructed, the probability value of each latent variable under predetermined conditions is calculated, and probability distribution data is generated.
[0047] In this embodiment, an improved hierarchical conditional variational autoencoder is used for encoding, specifically: receiving an input feature set, inputting it into the encoder network, and generating initial latent variables. Obtaining conditional information such as the scene type and motion complexity of the current frame from the scene analysis module, combining this conditional information with the initial latent variables, and inputting it into a multi-level latent variable generation network. The network generates latent variables at different levels of abstraction layer by layer, and the output of each layer serves as the input of the next layer. At the same time, an adaptive prior network is used to dynamically adjust the prior distribution parameters of each layer according to the current input. The latent variables of all levels are combined to form a final multi-level potential representation.
[0048] Map the multi-level latent representation to a predefined algebraic cluster to construct a geometric structure representation of the probability distribution. Obtain the encoding results of the most recent N frames from the data cache as observation data, and use algebraic statistical methods to estimate the parameters on the algebraic cluster based on these observation data. Apply semi-algebraic set theory and use pre-set constraints to constrain the parameter space to ensure the rationality of the estimation results. Use the estimated parameters to construct a conditional probability model of the latent variables. Calculate the probability value of each latent variable under given conditions, generate probability distribution data, and pass it together with the multi-level latent representation to the following steps for adaptive quantization.
[0049] In one embodiment of the present application, an improved hierarchical conditional variational autoencoder (HCVAE) model structure is constructed: the encoder is input features ->Dense layers ->μ, σ; sampling layer: ε~ N(0, I), z =μ+σ*ε; decoder: z->Dense layers ->reconstructed features . Condition information processing: scene type is scene type =classify scene (F t_resized ); the motion complexity is motion complexity =estimatemotion (F t , F t_1 ). Generate multi-level latent variables: for l in range(num layers ):μ l , σ l = encoder l (z l1 , scene type , motion complexity );z l = sample(μ l , σ l ). Adaptive prior network: p z = prior network (scene type , motion complexity ). Training HCVAE: loss = reconstruction loss + KL divergence (q(z|x), p z ); optimize(loss). Where μ, σ represent the mean and standard deviation of the latent space; ε represents random noise; z represents the sampled latent variable; scene type Indicates the scene type; motion complexity Indicates the complexity of motion; z l represents the latent variable at the lth level; p z Represents adaptive prior distribution, Dense layers represent dense layers or fully connected layers; reconstructed features Represents reconstruction features; classify scene Indicates scene classification; estimate motion Represents motion estimation, which refers to the function of calculating motion information between adjacent frames; num layers Indicates the number of layers; encoder l ( ) represents the encoder function, which represents the encoder of the lth layer and is used to generate the mean and standard deviation of the latent variables; scene type Indicates the scene type; motion complexity Indicates motion complexity; sample() indicates sampling function; prior network Represents a priori network; reconstruction loss represents the reconstruction loss; KL divergenceis the KL divergence, which represents the difference between two probability distributions and is used to measure the difference between the model distribution and the prior distribution; q(z|x) is the posterior distribution, which represents the conditional probability distribution of the latent variable z given the input x; optimize(loss) represents the optimization loss.
[0050] Define an algebraic variety: V = {(x, y, z) ∈ R³| f(x, y, z) = 0}, where f is a polynomial function that represents the geometric structure of the probability distribution. Apply algebraic statistical methods: Use maximum likelihood estimation (MLE) to solve for the parameter: θ MLE =argmax θ Σ log p(x i |θ); apply semi-algebraic set theory to constrain the parameter space: S = {θ∈ R d | g 1 (θ) ≥ 0, ..., g k (θ) ≥ 0}, where g i is a polynomial inequality. Construct a conditional probability model: p(z | x, θ) = exp(f(x, z, θ)) / Σ z exp(f(x, z, θ)) ; Compute the conditional probability of each latent variable: for z i in z,p i = p(z i |x,θ MLE ). Where V represents the algebraic variety of probability distribution; f represents the polynomial function that defines the algebraic variety; θ represents the model parameters; θ MLE represents the parameters of the maximum likelihood estimation; S represents the semi-algebraic set of parameter space; g i represents the polynomial inequality defining a semialgebraic set; p(z|x,θ) represents the conditional probability of z given x and parameter θ.
[0051] In another embodiment of the present application, the HCVAE model structure is implemented: the encoder is a multi-layer perceptron, outputting μ and σ; the sampling layer is z = μ + σ * ε, where ε ~ N (0, I); the decoder is a multi-layer perceptron, reconstructing the input. Conditional information processing: the scene type is to implement a simple scene classifier (such as a decision tree); the motion complexity is to calculate the optical flow of adjacent frames and take the average value of their moduli. Multi-level latent variable generation: implement multi-level encoders and decoders, and add conditional information to each layer. The adaptive prior network is specifically to implement a small neural network to generate prior distribution parameters based on conditional information. When training HCVAE, the reconstruction loss is implemented: mean square error; the KL divergence is implemented: DKL = 0.5*Σ(μ²+σ - log(σ²) - 1); the total loss = reconstruction loss + β* KL divergence; and random gradient descent optimization is used. Where μ, σ represent the mean and standard deviation of the latent space, ε represents random noise; z represents the latent variable obtained by sampling; DKL represents the KL divergence; β represents the weight of the KL divergence.
[0052] Define an algebraic variety: Choose a quadratic surface as a simplified model: ax²+ by²+ cz²+ dxy + exz + fyz + gx + hy + iz + j = 0. Apply algebraic statistics: Collect data points (x i ,y i , z i ), construct the least squares problem: min Σ(ax i ²+ by i ²+ cz i ²+ dx i y i + ex i z i +fy i z i + gx i +hy i + iz i+ j)²; Use SVD to solve the linear equations and get the parameters (a, b, c, d, e, f, g, h, i, j). Apply semi-algebraic set theory to constrain the parameter space: Define inequality constraints: such as a > 0, b > 0, c > 0 (make sure the surface is an ellipsoid). Construct a conditional probability model: Use the distance from the point to the surface as a probability measure: p(z|x, y)∝exp(-k * distance((x, y, z), surface)), where k is the scaling factor. Calculate the conditional probability of each latent variable: For each point, calculate its distance to the surface and use the above formula to calculate the conditional probability. Where a, b, c, d, e, f, g, h, i, j represent the parameters of the algebraic cluster, k represents the scaling factor in the probability calculation, p(z|x, y) represents the conditional probability of the latent variable z given the inputs x and y; distance() represents the distance function; surface represents the surface.
[0053] This embodiment improves the coding efficiency and accuracy of the entropy model by combining an improved hierarchical conditional variational autoencoder (HCVAE) with algebraic geometry methods. The multi-level structure of HCVAE enables the system to capture the characteristics of video data at different abstract levels, thereby more comprehensively representing the complexity of video content. The introduction of conditional information, such as scene type and motion complexity, enables the encoding process to be dynamically adjusted according to the characteristics of the video content, improving the adaptability of encoding. The application of adaptive prior networks further enhances the system's modeling capabilities for different types of video data, making the generated potential representation more compact and information-rich. The introduction of algebraic geometry methods provides a theoretical basis for probability distribution estimation. By mapping probability distributions to algebraic clusters, the system can more accurately capture the intrinsic structure of data. Not only does it improve the accuracy of probability estimation, but it also enhances the model's ability to express complex data distributions. The application of semi-algebraic set theory ensures the rationality of parameter estimation, effectively avoids overfitting problems, and improves the generalization ability of the model. This probability model optimization based on algebraic geometry is particularly suitable for processing nonlinear and non-Gaussian distributions commonly found in video data, and can more accurately capture complex dependencies between video frames. In addition, by constructing a conditional probability model, the system can more accurately predict the probability distribution of each potential variable, providing more accurate prior information for subsequent entropy coding. This improvement is particularly effective for processing highly dynamic and complex video sequences, and can retain the detailed information of the video to the maximum extent while maintaining a high compression rate. This embodiment improves the coding efficiency and accuracy of the entropy model through advanced probability modeling and optimization technology, laying the foundation for achieving efficient video compression.
[0054] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:
[0055] S31, using a pre-trained sparse dictionary, projecting the multi-level latent variables into a sparse domain to obtain a sparse domain representation; based on the content features of the current frame data, selecting a quantization matrix from a preset quantization matrix library; based on the probability distribution data, using a context analysis module, dynamically adjusting the parameters of the quantization matrix to obtain adjusted quantization parameters;
[0056] S32. Perform a quantization operation based on the sparse domain representation and the adjusted quantization parameter to obtain a quantized discrete value, that is, quantized data.
[0057] According to one aspect of the present application, it also includes: S33, based on the quantized discrete values, using a convex optimization algorithm to reconstruct the original data; calculating the perceptual quality difference between the reconstructed original data and the multi-level latent variables, and based on the perceptual quality difference, further fine-tuning the adjusted quantization parameters to obtain the fine-tuned quantization parameters.
[0058] According to another aspect of the present application, in step S31, based on the probability distribution data, a context analysis module is used to dynamically adjust the parameters of the quantization matrix, and the adjusted quantization parameters are further:
[0059] S311. Based on multi-level latent variables, use convolutional neural networks to extract local features, including edges and textures; use long short-term memory networks to extract global features, including motion trajectories and scene changes;
[0060] S312, based on local features, global features and probability distribution data, use the attention mechanism for weighted fusion to obtain a comprehensive feature vector;
[0061] S313. Based on the comprehensive feature vector, a reinforcement learning algorithm is used to dynamically adjust the parameters of the quantization matrix to obtain adjusted quantization parameters.
[0062] In this embodiment, a pre-trained sparse dictionary is used to project the multi-level potential representation into a sparse domain. Based on the content features of the current frame, a quantization matrix that satisfies the principle of restricted synchronization is selected from the quantization matrix library. A context analysis module is applied to dynamically adjust the quantization parameters by considering local and global video content features and combining probability distribution data. A quantization operation is performed to convert the sparse domain data into discrete values. Convex optimization algorithms such as LASSO are applied to reconstruct the original data from the quantized discrete values. The perceptual quality difference between the reconstructed data and the original multi-level potential representation is calculated, and the quantization parameters are fine-tuned based on the difference to balance the bit rate and visual quality. The quantized data and the adjusted quantization parameters are passed to the following steps for entropy coding.
[0063] In one embodiment of the present application, a sparse representation dictionary is constructed: the dictionary D is learned using the K-SVD algorithm, and the latent variable z is sparsely encoded: α = argminα ||z - Dα|| 2 2 +λ||α|| 1 . Construct RIP quantization matrix: generate random Gaussian matrix Φ, orthogonalize Q = orth(Φ), scale A = sqrt(m / n) * Q, where m < n. Perform context analysis: extract local features: local features =extract local _ features (z); Extract global features: global features =extract global _ features (z); Fusion features: context = concatenate([local features ,global features ]). Dynamically adjust quantization parameters: Construct neural network f q :q = f q (context); Quantization operation: z q =round(z / q) * q. Reconstructed using the LASSO algorithm: z hat = argmin z ||Az - y|| 2 2 + λ||z|| 1 ; Calculate perceptual quality: SSIM = structural similarity (z, z hat ); Update quantization parameter: q = update quantization (q, SSIM). Where D represents the sparse representation dictionary; α represents the sparse coefficient; Φ represents the random Gaussian matrix; Q represents the orthogonalized matrix; A represents the RIP quantization matrix; z represents the original latent variable; z q represents the quantized latent variable; q represents the quantization step size; z hat represents the latent variable of reconstruction; SSIM represents the structural similarity index, extract local _ features Indicates the extraction of local features; extract global _ features Indicates the extraction of global features, and round() indicates the rounding function.
[0064] In another embodiment of the present application, a sparse representation dictionary is constructed to implement the K-SVD algorithm: initialize the dictionary D, iterative optimization: use the orthogonal matching pursuit (OMP) algorithm to update the sparse coding step; for each atom, use SVD to update the dictionary update step. Construct a RIP quantization matrix: generate a random Gaussian matrix Φ, implement the Gram-Schmidt orthogonalization process, and normalize the matrix columns. Perform context analysis: local features are: calculate the mean, variance, and gradient of the local area; global features are: calculate the color histogram and edge direction histogram of the entire frame. Dynamically adjust the quantization parameters to implement a simple feedforward neural network, with context features as input; output as the quantization step size. Reconstruct and optimize: use the proximal gradient descent method to solve the optimization problem; implement the calculation of the mean, variance, and covariance, calculate the similarity according to the SSIM formula, and adjust the quantization step size according to SSIM.
[0065] This embodiment improves the efficiency and adaptability of the quantization process by introducing a context-aware adaptive quantization (CS-CAAQ) method based on compressed sensing theory. The application of compressed sensing theory enables the system to represent data in a sparse domain, reducing the amount of information that needs to be encoded. It is particularly suitable for processing natural video data because video signals usually exhibit sparse characteristics in certain transform domains (such as wavelet domain or DCT domain). By constructing a quantization matrix that satisfies restricted synchronization (RIP), the system can reduce the sampling rate while ensuring the reconstruction quality, thereby improving the compression efficiency. The introduction of the context-aware mechanism enables the quantization process to be dynamically adjusted according to the local and global characteristics of the video content, which is particularly important for processing complex and changing video scenes. For areas with intense motion, the system can adopt a more detailed quantization strategy to retain more details; while for static background areas, a coarser quantization can be used to improve the compression rate. The application of convex optimization algorithms such as LASSO ensures high-quality reconstruction from quantized discrete values to original data, effectively reducing quantization errors. This reconstruction process is particularly suitable for processing texture details and edge information in videos, and can maintain good visual quality at high compression rates. The introduction of perceptual quality indicators enables the system to achieve a better balance between bit rate and visual quality. This optimization based on the characteristics of the human visual system is particularly suitable for video compression applications because it can maximize compression efficiency while ensuring subjective visual quality. In addition, the dynamic adjustment mechanism of the adaptive quantization parameters enables the system to quickly respond to changes in video content, such as scene switching or sudden changes in lighting conditions, thereby maintaining stable compression performance throughout the video sequence. This embodiment combines compressed sensing theory and context-aware technology to achieve an efficient and adaptive quantization process, improve the efficiency and quality of video compression, and especially performs well when processing complex and dynamically changing video content.
[0066] like Figure 5As shown, according to one aspect of the present application, step S4 is further:
[0067] S41, obtaining statistical characteristics of the most recent N coded frames from a pre-stored coding statistics cache, and dynamically updating a pre-configured symbol occurrence probability model based on the statistical characteristics and quantized data to obtain an updated symbol occurrence probability model;
[0068] S42, based on the updated symbol occurrence probability model, assigning an optimal coding length to each symbol; based on the coding length and the quantized data, using a context adaptive arithmetic coding algorithm to calculate the coded data; based on the coded data, constructing and optimizing a variable length coding strategy to obtain optimized coded data; organizing the optimized coded data into a bit stream according to a predefined format, including necessary header information and metadata, to form a final compressed data packet;
[0069] S43. Based on the final compressed data packet, a statistical manifold is constructed and the Fisher information matrix is calculated; based on the Fisher information matrix, an iterative optimization algorithm based on natural gradient is constructed, and based on the current parameters of the iterative optimization algorithm, the rate-distortion performance under the current parameter configuration is evaluated until a preset number of iterations is reached; based on the evaluation results, the optimal parameter configuration is output to obtain an optimized compressed data packet.
[0070] This embodiment receives quantized data and adjusted quantization parameters. The statistical characteristics of the most recent N coded frames are obtained from the coding statistics cache, and the symbol occurrence probability model is dynamically updated in combination with the data of the current frame. According to the updated probability model, the optimal coding length is assigned to each symbol. Considering local spatial and temporal correlations, the context-adaptive arithmetic coding algorithm is applied to further improve the compression efficiency. Based on the data characteristics of the current frame, the Huffman coding table is adjusted in real time to optimize the variable-length coding strategy. The encoded data is organized into a bit stream in a predefined format, including necessary header information and metadata, to form the final compressed data packet, and passed to the following steps for global optimization, and stored in the encoding result cache for subsequent frames.
[0071] Receive compressed data packets and collect intermediate data and adjustable parameters of each module from the parameter cache of each module. Construct a statistical manifold and map the rate-distortion problem onto the manifold. Calculate the Fisher information matrix to quantify the importance of each direction in the parameter space. Construct an iterative optimization algorithm based on natural gradient to search for the optimal parameter configuration on the statistical manifold. In each iteration, update the parameters of each module and re-execute steps S1 to S4 to evaluate the rate-distortion performance under the current parameter configuration. Adjust the search direction and step size based on the evaluation results. When the preset number of iterations is reached or the performance improvement is less than the threshold, output the optimal parameter configuration, update the parameter cache of each module, and pass the optimized compressed data packet to the following steps for decoding and reconstruction.
[0072] In one embodiment of the present application, the symbol probability model is dynamically updated: a sliding window W is maintained, which contains the statistical information of the latest N coded frames; for each symbol s, its occurrence frequency in W is calculated: P(s) = count(s) / Σcount(s i ); Using adaptive hybrid model: P new (s) =λP(s) + (1-λ)P prior (s), using the Shannon coding principle to allocate the optimal code length: L(s) = -log 2 (P new (s)). Perform context adaptive arithmetic coding: define context C(s) as the pixels around s, calculate conditional probability: P(s|C(s)) = P(s, C(s)) / P(C(s)); use arithmetic encoder to encode symbol s according to P(s|C(s)). Dynamic Huffman coding table update: according to P new (s) Build a Huffman tree and update the Huffman coding table regularly (e.g., every 100 frames). Organize the bitstream, including the frame header, which contains metadata such as frame type and quantization parameter; the coded data, which contains the Huffman coded or arithmetic coded bitstream; and the frame tail, which is used for synchronization and error detection. Where W represents the sliding window; P(s) represents the empirical probability of symbol s; P prior (s) represents the prior probability of symbol s; P new (s) represents the updated symbol probability; λ represents the mixing weight; L(s) represents the optimal coding length of symbol s; C(s) represents the context of symbol s; P(s|C(s)) represents the conditional probability of symbol s given the context C(s), and count( ) represents the counting function.
[0073] Constructing statistical manifold: Define parameter space Θ = {θ|θ∈R d}; define the probability distribution family P = {p(x|θ) |θ∈Θ}; the statistical manifold S = (Θ, g), where g is the Fisher information matrix; calculate the Fisher information matrix: g ij (θ) = E x [-Ψ² / Ψθ i Ψθ j log p(x|θ)]; construct the natural gradient optimization algorithm: θ new =θ old - η g -1 θ old ▽Lθ old , where L(θ) is the rate-distortion function and η is the learning rate. Iterative optimization process, when not converged: 1. Use the current parameter θ to perform steps S1 to S4; 2. Calculate the rate-distortion performance: RD = λR + D; where R is the bit rate, D is the distortion, and λ is the Lagrange multiplier; 3. Calculate the natural gradient: ▽ nat L = g -1 (θ) ▽L(θ); 4. Update parameters: θ=θ- η ▽ nat L; 5. Check the convergence conditions. Output the optimal parameter configuration. Where g ij represents the i-th and j-th elements of the Fisher information matrix; θ represents the model parameters; ▽L(θ) represents the gradient of the rate-distortion function; ▽ nat L represents the natural gradient of the rate-distortion function; RD represents the rate-distortion performance index, and Ψ represents the sign of the partial derivative.
[0074] In another embodiment of the present application, the symbol probability model is dynamically updated: a counter array is maintained to record the number of occurrences of each symbol; the probability is updated regularly: P(s) = count(s) / total count ; Implementing an adaptive hybrid model: P new (s) =λP(s) + (1-λ)P prior (s). Optimal code length allocation, achieving -log 2 (P new (s)). Perform context-adaptive arithmetic coding: implement the context model and use the surrounding pixel values as context; implement conditional probability calculation; implement arithmetic coding: maintain an interval [low, high); subdivide the interval according to the symbol probability, and output a binary number that can uniquely determine the interval. Dynamic Huffman coding table update: implement the Huffman tree construction algorithm and rebuild the Huffman tree regularly. Organize the bitstream: construct a frame header format that contains necessary metadata; organize the encoded data into a byte stream. Where P(s) represents the probability of symbol s; λ represents the mixing weight; P new(s) represents the updated symbol probability; [low, high) represents the current interval of arithmetic coding.
[0075] Construct statistical manifolds: select key parameters as optimization objects and define parameter space; implement probability distribution calculations: such as Gaussian distribution, Laplace distribution, etc. Calculate Fisher information matrix: implement second-order derivative calculations of log-likelihood; use Monte Carlo method to estimate and calculate expectations. Construct natural gradient optimization algorithm to implement inverse operation of Fisher information matrix (for low dimensions, direct inversion can be performed; for high dimensions, conjugate gradient method can be used for approximation); implement parameter update: θ new =θ old - η g -1 θ old ▽Lθ old Iterate the optimization process to calculate the rate-distortion function: the bit rate R is the number of bits after encoding; the distortion D is the MSE or perceptual quality index; the finite difference method is used to calculate the gradient; the parameter update and convergence check are implemented. Output the optimal parameter configuration.
[0076] This embodiment improves the efficiency and adaptability of bitstream coding by introducing a dynamic entropy coding optimization (DECO) method. The application of a dynamic symbol probability model enables the system to adapt to changes in the statistical characteristics of video content in real time, which is particularly effective for processing long sequence videos or videos with changing content. By analyzing the statistical characteristics of the last N coded frames, the system can accurately capture the local temporal correlation of video data, thereby assigning the optimal coding length to each symbol. This adaptive coding strategy not only improves compression efficiency, but also enhances the system's ability to respond to changes in video content. The introduction of context-adaptive arithmetic coding further utilizes the local spatial and temporal correlation of video data to improve coding efficiency. It is particularly suitable for processing texture areas and motion sequences in videos, and can effectively capture the conditional probability relationship between pixel values, thereby achieving a more compact representation. The real-time adjusted Huffman coding table provides another layer of adaptability for the system, enabling the coding process to quickly respond to sudden changes in video content, such as scene switching or lighting changes. This dynamic coding table strategy not only improves coding efficiency, but also enhances the system's ability to adapt to different types of video content. In addition, by organizing the encoded data into a predefined bitstream format, including necessary header information and metadata, the system ensures the integrity and decodability of the compressed data. This structured data organization is not only convenient for subsequent transmission and storage, but also provides the necessary context information for the decoding end, which helps to improve the accuracy and efficiency of decoding. This embodiment realizes an efficient and flexible bitstream generation process through a multi-level adaptive coding strategy. It can effectively handle the spatiotemporal correlation and content diversity of video data, and retain the information content of the video to the maximum extent while ensuring a high compression rate. Especially for videos with long time series or complex and changeable content, this dynamic optimization method can continuously maintain efficient coding performance, providing key support for achieving high-quality, low-bitrate video compression.
[0077] By introducing a rate-distortion optimization method based on information geometry, global optimization of the entire video compression process is achieved, and the overall performance of the system is improved. Modeling the rate-distortion problem as an optimization problem on a statistical manifold enables the system to understand and optimize the compression process from a higher level of abstraction. By constructing a statistical manifold, the system can capture the geometric structure in the parameter space, which is crucial for understanding the impact of parameter changes on compression performance. The application of the Fisher information matrix further quantifies the importance of each direction in the parameter space, allowing the optimization process to allocate computing resources more effectively and focus on optimizing those parameters that have the greatest impact on performance. The construction of an iterative optimization algorithm based on natural gradient enables the system to perform efficient parameter search on the statistical manifold. It is particularly suitable for processing high-dimensional parameter spaces and can quickly converge to the optimal solution, avoiding the local optimal problem that may be encountered in traditional gradient descent methods. By re-executing steps S1 to S4 in each iteration and evaluating the rate-distortion performance, the system can fully consider the mutual influence between various modules and achieve true end-to-end optimization. This global optimization strategy is particularly effective for processing complex video compression tasks and can find the best parameter configuration under different compression rate requirements. The mechanism of dynamically adjusting the search direction and step size further enhances the adaptability of the optimization process, so that the system can quickly respond to changes in video content, such as scene switching or changes in motion complexity. This adaptive optimization is particularly suitable for processing long-time series of videos or video streams with variable content. By setting reasonable termination conditions, such as preset iteration times or performance improvement thresholds, the system can achieve a good balance between optimization effect and computational overhead. This is particularly important for real-time or near-real-time video compression applications, and can achieve optimal compression performance under limited computing resources. The present embodiment realizes the coordinated optimization of the entire video compression system through a global optimization strategy, and improves the adaptability and performance of the system under different video content and compression requirements. It can effectively handle complex nonlinear relationships in video compression, and provides strong support for achieving efficient and high-quality video compression.
[0078] like Figure 6 As shown, according to one aspect of the present application, step S5 is further:
[0079] S51, based on the header information and metadata in the optimized compressed data packet, parsing the encoding parameters, and reversely executing steps S4 to S2 to obtain preliminarily reconstructed video frame data; using a pre-trained perceptual quality assessment module, assessing the perceptual quality of the preliminarily reconstructed video frame data to obtain an assessment result;
[0080] S52, based on the evaluation result, adaptively selecting and applying post-processing technology from a preset post-processing technology library, including deblocking filtering and detail enhancement, processing the preliminarily reconstructed video frame data to obtain a processed frame; inputting the processed frame into a pre-trained adversarial network to improve visual quality and obtain an enhanced frame;
[0081] S53, comparing the enhanced frame with the current frame data in step S1, and calculating the perceptual similarity; adjusting the post-processing parameters based on the perceptual similarity; and outputting the reconstructed video frame with the best perceptual quality based on the adjusted post-processing parameters.
[0082] This embodiment receives an optimized compressed data packet. According to the header information and metadata in the data packet, the encoding parameters are parsed, and the process from step S4 to step S2 is reversed to obtain preliminary reconstructed video frame data. A pre-trained perceptual quality assessment module is used to simulate the human visual system to evaluate the perceptual quality of the reconstructed frame. According to the evaluation results, post-processing techniques such as deblocking filtering, detail enhancement, etc. are adaptively selected and applied from the post-processing technology library. The processed frame is input into the pre-trained adversarial network to further improve the visual quality. The enhanced frame is compared with the original input frame, the perceptual similarity is calculated, and the post-processing parameters are adjusted according to the similarity. The reconstructed video frame with the best perceptual quality is output, stored in the reconstructed frame buffer for subsequent processing, and passed to the video output module for display or storage.
[0083] In one embodiment of the present application, a preliminary reconstruction is performed: the process from step S4 to step S2 is reversed to obtain a preliminary reconstructed frame F r . Perform perceptual quality assessment: Use the pre-trained perceptual quality assessment network Q, q = Q(F r ), q∈[0,1], where 1 indicates the highest quality. Adaptive post-processing: Deblocking: F d =deblock(F r, strength=f(q)); detail enhancement: F e = enhance(F d , factor=g(q)), where f and g are adaptive functions based on q. Adversarial network enhancement: Generator G: F g = G(F e ); Discriminator D: authenticity = D(F g ); Optimization target: min G max D E[log(D(F original )) + log(1 - D(G(F e)))]. Perform perceptual similarity calculation and parameter adjustment: Calculate SSIM: s = SSIM(F original , F g ); Adjust post-processing parameters: strength new = strength * (1 + α(1 - s)) factor new =factor * (1 + β(1 - s)), where α and β are adjustment coefficients. The frame with the highest perceptual quality is selected as the final output: F final = argmax F Q(F) for F in {F r , F d , F e , F g}. Where q represents the perceived quality score, F d Denotes the frame after deblocking, F e represents the frame after detail enhancement, F g represents the frame enhanced by the adversarial network, F original represents the original input frame, s represents the structural similarity index (SSIM), F final Represents the final output frame. The perceptual quality assessment network Q can use the pre-trained VGG19 network and adapt it to the video quality assessment task through fine-tuning. The deblocking effect function deblock() can be implemented as: def deblock(frame, strength): return cv2.fastNlMeansDenoisingColored(frame, None, strength, strength, 7, 21); The detail enhancement function enhance() can be implemented as: def enhance(frame, factor): kernel = np.array([[-1, -1, -1], [-1, 9, -1], [-1, -1, -1]]) returncv2.filter2D(frame, -1, kernel * factor); The adversarial networks G and D can adopt an architecture similar to SRGAN, but need to be adjusted for the video frame reconstruction task. SSIM calculation can use the skimage library: from skimage.metricsimport structural similarity as ssim s = ssim(F original , F g , multichannel=True); the parameter adjustment function can be implemented as: def adjust params(strength, factor, ssim, alpha=0.1, beta=0.2): strength new = strength * (1 + alpha * (1 - ssim)) factor new = factor * (1 +beta * (1 - ssim)) return strength new , factor new .
[0084] In another embodiment of the present application, a preliminary reconstruction is performed: the previous steps are reversed to obtain a preliminary reconstructed frame. Perceptual quality assessment is performed: a simplified version of the perceptual quality assessment network is implemented, a pre-trained VGG feature extractor is used, and several fully connected layers are added for quality scoring. Adaptive post-processing is performed: a deblocking filter is implemented, and an adaptive window mean filter is constructed; detail enhancement is implemented: an adaptive sharpening filter is constructed. Adversarial network enhancement is performed: a simplified version of the generator G is implemented: a U-Net structure is used, including jump connections; a simplified version of the discriminator D is implemented: a PatchGAN structure is used to implement an adversarial training process. Perceptual similarity calculation and parameter adjustment are performed: SSIM calculation is implemented: local mean, variance, and covariance are calculated; similarity is calculated according to the SSIM formula; a parameter adjustment function is implemented: the processing intensity is linearly adjusted according to the SSIM. All candidate frames are scored using the perceptual quality assessment network, and the frame with the highest score is selected as the final output.
[0085] According to one aspect of the present application, a pre-trained perceptual quality assessment module is used to assess the perceptual quality of the pre-reconstructed video frame data, and the assessment result is specifically as follows: the pre-reconstructed video frame data is standardized to ensure that the input data meets the requirements of the pre-trained model. Standardization may include adjusting the image size, normalizing pixel values, etc. The high-level features of the video frame are extracted using the feature extraction layer of the pre-trained model. These features may include edges, textures, color distribution, etc. The pre-processed features are input into the pre-trained perceptual quality assessment module. The module is usually a deep neural network that has been trained on a large amount of labeled data. The model outputs a perceptual quality score, which is usually a scalar value representing the visual quality of the video frame. The score can be based on a variety of indicators, such as structural similarity (SSIM), peak signal-to-noise ratio (PSNR), etc. The video frame is multi-scale processed to extract features at different scales. This can be achieved using a pyramid structure or a multi-scale convolutional neural network (MSCNN). The perceptual quality scores at different scales are weighted averaged to obtain a comprehensive perceptual quality score. This step can improve the robustness and accuracy of the evaluation. Detect and process possible abnormal scores, such as scores that are too high or too low. Statistical methods or rule-based detection algorithms can be used. Smooth the perceptual quality scores of consecutive frames to reduce the fluctuation of the scores. Methods such as sliding average or Kalman filtering can be used. Store the final perceptual quality score in a specified data structure for subsequent processing and analysis. If necessary, the evaluation results can be visualized, such as drawing a quality score curve or generating a quality report.
[0086] According to one aspect of the present application, the enhanced frame is compared with the current frame data in step S1, and the perceptual similarity is calculated specifically as follows: features are extracted from the enhanced frame and the current frame data. A deep learning model such as a convolutional neural network (CNN) can be used to extract high-level visual features. The extracted features are matched, and the similarity between the features is calculated using metrics such as Euclidean distance and cosine similarity. Based on the results of feature matching, the overall perceptual similarity is calculated. A weighted average method can be used to combine the similarities of different features to obtain a comprehensive perceptual similarity score. The calculated perceptual similarity score is compared with a preset threshold to evaluate whether the similarity is within an acceptable range. If the similarity is low, post-processing parameters may need to be adjusted. Adjust the post-processing parameters based on the results of the similarity evaluation. Optimization algorithms such as gradient descent can be used to gradually adjust parameters to improve perceptual similarity.
[0087] This embodiment improves the quality of video decoding and reconstruction by introducing the perception-guided reconstruction enhancement (PGRE) technology. By reversely executing the encoding steps, the system can accurately parse the encoding parameters and data in the bitstream, providing a reliable foundation for the subsequent reconstruction process. The introduction of the pre-trained perceptual quality assessment module simulates the characteristics of the human visual system and can accurately assess the perceptual quality of the reconstructed frame. This evaluation method based on human eye perception is particularly suitable for video applications because it can capture the most sensitive visual features of the human eye, such as edge clarity, texture details, and motion coherence. The adaptive post-processing technology selection mechanism based on the evaluation results enables the system to adopt the most appropriate recovery strategy for different types of video content and compression artifacts. For example, for areas containing a lot of texture, the system may select a stronger deblocking effect filter; while for areas containing clear edges, detail enhancement technology may be applied. This adaptive processing strategy ensures that while improving the overall visual quality, important image details are not over-smoothed. The application of the pre-trained adversarial network further improves the visual quality of the reconstructed video. The adversarial learning mechanism can generate high-frequency details that are closer to the original video, especially when processing highly compressed videos, it can effectively restore the texture and detail information lost by the compression process. While maintaining the overall structure of the video, a more realistic and natural visual effect can be generated. By comparing the enhanced frame with the original input frame and calculating the perceptual similarity, the system can objectively evaluate the reconstruction quality and dynamically adjust the post-processing parameters accordingly. This closed-loop feedback mechanism ensures the stability and consistency of the reconstruction process, especially when processing long-time series of videos, it can maintain stable reconstruction quality. In addition, this quality assessment and parameter adjustment mechanism based on the original frame can also effectively prevent artifacts caused by over-processing, such as over-sharpening or unnatural textures. The strategy of storing the reconstructed video frames in the cache for subsequent processing provides additional support for processing video sequences with time correlation. This cache mechanism enables the system to use the information of the previous and next frames to improve the reconstruction quality of the current frame, especially when processing motion sequences or scene transitions, it can provide more coherent and natural visual effects. This embodiment achieves a high-quality video reconstruction process by combining advanced perceptual quality assessment, adaptive post-processing and adversarial learning technology. It can not only effectively restore the visual information lost by the compression process, but also optimize the reconstruction results according to the visual characteristics of the human eye, ensuring the high perceptual quality of the reconstructed video. It is particularly suitable for video applications that require high visual quality, such as high-definition video streaming, professional video production, etc., and can maintain excellent visual effects at high compression rates.
[0088] According to one aspect of the present application, a video compression system based on a variational autoencoder improved entropy model includes:
[0089] at least one processor; and,
[0090] a memory communicatively connected to at least one of the processors; wherein,
[0091] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the video compression method based on the variational autoencoder improved entropy model described in any of the above embodiments.
[0092] In another embodiment of the present application, the potential features of the current frame are modulated by learning a quantization scaling mechanism through a variational autoencoder to support a wider quality range. A new spatial-channel intelligent quantization mechanism is constructed to adaptively determine the quantization step size based on the cross entropy and KL divergence mixed loss function to improve compression efficiency and retain more useful information. At the same time, a lightweight framework is proposed, which achieves a better quality and complexity trade-off in video compression compared to previous state-of-the-art methods.
[0093] like Figure 7 As shown, the basic model of this embodiment is divided into two parts, the upper part of the structure diagram above is the basic structure of general video compression codec, and the lower part, called the entropy model, is the main improvement of this patent, which is an entropy model improved based on the variational autoencoder. Both neural network parts use the corresponding residual network to replace the original network, reduce irrelevant connections between neurons, and achieve lightweight improvement of the model.
[0094] (1) Inputs of Entropy Model: The inputs of the entropy model include the hyperprior z' t , temporal context prior C t and the latent prior y' t-1 , these inputs provide rich contextual information, helping the model to better predict the probability distribution.
[0095] (2) Outputs of Entropy Model: In addition to the probability distribution parameters, the entropy model also outputs quantization steps, which are determined at different granularity levels, including global, channel level, and spatial-channel level.
[0096] (3) Encoding and Decoding Process: In the encoding process, the current frame is converted into a potential representation, then quantized and encoded into a bitstream. In the decoding process, the bitstream is decoded and inverse quantized to reconstruct a high-quality frame.
[0097] According to the design of video compression, the neural compression module consists of a variational autoencoder, a quantization process, and a priori model. First, the temporal feature C t is encoded into a compact latent variable e by the feature encoder t :e t =Encoder(C t ).
[0098] In order to achieve robustness to noise, the latent variable e t Applied quantization. Most existing quantization solutions in video compression only use a fixed quantization step size. In fact, content features vary greatly in space. A fixed quantization step size cannot handle various complex contents well. For example, a fixed small quantization step size cannot effectively remove noise information, while a fixed large quantization step size will result in large information loss (i.e., inherent quantization noise). Therefore, an adaptive quantization mechanism is proposed, and the quantization step size is learnable. Currently, the latest learnable quantization step size method is implemented by simple addition, subtraction, multiplication, division, and rounding, and finally calculating the cross entropy loss. However, since the purpose of the entropy model is to find a latent variable to guide the learning of the quantization step size, this simple operation cannot learn the latent variable well, so a variational autoencoder (VAE) with a residual neural network module is used here. According to the input latent variable e t and the probability distribution after parameter processing (μ t , σ t ) allows it to automatically learn the noise-resistant latent variable e' t At the same time, in order to achieve noise robustness, the data distribution and quantization step size need to be properly learned. In practice, the data distribution is unknown, so a priori model is used to estimate it, and then a joint loss is used to guide the learning of data distribution and quantization step size. The loss formula is as follows:
[0099] Loss = Loss CE + Loss KL =-log 2 p(e' t )+∑ x p(e' t ) log 2 (q(e' t ) / p(e' t ));
[0100] Where p(e' t ) is the noise-resistant latent variable e' t The estimated probability mass function follows the Laplace distribution. The prior model composed of a neural network is used to estimate the distribution parameters. The joint loss guides the compression module to learn the appropriate data distribution and quantization step size, and then achieves robustness to noise. In the framework of this model, the quantization step size qsc It is learnable in terms of spatial-channel intelligence and can adapt to regions with different content characteristics. Finally, the processed quantization step size q sc The video compression codec structure is returned and after synchronous processing, the video compression operation is finally realized.
[0101] This embodiment supports a wider PSNR range through a learning quantization scaling mechanism, which exceeds the average range of the prior art; by learning quantization scaling through a variational autoencoder, the present invention can dynamically adjust the quantization step size according to different video contents, thereby obtaining better compression effects at different quality levels; the spatial-channel intelligent quantization mechanism allows the model to apply different quantization strategies to different areas of the video, which can reduce the bit rate while maintaining image quality; by optimizing the network structure, the amount of computation and model parameters are reduced, making the model more lightweight and easy to deploy in practical applications while maintaining efficient compression performance; it not only performs well on synthetic datasets, but also achieves good performance in real-world video compression tasks.
[0102] The present invention has a wide range of application prospects. The following are some known and potential technology / product application fields and their application methods: Using the video compression technology of the present invention, the streaming media platform can transmit high-quality video content at a lower data rate, thereby reducing bandwidth costs and improving user experience. The security monitoring field can use the technology of the present invention to efficiently store and transmit video data while maintaining high-definition requirements for details to facilitate post-analysis and evidence preservation. In a mobile network environment, the use of the video compression technology of the present invention can optimize the quality and fluency of video calls, especially under bandwidth-limited or unstable network conditions. Video editing software can integrate the technology of the present invention to support more efficient video file processing and storage, speed up the editing process and reduce hardware resource consumption. Enterprises and cloud service providers can use the video compression technology of the present invention to provide users with a more cost-effective solution for data storage and reduce storage costs. By optimizing the compression of video content through the present invention, the video transmission and distribution network (CDN) can cache and distribute video data more efficiently, improving the overall performance and reliability of the system. Virtual reality (VR) and augmented reality (AR) applications can integrate the video compression technology of the present invention to support higher-quality image transmission and rendering, and enhance the user's immersive experience. In intelligent traffic monitoring and management, it can be used to optimize the compression and analysis of video data captured by traffic cameras and improve processing efficiency. Independent video producers and content creators can use the present invention to produce and share high-quality video works while controlling the file size for easy dissemination on social platforms. Educational institutions can use the video compression technology of the present invention to provide high-quality video teaching resources and optimize the performance of online teaching platforms. In the medical field, the present invention can be used to efficiently store and transmit high-resolution medical imaging data, such as MRI and CT scans, to facilitate remote diagnosis and medical cooperation. Cloud gaming platforms can use the video compression technology of the present invention to provide high-quality game streaming services, reduce latency and improve player experience. In satellite imagery and geographic information systems (GIS), the present invention can reduce the required storage space and transmission bandwidth when processing and transmitting high-resolution satellite image data. Smart devices and Internet of Things (IoT) devices can integrate the video compression technology of the present invention to optimize the transmission of video data, extend the battery life of the device and improve network efficiency. These application areas demonstrate the diversity and flexibility of the present invention, which is expected to have a significant impact on the field of video processing and transmission. With further development and optimization of the technology, its scope of application may be further expanded.
[0103] The present invention combines variational autoencoders with advanced mathematical and computer science methods. This video compression method based on the variational autoencoder improved entropy model improves the performance and efficiency of video compression in many aspects. In the feature extraction stage, the application of spatiotemporal pyramid structure and continuous homology algorithm enables the system to capture the multi-scale spatiotemporal features and topological structure of video data, which enhances the expression ability of complex video content. This high-quality feature extraction lays a solid foundation for the subsequent encoding process, enabling the system to more accurately capture the temporal correlation and spatial structure in the video, thereby maximizing the retention of video detail information while maintaining a high compression rate. In the entropy model encoding stage, the combination of improved hierarchical conditional variational autoencoder (HCVAE) and algebraic geometry methods improves the accuracy and flexibility of probability distribution estimation. The multi-level structure of HCVAE enables the system to capture video features at different levels of abstraction, while algebraic geometry methods provide powerful tools for processing complex nonlinear data distributions. This combination not only improves the encoding efficiency, but also enhances the system's adaptability to different types of video content, especially when processing highly dynamic and complex video sequences. The context-aware adaptive quantization (CS-CAAQ) method based on compressed sensing theory introduced in the adaptive quantization stage improves the efficiency and adaptability of the quantization process. It can dynamically adjust the quantization strategy according to the local and global characteristics of the video content, maximizing the compression efficiency while ensuring visual quality. This adaptive quantization strategy can improve compression performance, especially when dealing with complex and changing video scenes. The dynamic entropy coding optimization (DECO) method in the bitstream encoding stage further improves the coding efficiency, especially when dealing with long sequence videos or videos with changing content. By dynamically adjusting the symbol probability model and coding strategy, the system can continuously maintain efficient coding performance and effectively respond to changes in video content. The rate-distortion optimization method based on information geometry introduced in the global optimization stage realizes end-to-end optimization of the entire compression process. This global optimization strategy can coordinate the mutual influence between various modules, find the best parameter configuration under different compression rate requirements, and improve the overall performance and adaptability of the system. In the decoding and reconstruction stage, the application of perception-guided reconstruction enhancement (PGRE) technology improves the visual quality of the reconstructed video. By combining perceptual quality assessment, adaptive post-processing and adversarial learning techniques, the system can effectively restore visual information lost in the compression process and optimize the reconstruction results based on the visual characteristics of the human eye. While maintaining a high compression rate, it ensures high perceptual quality of the reconstructed video, which is particularly suitable for video applications that require high visual quality.
[0104] The video compression method based on the variational autoencoder improved entropy model of the present invention achieves a comprehensive improvement in video compression performance. It not only improves compression efficiency and reconstruction quality, but also enhances the system's adaptability to different types of video content. It is particularly suitable for processing complex video content with high resolution and high frame rate, and can achieve high compression rate while maintaining high visual quality, providing strong technical support for application fields such as video streaming, video surveillance, and telemedicine that require efficient compression and high-quality reconstruction. In addition, the adaptability and scalability of the method enable it to adapt to the development of future video technologies, such as emerging video formats such as 8K resolution and high dynamic range (HDR), laying the foundation for the development of next-generation video compression technology.
[0105] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.
Claims
1. A video compression method based on a variational autoencoder improved entropy model, characterized in that: The steps include: S1. Receive the current frame data in the video stream, and construct a spatiotemporal pyramid structure based on the current frame data; apply the continuous homology algorithm to the spatiotemporal pyramid structure to generate a multi-scale persistence map; fuse the multi-scale persistence map with the preset traditional visual features to obtain enhanced temporal context features; extract super prior data and potential prior data based on the current frame data, and splice the temporal context features, super prior data and potential prior data to form an input feature set; S2, based on the input feature set, an improved hierarchical conditional variational autoencoder is used to generate multi-level latent variables; Based on multi-level latent variables, the probability distribution parameters are estimated using algebraic geometry methods; based on the probability distribution parameters, the probability value of each latent variable under predetermined conditions is calculated to generate probability distribution data; S3, based on multi-level latent variables and probability distribution data, context-aware adaptive quantization based on compressed sensing theory is performed to obtain quantized data; S4, based on the quantized data, dynamic entropy coding optimization is performed to obtain a compressed data packet; Based on the final compressed data packet, performing rate-distortion optimization based on information geometry to obtain an optimized compressed data packet; S5. Based on the optimized compressed data packet, a perception-guided decoding reconstruction is performed to obtain a reconstructed video frame with optimal perceptual quality.
2. The video compression method based on the variational autoencoder improved entropy model according to claim 1, characterized in that: Step S1 is further as follows: S11, reading the original video data stream from the video input device or storage medium, extracting the video frame data at the current moment, and obtaining the current frame data; temporarily storing the current frame data in a buffer, and recording its timestamp information; preprocessing the current frame data in the buffer, including denoising, color space conversion, and resolution adjustment, to obtain the preprocessed current frame data; S12, constructing a spatiotemporal pyramid structure based on the preprocessed current frame data, wherein each level of the spatiotemporal pyramid structure contains data of different time scales and spatial resolutions; Apply the persistence homology algorithm to the data at each level to calculate the topological features; summarize the topological features to generate a multi-scale persistence graph; S13, based on the preprocessed current frame data, using the pre-trained convolutional neural network, extracting traditional visual features; using a weighted fusion method, combining the multi-scale persistence map and the traditional visual features to obtain enhanced temporal context features; S14. Based on the preprocessed current frame data, extract super prior data from the pre-trained neural network model; based on the encoding results of the pre-stored historical frames, extract potential prior data; standardize the super prior data and potential prior data to obtain standardized super prior data and potential prior data; splice the standardized super prior data and potential prior data with the enhanced temporal context features to form a complete input feature set.
3. The video compression method based on the variational autoencoder improved entropy model according to claim 2, characterized in that: Step S2 is further as follows: S21, inputting the input feature set into the encoder network to generate initial latent variables; obtaining conditional information of the current frame data, combining the conditional information with the initial latent variables, and constructing a multi-level latent variable generation network model; The condition information includes scene type and motion complexity; S22. Based on the conditional information and the initial latent variables, an adaptive prior network is used to dynamically adjust the prior distribution parameters of each layer in the multi-level latent variable generation network model to obtain an adjusted multi-level latent variable generation network model; using the adjusted multi-level latent variable generation network model, latent variables of different abstraction levels are generated layer by layer; and latent variables of all levels are combined to form a multi-level latent variable; S23, mapping multi-level latent variables to predefined algebraic clusters to construct a geometric structure representation of probability distribution; Obtain the encoding results of the most recent N frames from the pre-stored data cache as observation data, and estimate the parameters on the algebraic cluster using algebraic statistical methods based on the observation data and geometric structure representation; use semi-algebraic set theory to constrain the space of parameters on the algebraic cluster using pre-set constraints to obtain the constrained parameters; wherein N is a natural number greater than 0; S24. Based on the constrained parameters, a conditional probability model of the latent variables is constructed, the probability value of each latent variable under predetermined conditions is calculated, and probability distribution data is generated.
4. The video compression method based on the variational autoencoder improved entropy model according to claim 3, characterized in that: Step S3 is further as follows: S31, using a pre-trained sparse dictionary, projecting the multi-level latent variables into a sparse domain to obtain a sparse domain representation; based on the content features of the current frame data, selecting a quantization matrix from a preset quantization matrix library; Based on the probability distribution data, a context analysis module is used to dynamically adjust the parameters of the quantization matrix to obtain the adjusted quantization parameters; S32. Perform a quantization operation based on the sparse domain representation and the adjusted quantization parameter to obtain a quantized discrete value, that is, quantized data.
5. The video compression method based on the variational autoencoder improved entropy model according to claim 4, characterized in that: Step S4 is further as follows: S41, obtaining statistical characteristics of the most recent N coded frames from a pre-stored coding statistics cache, and dynamically updating a pre-configured symbol occurrence probability model based on the statistical characteristics and quantized data to obtain an updated symbol occurrence probability model; S42, assigning an optimal coding length to each symbol based on the updated symbol occurrence probability model; Based on the coding length and the quantized data, a context adaptive arithmetic coding algorithm is used to calculate the coded data; Based on the encoded data, a variable-length encoding strategy is constructed and optimized to obtain optimized encoded data; Organize the optimized coded data into a bit stream according to a predefined format, including necessary header information and metadata, to form the final compressed data packet; S43, based on the final compressed data packet, construct a statistical manifold and calculate the Fisher information matrix; Based on the Fisher information matrix, an iterative optimization algorithm based on natural gradient is constructed. Based on the current parameters of the iterative optimization algorithm, the rate-distortion performance under the current parameter configuration is evaluated until the preset number of iterations is reached; Based on the evaluation results, the optimal parameter configuration is output to obtain the optimized compressed data packet.
6. The video compression method based on the variational autoencoder improved entropy model according to claim 5, characterized in that: Step S5 is further as follows: S51, based on the header information and metadata in the optimized compressed data packet, parsing the encoding parameters, and reversely executing steps S4 to S2 to obtain preliminarily reconstructed video frame data; using a pre-trained perceptual quality assessment module, assessing the perceptual quality of the preliminarily reconstructed video frame data to obtain an assessment result; S52, based on the evaluation result, adaptively selecting and applying a post-processing technology from a preset post-processing technology library, including deblocking filtering and detail enhancement, to process the preliminarily reconstructed video frame data to obtain a processed frame; Input the processed frames into the pre-trained adversarial network to improve the visual quality and obtain enhanced frames; S53, comparing the enhanced frame with the current frame data in step S1, calculating the perceptual similarity; adjusting the post-processing parameters based on the perceptual similarity; Based on the adjusted post-processing parameters, the reconstructed video frames with optimal perceptual quality are output.
7. The video compression method based on the variational autoencoder improved entropy model according to claim 6, characterized in that: Step S3 also includes: S33. Based on the quantized discrete values, a convex optimization algorithm is used to reconstruct the original data; the perceptual quality difference between the reconstructed original data and the multi-level latent variables is calculated, and based on the perceptual quality difference, the adjusted quantization parameters are further fine-tuned to obtain the fine-tuned quantization parameters.
8. The video compression method based on the variational autoencoder improved entropy model according to claim 6, characterized in that: In step S31, based on the probability distribution data, the context analysis module is used to dynamically adjust the parameters of the quantization matrix, and the adjusted quantization parameters are further obtained as follows: S311. Based on multi-level latent variables, use convolutional neural networks to extract local features, including edges and textures; use long short-term memory networks to extract global features, including motion trajectories and scene changes; S312, based on local features, global features and probability distribution data, use the attention mechanism for weighted fusion to obtain a comprehensive feature vector; S313. Based on the comprehensive feature vector, a reinforcement learning algorithm is used to dynamically adjust the parameters of the quantization matrix to obtain adjusted quantization parameters.
9. A video compression system based on a variational autoencoder improved entropy model, characterized in that: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the video compression method based on the variational autoencoder improved entropy model as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Method and system for image processing
CA3152644A1
Method for compressed sensing video reconstruction based on recursive convolution neural network
CN106911930A
Image encoding and decoding, video encoding and decoding: methods, systems, and training methods
CN116584098A
Factorized variational autoencoders
US20190026631A1
Cited By
Video data compression and reconstruction method, system, equipment and medium
CN119254971A
Video monitoring data compression and transmission method and system
CN119583800A
A video surveillance data compression transmission method and system
CN119583800B
Video frame rate processing method and device, equipment and storage medium
CN119767052A
Video reconstruction method based on video variational auto-encoder
CN119788860A
