H.266 dynamic code rate control method based on machine learning

Abstract

The invention relates to the technical field of video coding, and discloses an H.266 dynamic code rate control method based on machine learning, which comprises the following steps of: 1, extracting spatio-temporal characteristics of 15 different types of videos from an input video sequence; 2, monitoring the network state in real time, and grading; step 3, constructing a code rate prediction model according to the spatio-temporal features and the network features; performing training prediction on the coded video, and outputting a predicted target code rate; step 4, inputting the current video spatial-temporal characteristics and the network state characteristics into the code rate prediction model to obtain a basic code rate; correcting the basic code rate according to the network state grading to obtain a final code rate; and step 5, dynamically adjusting the H.266 coding core parameters according to the final code rate based on the parameter mapping relation of the code rate. According to the invention, the adaptive capability, the code rate adjustment real-time performance, the image quality stability and the coding efficiency are improved, the resolution adaptation is better, and the H.266 coding efficiency advantage is fully played.

Description

Technical Field

[0001] This invention relates to the field of video encoding and processing technology, and specifically to a machine learning-based H.266 dynamic bitrate control method. Background Technology

[0002] H.266 (VVC), as a next-generation video coding standard, leverages its advanced compression tools to achieve approximately 50% bitrate savings compared to its predecessor, H.265 / HEVC, while maintaining the same visual quality. This provides a crucial technological foundation for the storage and transmission of high-definition and ultra-high-definition video. However, this increased compression efficiency comes at the cost of significantly increased coding complexity. Especially in real-world applications with dynamic network environments and diverse video content, achieving precise and adaptive bitrate control becomes a critical challenge. In actual video transmission, network bandwidth often fluctuates, and packet loss rates change dynamically. Furthermore, video content itself exhibits significant differences in time and space, ranging from static backgrounds to high-speed motion scenes, and from simple, smooth areas to complex textures. These factors collectively place higher demands on the rationality, timeliness, and stability of bitrate allocation.

[0003] Faced with the aforementioned complex conditions, traditional H.266 bitrate control methods typically allocate bitrate based on fixed formulas or empirical models, resulting in limited adaptability. These methods generally rely on preset parameters and strategies, making it difficult to fully adapt to the diverse and time-varying characteristics of video content and network conditions, leading to poor performance under real-world dynamic bandwidth conditions. For example, when network bandwidth suddenly drops, the lag in bitrate adjustment may cause buffer overflows, resulting in video playback stuttering; conversely, when available bandwidth recovers, the inability to fully utilize resources in a timely manner may prevent the improvement of visual quality. Furthermore, existing methods do not adequately and comprehensively characterize the complexity of video content, resulting in poor image quality stability for content units with different spatiotemporal characteristics at the same target bitrate, especially in areas with rapid motion or complex textures, where blurring, blockiness, and other distortions are prone to occur.

[0004] Therefore, although H.266 has significant advantages in compression performance, its actual benefits will be difficult to fully realize without a matching intelligent and adaptive bitrate control mechanism. Summary of the Invention

[0005] The present invention aims to provide a machine learning-based H.266 dynamic bitrate control method to solve the problems of poor adaptability, slow adjustment, and unstable image quality of existing bitrate control methods.

[0006] To achieve the above objectives, the present invention employs the following technical solution: a machine learning-based H.266 dynamic bitrate control method, comprising the following steps: Step 1: Perform frame segmentation on the input video sequence, extract features for each frame and the relationship between frames, obtain spatiotemporal features of 15 different types of video, and perform normalization processing. Step 2: Monitor the network status in real time and classify the network status as a constraint for bitrate adjustment. Step 3: Construct a bitrate prediction model based on spatiotemporal features and network features; train and predict the encoded video, and output the predicted target bitrate. Step 4: Input the current video spatiotemporal features and network state features into the bitrate prediction model to obtain the base bitrate; adjust the base bitrate according to the network state classification to obtain the final bitrate; Step 5: Based on the parameter mapping relationship of the bitrate, dynamically adjust the core parameters of H.266 encoding according to the final bitrate.

[0007] The principles and advantages of this scheme are: This solution abandons the traditional bitrate allocation framework that relies on fixed formulas, and instead constructs a data-driven intelligent control closed loop with dynamic learning capabilities.

[0008] First, this solution employs a multi-feature fusion bitrate prediction model. This model not only deeply analyzes the spatial and temporal complexity of video content but also incorporates real-time network status information and integrates rate-distortion performance data from historical coding units. Through joint training and inference using a machine learning model, the model can accurately depict the complex nonlinear relationships between "content-network-bitrate-quality" in the current coding scenario, thereby predicting a more reasonable initial target bitrate for each video block or frame to be encoded.

[0009] Secondly, building upon this foundation, this solution further designs an adaptive bitrate adjustment mechanism tightly coupled with the prediction model. This mechanism is not a passive response, but rather based on forward-looking predictions, combined with the real-time buffer fullness status, to perform microsecond-level feedforward and feedback-based coordinated adjustments to the encoding parameters. When a decrease in network bandwidth is detected, the mechanism can reduce the bitrate allocation in advance and smoothly, effectively preventing buffer overflows and stuttering; when network resources become abundant, it can quickly guide the encoder to increase the bitrate to improve image quality, fully utilizing bandwidth advantages. The entire process achieves a fundamental shift from "delayed response" to "proactive prediction and real-time adjustment."

[0010] Compared to existing technologies, this solution addresses the poor adaptability of traditional methods due to their reliance on empirical formulas. It achieves precise adaptation to ever-changing video content and dynamic network environments through machine learning models. In terms of response speed, the model's real-time feature processing and prediction capabilities, combined with an adjustment mechanism, overcome the lag in bitrate adjustments, ensuring smoothness while avoiding waste of network resources. More importantly, due to its more refined and comprehensive description of video content complexity, this solution can achieve more optimized bit resource allocation within a given bitrate constraint. This improves image quality stability in different scenarios, effectively suppresses blurring and blockiness in high-speed motion or complex texture areas, and ultimately provides a superior and more stable subjective visual experience in dynamic network environments. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a machine learning-based H.266 dynamic bitrate control method according to the present invention. Figure 2 This is a system architecture diagram of an H.266 dynamic bitrate control method based on machine learning according to the present invention; Figure 3 This is a diagram of the bitrate prediction model structure in the H.266 dynamic bitrate control method based on machine learning of the present invention. Figure 4 This is a detailed diagram of the bitrate prediction model in the H.266 dynamic bitrate control method based on machine learning of the present invention; Figure 5 This is a flowchart of the dynamic bitrate adjustment process in the H.266 dynamic bitrate control method based on machine learning of the present invention. Detailed Implementation

[0012] The following detailed description illustrates the specific implementation method: This embodiment presents a machine learning-based H.266 dynamic bitrate control method that integrates video content features, network status information, and encoding history data. Through an intelligent model, it achieves precise dynamic bitrate control to fully leverage the encoding efficiency advantages of H.266.

[0013] A machine learning-based H.266 dynamic bitrate control method is provided, as shown in the attached figure. Figure 1 and attached Figure 2 As shown, it includes the following steps: Step 1: Perform frame segmentation on the input video sequence, extract features for each frame and the relationship between frames, obtain spatiotemporal features of 15 different types of video, and normalize the features.

[0014] In this embodiment, the spatiotemporal features include spatial features and temporal features. Among them, the spatial features include intra-frame luminance / chrominance variance, mean gradient magnitude (reflecting texture complexity), edge density (detected by the Canny operator), and the proportion of moving regions (determined by the frame difference method).

[0015] Temporal features include the mean amplitude of inter-frame motion vectors (reflecting the intensity of motion), macroblock partitioning depth distribution (the proportion of CU partitioning levels in H.266), and the number and weight of reference frames.

[0016] Then, Z-score normalization is used to map all features to a distribution with zero mean and unit variance, represented as: ; In the formula, The characteristic mean; The standard deviation is the feature value; all values ​​are obtained based on statistics from the offline training set.

[0017] Step 2: Monitor the network status in real time and classify the network status as a constraint for bitrate adjustment.

[0018] In this embodiment, network parameters are collected in real time through the RTP / RTCP protocol. The main parameters include the current available bandwidth, bandwidth fluctuation value (standard deviation), packet loss rate, round-trip time, and buffer occupancy rate (current cached data volume / maximum cache capacity).

[0019] Then, the network status is classified into five levels based on monitoring indicators: smooth, relatively smooth, normal, relatively congested, and congested. These levels serve as constraints for bitrate adjustment.

[0020] Step 3: Construct a bitrate prediction model based on spatiotemporal and network features; train and predict the encoded video, and output the predicted target bitrate.

[0021] In this embodiment, the bitrate prediction model is trained primarily based on video features and network features to predict the encoded video and output the predicted target bitrate, thereby improving the encoding parameters. In this embodiment, the bitrate prediction model employs a multi-scale attention fusion CNN-LSTM hybrid model, as shown in the attached figure. Figure 3 As shown, it includes an input layer, a multi-branch feature extraction module, a cross-attention fusion layer, and a prediction output layer, and includes a model regularization mechanism.

[0022] Specifically, in conjunction with the appendix Figure 4 As shown, the input layer is used to receive a 13-dimensional feature vector (10-dimensional video features + 3-dimensional network state features). The embedding layer converts the discrete features (such as network state levels) into an 8-dimensional continuous vector, which is then concatenated with the original continuous features to form a 20-dimensional input vector.

[0023] The multi-branch feature extraction module includes a texture branch, a motion branch, and a temporal branch. The texture branch comprises three 1D convolutional layers (kernel sizes 3 / 5 / 3, channel counts 32 / 64 / 128) and residual connections, specifically designed to extract video texture complexity features. Each convolutional layer is followed by normalization and LeakyReLU activation. .

[0024] The motion branch consists of two layers of dilated convolutions, with a dilation rate of 2 / 4, a kernel size of 5 / 5, and 64 / 128 channels. This is used to capture long-range dependencies in video motion and works in conjunction with the Squeeze-Excitation attention module (SE module) to enhance key motion features.

[0025] The temporal branch consists of two bidirectional LSTM layers (64 / 32 hidden units). The first layer is used to return the sequence output. Each LSTM layer is followed by Dropout (rate=0.3) and layer normalization. The second layer is used to output a 32-dimensional temporal feature vector.

[0026] The cross-attention fusion layer is used to exchange feature information between the texture branch and the motion branch through a mutual attention mechanism to calculate the attention weight of texture features to motion features, and vice versa. The fused spatial features (256 dimensions) and temporal branch features (32 dimensions) are then fused across modally through a self-attention layer (8-head attention, 36 key-value dimensions) to output a 128-dimensional fused feature.

[0027] The prediction output layer consists of three fully connected network layers (128 / 64 / 1 nodes), each using the GELU activation function. It also outputs two sub-tasks in parallel: the main task is target bitrate prediction (1-dimensional output, linear activation), and the auxiliary task is network state prediction (5-class classification, Softmax activation). Knowledge distillation is used to improve the generalization ability of the main task.

[0028] In this embodiment, the model regularization mechanism is to add a random depth mechanism between fully connected layers, randomly discarding 5% of the network layers during training to suppress overfitting.

[0029] In addition, the training prediction process also includes building a training dataset, which includes video samples, network environment, and label data.

[0030] The video samples cover 15 different scenarios, with resolutions ranging from 144p to 8K and frame rates from 15 to 120fps, including 30% dynamic bitrate encoded samples. The network environment was simulated using a 5G network simulator, generating eight bandwidth fluctuation modes (such as sudden drops and stepped increases), expanding the packet loss rate range to 0% to 10%, and adding 10% more network jitter samples. The label data uses reinforcement learning to annotate the optimal bitrate, with a reward function of "encoding efficiency - image quality stability - transmission smoothness" (weight ratio of 4:3:3), and the labels are iteratively optimized using the PPO algorithm.

[0031] The loss function is constructed using multi-task weighting, and is expressed as follows: ; In the formula, For bitrate prediction loss; Loss is classified as network state classification. The loss is for regularization; and the Lookahead optimizer is used to optimize the model.

[0032] Among them, bitrate prediction loss An improved Huber loss is used, with the threshold for the Huber loss set as follows: Used to distinguish squared loss (deviation ≤ (Time) and linear loss (deviation > The boundary of (time) is used to reduce the penalty weight for extreme bitrate deviations, which is then expressed as: ; In the formula, The output is the predicted target bitrate; The optimal bitrate is determined through reinforcement learning annotation and used to calculate the bitrate prediction loss.

[0033] Network state classification loss The cross-entropy loss is labeled and smoothed (smoothing coefficient 0.1).

[0034] Regularization loss For L2 regularization (weight decay 1e) -5 + Attention sparsity penalty (mean of attention weight entropy).

[0035] Finally, the Lookahead optimizer (primary optimizer AdamW, learning rate 5e) was used. -4 Weight decay 1e -4 The model is optimized according to the set training strategy. The step size is 5 and the learning rate is 0.5. Mixed precision training (FP16+FP32) is implemented to accelerate convergence. Gradient clipping is used to prevent gradient explosion.

[0036] In this embodiment, the training strategy is to freeze the temporal branches in the first 10 rounds to pre-train the spatial feature extractor, jointly train all layers in the middle 20 rounds, and unfreeze all layers in the last 10 rounds using cosine annealing with a minimum learning rate of 5e. -6 The final trained bitrate prediction model is obtained.

[0037] Step 4: Input the current video spatiotemporal features and network state features into the bitrate prediction model to obtain the base bitrate; adjust the base bitrate according to the network state classification to obtain the final bitrate.

[0038] As attached Figure 5 As shown, in this embodiment, when the network status is congested (level 4-5) or the buffer occupancy rate is >80%, the bitrate is reduced in the following way: ; When the network is smooth (level 1-2) and the buffer occupancy rate is less than 30%, the following methods can be used to increase the bitrate: ; Level 3 is the normal state, in which the bitrate remains unchanged.

[0039] Meanwhile, to avoid image quality fluctuations caused by sudden changes in bitrate, exponential smoothing is used to filter the adjusted bitrate, as shown below: ; In the formula, Base bitrate; This is the corrected bitrate; This is the final bitrate; This represents the network status level (values ​​1-5, corresponding to smooth to congested), used for adjusting bitrate constraints. This is a smoothing factor, with a value ranging from 0.3 to 0.6. A larger value is used for scenes with intense motion to quickly respond to changes in bitrate and avoid image quality fluctuations.

[0040] Step 5: Based on the parameter mapping relationship of the bitrate, dynamically adjust the core parameters of H.266 encoding according to the final bitrate.

[0041] In this embodiment, based on the final bitrate Dynamic adjustments mainly include the following sub-steps: Step 5.1, Encoding resolution adjustment: Establish a bitrate-resolution mapping relationship and dynamically adapt it in combination with video content features.

[0042] In this embodiment, multiple resolution levels are set (such as 8K, 4K, 2K, 1080p, 720p, 480p), and the corresponding bitrate range for each level is determined through offline testing (e.g., 8K corresponds to 40~100Mbps, 4K corresponds to 15~40Mbps, etc.). The final bitrate... When the video falls into a certain range, the corresponding resolution level is initially determined, and then fine-tuned according to the characteristics of the video content: for videos with complex textures and a high proportion of static areas, a high resolution level is preferred within the bitrate range to preserve details; for videos with intense motion and a high proportion of dynamic areas (such as sports events), the resolution level can be appropriately reduced to ensure smoothness.

[0043] Step 5.2, Quantization Parameter (QP): Establish a bitrate-QP mapping table, with different bitrate ranges corresponding to initial QP values, and then fine-tune them according to the features of the current frame.

[0044] In this embodiment, QP is reduced by 2-3 for frames with complex textures and increased by 1-2 for frames with intense motion.

[0045] Step 5.3, Change Block Size (TU): When the bit rate is high, a larger TU (32×32) is preferred to improve compression efficiency; when the bit rate is low, a smaller TU (4×4) is used to preserve details.

[0046] Step 5.4, Loop Filter Strength: Adjust the Sample Adaptive Offset (SAO) and Adaptive Loop Filter Parameters (ALF) according to the bit rate and intra-frame variance, and enhance the filter strength at high bit rates to improve subjective quality.

[0047] Step 5.5, Feedback Optimization: After encoding, calculate the deviation between the actual output bitrate and the target bitrate. If the deviation is >10%, adjust the model input weights in the next frame to achieve closed-loop optimization. Simultaneously, if the Subjective Quality of Performance (MOS) drops by more than 0.5 points after resolution adjustment, or the PSNR decreases by more than 2dB at the same bitrate, prioritize reverting to the previous resolution setting in the next frame and recalibrate the bitrate-resolution mapping relationship.

[0048] In this embodiment, by constructing a multi-feature fusion bitrate prediction model and an adaptive bitrate adjustment mechanism, accurate adaptation to different video content and network environments is achieved. While ensuring smooth video transmission, the subjective quality of the video is maximized, solving the problems of poor adaptability, adjustment lag, and unstable image quality of traditional bitrate control methods.

[0049] This solution utilizes a machine learning model that integrates video content features and network conditions to achieve precise adaptation to diverse scenarios, addressing the poor adaptability of traditional methods to complex content and dynamic networks. Its CNN-LSTM-based prediction mechanism can anticipate bitrate requirements 5-10 frames in advance, and combined with hierarchical adjustments based on network conditions, effectively avoiding the lag inherent in traditional methods. Compared to traditional bitrate control, it saves bitrate while maintaining the same subjective quality, fully leveraging the compression potential of H.266 and reducing storage and transmission costs. Furthermore, its encoding resolution adjustment mechanism dynamically matches the most suitable resolution based on bitrate and video content features, ensuring smooth video transmission while further enhancing image quality and viewing experience, avoiding the waste or inadequacy of image quality that occurs with fixed resolutions when bitrate fluctuates.

[0050] The above descriptions are merely embodiments of the present invention, and common knowledge such as specific technical solutions and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solutions of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A machine learning-based H.266 dynamic bitrate control method, characterized in that, Includes the following steps: Step 1: Perform frame segmentation on the input video sequence, extract features for each frame and the relationship between frames, obtain spatiotemporal features of 15 different types of video, and perform normalization processing. Step 2: Monitor the network status in real time and classify the network status as a constraint for bitrate adjustment. Step 3: Construct a bitrate prediction model based on spatiotemporal features and network features; train and predict the encoded video, and output the predicted target bitrate. Step 4: Input the current video spatiotemporal features and network state features into the bitrate prediction model to obtain the base bitrate; adjust the base bitrate according to the network state classification to obtain the final bitrate; Step 5: Based on the parameter mapping relationship of the bitrate, dynamically adjust the core parameters of H.266 encoding according to the final bitrate.

2. The H.266 dynamic bitrate control method based on machine learning according to claim 1, characterized in that: In step 1, the spatiotemporal features include spatial features and temporal features; the spatial features include intra-frame luminance / chrominance variance, mean gradient magnitude, edge density, and percentage of motion regions; the temporal features include mean inter-frame motion vector magnitude, macroblock partitioning depth distribution, and the number and weight of reference frames.

3. The H.266 dynamic bitrate control method based on machine learning according to claim 1, characterized in that: In step 2, the network status includes 5 levels: smooth, relatively smooth, normal, relatively congested, and congested.

4. The H.266 dynamic bitrate control method based on machine learning according to claim 1, characterized in that: In step 3, the bitrate prediction model adopts a multi-scale attention fusion CNN-LSTM hybrid model, which includes an input layer, a multi-branch feature extraction module, a cross-attention fusion layer and a prediction output layer, and includes a model regularization mechanism.

5. The H.266 dynamic bitrate control method based on machine learning according to claim 4, characterized in that: The input layer is used to receive 13-dimensional feature vectors. The discrete features are converted into 8-dimensional continuous vectors through the embedding layer and concatenated with the original continuous features to form a 20-dimensional input vector.

6. The H.266 dynamic bitrate control method based on machine learning according to claim 4, characterized in that: The multi-branch feature extraction module includes a texture branch, a motion branch, and a temporal branch. The texture branch includes three 1D convolutional layers and residual connections to extract video texture complexity features. Each convolutional layer is followed by normalization and LeakyReLU activation. The motion branch includes two dilated convolutional layers to capture long-range dependencies in video motion. The temporal branch includes two bidirectional LSTM layers. The first layer returns the sequence output, and each LSTM layer is followed by Dropout and layer normalization. The second layer outputs a 32-dimensional temporal feature vector.

7. A machine learning-based H.266 dynamic bitrate control method according to claim 6, characterized in that: The cross-attention fusion layer is used to exchange feature information between the texture branch and the motion branch through a mutual attention mechanism; and to perform cross-modal fusion of the fused spatial features and temporal branch features through a self-attention layer to output 128-dimensional fused features.

8. The H.266 dynamic bitrate control method based on machine learning according to claim 4, characterized in that: In step 3, during prediction training, a training dataset is constructed, which includes video samples, network environment data, and label data; and a multi-task weighted loss function is constructed, expressed as follows: ； In the formula, For bitrate prediction loss; Loss is classified as network state classification. The loss is for regularization; and the Lookahead optimizer is used to optimize the model.

9. The H.266 dynamic bitrate control method based on machine learning according to claim 3, characterized in that, In step 4, when the network is congested or the buffer occupancy rate is >80%, the bitrate is reduced in the following ways: ； When the network is smooth and the buffer occupancy rate is less than 30%, the following methods can be used to increase the bitrate: ； The adjusted bitrate is then filtered using exponential smoothing, as shown below: ； In the formula, Base bitrate; This is the corrected bitrate; This is the final bitrate; Network status level; This is the smoothing coefficient.

10. The H.266 dynamic bitrate control method based on machine learning according to claim 1, characterized in that, Step 5 also includes the following sub-steps: Step 5.1: Establish the bitrate-resolution mapping relationship and perform dynamic adaptation based on video content features; Step 5.2: Establish a bitrate-QP mapping table, where different bitrate ranges correspond to initial QP values, which are then fine-tuned based on the characteristics of the current frame. Step 5.3: When the bit rate is high, prioritize using a large-size TU to improve compression efficiency; When using a small TU to preserve details at low bitrates; Step 5.4: Adjust the sample adaptive offset and adaptive loop filtering parameters according to the bit rate and intra-frame variance, and enhance the filtering strength at high bit rates; Step 5.5: After encoding is completed, calculate the deviation between the actual output bitrate and the target bitrate. If the deviation is greater than 10%, adjust the model input weights in the next frame to achieve closed-loop optimization.

Images