Rate-distortion prediction based method and system for rate control of depth video encoder

Abstract

The application provides a rate distortion prediction-based deep video encoder code rate control method and system, which comprises the following steps: step 1, training a prediction module; step 2, inputting a video frame into the prediction module to obtain a prediction point set; step 3, fitting a code rate and quality model according to the prediction point set; step 4, obtaining a frame-level code rate allocation ratio through a code rate control algorithm; and step 5, determining the corresponding encoding parameters of each frame and inputting the encoding parameters into an encoder for encoding. The application directly utilizes a neural network and an original video to predict the code rate model and the quality model of each frame for the first time, without pre-encoding; the video frame is down-sampled to a fixed small resolution before being inputted into the neural network, so that the efficiency is improved and the generalization is enhanced; and the application realizes code rate control at a mini-GOP level for the first time. Compared with the existing code rate control methods, the application can realize the same code rate control accuracy and finer code rate control granularity at a faster speed.

Description

Technical Field

[0001] This invention relates to the field of multimedia communication technology, and more specifically to a method and system for rate-distortion prediction-based bitrate control of a depth video encoder. Background Technology

[0002] Deep learning-based video encoders have developed rapidly in recent years and have enormous potential. With the rise of video platforms, video traffic now accounts for over 80% of internet traffic and continues to grow. Therefore, designing efficient encoding technologies for video storage and transmission is crucial. Traditional video encoders, including AVC, HEVC, and VVC, encode video based on manually designed rules. In recent years, with the development of artificial intelligence-related technologies, deep learning-based video coding has received increasing attention. Current methods can be divided into two categories: motion & residual hybrid coding and probabilistic predictive coding, and their performance has surpassed that of traditional video encoders in some aspects.

[0003] Existing deep video encoders are typically trained by minimizing the RD loss function R+λ·D, where λ is the encoding parameter corresponding to a fixed bitrate. Therefore, to achieve different bitrates for the same video sequence, it is necessary to retrain the model by changing λ. To address this issue, multiple bitrate models can be implemented under a single model by downsampling the original input video frames and upsampling the output (Chen J, Wang M, Zhang P, et al. Sparse-to-Dense: High Efficiency Rate Control for End-to-end Scale-Adaptive Video Coding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023.). Alternatively, a content-adaptive quantization mechanism can be used to implement a variable bitrate model (Li J, Li B, Lu Y. Hybrid spatial-temporal entropy modelling for neural video compression[C] / / Proceedings of the 30th ACM International Conference on Multimedia. 2022:1503-1511.). Another approach is to map the coding parameter λ to the encoder via a neural network to achieve a variable bitrate model (Lin J, Liu D, Liang J, et al. A deeply modulated scheme for variable-rate video compression[C] / / 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021:3722-3726.).

[0004] Traditional video encoder rate control methods can be divided into three main categories: RQ model, R-ρ model, and R-λ model (WeiX, Zhou M, Wang H, et al. Recent Advances in Rate Control: From Optimisation to Implementation and Beyond[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023.). These methods directly or indirectly establish a mapping relationship between bitrate and encoding parameters to achieve bitrate control. Among them, the R-λ model (JCTVC-K0103, "Rate control by R-lambda model for HEVC," Shanghai, CN, 10–19 Oct. 2012.) has the best bitrate control accuracy and is still used in traditional video coding standards.

[0005] Most existing work on depth video encoders focuses on designing and improving compression architectures to achieve better rate-distortion performance, but there is relatively little research on corresponding bitrate control techniques. To address this issue, one approach is to borrow the bitrate control methods from traditional video encoders and redesign the bitrate allocation and hyperparameter update mechanisms. This can achieve high bitrate accuracy without pre-coding (Yanghao Li, Xinyao Chen, Jisheng Li, Jiangtao Wen, Yuxing Han, Shan Liu, and Xiaozhong Xu, “Rate control for learned video compression,” in ICASSP2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 2829–2833.). Alternatively, bitrate accuracy can be improved through multiple pre-coding operations and fitting a bitrate model, but this comes at the cost of significantly increased time consumption (Jiancong Chen, Meng Wang, Pingping Zhang, Shurun ​​Wang, and Shiqi Wang, “Sparse-to-dense: High efficiency rate control for end-to-end scale-adaptive video coding,” IEEE Transactions on Circuits and Systems for Video). Technology, vol.34, no.5, pp.4027-4039, 2024.). Both of these methods can only achieve bitrate control at the Group of Pictures (GOP) level, at the cost of relatively low accuracy or high time consumption. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to overcome the shortcomings of existing bitrate control schemes for depth video encoders, including low accuracy, slow speed, only able to achieve group-level (GOP) bitrate control, and large inter-frame quality fluctuations. The invention provides a bitrate control algorithm that balances accuracy and speed, can achieve high bitrate accuracy at the mini-GOP level, and effectively control inter-frame quality fluctuations.

[0007] This invention provides a method and system for bitrate control of a depth video encoder based on rate-distortion prediction, wherein the method includes the following steps:

[0008] Step 1: Train the prediction module, which includes a downsampling module, a bitrate predictor, and a quality predictor;

[0009] Step 2: Downsample each two adjacent frames of the current mini-GOP to be encoded to a low resolution (e.g., 240P), and after concatenation in the channel dimension, feed them into a pre-designed rate predictor and quality predictor to obtain a set of (R,λ) points and a set of (D,λ) points, where R represents the rate, D represents the quality, and λ represents the coding parameters.

[0010] Step 3: Using the least squares method, fit the (R,λ) point set and (D,λ) point set obtained in Step 2 to the bitrate model and quality model, and obtain the bitrate distortion curve;

[0011] Step 4: Input the target bitrate, (R,λ) point set, (D,λ) point set, and the fitted bitrate model, quality model, and bitrate distortion curve into the bitrate control algorithm, and output the frame-level bitrate allocation ratio of the mini-GOP to be encoded.

[0012] Step 5: Determine the encoding parameters of each frame within the mini-GOP based on the frame-level bitrate allocation ratio obtained in Step 4 and the bitrate model obtained in Step 3, and input them into the video encoder to start encoding.

[0013] In step 1, the bitrate predictor and quality predictor in the prediction module are trained separately, using the exact same neural network architecture with identical inputs and different outputs; the neural network architecture includes an embedding layer, a pooling layer, an interpolation upsampling layer, a convolutional layer, an adaptive pooling layer, and a multilayer perceptron.

[0014] The specific structure of the embedding layer is as follows: the feature map passes through two convolutional layers; the downsampling ratio of the video frame is processed by sinusoidal embedding, then passed through a multilayer perceptron, and finally multiplied with the output of the two convolutional layers;

[0015] The neural network architecture employs a symmetrical structure. First, the input feature map passes through an embedding layer and a pooling layer twice, i.e., a downsampling operation. Then, it passes through a residual network to extract deeper features. Next, it passes through an interpolation upsampling layer and an embedding layer twice, i.e., an upsampling operation. In the downsampling operation, the output of each embedding layer is concatenated with the output of the corresponding embedding layer in the upsampling operation along the channel dimension. Finally, it passes through two convolutional layers, one adaptive pooling layer, and one multilayer perceptron. Ultimately, the bitrate predictor outputs the predicted bitrate R for the corresponding encoding parameter set Λ, and the quality predictor outputs the predicted quality D for the corresponding encoding parameter set Λ.

[0016] In step 1, the set of encoding parameters Λ is:

[0017] Λ={λ1,λ2,…,λM} (1)

[0018] Where Λ is the set of M coding parameters obtained by logarithmic interpolation between the maximum and minimum values ​​of the coding parameters, λ1 is the minimum value of the coding parameters that can be input to the depth video encoder, and λ M This represents the maximum value of the encoding parameters that can be input to the depth video encoder.

[0019] In step 1, the training prediction module includes: after downloading the training dataset, loading the training dataset and starting training. During training, each iteration selects consecutive video frames as a group (e.g., selecting 5 consecutive video frames as a group each iteration), randomly downsamples the selected consecutive video frames to a resolution (e.g., downsampled to any resolution among 1080P, 720P, 480P, and 240P), and then starts frame-by-frame encoding. During each frame encoding, the current frame to be encoded and the reference frame are input into the depth video encoder, and encoded using each encoding parameter in the set Λ. The resulting bitrate or quality is concatenated into a vector.

[0020]

[0021] Where R real and D real These represent the ground truth labels when training the bitrate predictor and the ground truth labels when training the quality predictor, respectively. This represents the i-th encoding parameter λ that can be input to the depth video encoder for the current frame to be encoded and the reference frame. i The bit rate (bits per pixel) obtained from encoding. This represents the i-th encoding parameter λ that can be input to the depth video encoder for the current frame to be encoded and the reference frame. i The distortion obtained from encoding (mean squared error); i takes values ​​from 1 to M;

[0022] After encoding the current frame to be encoded using each encoding parameter in set Λ, the reconstructed frame obtained by selecting any encoding parameter is used as the reference frame for the next frame to be encoded.

[0023] After obtaining the ground truth label, the downsampled video frames are downsampled again to a predetermined low resolution (i.e., 240P resolution), and then concatenated along the channel dimension. The result is then fed into the bitrate predictor to obtain the corresponding bitrate prediction vector. The corresponding quality prediction vector is obtained by feeding it into the quality predictor.

[0024] In step 1, during the training of the bitrate predictor, the loss function L of the bitrate predictor is... R for:

[0025]

[0026] in This is the i-th bitrate prediction value output by the bitrate predictor;

[0027] When training the quality predictor, the loss function L of the quality predictor is... D for:

[0028]

[0029] in This is the i-th quality prediction value output by the quality predictor.

[0030] Each frame in a set of video frames can have its loss value obtained by formula (4) or formula (5). The loss values ​​of each set of video frames are accumulated, and then gradient backpropagation and network parameter updates are performed to complete one iteration.

[0031] In step 2, each pair of adjacent frames in the current mini-GOP to be encoded is downsampled (e.g., downsampled to 240P resolution), and then input into the bitrate predictor trained in step 1 to obtain a set of (R, λ) points for the corresponding set Λ. At the same time, it is input into the quality predictor trained in step 1 to obtain a set of (D, λ) points for the corresponding set Λ; where R is the bitrate, λ is the coding parameter, and D is the quality.

[0032] After obtaining the set of (D, λ) points for each frame, the interference value D for the next frame prediction is obtained according to the following formula. add :

[0033]

[0034] Where T is a tensor of the same size as the resolution of the encoded frame of the input encoder and conforms to a standard normal distribution, and e is a natural constant;

[0035] When predicting bitrate and quality for the next frame, the uncoded reference frame is first given an interference value D. add Then, downsampling, splicing, and prediction are performed.

[0036] In step 3, based on the (R, λ) point set obtained in step 2, a bitrate model is fitted to each frame to be coded within the mini-GOP using the least squares method. The expression is:

[0037]

[0038] Where α1 and β1 are hyperparameters of the code rate model;

[0039] Based on the (D, λ) point set obtained in step 2, a quality model is fitted to each frame to be encoded within the mini-GOP using the least squares method. The expression is:

[0040]

[0041] Where α2 and β2 are the hyperparameters of the mass model;

[0042] According to formulas (7) and (8), the rate-distortion curve of each frame to be encoded within the mini-GOP is obtained, and the expression is:

[0043]

[0044] In step 4, the rate control algorithm includes the following steps:

[0045] Step 4-1: Determine if the target bit rate exceeds the encoder's encoding range. If it exceeds the maximum range value, then use the maximum encoding parameter λ. M If the value is less than the minimum value, then use the minimum encoding parameter λ1 for encoding.

[0046] Step 4-2: Determine the upper and lower bounds of the target quality of the mini-GOP based on the (D, λ) point set obtained in Step 2 (the target quality is the same for each frame within the mini-GOP): Assume the quality prediction vector D for each frame... pred The minimum values ​​of the M elements are grouped into a set. The maximum value of the data in the set is taken as the lower bound of the target quality. The quality prediction vector D of each frame is then used. pred The maximum values ​​of the M elements are formed into a set, and the minimum value of the data in the set is taken as the upper bound of the target quality.

[0047] Step 4-3: Use the binary search method to iterate continuously and change the value of the target quality. Each iteration is based on the target quality and uses formula (9) to obtain the bit rate of each frame in the corresponding mini-GOP. Calculate the total bit rate of the mini-GOP until the number of iterations of the binary search method is greater than the maximum value K or the target quality that meets the bit rate accuracy requirements is obtained.

[0048] Step 4-4: Use the proportion of bitrate of each frame within the mini-GOP obtained in the last iteration as the bitrate allocation ratio (bits). ratio The expression is:

[0049] bit ratio ={r1, r2, ..., r N} (10)

[0050] Where N is the number of frames within a mini-GOP, r1 represents the bitrate allocated to the first frame within the mini-GOP, and r N This represents the bitrate allocated to the Nth frame within a mini-GOP.

[0051] In step 5, the original target bitrate of each frame within the mini-GOP is obtained based on the total bitrate of the mini-GOP and formula (10):

[0052]

[0053] in, R represents the raw target bitrate of the j-th frame within a mini-GOP. mini-GOP This represents the total bitrate of a mini-GOP, where j ranges from 1 to N;

[0054] The actual target bitrate for each frame is:

[0055]

[0056] Among them, among them, This represents the actual target bitrate of the j-th frame within the mini-GOP. Buffer represents the buffer, which is initialized to 0 before each mini-GOP encoding.

[0057] Substitute the actual target bitrate of each frame obtained from formula (12) into formula (7) to obtain the encoding parameters of each frame within the mini-GOP. Then, send the encoding parameters of each frame and the corresponding frame to be encoded into the encoder to start encoding. After each frame is encoded, the value of the buffer is set as follows:

[0058]

[0059] in, The actual encoded bitrate of the j-th frame;

[0060] When encoding the next frame, the actual target bitrate of the next frame. The original target bitrate obtained by formula (11) is obtained by adding the buffer.

[0061] The present invention also provides a rate-distortion prediction-based deep video encoder bitrate control system, including a training module, a prediction module, a fitting module, a bitrate control algorithm calculation module, and an encoding module;

[0062] The training module is used to train the prediction module. The prediction module includes a downsampling module, a bitrate predictor, and a quality predictor. The downsampling module is used to downsample each two adjacent frames of the current mini-GOP to be encoded to a low resolution. Each two adjacent frames after downsampling are concatenated in the channel dimension and then fed into the pre-designed bitrate predictor and quality predictor to obtain a set of (R, λ) points and a set of (D, λ) points.

[0063] The fitting module uses the least squares method to fit the bitrate model and the quality model using the (R, λ) point set and (D, λ) point set obtained in step 2, and obtains the bitrate distortion curve.

[0064] The bitrate control algorithm calculation module inputs the target bitrate, the (R,λ) point set, the (D,λ) point set, the fitted bitrate model, the quality model, and the bitrate distortion curve into the bitrate control algorithm, and outputs the frame-level bitrate allocation ratio of the mini-GOP to be encoded.

[0065] The encoding module determines the encoding parameters of each frame within the mini-GOP based on the obtained frame-level bitrate allocation ratio and the bitrate model obtained in step 3, and inputs them into the video encoder to start encoding.

[0066] Beneficial effects:

[0067] 1) This invention is the first to directly predict the bitrate and quality model for each frame using a neural network and the original video, without pre-encoding; 2) It downsamples video frames to a fixed small resolution before inputting them into the neural network, improving efficiency and enhancing generalization; 3) It proposes an interference mechanism to improve prediction accuracy; 4) It is the first to achieve bitrate control at the mini-GOP level; 5) It is the first to achieve smooth inter-frame quality fluctuations in bitrate control. Compared to existing bitrate control methods, this invention can achieve comparable bitrate control accuracy and finer bitrate control granularity at a faster speed. Attached Figure Description

[0068] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0069] Figure 1 This is the system architecture diagram of the present invention.

[0070] Figure 2 This is a structural diagram of the prediction module of the present invention.

[0071] Figure 3 This is a comparison of the accuracy and efficiency of the present invention with the most advanced existing solutions in terms of bit rate control.

[0072] Figure 4This is a comparison of the inter-frame quality fluctuation between the present invention and the most advanced existing solutions.

[0073] Figure 5a and Figure 5b This is a graph showing the bitrate control accuracy and quality fluctuation results obtained from test videos using the present invention. Detailed Implementation

[0074] like Figure 1 As shown, this embodiment of the invention provides a bitrate control method for a depth video encoder based on rate-distortion prediction, comprising the following steps:

[0075] Step 1, Training as follows Figure 2 The prediction module shown includes a downsampling module, a bitrate predictor, and a quality predictor.

[0076] Step 2: Downsample each two adjacent frames of the current mini-GOP to be encoded to a low resolution (e.g., 240P), and after concatenation in the channel dimension, feed them into the pre-trained rate predictor and quality predictor to obtain a set of (R,λ) points and a set of (D,λ) points.

[0077] Step 3: Using the least squares method, fit the bitrate model and quality model using the (R, λ) point set and (D, λ) point set obtained in Step 2, and obtain the bitrate distortion curve;

[0078] Step 4: Input the target bitrate, the (R, λ) point set, the (D, λ) point set, and the fitted bitrate model, quality model, and bitrate distortion curve into... Figure 1 In the rate control algorithm, the frame-level rate allocation ratio of the mini-GOP (mini-group of frames) to be encoded is output.

[0079] Step 5: Determine the encoding parameters of each frame within the mini-GOP based on the frame-level bitrate allocation ratio obtained in Step 4 and the bitrate model obtained in Step 3, and input them into the video encoder to start encoding.

[0080] In embodiments of the present invention, a variable bitrate version of DVC is used as a depth video encoder.

[0081] According to an embodiment of the present invention, in step 1, the bitrate predictor and quality predictor in the prediction module are trained separately, using the exact same neural network architecture, with identical inputs and different outputs. First, the set of encoding parameters is determined as follows:

[0082] Λ={λ1,λ2,...,λ M} (1)

[0083] Where Λ is the set of M coding parameters obtained by logarithmic interpolation between the maximum and minimum values ​​of the coding parameters, λ1 is the minimum value of the coding parameters that can be input to the depth video encoder, and λ M M represents the maximum value of the encoding parameters that can be input to the depth video encoder; in this example, M is set to 8.

[0084] In this embodiment of the invention, the training dataset uses high-resolution Vimeo (https: / / vimeo.com). After downloading the training dataset, load the dataset and begin training. During training, each iteration selects 5 consecutive video frames as a group, randomly downsamples these selected consecutive video frames to any resolution among 1080P, 720P, 480P, and 240P, and then begins frame-by-frame encoding. During each frame encoding, the current frame to be encoded and the reference frame are input into the depth video encoder, and encoded using each encoding parameter in Λ. The resulting bitrate or quality is concatenated into a vector:

[0085]

[0086] Equations (2) and (3) represent the ground truth labels when training the bitrate predictor and the quality predictor, respectively. This indicates that the current frame to be encoded and the reference frame are determined by λ. i The bitrate (bits per pixel) obtained from encoding. This indicates that the current frame to be encoded and the reference frame are determined by λ. i The distortion obtained from encoding (mean squared error). After encoding the current frame to be encoded using each encoding parameter in Λ, the reconstructed frame obtained by any encoding parameter is selected as the reference frame for the next frame to be encoded.

[0087] After obtaining the truth label, such as Figure 2 As shown, the downsampled video frames are downsampled again to 240P resolution and stitched together along the channel dimension, then fed into the bitrate predictor to obtain the corresponding bitrate prediction vector. The corresponding quality prediction vector is obtained by feeding it into the quality predictor.

[0088] When training the bitrate predictor, the loss function L of the bitrate predictor is... R for:

[0089]

[0090] in This is the bitrate prediction value output by the bitrate predictor.

[0091] When training the quality predictor, the loss function L of the quality predictor is... D for:

[0092]

[0093] in The quality prediction value output by the quality predictor.

[0094] Each frame in a set of video frames can have its loss value obtained by formula (4) or formula (5). The loss values ​​of each set of video frames are accumulated, and then gradient backpropagation and network parameter updates are performed. This is one iteration.

[0095] According to an embodiment of the present invention, in step 2, as follows: Figure 2 As shown, each pair of adjacent frames within a mini-GOP is downsampled to a resolution of 240P and then stitched together in the channel dimension. In this embodiment, bilinear interpolation is used for downsampling. The data is then input into the bitrate predictor that has been trained in step 1 to obtain a set of (R,λ) points corresponding to Λ. At the same time, the data is input into the quality predictor that has been trained in step 1 to obtain a set of (D,λ) points corresponding to Λ.

[0096] After obtaining the set of (D,λ) points for each frame, the interference value D for the next frame prediction is obtained according to the following formula. add :

[0097]

[0098] Here, T is a tensor with the same size as the resolution of the encoded frame of the input encoder and conforming to a standard normal distribution. When predicting the bitrate and quality of the next frame, this interference value is first added to the uncoded reference frame before downsampling, splicing, and prediction, which can improve prediction accuracy.

[0099] According to an embodiment of the present invention, in step 3, based on the (R,λ) point set obtained in step 2, a bitrate model is fitted to each frame to be encoded within the mini-GOP using the least squares method, and the expression is:

[0100]

[0101] Similarly, based on the (D,λ) point set obtained in step 2, a quality model is fitted to each frame to be encoded within the mini-GOP using the least squares method, and the expression is:

[0102]

[0103] According to formulas (7) and (8), the rate-distortion curve of each frame to be encoded within a mini-GOP can be obtained, and the expression is:

[0104]

[0105] According to an embodiment of the present invention, step 4 of the rate control algorithm includes the following steps:

[0106] (1) Determine whether the target bit rate exceeds the encoder's encoding range. If it is greater than the maximum value of the range, then use the maximum encoding parameter λ. M If the value is less than the minimum value, then use the minimum encoding parameter λ1 for encoding.

[0107] (2) Determine the upper and lower bounds of the target quality of the mini-GOP based on the (D,λ) point set obtained in step 2 (the target quality of each frame within the mini-GOP is the same). Specifically, the method is to set the D of each frame... pred The minimum values ​​of the M elements are grouped into a set, and the maximum value of the data in the set is taken as the lower bound of the target quality. The D values ​​of each frame are then... pred The maximum values ​​of the M elements are formed into a set, and the minimum value of the data in the set is taken as the upper bound of the target quality.

[0108] (3) Use the binary search method to iterate continuously, change the value of the target quality, and obtain the bit rate of each frame in the corresponding mini-GOP based on the target quality through formula (9) in each iteration, and calculate the total bit rate of the mini-GOP until the number of iterations of the binary search method is greater than the maximum value K (in the embodiment of the present invention, K is 100) or the target quality that meets the bit rate accuracy requirement (in the embodiment of the present invention, the bit rate accuracy requirement is that the bit rate error is less than 1%) is obtained.

[0109] (4) The proportion of bitrate in each frame within the mini-GOP obtained in the last iteration is used as the bitrate allocation ratio, expressed as:

[0110] bit ratio ={r1,r2,…,r N} (10)

[0111] Where N is the number of frames within a mini-GOP, and in the embodiments of the present invention, N is 4.

[0112] According to an embodiment of the present invention, in step 5, the original target bitrate of each frame within the mini-GOP is obtained based on the total bitrate of the mini-GOP and formula (10):

[0113]

[0114] The actual target bitrate for each frame is:

[0115]

[0116] Among them, such as Figure 1 As shown, buffer represents the buffer, which is initialized to 0 before each mini-GOP encoding.

[0117] Substitute the actual target bitrate of each frame obtained from formula (12) into formula (7) to obtain the encoding parameters of each frame within the mini-GOP, and then send them along with the corresponding frame to be encoded into the encoder to begin encoding. After each frame is encoded, the value of the buffer is set to:

[0118]

[0119] in, This is the actual encoded bitrate for this frame. When encoding the next frame, the actual target bitrate for that frame will be... Add a buffer to the original target bitrate obtained by formula (11).

[0120] This invention embodiment is tested based on a variable bitrate version of the Deep Video Encoder (DVC). The test dataset includes validation sets of HEVC B, HEVC C, HEVC D, HEVC E, and the Vimeo-90K septuplet. The experimental results are compared with two existing state-of-the-art bitrate control schemes, and the comparison metrics include:

[0121] 1) Relative bitrate error:

[0122]

[0123] 2) Bitrate control efficiency:

[0124]

[0125] Where P1 represents the bitrate control duration and P2 represents the encoding duration;

[0126] 3) Inter-frame quality fluctuation ratio Q ratio for:

[0127]

[0128] Among them, Q F Indicates inter-frame quality fluctuation, MSE i Q represents the distortion of each frame, μ represents the mean distortion of each frame, and Q represents the distortion of each frame. fixed_λ This indicates the Q generated by encoding with fixed encoding parameters. F .

[0129] The comparison algorithm is implemented as follows:

[0130] (1) Multi-pass rate control (Reference: Jiancong Chen, Meng Wang, Pingping Zhang, Shurun ​​Wang, and Shiqi Wang, “Sparse-to-dense: High efficiency rate control for end-to-end scale-adaptive video coding,” IEEE Transactions on Circuits and Systems for Video Technology, vol.34, no.5, pp.4027-4039, 2024.): Each GOP is encoded with the minimum and maximum coding parameters. The rate model is fitted using the least squares method. Then, the frame-level rate is allocated according to the rate allocation strategy in HEVC. The coding parameters of each frame are determined by the fitted rate model.

[0131] (2) One-pass rate control (Reference: Yanghao Li, Xinyao Chen, Jisheng Li, Jiangtao Wen, Yuxing Han, Shan Liu, and Xiaozhong Xu, “Rate control for learned video compression,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 2829–2833.): Initialize the rate model and quality model, then perform frame-level rate allocation for each frame within the GOP, and determine the encoding parameters for each frame through the rate model. After each frame is encoded, update the hyperparameters of the rate model and quality model according to the actual rate and actual quality.

[0132] Figure 3 This demonstrates the relative bit rate error (ΔR) and bit rate control efficiency (T). RC The comparison results show that the Vimeo test represents the validation set of the Vimeo-90K septuplet. This invention achieved high accuracy on all test sequences, with accuracy only slightly lower than the comparative multi-pass scheme (average accuracy only 2.54% lower), and significantly improved efficiency compared to the multi-pass scheme. Figure 5aIn the figure, "benchmark" represents the comparison scheme multi-pass. The figure further shows that the present invention achieves a high bitrate accuracy at the mini-GOP level, but the comparison scheme multi-pass has a large gap with the target bitrate at the mini-GOP level, and only achieves high accuracy at the group GOP level.

[0133] Figure 4 This demonstrates the inter-frame quality fluctuation ratio (Q). ratio The comparison results are shown below. The Vimeo test represents the validation set of the Vimeo-90K septuplet. All data in the figure are expressed as a percentage. The inter-frame quality fluctuation ratio Q of this invention... ratio Both are less than 100%, thus significantly reducing quality fluctuations compared to fixed coding parameters. In contrast, the inter-frame quality fluctuation ratios of the comparison schemes multi-pass and one-pass are both greater than 100%, which means that the quality fluctuations are increased compared to fixed coding parameters. Figure 5b This further demonstrates the effectiveness of the present invention in controlling inter-frame quality fluctuations in test videos. It can be seen that, in the test videos, the inter-frame quality fluctuations of the present invention are significantly less than those produced by encoding with fixed coding parameters, and the average quality is comparable.

[0134] This invention provides a method and system for rate-distortion prediction-based bitrate control of a depth video encoder. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A rate control method for a depth video encoder based on rate-distortion prediction, characterized in that, Includes the following steps: Step 1: Train the prediction module, which includes a downsampling module, a bitrate predictor, and a quality predictor; Step 2: Downsample each two adjacent frames of the current mini-GOP to be encoded to a low resolution, and then concatenate them in the channel dimension and send them to the pre-designed rate predictor and quality predictor to obtain a set of (R,λ) points and a set of (D,λ) points, where R represents the rate, D represents the quality, and λ represents the coding parameters. Step 3: Using the least squares method, fit the (R,λ) point set and (D,λ) point set obtained in Step 2 to the bitrate model and quality model, and obtain the bitrate distortion curve; Step 4: Input the target bitrate, (R,λ) point set, (D,λ) point set, and the fitted bitrate model, quality model, and bitrate distortion curve into the bitrate control algorithm, and output the frame-level bitrate allocation ratio of the mini-GOP to be encoded. Step 5: Determine the encoding parameters of each frame within the mini-GOP based on the frame-level bitrate allocation ratio obtained in Step 4 and the bitrate model obtained in Step 3, and input them into the video encoder to start encoding. In step 3, based on the (R,λ) point set obtained in step 2, the bitrate model is fitted to each frame to be coded within the mini-GOP using the least squares method. The expression is: Where α1 and β1 are hyperparameters of the code rate model; Based on the (D,λ) point set obtained in step 2, a quality model is fitted to each frame to be encoded within the mini-GOP using the least squares method. The expression is: Where α2 and β2 are the hyperparameters of the mass model; According to formulas (7) and (8), the rate-distortion curve of each frame to be encoded within the mini-GOP is obtained, and the expression is: In step 4, the rate control algorithm includes the following steps: Step 4-1: Determine if the target bit rate exceeds the encoder's encoding range. If it exceeds the maximum range value, then use the maximum encoding parameter λ. M If the value is less than the minimum value, then use the minimum encoding parameter λ1 for encoding. Step 4-2: Determine the upper and lower bounds of the target quality of the mini-GOP for each frame based on the (D,λ) point set obtained in Step 2: [This involves] dividing the quality prediction vector D of each frame... pred The minimum values ​​of the M elements are grouped into a set. The maximum value of the data in the set is taken as the lower bound of the target quality. The quality prediction vector D of each frame is then used. pred The maximum values ​​of the M elements are formed into a set, and the minimum value of the data in the set is taken as the upper bound of the target quality. Step 4-3: Use the binary search method to iterate continuously and change the value of the target quality. Each iteration is based on the target quality and uses formula (9) to obtain the bit rate of each frame in the corresponding mini-GOP. Calculate the total bit rate of the mini-GOP until the number of iterations of the binary search method is greater than the maximum value K or the target quality that meets the bit rate accuracy requirements is obtained. Step 4-4: Use the proportion of bitrate of each frame within the mini-GOP obtained in the last iteration as the bitrate allocation ratio (bits). ratio The expression is: bit ratio ={r1,r2,…,r N } (10) Where N is the number of frames within a mini-GOP, r1 represents the bitrate allocated to the first frame within the mini-GOP, and r N This indicates the bitrate allocated to the Nth frame within a mini-GOP; In step 5, the original target bitrate of each frame within the mini-GOP is obtained based on the total bitrate of the mini-GOP and formula (10): in, R represents the raw target bitrate of the j-th frame within a mini-GOP. mini-GOP This represents the total bitrate of a mini-GOP, where j ranges from 1 to N; The actual target bitrate for each frame is: Among them, among them, This represents the actual target bitrate of the j-th frame within the mini-GOP. Buffer represents the buffer, which is initialized to 0 before each mini-GOP encoding. Substitute the actual target bitrate of each frame obtained from formula (12) into formula (7) to obtain the encoding parameters of each frame within the mini-GOP. Then, send the encoding parameters of each frame and the corresponding frame to be encoded into the encoder to start encoding. After each frame is encoded, the value of the buffer is set as follows: in, The actual encoded bitrate of the j-th frame; When encoding the next frame, the actual target bitrate of the next frame. The original target bitrate obtained by formula (11) is obtained by adding the buffer.

2. The method according to claim 1, characterized in that, In step 1, the bitrate predictor and quality predictor in the prediction module are trained separately, using the same neural network architecture; the neural network architecture includes an embedding layer, a pooling layer, an interpolation upsampling layer, a convolutional layer, an adaptive pooling layer, and a multilayer perceptron; The specific structure of the embedding layer is as follows: the feature map passes through two convolutional layers; the downsampling ratio of the video frame is processed by sinusoidal embedding, then passed through a multilayer perceptron, and finally multiplied with the output of the two convolutional layers; The neural network architecture employs a symmetrical structure. First, the input feature map passes through an embedding layer and a pooling layer twice, i.e., a downsampling operation. Then, it passes through a residual network to extract deeper features. Next, it passes through an interpolation upsampling layer and an embedding layer twice, i.e., an upsampling operation. In the downsampling operation, the output of each embedding layer is concatenated with the output of the corresponding embedding layer in the upsampling operation along the channel dimension. Finally, it passes through two convolutional layers, one adaptive pooling layer, and one multilayer perceptron. Ultimately, the bitrate predictor outputs the predicted bitrate R for the corresponding encoding parameter set Λ, and the quality predictor outputs the predicted quality D for the corresponding encoding parameter set Λ.

3. The method according to claim 2, characterized in that, In step 1, the set of encoding parameters Λ is: Λ={λ1,λ2,…,λ M } (1) Where Λ is the set of M coding parameters obtained by logarithmic interpolation between the maximum and minimum values ​​of the coding parameters, λ1 is the minimum value of the coding parameters that can be input to the depth video encoder, and λ M This represents the maximum value of the encoding parameters that can be input to the depth video encoder.

4. The method according to claim 3, characterized in that, In step 1, the training prediction module includes: after downloading the training dataset, loading the training dataset and starting training. During training, in each iteration, consecutive video frames are selected as a group, and the selected consecutive video frames are randomly downsampled to a resolution. Then, frame-by-frame encoding begins. During each frame encoding, the current frame to be encoded and the reference frame are input into the depth video encoder, and encoded using each encoding parameter in the set Λ. The resulting bitrate or quality is concatenated into a vector. Where R real and D real These represent the ground truth labels when training the bitrate predictor and the ground truth labels when training the quality predictor, respectively. This represents the i-th encoding parameter λ that can be input to the depth video encoder for the current frame to be encoded and the reference frame. i The bitrate obtained from encoding, This represents the i-th encoding parameter λ that can be input to the depth video encoder for the current frame to be encoded and the reference frame. i The distortion obtained from encoding; i takes values ​​from 1 to M; After encoding the current frame to be encoded using each encoding parameter in set Λ, the reconstructed frame obtained by selecting any encoding parameter is used as the reference frame for the next frame to be encoded. After obtaining the ground truth labels, the downsampled video frames are downsampled again to a predetermined low resolution, and then concatenated along the channel dimension. The result is then fed into the bitrate predictor to obtain the corresponding bitrate prediction vector. The corresponding quality prediction vector is obtained by feeding it into the quality predictor.

5. The method according to claim 4, characterized in that, In step 1, during the training of the bitrate predictor, the loss function L of the bitrate predictor is... R for: in This is the i-th bitrate prediction value output by the bitrate predictor; When training the quality predictor, the loss function L of the quality predictor is... D for: in This is the i-th quality prediction value output by the quality predictor; Each frame in a set of video frames can have its loss value obtained by formula (4) or formula (5). The loss values ​​of each set of video frames are accumulated, and then gradient backpropagation and network parameter updates are performed to complete one iteration.

6. The method according to claim 5, characterized in that, In step 2, downsample every two adjacent frames of the current mini-GOP to be encoded, and then input them into the bitrate predictor that has been trained in step 1 to obtain a set of (R,λ) points in the corresponding set Λ. At the same time, input them into the quality predictor that has been trained in step 1 to obtain a set of (D,λ) points in the corresponding set Λ. Where R is the bitrate, λ is the encoding parameter, and D is the quality; After obtaining the set of (D,λ) points for each frame, the interference value D for the next frame prediction is obtained according to the following formula. add : Where T is a tensor of the same size as the resolution of the encoded frame of the input encoder and conforms to a standard normal distribution, and e is a natural constant; When predicting bitrate and quality for the next frame, the uncoded reference frame is first given an interference value D. add Then, downsampling, splicing, and prediction are performed.

7. A depth video encoder bitrate control system based on rate-distortion prediction, implemented using the method described in any one of claims 1 to 6, characterized in that, It includes a training module, a prediction module, a fitting module, a bitrate control algorithm calculation module, and an encoding module; The training module is used to train the prediction module. The prediction module includes a downsampling module, a bitrate predictor, and a quality predictor. The downsampling module is used to downsample each two adjacent frames of the current mini-GOP to be encoded to a low resolution. Each two adjacent frames after downsampling are concatenated in the channel dimension and then fed into the pre-designed bitrate predictor and quality predictor to obtain a set of (R,λ) points and a set of (D,λ) points. The fitting module uses the least squares method to fit the bitrate model and the quality model using the (R,λ) point set and (D,λ) point set obtained in step 2, and obtains the bitrate distortion curve; The bitrate control algorithm calculation module inputs the target bitrate, the (R,λ) point set, the (D,λ) point set, the fitted bitrate model, the quality model, and the bitrate distortion curve into the bitrate control algorithm, and outputs the frame-level bitrate allocation ratio of the mini-GOP to be encoded. The encoding module determines the encoding parameters of each frame within the mini-GOP based on the obtained frame-level bitrate allocation ratio and the bitrate model obtained in step 3, and inputs them into the video encoder to start encoding.

Images