Depth video encoder code rate control method and system based on rate distortion prediction

Abstract

The invention provides a depth video encoder code rate control method and system based on rate distortion prediction, and the method comprises the steps: 1, training a prediction module, 2, inputting a video frame into the prediction module to obtain a prediction point set, 3, fitting a code rate and quality model according to the prediction point set, and 4, obtaining a frame-level code rate distribution proportion through a code rate control algorithm, and 5, determining encoding parameters corresponding to each frame, and inputting the encoding parameters into an encoder for encoding. According to the method, the neural network and the original video are directly used for predicting the code rate model and the quality model of each frame for the first time, and pre-coding is not needed; video frames are downsampled to a fixed small resolution and then are input into the neural network, so that the efficiency is improved and the generalization is enhanced; according to the method, the code rate control is realized at the mini-GOP level of the small group frame for the first time. Compared with the existing code rate control method, the method provided by the invention can realize considerable code rate control precision and finer code rate control granularity at a faster speed.

Description

Technical Field

[0001] The present invention relates to the field of multimedia communication technology, and in particular to a rate control method and system for a deep video encoder based on rate-distortion prediction. Background Art

[0002] Deep learning-based video encoders have developed rapidly in recent years and have great potential. With the prevalence of video platforms, video traffic has accounted for more than 80% of Internet traffic and continues to grow. Therefore, it is crucial to design efficient coding technologies for video storage and transmission. Traditional video encoders, including AVC, HEVC, and VVC, encode videos based on manually designed rules. In recent years, with the development of artificial intelligence-related technologies, deep learning-based video coding has received more and more attention. The existing methods can be divided into two categories: motion & residual hybrid coding and probabilistic prediction coding. Their performance has surpassed traditional video encoders in some aspects.

[0003] Existing deep video encoders are usually trained by minimizing the RD loss function R+λ·D, where λ is a coding parameter corresponding to a fixed bit rate. Therefore, to achieve different bit rates for the same video sequence, it is necessary to retrain the model by changing λ. To solve this problem, multiple bitrates can be achieved under a single model by downsampling the original frame of the input video and upsampling it at the output (Chen J, Wang M, Zhang P, et al. Sparse-to-Dense: High Efficiency Rate Control for End-to-end Scale-Adaptive Video Coding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023.), or a variable bitrate model can be achieved through a content-adaptive quantization mechanism (Li J, Li B, Lu Y. Hybrid spatial-temporal entropy modelling for neural video compression[C] / / Proceedings of the 30th ACM International Conference on Multimedia. 2022: 1503-1511.), and there is also a method of mapping the encoding parameter λ to the inside of the encoder through a neural network to achieve a variable bitrate model (Lin J, Liu D, Liang J, et al. A deeply modulated scheme for variable-rate video compression[C] / / 2021IEEE International Conference on Image Processing (ICIP). IEEE, 2021:3722-3726.).

[0004] The rate control methods of traditional video encoders can be divided into three categories: RQ model, R-ρ model, and R-λ model (WeiX, Zhou M, Wang H, et al. Recent Advances in Rate Control: From Optimisation to Implementation and Beyond [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023.), which directly or indirectly establish a mapping relationship between bit rate and coding parameters to achieve bit rate control. Among them, the R-λ model (JCTVC-K0103, "Rate control by R-lambda model for HEVC," Shanghai, CN, 10–19 Oct. 2012.) has the best rate control scheme accuracy and is still used by traditional video coding standards.

[0005] Most of the existing work on deep video encoders focuses on designing and improving compression architectures to achieve better rate-distortion performance, but there is less research on corresponding rate control techniques. To solve this problem, we can borrow the rate control method of traditional video encoders and redesign the rate allocation and hyperparameter update mechanism to achieve higher rate accuracy without pre-encoding (Yanghao Li, Xinyao Chen, Jisheng Li, Jiangtao Wen, Yuxing Han, ShanLiu, and Xiaozhong Xu, "Rate control for learned video compression," in ICASSP2022-2022IEEE International Conference on Acoustics, Speech and SignalProcessing (ICASSP), 2022, pp. 2829–2833.), or we can improve the rate accuracy by pre-encoding multiple times and fitting the rate model, but the cost is a great increase in time consumption (Jiancong Chen, Meng Wang, Pingping Zhang, Shurun ​​Wang, and Shiqi Wang, "Sparse-to-dense: High efficiency rate control for end-to-end scale-adaptive video coding," IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 5, pp. 4027-4039, 2024.). Both methods can only achieve rate control at the level of group of pictures (GOP), and at the cost of relatively low accuracy or greater time consumption. Summary of the invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is: to overcome the shortcomings of the existing rate control schemes of deep video encoders, including low accuracy, slow speed, only being able to achieve group frame GOP level code control, and large fluctuations in quality between frames, and to provide a rate control algorithm that takes into account both accuracy and speed, can achieve high rate accuracy at the mini-GOP level, and effectively control fluctuations in quality between frames.

[0007] The present invention provides a method and system for bit rate control of a deep video encoder based on rate distortion prediction, wherein the method comprises the following steps:

[0008] Step 1: training a prediction module, wherein the prediction module includes a downsampling module, a bit rate predictor and a quality predictor;

[0009] Step 2: downsample every two adjacent frames of the current mini-GOP to be encoded to a low resolution (such as 240P), and send them to the pre-designed rate predictor and quality predictor after splicing in the channel dimension to obtain a set of (R, λ) point sets and a set of (D, λ) point sets, where R represents the rate, D represents the quality, and λ represents the encoding parameter;

[0010] Step 3: Fit the rate model and the quality model using the (R, λ) point set and the (D, λ) point set obtained in step 2 by the least square method, and obtain the rate distortion curve;

[0011] Step 4: Input the target bit rate, (R,λ) point set, (D,λ) point set, and the fitted bit rate model, quality model, and bit rate distortion curve into the bit rate control algorithm, and output the frame level bit rate allocation ratio of the mini-GOP to be encoded;

[0012] Step 5: Determine the encoding parameters of each frame in the mini-GOP according to the frame level bit rate allocation ratio obtained in step 4 and the bit rate model obtained in step 3, and input them into the video encoder to start encoding.

[0013] In step 1, the bit rate predictor and the quality predictor in the prediction module are trained separately, using exactly the same neural network architecture, with the same input and only different outputs; the neural network architecture includes an embedding layer, a pooling layer, an interpolation upsampling layer, a convolution layer, an adaptive pooling layer, and a multi-layer perceptron;

[0014] The specific structure of the embedding layer is as follows: the feature map passes through two convolutional layers; the downsampling ratio of the video frame is processed by sinusoidal embedding, then passes through a multi-layer perceptron, and finally multiplied with the output of the two convolutional layers;

[0015] The neural network architecture adopts a symmetrical structure. First, the input feature map passes through the embedding layer and the pooling layer twice, that is, the downsampling operation; then passes through the residual network to extract deeper features; then passes through the interpolation upsampling layer and the embedding layer twice, that is, the upsampling operation; the output of each embedding layer in the downsampling operation must be spliced ​​with the output of the corresponding embedding layer in the upsampling operation in the channel dimension; finally, it passes through two convolutional layers, one adaptive pooling layer, and a multi-layer perceptron, and finally the rate predictor outputs the predicted value of the rate R corresponding to the coding parameter set Λ, and the quality predictor outputs the predicted value of the quality D corresponding to the coding parameter set Λ.

[0016] In step 1, the encoding parameter set Λ is:

[0017] Λ={λ1,λ2,…,λM} (1)

[0018] Where Λ is a set of M coding parameters obtained by logarithmic interpolation between the maximum and minimum values ​​of the coding parameters, λ1 is the minimum value of the coding parameter that can be input to the deep video encoder, and λ M The maximum value of the encoding parameters that can be input to the deep video encoder.

[0019] In step 1, the training prediction module includes: after downloading the training data set, loading the training data set and starting training. During the training, each iteration selects continuous video frames as a group (for example, each iteration selects 5 continuous video frames as a group), and randomly downsamples the selected continuous video frames to a resolution (for example, downsampled to any resolution of 1080P, 720P, 480P and 240P), and then starts encoding frame by frame. When encoding each frame, the current frame to be encoded and the reference frame are input into the deep video encoder, and encoded with each encoding parameter in the set Λ, and the obtained bit rate or quality is spliced ​​into a vector:

[0020]

[0021] Where R real and D real They represent the true value labels when training the bitrate predictor and the true value labels when training the quality predictor respectively; Represents the i-th encoding parameter λ that can be input by the deep video encoder for the current frame to be encoded and the reference frame i The bit rate obtained by encoding (bits per pixel), Represents the i-th encoding parameter λ that can be input by the deep video encoder for the current frame to be encoded and the reference frame i The distortion obtained by encoding (mean squared error); i ranges from 1 to M;

[0022] After encoding the current frame to be encoded using each encoding parameter in the set Λ, a reconstructed frame obtained by selecting any encoding parameter is used as a reference frame for the next frame to be encoded;

[0023] After obtaining the true value label, the downsampled video frame is downsampled again to a predetermined low resolution (i.e., 240P resolution), spliced ​​in the channel dimension, and then sent to the bit rate predictor to obtain the corresponding bit rate prediction vector Send it to the quality predictor to get the corresponding quality prediction vector

[0024] In step 1, when training the bitrate predictor, the loss function L of the bitrate predictor is R for:

[0025]

[0026] in is the i-th bit rate prediction value output by the bit rate predictor;

[0027] When training the quality predictor, the loss function L of the quality predictor is D for:

[0028]

[0029] in is the i-th quality prediction value output by the quality predictor.

[0030] The loss value of each frame in a group of video frames can be obtained by formula (4) or formula (5). The loss value of each group of video frames is accumulated, and then the gradient is returned and the network parameters are updated to complete an iteration.

[0031] In step 2, downsample every two adjacent frames of the current mini-GOP to be encoded (for example, downsample to a resolution of 240P), and then input the rate predictor trained in step 1 to obtain a set of (R, λ) points corresponding to the set Λ, and input the quality predictor trained in step 1 to obtain a set of (D, λ) points corresponding to the set Λ; where R is the rate, λ is the encoding parameter, and D is the quality;

[0032] After obtaining the (D, λ) point set of each frame, the interference value D for the next frame prediction is obtained according to the following formula add :

[0033]

[0034] Where T is a tensor with the same size as the encoded frame resolution of the input encoder and conforms to the standard normal distribution, and e is a natural constant;

[0035] When predicting the bit rate and quality of the next frame, the uncoded reference frame is first added with the interference value D add Then down-sample, concatenate and predict.

[0036] In step 3, based on the (R, λ) point set obtained in step 2, a bit rate model is fitted for each frame to be encoded in the mini-GOP using the least squares method, and the expression is:

[0037]

[0038] Among them, α1 and β1 are the hyperparameters of the bit rate model;

[0039] Based on the (D, λ) point set obtained in step 2, the quality model is fitted for each frame to be encoded in the mini-GOP using the least squares method. The expression is:

[0040]

[0041] Among them, α2 and β2 are the hyperparameters of the quality model;

[0042] According to formula (7) and formula (8), the rate distortion curve of each frame to be encoded in the mini-GOP is obtained, and the expression is:

[0043]

[0044] In step 4, the rate control algorithm includes the following steps:

[0045] Step 4-1: Determine whether the target bit rate exceeds the encoder encoding range. If it is greater than the maximum value of the range, use the maximum encoding parameter λ M Encoding, if it is less than the minimum value, it is encoded with the minimum encoding parameter λ1;

[0046] Step 4-2, determine the upper and lower bounds of the target quality of the mini-GOP according to the (D, λ) point set obtained in step 2 (the target quality of each frame in the mini-GOP is the same): pred The minimum values ​​of the M elements of form a set, take the maximum value of the data in the set as the lower bound of the target quality, and transform the quality prediction vector D of each frame into pred The maximum values ​​of the M elements of form a set, and the minimum value of the data in the set is taken as the upper bound of the target quality;

[0047] Step 4-3, using the binary search method to iterate continuously, changing the value of the target quality, obtaining the bit rate of each frame in the corresponding mini-GOP based on the target quality through formula (9) in each iteration, and calculating the total bit rate of the mini-GOP, until the number of binary search iterations is greater than the maximum value K or the target quality that meets the bit rate accuracy requirement is obtained;

[0048] Step 4-4: The bit rate ratio of each frame in the mini-GOP obtained in the last iteration is used as the bit rate allocation ratio bit ratio , the expression is:

[0049] bit ratio ={r1, r2, ..., r N} (10)

[0050] Where N is the number of frames in a mini-GOP, r1 represents the bit rate assigned to the first frame in the mini-GOP, and r N Indicates the code rate assigned to the Nth frame in the mini-GOP.

[0051] In step 5, the original target bit rate of each frame in the mini-GOP is obtained according to the total bit rate of the mini-GOP and formula (10):

[0052]

[0053] in, represents the original target bit rate of the jth frame in the mini-GOP, R mini-GOP Indicates the total bit rate of the mini-GOP, j ranges from 1 to N;

[0054] The actual target bitrate for each frame is:

[0055]

[0056] Among them, among them, It represents the actual target bit rate of the jth frame in the mini-GOP. Buffer represents the buffer. Before encoding each mini-GOP, the buffer is initialized to 0.

[0057] Substitute the actual target bit rate of each frame obtained by formula (12) into formula (7) to obtain the encoding parameters of each frame in the mini-GOP, and send the encoding parameters of each frame and the corresponding frame to be encoded to the encoder to start encoding. After encoding each frame, the value of the buffer is set to:

[0058]

[0059] in, is the actual encoding bit rate of the jth frame;

[0060] When encoding the next frame, the actual target bitrate for the next frame The original target bit rate obtained by formula (11) plus the buffer buffer is obtained.

[0061] The present invention also provides a rate control system of a deep video encoder based on rate distortion prediction, including a training module, a prediction module, a fitting module, a rate control algorithm calculation module, and an encoding module;

[0062] The training module is used to train the prediction module, and the prediction module includes a downsampling module, a rate predictor and a quality predictor. The downsampling module is used to downsample every two adjacent frames of the current group frame mini-GOP to be encoded to a low resolution. After the downsampled two adjacent frames are spliced ​​in the channel dimension, they are sent to the pre-designed rate predictor and quality predictor to obtain a set of (R, λ) point sets and a set of (D, λ) point sets;

[0063] The fitting module fits the rate model and the quality model using the (R, λ) point set and the (D, λ) point set obtained in step 2 by the least square method, and obtains a rate-distortion curve;

[0064] The rate control algorithm calculation module inputs the target rate, (R, λ) point set, (D, λ) point set and the fitted rate model, quality model and rate distortion curve into the rate control algorithm, and outputs the frame level rate allocation ratio of the mini-GOP frame to be encoded;

[0065] The encoding module determines the encoding parameters of each frame in the mini-GOP according to the obtained frame level bit rate allocation ratio and the bit rate model obtained in step 3, and inputs the parameters into the video encoder to start encoding.

[0066] Beneficial effects:

[0067] 1) For the first time, the neural network and the original video are directly used to predict the bit rate and quality model of each frame without pre-encoding; 2) The video frame is downsampled to a fixed small resolution before inputting into the neural network, which improves efficiency and enhances generalization; 3) An interference mechanism is proposed to improve prediction accuracy; 4) Bit rate control is implemented at the mini-GOP level for the first time; 5) Smooth inter-frame quality fluctuations are achieved in bit rate control for the first time. Compared with existing bit rate control methods, the present invention can achieve comparable bit rate control accuracy and finer bit rate control granularity at a faster speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0069] Figure 1 It is a system architecture diagram of the present invention.

[0070] Figure 2 It is a structural diagram of the prediction module of the present invention.

[0071] Figure 3 This is the comparison result between the present invention and the most advanced existing solutions in terms of rate control accuracy and efficiency.

[0072] Figure 4This is the comparison result between the present invention and the most advanced existing solution in terms of inter-frame quality fluctuation.

[0073] Figure 5a and Figure 5b This is a graph showing the bit rate control accuracy and quality fluctuation results obtained by the present invention on a test video. DETAILED DESCRIPTION

[0074] like Figure 1 As shown, an embodiment of the present invention provides a rate control method for a deep video encoder based on rate-distortion prediction, comprising the following steps:

[0075] Step 1: Training Figure 2 The prediction module shown includes a downsampling module, a bit rate predictor and a quality predictor;

[0076] Step 2: downsample every two adjacent frames of the current mini-GOP to be encoded to a low resolution (such as 240P), and send them to the trained rate predictor and quality predictor after splicing them in the channel dimension to obtain a set of (R, λ) point sets and a set of (D, λ) point sets;

[0077] Step 3: Fit the rate model and the quality model using the (R, λ) point set and the (D, λ) point set obtained in step 2 by the least square method, and obtain the rate distortion curve;

[0078] Step 4: Input the target bit rate, (R, λ) point set, (D, λ) point set, and the fitted bit rate model, quality model, and bit rate distortion curve into Figure 1 In the bit rate control algorithm, the frame level bit rate allocation ratio of the mini-GOP frame to be encoded is output;

[0079] Step 5: Determine the encoding parameters of each frame in the mini-GOP according to the frame level bit rate allocation ratio obtained in step 4 and the bit rate model obtained in step 3, and input them into the video encoder to start encoding.

[0080] The embodiment of the present invention uses a variable bit rate version of DVC as a deep video encoder.

[0081] According to an embodiment of the present invention, in step 1, the rate predictor and the quality predictor in the prediction module are trained separately, using exactly the same neural network architecture, with the same input and only different outputs. First, the encoding parameter set is determined as:

[0082] Λ={λ1,λ2,...,λ M} (1)

[0083] Where Λ is a set of M coding parameters obtained by logarithmic interpolation between the maximum and minimum values ​​of the coding parameters, λ1 is the minimum value of the coding parameter that can be input to the deep video encoder, and λ M It is the maximum value of the encoding parameter that can be input to the depth video encoder. In this embodiment, M is 8.

[0084] In an embodiment of the present invention, the training data set uses a large-resolution Vimeo (https: / / vimeo.com). After downloading the training data set, load the training data set and start training. During training, 5 consecutive video frames are selected as a group in each iteration, and the selected consecutive video frames are randomly downsampled to any resolution of 1080P, 720P, 480P and 240P, and then frame-by-frame encoding is started. When encoding each frame, the current frame to be encoded and the reference frame are input into the deep video encoder, and encoded with each encoding parameter in Λ, and the obtained bit rate or quality is spliced ​​into a vector:

[0085]

[0086] Formula (2) and Formula (3) represent the true value labels when training the bitrate predictor and the true value labels when training the quality predictor, respectively. Indicates that the current frame to be encoded and the reference frame are represented by λ i The bit rate obtained by encoding (bits perpixel), Indicates that the current frame to be encoded and the reference frame are represented by λ i After encoding the current frame to be encoded using each encoding parameter in Λ, a reconstructed frame obtained by any encoding parameter is selected as a reference frame for the next frame to be encoded.

[0087] After getting the true value label, such as Figure 2 As shown, the downsampled video frames are downsampled again to 240P resolution and spliced ​​in the channel dimension, and then sent to the bit rate predictor to obtain the corresponding bit rate prediction vector Send it to the quality predictor to get the corresponding quality prediction vector

[0088] When training the bitrate predictor, the loss function L of the bitrate predictor is R for:

[0089]

[0090] in It is the bit rate prediction value output by the bit rate predictor.

[0091] When training the quality predictor, the loss function L of the quality predictor is D for:

[0092]

[0093] in is the quality prediction value output by the quality predictor.

[0094] The loss value of each frame in a group of video frames can be obtained by formula (4) or formula (5). The loss value of each group of video frames is accumulated, and then the gradient is returned and the network parameters are updated. This is one iteration.

[0095] According to an embodiment of the present invention, in step 2, if Figure 2 As shown, every two adjacent frames in the mini-GOP are downsampled to a resolution of 240P and then spliced ​​in the channel dimension. In the embodiment, bilinear interpolation is used for downsampling, and then the trained bit rate predictor in step 1 is input to obtain a set of (R, λ) points corresponding to Λ. At the same time, the trained quality predictor in step 1 is input to obtain a set of (D, λ) points corresponding to Λ.

[0096] After obtaining the (D,λ) point set of each frame, the interference value D for the next frame prediction is obtained according to the following formula add :

[0097]

[0098] Where T is a tensor with the same size as the encoding frame resolution of the input encoder and conforms to the standard normal distribution. When predicting the bit rate and quality of the next frame, the unencoded reference frame is first added with this interference value, and then downsampled, spliced, and predicted, which can improve the prediction accuracy.

[0099] According to an embodiment of the present invention, in step 3, based on the (R, λ) point set obtained in step 2, a bit rate model is fitted for each frame to be encoded in the mini-GOP using the least squares method, and the expression is:

[0100]

[0101] Similarly, based on the (D,λ) point set obtained in step 2, the quality model is fitted for each frame to be encoded in the mini-GOP by the least squares method, and the expression is:

[0102]

[0103] According to formula (7) and formula (8), the rate distortion curve of each frame to be encoded in the mini-GOP can be obtained, and the expression is:

[0104]

[0105] According to an embodiment of the present invention, in step 4, the rate control algorithm includes the following steps:

[0106] (1) Determine whether the target bit rate exceeds the encoder encoding range. If it is greater than the maximum value of the range, use the maximum encoding parameter λ M Encoding, if it is less than the minimum value, it is encoded with the minimum encoding parameter λ1;

[0107] (2) According to the (D,λ) point set obtained in step 2, the upper and lower bounds of the target quality of the mini-GOP are determined (the target quality of each frame in the mini-GOP is the same). The specific method is to set the D of each frame to pred The minimum values ​​of the M elements of the set are used to form a set, and the maximum value of the data in the set is taken as the lower bound of the target quality. pred The maximum values ​​of the M elements of form a set, and the minimum value of the data in the set is taken as the upper bound of the target quality;

[0108] (3) The binary search method is used to iterate continuously to change the value of the target quality. In each iteration, the bit rate of each frame in the corresponding mini-GOP is obtained based on the target quality through formula (9), and the total bit rate of the mini-GOP is calculated, until the number of binary search iterations is greater than a maximum value K (in the embodiment of the present invention, K is 100) or the target quality that meets the bit rate accuracy requirement (in the embodiment of the present invention, the bit rate accuracy requirement is that the bit rate error is less than 1%) is obtained;

[0109] (4) The bit rate ratio of each frame in the mini-GOP obtained in the last iteration is used as the bit rate allocation ratio, which is expressed as:

[0110] bit ratio ={r1,r2,…,r N} (10)

[0111] Wherein N is the number of frames in a mini-GOP. In the embodiment of the present invention, the value of N is 4.

[0112] According to an embodiment of the present invention, in step 5, the original target bit rate of each frame in the mini-GOP is obtained according to the total bit rate of the mini-GOP and formula (10):

[0113]

[0114] The actual target bitrate for each frame is:

[0115]

[0116] Among them, Figure 1 As shown, buffer represents a buffer zone, and the buffer zone is initialized to 0 before encoding each mini-GOP frame.

[0117] Substitute the actual target bit rate of each frame obtained by formula (12) into formula (7) to obtain the encoding parameters of each frame in the mini-GOP and send it and the corresponding frame to be encoded to the encoder to start encoding. After encoding each frame, the value of the buffer is set to:

[0118]

[0119] in, The actual encoding bitrate of this frame. When encoding the next frame, the actual target bitrate of the next frame The original target bit rate obtained by formula (11) plus the buffer buffer.

[0120] The embodiment of the present invention is tested based on the variable bit rate version of the deep video encoder DVC. The test data set includes the validation set of HEVC B, HEVC C, HEVC D, HEVC E and Vimeo-90K septuplet. The experimental results are compared with two existing most advanced bit rate control schemes. The comparison indicators include:

[0121] 1)Relative bit rate error:

[0122]

[0123] 2) Rate control efficiency:

[0124]

[0125] Where P1 represents the bit rate control duration, and P2 represents the encoding duration;

[0126] 3) Inter-frame quality fluctuation ratio Q ratio for:

[0127]

[0128] Among them, Q F Indicates the quality fluctuation between frames, MSE i represents the distortion of each frame, μ represents the mean value of the distortion of each frame, Q fixed_λ Indicates Q generated by encoding with fixed coding parameters F .

[0129] The implementation of the comparison algorithm is as follows:

[0130] (1) Multi-pass rate control (Reference: Jiancong Chen, Meng Wang, Pingping Zhang, Shurun ​​Wang, and Shiqi Wang, “Sparse-to-dense: High efficiency rate control for end-to-end scale-adaptive video coding,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 5, pp. 4027-4039, 2024.): Each GOP is encoded with the minimum coding parameter and the maximum coding parameter, and the rate model is fitted using the least squares method. Then, frame-level rate allocation is performed according to the rate allocation strategy in HEVC, and the coding parameters of each frame are determined by the fitted rate model.

[0131] (2) One-pass rate control (Reference: Yanghao Li, Xinyao Chen, Jisheng Li, Jiangtao Wen, Yuxing Han, Shan Liu, and Xiaozhong Xu, “Rate control for learned video compression,” in ICASSP 2022-2022IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 2829–2833.): Initialize the rate model and quality model, then allocate frame-level rate for each frame in the GOP, and determine the encoding parameters of each frame through the rate model. After each frame is encoded, the hyperparameters of the rate model and quality model are updated according to the actual rate and actual quality.

[0132] Figure 3 The relative rate error (ΔR) and rate control efficiency (T RC ), Vimeo test represents the validation set of Vimeo-90K septuplet. The present invention achieves high accuracy on all test sequences, and its accuracy is only slightly lower than that of the comparison scheme multi-pass, which is only 2.54% lower on average, and its efficiency is significantly improved compared with the comparison scheme multi-pass. Figure 5aIn the figure, "benchmark" represents the comparison scheme multi-pass. The figure further shows that the present invention achieves a very high bit rate accuracy at the small group frame mini-GOP level, but the comparison scheme multi-pass has a large gap with the target bit rate at the small group frame mini-GOP level and only achieves high accuracy at the group frame GOP level.

[0133] Figure 4 Shows the inter-frame quality fluctuation ratio (Q ratio ), Vimeo test represents the validation set of Vimeo-90Kseptuplet, and the data in the figure are all in %. ratio Both are less than 100%, so the quality fluctuation is significantly reduced compared with the fixed coding parameter encoding. In contrast, the inter-frame quality fluctuation ratios of the comparison scheme multi-pass and the comparison scheme one-pass are both greater than 100%, that is, the quality fluctuation is increased compared with the fixed coding parameter encoding. Figure 5b The control effect of the present invention on the inter-frame quality fluctuation on the test video is further demonstrated. It can be seen that on the test video, the inter-frame quality fluctuation of the present invention is significantly smaller than the inter-frame quality fluctuation generated by the fixed encoding parameter encoding, and the average quality is equivalent.

[0134] The present invention provides a rate control method and system for a deep video encoder based on rate distortion prediction. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A rate control method for a deep video encoder based on rate-distortion prediction, characterized in that: The following steps are involved: Step 1: training a prediction module, wherein the prediction module includes a downsampling module, a bit rate predictor and a quality predictor; Step 2: downsample every two adjacent frames of the current mini-GOP to be encoded to a low resolution, and send them to the pre-designed rate predictor and quality predictor after splicing them in the channel dimension to obtain a set of (R, λ) point sets and a set of (D, λ) point sets, where R represents the rate, D represents the quality, and λ represents the encoding parameter; Step 3: Fit the rate model and the quality model using the (R, λ) point set and the (D, λ) point set obtained in step 2 by the least square method, and obtain the rate distortion curve; Step 4: Input the target bit rate, (R,λ) point set, (D,λ) point set, and the fitted bit rate model, quality model, and bit rate distortion curve into the bit rate control algorithm, and output the frame level bit rate allocation ratio of the mini-GOP to be encoded; Step 5: Determine the encoding parameters of each frame in the mini-GOP according to the frame level bit rate allocation ratio obtained in step 4 and the bit rate model obtained in step 3, and input them into the video encoder to start encoding.

2. The method according to claim 1, characterized in that In step 1, the bit rate predictor and the quality predictor in the prediction module are trained separately, using exactly the same neural network architecture; the neural network architecture includes an embedding layer, a pooling layer, an interpolation upsampling layer, a convolution layer, an adaptive pooling layer, and a multi-layer perceptron; The specific structure of the embedding layer is as follows: the feature map passes through two convolutional layers; the downsampling ratio of the video frame is processed by sinusoidal embedding, then passes through a multi-layer perceptron, and finally multiplied with the output of the two convolutional layers; The neural network architecture adopts a symmetrical structure. First, the input feature map passes through the embedding layer and the pooling layer twice, that is, the downsampling operation; then passes through the residual network to extract deeper features; then passes through the interpolation upsampling layer and the embedding layer twice, that is, the upsampling operation; the output of each embedding layer in the downsampling operation must be spliced ​​with the output of the corresponding embedding layer in the upsampling operation in the channel dimension; finally, it passes through two convolutional layers, one adaptive pooling layer, and a multi-layer perceptron, and finally the rate predictor outputs the predicted value of the rate R corresponding to the coding parameter set Λ, and the quality predictor outputs the predicted value of the quality D corresponding to the coding parameter set Λ.

3. The method according to claim 2, characterized in that In step 1, the encoding parameter set Λ is: Λ={λ1,λ2,…,λ M } (1) Where Λ is a set of M coding parameters obtained by logarithmic interpolation between the maximum and minimum values ​​of the coding parameters, λ1 is the minimum value of the coding parameter that can be input to the deep video encoder, and λ M The maximum value of the encoding parameters that can be input to the deep video encoder.

4. The method according to claim 3, characterized in that In step 1, the training prediction module includes: after downloading the training data set, loading the training data set and starting training. During the training, each iteration selects continuous video frames as a group, randomly downsamples the selected continuous video frames to a resolution, and then starts encoding frame by frame. When encoding each frame, the current frame to be encoded and the reference frame are input into the deep video encoder, and encoded with each encoding parameter in the set Λ, and the obtained bit rate or quality is spliced ​​into a vector: Where R real and D real They represent the true value labels when training the bitrate predictor and the true value labels when training the quality predictor respectively; Represents the i-th encoding parameter λ that can be input by the deep video encoder for the current frame to be encoded and the reference frame i The bit rate obtained by encoding, Represents the i-th encoding parameter λ that can be input by the deep video encoder for the current frame to be encoded and the reference frame i The distortion obtained by encoding; i ranges from 1 to M; After encoding the current frame to be encoded using each encoding parameter in the set Λ, a reconstructed frame obtained by selecting any encoding parameter is used as a reference frame for the next frame to be encoded; After obtaining the true value label, the downsampled video frame is downsampled again to a predetermined low resolution, spliced ​​in the channel dimension, and then sent to the bit rate predictor to obtain the corresponding bit rate prediction vector Send it to the quality predictor to get the corresponding quality prediction vector 5. The method according to claim 4, characterized in that In step 1, when training the bitrate predictor, the loss function l of the bitrate predictor is R for: in is the i-th bit rate prediction value output by the bit rate predictor; When training the quality predictor, the loss function l of the quality predictor is D for: in is the i-th quality prediction value output by the quality predictor; The loss value of each frame in a group of video frames can be obtained by formula (4) or formula (5). The loss value of each group of video frames is accumulated, and then the gradient is returned and the network parameters are updated to complete an iteration.

6. The method according to claim 5, characterized in that In step 2, downsample every two adjacent frames of the current mini-GOP to be encoded, and then input the rate predictor trained in step 1 to obtain a set of (R, λ) points corresponding to the set Λ. At the same time, input the quality predictor trained in step 1 to obtain a set of (D, λ) points corresponding to the set Λ. Where R is the bit rate, λ is the encoding parameter, and D is the quality; After obtaining the (D,λ) point set of each frame, the interference value d for the next frame prediction is obtained according to the following formula add : Where T is a tensor with the same size as the encoded frame resolution of the input encoder and conforms to the standard normal distribution, and e is a natural constant; When predicting the bit rate and quality of the next frame, the uncoded reference frame is first added with the interference value d add Then down-sample, concatenate and predict.

7. The method according to claim 6, characterized in that In step 3, based on the (R,λ) point set obtained in step 2, the bit rate model is fitted for each frame to be encoded in the mini-GOP by the least squares method, and the expression is: Among them, α1 and β1 are the hyperparameters of the bit rate model; Based on the (D,λ) point set obtained in step 2, the quality model is fitted for each frame to be encoded in the mini-GOP using the least squares method. The expression is: Among them, α2 and β2 are the hyperparameters of the quality model; According to formula (7) and formula (8), the rate distortion curve of each frame to be encoded in the mini-GOP is obtained, and the expression is:

8. The method according to claim 7, characterized in that In step 4, the rate control algorithm includes the following steps: Step 4-1: Determine whether the target bit rate exceeds the encoder encoding range. If it is greater than the maximum value of the range, use the maximum encoding parameter λ M Encoding, if it is less than the minimum value, it is encoded with the minimum encoding parameter λ1; Step 4-2, determine the upper and lower bounds of the target quality of the mini-GOP frame according to the (D, λ) point set obtained in step 2: transform the quality prediction vector D of each frame into Pred The minimum values ​​of the M elements of form a set, take the maximum value of the data in the set as the lower bound of the target quality, and transform the quality prediction vector D of each frame into pred The maximum values ​​of the M elements of form a set, and the minimum value of the data in the set is taken as the upper bound of the target quality; Step 4-3, using the binary search method to iterate continuously, changing the value of the target quality, obtaining the bit rate of each frame in the corresponding mini-GOP based on the target quality through formula (9) in each iteration, and calculating the total bit rate of the mini-GOP, until the number of binary search iterations is greater than the maximum value K or the target quality that meets the bit rate accuracy requirement is obtained; Step 4-4: The bit rate ratio of each frame in the mini-GOP obtained in the last iteration is used as the bit rate allocation ratio bit ratio , the expression is: bit ratio ={r1,r2,…,r N } (10) Where N is the number of frames in a mini-GOP, r1 represents the bit rate assigned to the first frame in the mini-GOP, and r N Indicates the code rate assigned to the Nth frame in the mini-GOP.

9. The method according to claim 8, characterized in that In step 5, the original target bit rate of each frame in the mini-GOP is obtained according to the total bit rate of the mini-GOP and formula (10): in, represents the original target bit rate of the jth frame in the mini-GOP, R mini-GOP Indicates the total bit rate of the mini-GOP, j ranges from 1 to N; The actual target bitrate for each frame is: Among them, among them, It represents the actual target bit rate of the jth frame in the mini-GOP. Buffer represents the buffer. Before encoding each mini-GOP, the buffer is initialized to 0. Substitute the actual target bit rate of each frame obtained by formula (12) into formula (7) to obtain the encoding parameters of each frame in the mini-GOP, and send the encoding parameters of each frame and the corresponding frame to be encoded to the encoder to start encoding. After encoding each frame, the value of the buffer is set to: in, is the actual encoding bit rate of the jth frame; When encoding the next frame, the actual target bitrate for the next frame The original target bit rate obtained by formula (11) plus the buffer buffer is obtained.

10. A deep video encoder bit rate control system based on rate distortion prediction implemented by the method according to any one of claims 1 to 9, characterized in that: It includes training module, prediction module, fitting module, bit rate control algorithm calculation module and encoding module; The training module is used to train the prediction module, and the prediction module includes a downsampling module, a rate predictor and a quality predictor. The downsampling module is used to downsample every two adjacent frames of the current group frame mini-GOP to be encoded to a low resolution. After the downsampled two adjacent frames are spliced ​​in the channel dimension, they are sent to the pre-designed rate predictor and quality predictor to obtain a set of (R, λ) point sets and a set of (D, λ) point sets; The fitting module fits the rate model and the quality model using the (R, λ) point set and the (D, λ) point set obtained in step 2 by the least square method, and obtains a rate-distortion curve; The rate control algorithm calculation module inputs the target rate, (R, λ) point set, (D, λ) point set and the fitted rate model, quality model and rate distortion curve into the rate control algorithm, and outputs the frame level rate allocation ratio of the mini-GOP frame to be encoded; The encoding module determines the encoding parameters of each frame in the mini-GOP according to the obtained frame level bit rate allocation ratio and the bit rate model obtained in step 3, and inputs the parameters into the video encoder to start encoding.

Images