Method and system for processing video
By adopting DISTS-based quality control scheme in VVC encoder, the problems of high computational complexity and difficulty in quality control in video encoding technology are solved, and more efficient video encoding performance and perceived quality improvement are achieved.
Patent Information
- Application Number
- CN202410602325.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-04-30
- Filing Date
- 2024-05-15
- Publication Date
- 2024-11-22
AI Technical Summary
Although existing video encoding technology, especially VVC, improves compression efficiency, it has high computational complexity and is difficult to effectively control video quality, resulting in problems of encoding speed and bit rate loss.
Adopting a quality control scheme based on depth image structure and texture similarity (DISTS), by establishing a DISTS-quantization model and rate-distortion model, the encoding parameters are adaptively adjusted to achieve the best rate-distortion performance, and open source encoding in VVC This solution is implemented on VVenc 1.6.0.
The R-D performance and perceived quality of video encoding are improved, and the bit rate is saved compared to the existing methods, which significantly improves the subjective quality of video.
Smart Images

Figure CN119011833A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method and system for processing video. Background Art
[0002] In recent years, the demand for multimedia applications has continued to increase, and a number of video coding standards have emerged that aim to improve video compression efficiency. One of the most advanced video coding standards is Versatile Video Coding (VVC) [1], which improves compression efficiency compared to High Efficiency Video Coding (HEVC) [2]. VVC uses a series of advanced coding tools to improve coding performance (but at the cost of high computational complexity). In the VVC test model (VTM-17.2), the coding complexity is increased by more than six times compared to the HEVC reference software (HM-16.24). In practical applications, the Fraunhofer Versatile Video Coder (VVenC) [3] is an open source VVC encoder implementation developed by Fraunhofer HHI with system optimization. In VVenc 1.6.0, the encoding speed is more than 15 times faster than VTM-17.2, and at slower preset values, the BD-Rate ( The loss of rate is less than 2%. VVenc provides a feasible way for academia and industry to access VVC, enabling researchers to work with higher efficiency.
[0003] As people's demand for high-quality video grows, there is a need to improve the quality control of video encoding.
[0004] References
[0005] The following references are cited throughout this specification as indicated by the numbers contained in square brackets, and their entireties are incorporated herein by reference:
[0006] [1] B.Bross, Y.-K.Wang, Y.Ye, S.Liu, J.Chen, GJSullivan, and J.-R.Ohm, "Overview of the versatile video coding (VVC) standard and its applications," IEEE Transactions on Circuits and Systems for Video Technology, vol.31, no.10, pp.3736-3764, 2021.
[0007] [2]G.J.Sullivan,J.Ohm,W.Han,and T.Wiegand,“Overview of the highefficiency video coding(HEVC)standard,"IEEE Transactions on Circuits andSystems for Video Technology,vol.22,no.12,pp.1649-1668,2012.
[0008] [3]A.Wieckowski,J.Brandenburg,T.Hinz,C.Bartnik,V.George,Hege,C.Helmrich,A.Henkel,C.Lehmann,C.Stoffers,I.Zupancic,B.Bross,and D.Marpe,“Vvenc:An open and optimized Vvc encoder implementation,"in 2021 IEEEInternational Conference on Multimedia Expo Workshops(ICMEW),pp.1-2.
[0009] [4]L.Li,B.Li,H.Li,and C.W.Chen,“λ-domain optimal bit allocationalgorithm for high efficiency video coding,"IEEE Transactions on Circuits andSystems for Video Technology,vol.28,no.1,pp.130-142,2018.
[0010] [5]M.Zhou,X.Wei,S.Kwong,W.Jia,and B.Fang,“Rate control method basedon deep reinforcement learning for dynamic video sequences in HEVC,"IEEETransactions on Multimedia,vol.23,pp.1106-1121,2020.
[0011] [6]Z.He,Y.Kim,and S.K.Mitra,“Low-delay rate control for DCT videocoding viaρ-domain source modeling,"IEEE Transactions on Circuits and Systemsfor Video Technology,vol.11,no.8,pp.928-940,2001.
[0012] [7]B.Li,H.Li,L.Li,and J.Zhang,“λ-domain rate control algorithm forhigh efficiency video coding,"IEEE Transactions on Image Processing,vol.23,no.9,pp.3841-3854,2014.
[0013] [8]Y.Mao,M.Wang,S.Wang,and S.Kwong,“High efficiency rate control forversatile video coding based on composite cauchy distribution,"IEEETransactions on Circuits and Systems for Video Technology,2021.
[0014] [9]F.Liu and Z.Chen,“Multi-objective optimization of quality in vvcrate control for low-delay video coding,"IEEE Transactions on ImageProcessing,vol.30,pp.4706-4718,2021.
[0015]
[10] M.Zhou,X.Wei,C.Ji,T.Xiang,and B.Fang,“Optimum quality controlalgorithm for versatile video coding,"IEEE Transactions on Broadcasting,vol.68,no.3,pp.582-593,2022.
[0016]
[11] Z.Wang,A.C.Bovik,H.R.Sheikh,and E.P.Simoncelli,“Image qualityassessment:from error visibility to structural similarity,"IEEE transactionson image processing,vol.13,no.4,pp.600-612,2004.
[0017]
[12] K.Ding,K.Ma,S.Wang,and E.P.Simoncelli,“Image quality assessment:unifying structure and texture similarity,"IEEE transactions on patternanalysis and machine intelligence,2020.
[0018]
[13] H.Choi,J.Yoo,J.Nam,D.Sim,and I.V.Bajic,“Pixel-wise unified rate-quantization model for multi-level rate control,"IEEE Journal of SelectedTopics in Signal Processing,vol.7,no.6,pp.1112-1123,2013.
[0019]
[14] S.Ma,W.Gao,and Y.Lu,“Rate-distortion analysis for 264 / AVC videocoding and its application to rate control,"IEEE Transactions on Circuits andSystems for Video Technology,vol.15,no.12,pp.1533-1544,2005.
[0020]
[15] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, high-performance deep learning library," Advances in neural information processing systems, vol. 32, 2019.
[0021]
[16] M.Sugawara, S.-Y.Choi, and D.Wood, "ultra-high-definition television(Rec.ITU-R BT.2020): A generational leap in the evolution of television[standards in a nutshell]," IEEE Signal Processing Magazine, vol.31, no.3, pp.170-174, 2014.
[0022]
[17] F. Bossen, J. Boyce, K. Suehring, X. Li, and V. Seregin, "JVET common testconditions and software reference configurations for SDR video," JVET T2010, Oct. 2020. Summary of the invention
[0023] An exemplary embodiment of the present invention develops a quality control scheme based on deep image structure and texture similarity (DISTS) for VVC. More specifically, the frame-level relationship between the quantization parameter (QP) and the DISTS value (D) is first studied to establish a DISTS-quantization (DQ) model. Then, an adaptive quality control scheme is proposed to ensure optimal rate-distortion (RD) performance. The above-mentioned quality control scheme is implemented on VVC's open source encoder VVenc 1.6.0. Experimental results show that due to proper quality control, the quality control scheme proposed in the exemplary embodiment of the present invention achieves RD performance gain and better perceptual quality compared with some existing methods.
[0024] In a first aspect of the present invention, a computer-implemented method for processing a video is provided. The computer-implemented method comprises (a) determining a target frame-level quality required for a frame of a video to be encoded, the determination of the target frame-level quality being based at least on an RQ model associating a bit rate with a quantization step size and a quality-quantization model associating a quality measure with a quantization step size; and (b) determining one or more encoding parameters for encoding the frame based on the determined target frame-level quality. The target frame-level quality may be represented as a value. The target frame-level quality may be a target frame-level perceptual quality. The quality metric of the quality-quantization model may be a perceptual quality metric, such as a perceptual quality metric based on machine learning.
[0025] Optionally, the RQ model is a linear RQ model.
[0026] Optionally, the RQ model is composed of Defined as, where R is the bit rate, Q is the quantization step size, and γ is the model parameter of the RQ model.
[0027] Optionally, the quality-quantization model comprises a DQ model that relates a DISTS value to a quantization step size. The DISTS value is a measure of the perceived quality of an image.
[0028] Alternatively, the DQ model is defined as D = αQ β , where D is the DISTS value, Q is the quantization step size, and α and β are the model parameters of the DQ model.
[0029] Optionally, the computer-implemented method further comprises determining a target GOP-level quality required for a group of pictures (GOP) of the video. A GOP includes a plurality of frames, including frames to be encoded. Optionally, the determination of the target frame-level quality is further based on the determined target GOP-level quality. The target GOP-level quality may be represented as a value. The target GOP-level quality may vary as the GOP frames are encoded.
[0030] For example, the target frame level quality may be a target frame level DISTS value. For example, the target GOP level quality may be a target GOP level DISTS value.
[0031] Optionally, the computer-implemented method further comprises determining a sequence-level quality of a video sequence, the video sequence comprising a plurality of GOPs, including the GOP described above. The sequence-level quality may be represented as a value, such as a sequence-level DISTS value. Optionally, a target GOP-level quality required for the video GOP is determined based on the determined sequence-level quality. Optionally, the determination of the sequence-level quality (such as the DISTS value) is based on a plurality of CTU (coding tree unit)-level quality values (such as the DISTS values). For example, the sequence-level quality (such as the DISTS value) may be determined based on an average of a plurality of CTU-level DISTS values.
[0032] Optionally, determining a target frame-level quality (eg, a value) required for frames of the video includes allocating or distributing at least a portion of a target GOP-level quality (eg, a target GOP-level DISTS value) to a plurality of frames of the GOP.
[0033] Optionally, determining a target frame-level quality required for frames of the video includes determining the target frame-level quality (eg, a target frame-level DISTS value) while optimizing (eg, minimizing) a GOP-level RD cost function.
[0034] Optionally, a GOP-level RD cost function is defined based at least on a GOP-level Lagrangian multiplier of the GOP.
[0035] Optionally, the GOP-level Lagrangian multipliers are related to the target GOP-level quality at least through an RQ model and a DQ model.
[0036] Optionally, determining a target frame-level quality required for the video frame includes determining the target frame-level quality required for the video frame (eg, a target frame-level DISTS value) according to a GOP-level Lagrangian multiplier.
[0037] Optionally, the one or more encoding parameters used to encode the frame include a quantization parameter and a Lagrange multiplier.
[0038] Optionally, the quantization parameter is determined based on and QP = log X (Q)×A+B, where D is the target frame-level quality expressed as a target frame-level DISTS value, Q is the quantization step size, α and β are model parameters of the DQ model, QP is the quantization parameter, and A, B, and X are constants.
[0039] Optionally, the Lagrange multipliers are determined based on Where λ is the Lagrange multiplier, QP is the quantization parameter, and C, D, and E are constants.
[0040] Optionally, the above computer-implemented method further includes: (c) encoding the frame based on one or more determined encoding parameters.
[0041] Optionally, the above-mentioned computer-implemented method further includes: (d) determining the output bit rate and output quality (e.g., value, such as DISTS value) of the frame based on the encoding of the frame; and (e) updating one or more model parameters of the RQ model and the quality-quantization model (e.g., DQ model) based on the determined output bit rate and output quality (e.g., value, such as DISTS value).
[0042] Optionally, the updating in step (e) is performed based on a gradient descent update method.
[0043] Optionally, the frame is a first frame, and the computer-implemented method further comprises: determining a target frame-level quality required for a second frame of the video to be encoded, the target frame-level quality being determined based at least on an RQ model and a quality-quantization model and updated model parameters; and determining one or more encoding parameters for encoding the second frame based on the determined target frame-level quality. Optionally, the first frame and the second frame are consecutive frames of the video.
[0044] Optionally, the computer-implemented method further comprises: performing or repeating steps (a) to (e) for a plurality of (e.g., substantially all) frames of the video. Optionally, the computer-implemented method is performed for at least 50%, at least 60%, at least 70%, at least 80%, at least 90%, etc. of all frames of the video.
[0045] Optionally, the encoding in (c) is performed based on VVC technology, such as VVenc-based technology.
[0046] The computer-implemented method of the first aspect described above may be applicable to the VVC standard, among other standards.
[0047] In a second aspect of the present invention, a system for processing video is provided, the system comprising one or more processors and a memory storing one or more programs. The programs are configured to be executed by the one or more processors, and the programs include instructions for executing or facilitating the execution of the computer-implemented method of the first aspect described above.
[0048] In a third aspect of the present invention, a non-transitory computer-readable storage medium is provided. The medium stores one or more programs configured to be executed by one or more processors. The one or more programs include instructions for executing or facilitating the execution of the computer-implemented method of the first aspect described above.
[0049] Other features and aspects of the present invention will become apparent from the following description of specific embodiments and the accompanying drawings of the specification. Any feature described herein in relation to one aspect or embodiment may be combined with any other feature described herein in relation to any other aspect or embodiment, as appropriate and applicable. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Embodiments of the present invention will now be described, by way of example, with reference to the accompanying drawings.
[0051] Figure 1AThe DQ relationship of the DQ model used by the I slice of the video sequence "BQMall" in one embodiment of the present invention is shown.
[0052] Figure 1B The DQ relationship of the DQ model used by the B slice of the video sequence "BQMall" in one embodiment of the present invention is shown.
[0053] Figure 1C The DQ relationship of the DQ model used by the I slice of the video sequence "Cactus" in one embodiment of the present invention is shown.
[0054] Figure 1D The DQ relationship of the DQ model used by the B slice of the video sequence "Cactus" in one embodiment of the present invention is shown.
[0055] Figure 2 The operation of a video quality control algorithm in one embodiment of the present invention is demonstrated.
[0056] FIG. 3A to FIG. 3C Shows a fixed QP ( Figure 3A ), a method proposed in one embodiment of the present invention ( Figure 3B ) and Zhou et al.’s method ( Figure 3C ), subjective quality comparison on 400x400 patches of POC 230 selected from the video sequence “RitualDance”.
[0057] FIG. 4A to FIG. 4C Shows a fixed QP ( Figure 4A ), a method proposed in one embodiment of the present invention ( Figure 4B ) and Zhou et al.’s method ( Figure 4C ) Subjective quality comparison on 200x200 patches of POC 257 selected from the video sequence "BasketballDrill".
[0058] Figure 5 is a schematic diagram of an example of a data processing system for performing or facilitating the performance of one or more methods in accordance with one or more embodiments of the present invention.
[0059] Figure 6 is a flow chart of a computer-implemented method for processing video in certain embodiments of the present invention. and
[0060] Figure 7 is a flow chart of a computer-implemented method for processing video in some embodiments of the present invention.
[0061] Before explaining any embodiments of the present invention in detail, it should be understood that the application of the present invention is not limited to the implementation details and component arrangements described in the following description or shown in the following figures. The present invention may have other embodiments and may be implemented or performed in various ways. In addition, it should be understood that the words and terms used herein are for descriptive purposes only and should not be considered as limiting. DETAILED DESCRIPTION
[0062] Hereinafter, some exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.
[0063] VVC Perception Quality Control Based on DISTS
[0064] Rate control is an important operation in video coding, especially for providing the best / optimal video quality under bitrate constraints. Rate control generally consists of two main steps: bit allocation and parameter derivation. Considering the overall target bitrate, it is necessary to allocate the bitrate at the group of pictures (GOP) level and the frame level respectively. The principle is to find an allocation scheme that minimizes the total RD cost. Some existing rate control algorithms [4], [5] extend the bit allocation to the CTU level to obtain improved RD performance. Given the target bitrate, the quantization parameter (QP) and λ of the coded frame and each CTU can be derived. Here, the rate control algorithm establishes a robust model between the target bitrate and the coding parameters. According to the model type, the rate control algorithms can be divided into three categories. These three rate control algorithms include Q-domain rate control, ρ-domain rate control and λ-domain rate control. In order to establish the RQ model, Q-domain rate control has been studied since MPEG-2 adopted TM5. Since then, Q-domain rate control has been used in commercial codecs due to its high accuracy and ease of implementation. He et al. [6] developed ρ-domain rate control by establishing a linear relationship between the bit rate and the percentage of zero coefficients. ρ-domain rate control can achieve better coding performance. Li et al. [7] proposed λ-domain rate control, attempting to establish a rate-λ (R-λ) relationship. Later, λ-domain rate control was further developed by adopting an optimized bit allocation scheme [4]. Due to accurate parameter estimation and reasonable bit allocation, λ-domain rate control achieves good RD performance and is adopted by HEVC and VVC. The bit allocation scheme in λ-domain rate control can be further improved by considering frame dependency [8].
[0065] As people's demand for high-quality videos increases, the quality fluctuation of compressed videos has become an increasingly important issue, especially for videos with high resolution and fast frame rates. Similar to rate control algorithms, quality control also minimizes the consumed bit rate under limited video quality and can be applied to high-quality video encoding without bandwidth constraints. Liu[9] proposed a multi-objective optimization method that takes into account bit rate, overall distortion, and quality fluctuation to achieve a balance among the three multi-objectives. Zhou
[10] proposed a quality control algorithm that can accurately guarantee the objective quality and RD performance of each frame.
[0066] VVC uses peak signal-to-noise ratio (PSNR) as a measure of video quality. However, PSNR, derived from mean square error (MSE), compares the pixel-by-pixel differences between two frames, which may have poor correlation with the human visual system (HVS). The structural similarity (SSIM) index
[11] takes structural similarity into account. Recently, machine learning-based image quality assessment (IQA) methods have been explored that can learn the differences between texture features. Ding et al.
[12] proposed DISTS, which unifies structural and texture similarity and achieves good results compared to manual quality ratings.
[0067] In this example, a DISTS-based quality control scheme is designed to achieve better visual quality on the VVenc 1.6.0 platform. In this example, the DISTS-quantization (DQ) relationship is first studied. Then, a quality allocation scheme is designed at the image level to achieve better RD performance. Given a target DISTS value for a frame, the QP is derived based on the DQ model. Finally, experiments are conducted to verify the effectiveness of the proposed method in this example.
[0068] I. Rate and Distortion Model
[0069] This section will study the relationship between distortion indicators and coding parameters, and further establish a quality control scheme based on this. As mentioned earlier, according to the parameters selected for model building, the existing rate control algorithms include Q-domain rate control and λ-domain rate control. Considering the fitting accuracy and model robustness, the coding parameter QP is used as the key parameter for building DQ and RQ models.
[0070] A. DQ model used
[0071] In this example, two sequences “BQMall” and “Cactus” are first encoded with four different QPs in a random access (RA) configuration. After encoding each CTU, the original pixels and reconstructed pixels are sent to the neural network provided in
[12] for processing to obtain the corresponding CTU-level DISTS value. In one example, the overall sequence-level DISTS value can be obtained by calculating the average of all CTU-level DISTS values. The obtained sequence-level DISTS values and the corresponding quantization step size (Q) are shown in Figures 1A to 1D As shown. Figures 1A to 1D It can be seen that the hyperbolic DQ function has achieved high fitting accuracy for both the I band and the B band. At the same time, the parameter update of the hyperbolic function is relatively convenient. The DQ model used in this example can be written as
[0072] D=αQ β (1)
[0073] Among them, α and β are model parameters, and Q is the quantization step size.
[0074] B. Using the RQ model
[0075] A variety of RQ models have been proposed [8],
[13] ,
[14] . The RQ model used in this example is as follows:
[0076]
[0077] Where γ is a model parameter. In this example, formula (2) is used as the RQ model.
[0078] II. Quality Control Program
[0079] In view of the RQ and DQ models, a DISTS-based quality control scheme is established considering the overall RD performance. First, DISTS values are distributed at different granularities to obtain the target DISTS value for each frame. Then, the encoding parameters are derived according to the target DISTS value.
[0080] A. Quality Distribution
[0081] In this example, quality allocation is performed at both GOP level and frame level, with the goal of assigning an appropriate target DISTS value to each frame.
[0082] 1) GOP-level quality allocation
[0083] Given a sequence-level DISTS value D seq , a sliding window-based quality allocation scheme is used to smooth the target DISTS value of each GOP The calculation formula is:
[0084]
[0085] Among them, D C is the sum of the DISTS values of the coded frames. coded and N SW are the number of coded frames and the size of the sliding window, respectively. GOP is the number of frames in a GOP. GOP is the target average DISTS value for each GOP, calculated as:
[0086]
[0087] Among them, N GOP is the number of GOPs in a sequence.
[0088] 2) Frame-level quality allocation
[0089] Given a GOP-level target DISTS value The target DISTS value D needs to be assigned to the i-th frame i This mass allocation problem can be formulated as an optimization problem and can be written as:
[0090]
[0091] Among them, D i and R i denote the DISTS value and consumed bit rate of the i-th frame respectively. In [4], this constrained optimization problem can be transformed into an unconstrained problem using Lagrange multipliers. According to [4], it is necessary to minimize the GOP-level RD cost J, which is defined as:
[0092]
[0093] Among them, λ GOP is the GOP-level Lagrange multiplier used for this optimization problem, and N is the number of frames in a GOP. i is obtained for the i-th frame while minimizing J. The solution in equation (8) is given as follows:
[0094]
[0095] Among them, Q i is the quantization step size of the i-th frame, k i is the influence factor of the i-th frame, which is used to show the importance of the frame. Generally speaking, k i The higher frame will be used as low k i The reference frame of the frame. It is a predefined constant related to the temporal layer of the i-th frame. Combining formula (7), formula (1) and formula (2), we can get λ GOP -Qi The relationship is
[0096]
[0097] Among them, α i , β i and γ i are the model parameters of the i-th frame.
[0098] Combining formula (8) and formula (1), we get λ GOP -D i The model can be written as:
[0099]
[0100]
[0101] Considering D i The sum of can be written as:
[0102]
[0103] Combining formula (10) and formula (11), λ GOP This can be obtained by solving the following equation:
[0104]
[0105] Considering that the right side of formula (12) is about λ GOP is a monotonically increasing function, so the method in [4] is used to solve formula (12). We get λ GOP Then, the corresponding D of the i-th frame is calculated according to formula (10): i .
[0106] B. Encoding parameters
[0107] Given the target DISTS value D of the i-th frame i , the corresponding quantization step size can be obtained as follows:
[0108]
[0109] Finally, the relevant coding parameter QP of the i-th frame can be obtained i and λ i for:
[0110] QP i =log2(Q i )×6+4 (14)
[0111]
[0112] C. Update plan
[0113] After the i-th frame is encoded, the model parameters are updated according to the actual encoding results. Assume that the i-th frame is encoded with the parameter QP i and λ i , encoding, the output bit rate and DISTS values are and α i β i and λ i The update of adopts the gradient descent update method proposed in the literature [7]. The update scheme is as follows:
[0114]
[0115]
[0116]
[0117]
[0118] in, and is the updated model parameter. Q i It is QP i The corresponding quantization step size. α and δ β is the update ratio of α and β, which are predefined constants. and Quality allocation and coding parameter derivation for the next frame in the same spatiotemporal layer.
[0119] D. Implementation Details
[0120] In an example, the proposed quality control scheme is implemented on VVenc 1.6.0. The DISTS neural network
[12] is integrated with LibTorch
[15] supporting both CPU and GPU platforms. The workflow is as follows Figure 2 As shown in the figure, the DISTS value estimation is performed in the RGB domain, while the input sequence is a YUV file. The input sequence is converted to the RGB domain according to the Rec.BT.2020
[16] configuration.
[0121] III. Experiment
[0122] The proposed quality control algorithm was tested in the Common Test Conditions (CTCs)
[17] settings under RA configuration (GOP size = 32) and fast preset. The test sequences are also specified in CTCs
[17] . All experiments were performed on Intel Core i7-8700K and GeForce RTX 2080Ti platforms without frame-level parallelism. The test sequences were first encoded using a fixed QP configuration, with QP set to 22, 27, 32 and 37 respectively. The corresponding DISTS value was calculated and set as the target DISTS value for quality control. To evaluate the encoding performance, the DISTS-based BD-Rate was used, which represents the bitrate saved at the same DISTS quality. The calculation formula for Err(DISTS) is:
[0123]
[0124] Among them, D (o) is the DISTS value of the output sequence, D (t) is the corresponding target DISTS value.
[0125] A. Coding Performance
[0126] Table 1 shows the encoding performance of the proposed algorithm compared with the fixed QP and Zhou et al.'s method, respectively. Due to the optimal quality allocation scheme, the proposed method saves 2.78% BD-Rate compared with Zhou et al.'s method in VVenc 1.6.0. In particular, for level A1 sequences, the proposed method can save 3.89% BD rate. Compared with the fixed QP configuration, the BD-Rate is improved by 4.89%. It can be seen that since the calculation of the DISTS value is based on 64x 64 blocks, the proposed method will suffer BD-Rate loss in high-resolution sequences. For frames composed of more CTUs, the prediction accuracy of the updated DQ model will decrease. For these sequences, a CTU-level quality control scheme is needed to improve the encoding performance.
[0127] Table 1 – Description of the BD rates of the proposed rate control algorithm compared with the method of Zhou et al.
[10] and fixed QP configuration
[0128]
[0129]
[0130] B. Quality Control
[0131] The quality control accuracy of the proposed method and the method of Zhou et al. are shown in Table II. It is found that the overall quality control accuracy decreases by 0.62%. This accuracy loss is mainly caused by the RD optimization (RDO) process in mass assignment. The main goal of mass assignment is to minimize the total RD cost (Eq. (6)) rather than to minimize the quality control error. In addition to the RDO process, the rounding process, D i Derivation of QP i λ i The calculation of can also lead to quality control errors.
[0132] Table 2 - Illustration of the quality control accuracy of the proposed rate control algorithm compared with the method of Zhou et al.
[10]
[0133]
[0134] C. Subjective quality
[0135] Figures 3A to 4C shows a fixed QP (see Figure 3A and 4A ), the proposed quality control algorithm (see Figure 3B and 4B ) and the method of Zhou et al. (see Figure 3C and 4C ) subjective quality. The pictures shown are the reconstructed frames of the three methods. The three pictures in the first row (top) (i.e. FIG. 3A to FIG. 3C ) is a 400x 400 block selected from POC 230 of "RitualDance", with a target DISTS value equal to 0.218. The three images in the second row (below) (i.e. Figures 4A to 4C ) is a 200x 200 patch selected from POC257 of “BasketballDrill” with a target DISTS value equal to 0.177. It can be seen that in this example, the proposed method shows the best subjective quality of the textures of the clothes and the basketball.
[0136] As mentioned above, in this example, a DISTS
[12] -based quality control method is established for VVenc. By simulating the relationship between DQ and RQ, an optimal target DISTS value is assigned to each frame. Then, the encoding parameters are derived based on the target DISTS value and the corresponding DQ model. The proposed quality control algorithm is implemented in VVenc 1.6.0. Experiments show that compared with the method of Zhou et al.
[10] , the proposed quality control algorithm can achieve a 2.78% DISTS-based BD-Rate saving with tolerable quality errors. Subjective experiments also verify the effectiveness of the proposed method in this example.
[0137] IV. System
[0138] Figure 5 An example of a data processing system 500 is shown, which may be used as a server or other type of information or data processing system in one embodiment of the present invention. Data processing system 500 may be used to perform one or more method embodiments of the present invention.
[0139] The data processing system 500 generally includes appropriate components required to receive, store and execute appropriate computer instructions, commands and / or codes. The main components of the data processing system 500 are a processor 502 and a memory (storage) 504. The processor 502 may include one or more of the following: one or more CPUs, one or more MCUs, one or more GPUs, one or more logic circuits, one or more Raspberry Pi chips, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other digital or analog circuits / circuits configured to interpret and / or execute program instructions and / or process signals and / or information and / or data. The memory 504 may include one or more volatile memories (such as RAM, DRAM, SRAM, etc.), one or more non-volatile memories (such as ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM, etc.) or any combination thereof. Appropriate computer instructions, commands, codes, information and / or data may be stored in the memory 504. Computer instructions for executing or facilitating the execution of the method embodiments of the present invention may be stored in the memory 504. The video file and related data may be stored in the memory 504. The RQ model, DQ model, etc. may be stored in the memory. The processor 502 and the memory (storage) 504 may be integrated or separated (and operably connected). Optionally, the data processing system 500 further includes one or more input devices 506. Examples of such input devices 506 include: keyboard, mouse, stylus, image scanner, microphone, tactile / touch input device (such as touch sensitive screen), image / video input device (such as camera), etc. The input device 506 may be used to receive user input on the quality of images, frames, GOP, etc. Optionally, the data processing system 500 further includes one or more output devices 508. Examples of such output devices 508 include: display (such as monitor, screen, projector, etc.), speaker, earphone, headset, printer, additive manufacturing machine (such as 3D printer), etc. The display screen may include an LCD display screen, an LED / OLED display screen, or other suitable display screen, which may be touch sensitive or non-touch sensitive. The processed image or video (frame, GOP, etc.) may be displayed on the display. Data processing system 500 may further include one or more disk drives 512, which may include one or more of a solid-state drive, a hard disk drive, an optical drive, a flash drive, a tape drive, etc. A suitable operating system may be installed in data processing system 500, for example, on disk drive 512 or in memory 504. Memory 504 and disk drive 512 may be operated by processor 502.Optionally, the data processing system 500 further includes a communication device 510 for establishing one or more communication links (not shown) with one or more other computing devices (such as servers, personal computers, terminals, tablet computers, mobile phones, watches, IoT devices or other wireless computing devices). The communication device 510 may include one or more of the following devices: a modem, a network interface card (NIC), an integrated network interface, an NFC transceiver, a ZigBee transceiver, a Wi-Fi transceiver, etc. Transceiver, radio frequency transceiver, cellular (2G, 3G, 4G, 5G, 5G or above or similar) transceiver, optical port, infrared port, USB connection or other wired or wireless communication interface. The transceiver can be implemented by one or more devices (integrated transmitter and receiver, independent transmitter and receiver, etc.). The communication link can be wired or wireless and is used to communicate commands, instructions, information and / or data. In one example, the processor 502, memory 504 (optionally input device 506, output device 508, communication device 510 and disk drive 512 (if any)) are directly or indirectly connected to each other via a bus, peripheral component interconnect (PCI) (such as PCI Express), universal serial bus (USB), optical bus or other similar bus structure. In one embodiment, at least some of these components can be connected wirelessly, for example via a network such as the Internet or a cloud-based computing network. It will be appreciated by those skilled in the art that Figure 5 The data processing system 500 shown is merely an example, and in other embodiments, data processing system 500 may have a different configuration (eg, include additional components, have fewer components, etc.).
[0140] V. Examples
[0141] Figure 6 A computer-implemented method 600 for processing video in accordance with certain embodiments of the present invention is shown.
[0142] Method 600 begins in step 602, where a target frame-level quality required for a video frame to be encoded is determined based on an RQ model and a quality quantization model. The target frame-level quality may be related to a target frame-level perceptual quality (quality perceivable by humans), which may be quantified or expressed as a value.
[0143] In these embodiments, the RQ model is a model that relates the bit rate to the quantization step size (i.e., determines the relationship between the bit rate and the quantization step size). In some embodiments, the RQ model is a linear RQ model. In some embodiments, the RQ model is composed of Defined as, where R is the bit rate, Q is the quantization step size, and γ is the model parameter of the RQ model.
[0144] In these embodiments, the quality quantization model is a model that relates the quality metric to the quantization step size (i.e., determines the relationship between the quality metric and the quantization step size). The quality metric of the quality quantization model can be a perceptual quality metric, such as a perceptual quality metric based on machine learning. In some embodiments, the quality quantization model is a DQ model that relates the DISTS value to the quantization step size. In some embodiments, the DQ model is defined as D=αQ β , where D is the DISTS value, Q is the quantization step size, and α and β are model parameters of the DQ model. In some embodiments, the target frame-level quality may be a target frame-level DISTS value.
[0145] In some embodiments, the determination of the target frame-level quality is further based on the determined target GOP-level quality. Thus, in some embodiments, method 600 may further include determining a target GOP-level quality required for a group of pictures (GOP) of the video. A GOP includes a plurality of frames, including a frame to be encoded. For example, the target GOP-level quality may be a target GOP-level DISTS value. The target GOP-level quality may vary with the encoding of the GOP frame.
[0146] In some embodiments, determining a target GOP-level quality required for a video GOP is based on a determined sequence-level quality. Thus, in some embodiments, method 600 may further include determining a sequence-level quality for a video sequence. Here, the video sequence includes a plurality of GOPs, the plurality of GOPs including a GOP in which a frame to be encoded is located. The sequence-level quality may be represented as a value, such as a sequence-level DISTS value. For example, the determination of the sequence-level quality (such as a DISTS value) may be based on a plurality of CTU-level quality values (such as a DISTS value). For example, the determination of the sequence-level quality (such as a DISTS value) may be based on an average of the CTU-level DISTS values.
[0147] In some embodiments, determining a target frame-level quality (e.g., a value) required for a frame of the video includes allocating or assigning at least a portion of a target GOP-level quality (e.g., a value) to a plurality of frames of the GOP. In some embodiments, determining a target frame-level quality required for a frame of the video includes determining the target frame-level quality (e.g., a value) while optimizing (e.g., minimizing) a GOP-level RD cost function. In some embodiments, the GOP-level RD cost function is defined based on at least a GOP-level Lagrangian multiplier of the GOP. In some embodiments, the GOP-level Lagrangian multiplier is related to the target GOP-level quality by an RQ model and a DQ model. In some embodiments, determining a target frame-level quality required for a frame of the video includes determining a target frame-level quality (e.g., a value) required for a frame of the video based on the GOP-level Lagrangian multiplier.
[0148] After step 602, in step 604, encoding parameters for encoding the frame are determined according to the target frame-level quality determined in step 602. In some embodiments, the encoding parameters include quantization parameters and Lagrangian multipliers (for the frame to be encoded), so in step 604, determining the encoding parameters for encoding the frame includes determining the quantization parameters and Lagrangian multipliers for encoding the frame. In some embodiments, the quantization parameters are determined based on and QP = log X (Q)×A+B, where D is the target frame-level quality expressed as a target frame-level DISTS value, Q is the quantization step size, α and β are model parameters of the DQ model, QP is the quantization parameter, and A, B, and X are constants. In some embodiments, the Lagrangian multiplier is determined based on Where λ is the Lagrange multiplier, QP is the quantization parameter, and C, D, and E are constants.
[0149] In step 606, the encoder encodes the frame according to the determined encoding parameters. In some embodiments, the encoding in step 606 is performed based on VVC technology, such as VVenc-based technology.
[0150] In step 608, the output bit rate and output quality of the frame (encoded in step 606) are determined based on the encoding of the frame. The output quality of the frame may be represented as a value, such as a DISTS value.
[0151] In step 610, model parameters of the RQ model and the quality quantization model (eg, DQ model) are updated according to the determined output bit rate and output quality. In some embodiments, the updating is performed based on a gradient descent updating method.
[0152] In step 612, it is determined whether there are more video frames to be processed. If there are no video frames to be processed, the method 600 ends. If there are still one or more frames of video to be processed, the method 600 returns to step 602 to process another frame of video (using the model with updated model parameters). The other frame of video can be a frame that is continuous with the previous frame being encoded.
[0153] Method 600 may be performed for at least some (e.g., most, if not all) frames of a video. For example, method 600 may be performed for at least 50%, at least 60%, at least 70%, at least 80%, or at least 90% of all frames of a video.
[0154] It should be understood that in some other embodiments, the order of the method steps of method 600 may be modified. For example, two or more method steps may be performed in a different order or substantially simultaneously, as long as the skilled person understands that this is possible and feasible. For example, step 612 may be performed immediately after step 606, and if it is determined that no frames need to be processed, method 600 may end without further performing step 608 and / or step 610.
[0155] Figure 7 A computer-implemented method 700 for processing video in accordance with certain embodiments of the present invention is shown.
[0156] The method 700 includes initializing the RQ model and the DQ model in step 701. The RQ model and the DQ model may be the same as those described above with reference to Figure 6 any of the models described, so for the sake of brevity, the details are not provided here.
[0157] In step 702, the method 700 further includes analyzing the content of the video to be processed. The video to be processed includes a video sequence having a plurality of GOPs, each GOP including a plurality of frames. In one embodiment, the analysis may include determining the content of the video to be processed. Figure 6 The sequence-level quality (eg, a value, such as a sequence-level DISTS value).
[0158] The method 700 further includes performing a GOP-level quality allocation in step 704 to allocate a quality (eg, a value) to a GOP of the video. Such allocation may be based on the sequence-level quality (eg, a sequence-level DISTS value) obtained in step 702.
[0159] The method 700 also includes performing a frame-level quality assignment in step 706 to assign a quality (e.g., a value) to one of the frames in the video GOP that has been assigned a GOP-level quality (e.g., a value) in step 704. This assignment may be based on the GOP-level quality (e.g., a value, such as a GOP-level DISTS value) of the GOP as well as the RQ model and the DQ model.
[0160] The method 700 further includes determining, at step 708, frame-level encoding parameters, including a quantization parameter QP and a Lagrangian multiplier λ, for the frame to which the frame-level quality (eg, value) has been assigned at step 706. This determination may be based on a DQ model.
[0161] The method 700 further includes encoding the frame according to the determined frame-level encoding parameters (quantization parameter QP and Lagrange multiplier λ) in step 710. The encoding may be performed using a VVC-based encoder according to the VVC encoding technique.
[0162] After encoding the frame, the method 700 determines, in step 712, an actual bit rate used to encode the frame and a quality (eg, a value, such as a DISTS value) of the encoded frame.
[0163] Subsequently, the method 700 updates one or more model parameters of the RQ model and the DQ model using the actual bit rate for the encoded frame and the quality of the encoded frame (eg, values such as DISTS values) in step 714 for processing the next frame of the video.
[0164] In step 716, method 700 determines whether there are any frames in the current GOP (i.e., the GOP associated with the frame processed in step 710) that have not been processed. If it is determined that at least one frame has not been processed, method 700 returns to step 704 to process the next frame. If it is determined that no additional frames in the GOP are to be processed, method 700 proceeds to step 718 to determine whether there is a frame in the video GOP that is to be processed. If it is determined that at least one GOP is to be processed, method 700 returns to step 704 to process the next GOP (of multiple frames). If it is determined that there are no more GOPs in the video that need to be processed, method 700 ends.
[0165] It should be understood that in some other embodiments, the order of the method steps of method 700 may be modified. For example, two or more method steps may be performed in a different order or substantially simultaneously, as long as the skilled person understands that this is possible and feasible. For example, steps 716 and 718 may be performed immediately after step 710, and if it is determined that no frames and GOPs remain to be processed, method 700 may be terminated without further performing other steps.
[0166] refer to Figures 1A to 1D , 2, 3A-3C and 4A-4C (entitled "DISTS-Based Perceptual Quality Control for VVC") provide disclosures that illustrate computer-implemented methods for processing video in certain embodiments of the present invention. In some embodiments, reference is made to Figures 1A to 1D , 2, 3A-3C and 4A-4C may be considered as example implementations of method 600 or method 700. However, it should be noted that method 600, method 700 or the present invention more generally may have other embodiments or implementations than those specifically illustrated and / or described.
[0167] Although not required, the embodiments described with reference to the accompanying drawings may be implemented as an application programming interface (API) or a series of libraries for developers to use, or may be included in another software application, such as a terminal or computer operating system or a portable computing device operating system. Generally speaking, since program modules include routines, programs, objects, components, and data files that assist in performing specific functions, technicians can understand that the functions of a software application can be distributed among multiple routines, objects, and / or components to achieve the same functions required herein.
[0168] In addition, it is also understood that if the methods and systems of the present invention are implemented entirely or in part by a computing system, any suitable computing system architecture may be used. This includes stand-alone computers, network computers, dedicated or non-dedicated hardware devices. When the terms "computing system" and "computing device" are used, these terms are intended to include (but not limited to) any suitable arrangement of computers or information processing hardware capable of implementing the described functions.
[0169] Some embodiments of the present invention provide a video codec to improve encoding performance and facilitate video transmission and storage under limited resources.
[0170] Some embodiments of the present invention provide a frame-level quality control scheme suitable for VVC. The quality control scheme in some embodiments of the present invention adopts perceptual quality. Some embodiments of the present invention provide or use a DQ model to predict the DISTS value of a coded frame. Some embodiments of the present invention provide an optimal quality allocation scheme for a quality control algorithm.
[0171] Some example features of certain embodiments of the invention include:
[0172] In some embodiments, quality control may minimize the consumed bit rate while maintaining perceived quality.
[0173] In certain embodiments, perceptual quality is utilized to overcome the shortcomings of traditional quality indicators.
[0174] The DISTS-Q model in certain embodiments may accurately predict the characteristics of a VVC encoder.
[0175] In certain embodiments, optimal mass allocation is achieved by utilizing estimated model parameters.
[0176] Certain embodiments may save bitrate and / or improve subjective quality, among other things, compared to other existing approaches.
[0177] Some embodiments of the present invention aim to minimize the bit rate consumption while ensuring the perceived quality. The scheme proposed in some embodiments can provide better RD performance and visual quality improvement.
[0178] Compared to existing algorithms, certain embodiments of the present invention can achieve better coding performance and visual quality improvements.
[0179] It is expected that the computational complexity of certain embodiments of the present invention can be reduced by neural network-based calculations or techniques. It is also conceivable that the additional time cost associated with neural network-based calculations or techniques can be reduced by implementing on a GPU and CPU hybrid platform.
[0180] It will be appreciated by those skilled in the art that the described and / or illustrated embodiments of the present invention may be varied and / or modified to provide other embodiments of the present invention. Therefore, the embodiments described and / or illustrated in the present invention should be considered in all respects as exemplary and non-restrictive. Examples of optional features of certain embodiments of the present invention are provided in the summary and description. Some embodiments of the present invention may include one or more of these optional features (some of which are not specifically described in the drawings). Some embodiments of the present invention may lack one or more of these optional features (some of which are not specifically described in the drawings). For example, some embodiments of the present invention may use other quality or perceptual quality measurement methods (except DISTS). For example, some embodiments of the present invention may be applied to other coding techniques, not limited to VVenc or more generally VVC.
Claims
1. A computer-implemented method for processing video, include: (a) determining a target frame-level quality required for a frame to be encoded of a video, the target frame-level quality being determined based at least on an RQ model associating a bit rate with a quantization step size and a quality-quantization model associating a quality metric with the quantization step size; and (b) determining one or more encoding parameters for encoding the frame based on the determined target frame level quality.
2. The computer-implemented method of claim 1, wherein the RQ model is defined as Wherein R is the bit rate, Q is the quantization step size, and γ is the model parameter of the RQ model. 3 . The computer-implemented method of claim 1 , wherein the quality-quantization model comprises a DQ model that relates a DISTS value to the quantization step size.
4. The computer-implemented method of claim 3, wherein the DQ model is defined as D = αQ β , where D is the DISTS value, Q is the quantization step size, and α and β are model parameters of the DQ model.
5. The computer-implemented method of claim 3 , further comprising determining a target GOP-level quality required for a GOP of the video, the GOP comprising a plurality of frames including the frame to be encoded, and in, The determination of the target frame level quality is further based on the determined target GOP level quality.
6. The computer-implemented method of claim 5, wherein determining the target frame-level quality required for the to-be-encoded frames of the video comprises allocating at least a portion of the target GOP-level quality to the plurality of frames of the GOP.
7. The computer-implemented method of claim 5, wherein determining the target frame-level quality required for the to-be-encoded frame of the video comprises determining the target frame-level quality while optimizing a GOP-level RD cost function.
8. The computer-implemented method of claim 7, wherein the GOP-level RD cost function is defined based on at least a GOP-level Lagrangian multiplier of the GOP.
9. The computer-implemented method of claim 8, in: The GOP-level Lagrangian multiplier is related to the target GOP-level quality through the RQ model and the DQ model; and / or Determining the target frame-level quality required for the frames of the video includes determining the target frame-level quality required for the frames of the video based on the GOP-level Lagrangian multipliers.
10. The computer-implemented method of claim 3, wherein the one or more encoding parameters include a quantization parameter and a Lagrangian multiplier.
11. The computer-implemented method of claim 10, wherein the determination of the quantization parameter in the step of determining one or more encoding parameters for encoding the frame is based on: and QP = log X (Q)×A+B in, D is the target frame-level quality represented by the target frame-level DISTS value; Q is the quantization step size; α and β are model parameters of the DQ model; QP is the quantization parameter; A, B and X are constants.
12. The computer-implemented method of claim 11, wherein the determination of the Lagrange multipliers in step (b) is based on: in, λ is the Lagrange multiplier, QP is the quantization parameter, and C, D and E are constants.
13. The computer-implemented method of claim 1, further comprising: include: (c) encoding the frame based on the determined one or more encoding parameters.
14. The computer-implemented method according to claim 13, wherein the encoding in step (c) is performed based on the general video coding (VVC) technology.
15. The computer-implemented method of claim 13, further comprising: include: (d) determining an output bit rate and an output quality of said frame based on said encoding of said frame; as well as (e) updating the model parameters of the RQ model and the quality-quantization model according to the determined output bit rate and output quality.
16. The computer-implemented method of claim 15, wherein the updating in step (e) is performed based on a gradient descent updating method.
17. The computer-implemented method of claim 15, further comprising: include: Steps (a) to (e) are performed or repeated for a plurality of frames of the video.
18. A system for processing video, include: one or more processors; as well as a memory storing one or more programs configured to be executed by the one or more processors; The one or more programs include instructions for executing or facilitating the execution of the computer-implemented method according to claim 1 .
19. A non-transitory computer-readable medium having instructions stored thereon; when executed by one or more processors, the instructions cause the one or more processors to perform the computer-implemented method of claim 1.
