VVC rate control optimization method based on hvs characteristics

Abstract

The present application discloses a VVC code rate control optimization method based on HVS characteristics from the two problems of bit allocation in VVC not matching human visual characteristics and the convergence step being too small when updating parameters and not considering absolute convergence. First, the spatial domain information is obtained by using the Sobel operator; the temporal domain information is adaptively obtained by using the MV information of the reference frame; a special case of temporal domain information extraction method is proposed; then, the frame target bits are divided into two parts, and the CTU level bit allocation is carried out by using the spatial domain information of each CTU; after the completion of a CTU or a frame coding, the quasi-Newton convergence algorithm is used for parameter updating, and the corresponding method is proposed for the convergence of special cases. The video sequence obtained based on the present application can better meet the needs of the human visual system, and the objective quality index and subjective quality can be greatly optimized, and the fluctuation degree of the output coded video is more stable.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video coding, and particularly relates to a method for rate control optimization of a versatile video coding (VVC) standard based on joint spatial-temporal domain information. BACKGROUND

[0002] In recent years, with the development of video towards ultra-clear and high dynamic range (HDR), H.264 and H.265 cannot fully meet the coding requirements of such videos. In order to solve this situation and cope with the development trend of video, the VVC standard is proposed. Experiments prove that it improves the coding efficiency and exceeds any previous video coding standard. In order to better adapt to high-definition video, VVC adopts a larger LCU (128x128) with flexible partitioning modes such as binary tree partitioning and ternary tree partitioning. In addition, new intra prediction modes are added, inter prediction techniques are enhanced, and prediction errors are reduced. In addition, it also optimizes the quantization step and uses multiple transform selection to compress the residual energy in the frequency domain to obtain better performance. These technologies are more suitable for the transmission of special videos such as panoramic videos and 3D videos, and the video coding tools and algorithms of H.266 are still being further optimized.

[0003] In the actual transmission process of video, the bandwidth is often limited. The purpose of the rate control model is to avoid overflow of the compressed coded stream in the buffer. It also ensures that the actual encoding bits are as close as possible to the target bits allocated, thereby minimizing the video quality distortion. The rate control model of H.265 adopts R-lambda algorithm model, R-Q algorithm model and R-p algorithm model, among which the bit fluctuation of the R-lambda algorithm model is smaller and the rate control effect is better. Therefore, the R-lambda algorithm model will also be used in the rate control algorithm of VVC.

[0004] The R-lambda rate control scheme is generally divided into two main processes: target bit allocation and model parameter update. The VVC rate control algorithm first allocates target bits for the GOP level according to the total number of bits, and then allocates target bits for the Frame level and the CTU level according to the position of each frame in the GOP and the complexity of the coding structure unit. After the target bits are allocated, the parameters inside the rate control model need to be updated so that the actual number of coded bits of the compressed video is close to the target number of bits allocated.

[0005] Firstly, the Mean Absolute Difference (MAD) in VVC is an important criterion for measuring the complexity of a coding unit, but the measurement standard of the human visual system (HVS) is inconsistent with the above-mentioned coding unit complexity criterion, resulting in a low subjective evaluation of the human eye for the encoded video. Moreover, VVC adopts the Least Mean Square (LMS) algorithm, and the convergence step of LMS is too small when updating the R-λ model parameters, which causes the updated model parameters to have a precision decline phenomenon for sequences with a dramatic displacement of moving targets and a sudden scene switching.

[0006] In view of the problems of the mismatch between the complexity measurement standard and the HVS in the R-λ model in VVC and the too small convergence step of the updating algorithm, the research group carried out exploratory research and improved two steps of the rate control model. Firstly, the human visual perception characteristics combined with distortion characteristics are introduced when the target bit allocation is performed, the space-time domain joint information is used to replace the original coding complexity, and finally a CTU-level bit allocation model is established. Secondly, the quasi-Newton convergence algorithm is used to replace the LM algorithm when updating the R-λ model parameters. This is because the quasi-Newton convergence algorithm is a second-order convergence, which has a better convergence step than LMS, and thus the parameters are accurately updated. After the above improvements to the R-λ model in VVC, the subjective quality of the encoded video and the overall coherence of the video sequence quality can be improved under the premise of accurate target bit allocation, and thus the purpose of rate control optimization is achieved, which is the task of the present application. SUMMARY

[0007] In view of the problems of the mismatch between the complexity measurement standard and the HVS in the R-λ model in VVC and the too small convergence step of the updating algorithm, the research group carried out exploratory research and improved two steps of the rate control model. Firstly, the human visual perception characteristics combined with distortion characteristics are introduced when the target bit allocation is performed, the space-time domain joint information is used to replace the original coding complexity, and finally a CTU-level bit allocation model is established. Secondly, the quasi-Newton convergence algorithm is used to replace the LM algorithm when updating the R-λ model parameters. This is because the quasi-Newton convergence algorithm is a second-order convergence, which has a better convergence step than LMS, and thus the parameters are accurately updated. After the above improvements to the R-λ model in VVC, the subjective quality of the encoded video and the overall coherence of the video sequence quality can be improved under the premise of accurate target bit allocation, and thus the purpose of rate control optimization is achieved, which is the task of the present application.

[0008] The regions in the image can be roughly divided into flat regions, texture regions, motion regions and non-motion regions. However, the degree of attention of the human visual system to these regions is not the same, that is, texture regions and motion regions are more likely to attract the attention of the human eye, and the encoding distortion in the texture region and the motion region is relatively larger than that in the flat region and the non-motion region. Therefore, more encoding bits are often required for the motion region and the texture region during encoding. However, VVC uses MAD as the measurement standard of the complexity of the encoding unit when allocating bits, and does not fully consider the characteristics of the human visual system and the bit allocation, resulting in a series of problems such as the decline of the subjective and objective quality of the output compressed video. The method proposed in the present application is to use the spatial and temporal information to replace the original MAD as the measurement standard of the encoding complexity in the process of VVC rate control based on the characteristics of the human visual system, so as to achieve the purpose of optimizing the CTU-level target bit allocation. Finally, the quasi-Newton convergence algorithm is used for parameter updating to improve the accuracy of parameter updating. Compared with VVC, the method proposed in the present application can improve the subjective quality of the encoded video and the overall coherence of the video sequence quality while ensuring the accuracy of the target bit allocation.

[0009] The present application patent proposes an optimization method for the R-λ rate control model of VVC. In the target bit allocation link, the gradient information of each CTU in the current frame is obtained by using the Sobel operator as the spatial information thereof, the temporal information of each CTU in the current frame is obtained by using the MV information of the reference frame, for the special case that the reference frame contains I frames and all I frames, the gradient information of each CTU in the reference frame is obtained by using the Sobel operator, and the difference between the spatial information of the current frame and the temporal information thereof is used as the temporal information thereof, and the joint spatial and temporal information is used for target bit allocation at the CTU level. In the parameter updating process, the quasi-Newton convergence algorithm is used to replace the original LMS algorithm to speed up the model convergence and improve the accuracy of parameter updating, which specifically includes the following process steps:

[0010] (1) The gradient information of the current CTU is obtained by using the Sobel operator, which is used as the spatial information Grad of the CTU curCTU ;

[0011] (2) The temporal information M of each CTU in the current frame is obtained by using the MV of the current frame and the reference frame, for the special case that the reference frame contains I frames and all I frames, the gradient information of each CTU in the reference frame is obtained by using the Sobel operator, and the difference between the spatial information of the current frame and the temporal information thereof is used as the temporal information M of each CTU curCTU ; curCTU , then the threshold value of the temporal information of the current frame is obtained φ time , and the CTU is divided into time domain sensitive and non-time domain sensitive CTU;

[0012] (3) Using the time domain information of the time domain sensitive CTU and the non-time domain sensitive CTU, the target frame bit number is divided into the time domain sensitive bit number and the non-time domain sensitive bit number;

[0013] (4) According to the spatial domain information of the time domain sensitive and non-sensitive CTU respectively, bit allocation is performed on all CTUs in the current frame;

[0014] (5) After the current CTU or frame coding is completed, the related parameters are updated using the quasi-Newton convergence algorithm, and for the extreme case in the quasi-Newton algorithm, the first-order convergence value is linearly enlarged to update the parameters;

[0015] (6) Return to (1) until the coding is completed.

[0016] In step (2), the time domain information of each CTU of the current frame is extracted using formula (1-3) for the B frame or P frame whose reference frame does not contain I frame.

[0017]

[0018] M curCTU = MV, P Frame (2)

[0019]

[0020] Wherein, M curCTU represents the time domain information of the current CTU in the frame, MV x and MV y respectively represent the horizontal and vertical components of the 4*4 block MV in the reference frame CTU, and MC1 and MC2 respectively represent the motion compensation of the two reference frames of the B frame.

[0021] In step (2), for the special case that the reference frame contains I frame or is I frame, it is divided into two specific types. When the reference frame of the B frame contains an I frame, the same extraction formula (2) as the P frame is used to extract the information of the other non-I frame reference frame of the B frame. When the reference frame of the P frame is I frame, formula (4) is used to extract the time domain information.

[0022] M curCTU = abs(Grad curCTU - Grad refCTU ) (4)

[0023] Wherein, Grad curCTU and Grad refCTU respectively represent the spatial domain information of the current frame CTU and the reference frame CTU extracted by the Sobel operator, then using formula (5), the time domain information threshold is obtained, and the CTU with time domain information greater than the threshold is classified as time domain sensitive CTU, and the CTU with time domain information less than the threshold is classified as non-time domain sensitive CTU.

[0024]

[0025] where N ctu is the total number of CTUs in the current frame.

[0026] In step (3), the frame-level target bits are first divided into time- sensitive bits and non-time-sensitive bits using equations (6-7)

[0027]

[0028]

[0029] where Bit curframe represents the number of bits allocated for the current frame, N sens and N nonsen represent the total number of time-sensitive CTUs and non-time- sensitive CTUs in the current frame, M sensCTU and M nonsensCTU represent the time information of the time-sensitive CTUs and non-time- sensitive CTUs, Bit sense and Bit nonsense represent the number of time-sensitive bits and non-time-sensitive bits, respectively.

[0030] In step (4), the CTU-level target bits are allocated using equations (8-9):

[0031]

[0032]

[0033] where Bit sensCTU and Bit nonsensCTU represent the number of bits allocated for each time-sensitive CTU and non-time-sensitive CTU in the current frame, respectively, Bit leftsens and Bit leftnonsens represent the remaining number of time-sensitive bits and non-time- sensitive bits, respectively. Grad sensCTU and Grad nonsensCTU represent the spatial information of the time-sensitive CTU and non-time- sensitive CTU, respectively.

[0034] In step (5), the quasi-Newton convergence algorithm based on bit distortion is used, and the model parameters a new , β new are updated as shown in equations (10-15):

[0035]

[0036]

[0037]

[0038]

[0039]

[0040]

[0041] wherein R' and R represent actual bit number and target bit number respectively.

[0042] For the extreme case in step (5), its updating condition is shown as formula (16-17):

[0043]

[0044]

[0045] Compared with the prior art, the present application has the advantages and beneficial technical effects:

[0046] (1) The VVC rate control optimization method based on HVS characteristics proposed in the present application has higher coding performance than the coding performance of the code control model corresponding to the VVC standard algorithm under the premise of controllable coding complexity compared with the traditional VVC standard coding;

[0047] (2) The VVC rate control optimization method based on HVS characteristics proposed in the present application utilizes the characteristics of the human visual system, uses gradient information as a measure of complexity, and jointly uses spatial and temporal information to guide the target bit allocation of CTU, thereby improving the subjective quality of compressed video;

[0048] (3) The VVC rate control optimization method based on HVS characteristics proposed in the present application corrects extreme coefficients when target bit allocation is performed and uses a quasi-Newton algorithm with faster convergence speed in the parameter updating part, thereby improving the accuracy of parameter updating, reducing the fluctuation degree of video quality, and being more consistent with the perception characteristics of the human eye. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The figure is the total flow chart of the VVC rate control optimization method based on HVS characteristics.

[0050] Figure 2 The figure is the rate distortion curve comparison chart of the sequence Cactus (1920x1080) under LDP configuration and PartyScene under RA configuration (832x480) based on the method of the present application and VVC.

[0051] Figure 3Fig. 4 is a comparison chart of the fluctuation of the bit consumption per frame of the sequence SlideEditing (1965kbps) in LDP configuration and BQMall (5292kbps) in RA configuration based on the method of the present application and VVC.

[0052] Figure 4 Fig. 5 is a comparison chart of the fluctuation of the PSNR per frame of the sequence SlideEditing (1965kbps) in LDP configuration and BQMall (5292kbps) in RA configuration based on the method of the present application and VVC.

[0053] Figure 5 Fig. 6 is a comparison chart of the subjective comparison of the 80th to 82nd frames of the sequence BasketballDrive based on the method of the present application and VVC ((a) is VVC and (b) is the optimization method of the present application).

[0054] Figure 6 Fig. 7 is a comparison chart of the subjective comparison of the red rectangular area of the 80th to 82nd frames of the sequence BasketballDrive based on the method of the present application and HEVC, wherein (c), (d) and (e) are the red area framed by the subjective chart respectively (left is VVC and right is the optimization method of the present application). DETAILED DESCRIPTION

[0055] The present application is further described in detail below with reference to the embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present application and cannot be understood as limiting the protection scope of the present application. The skilled in the art can make some non-essential improvements and adjustments to the specific implementation of the present application according to the above description, which should still belong to the protection scope of the present application.

[0056] (1) The method of the present application is carried out on the VVC standard test code VTM-9.1 platform. In the LDP configuration, the configuration file is selected as encoder_lowdelay_P_main.cfg, the GOP size is set to 8, IntraPeriod is set to 8, in the rate control parameter configuration, RateControl is set to 1, indicating that the rate control is turned on; in the RA configuration, the configuration file is selected as encoder_randomaccess_vtm.cfg, the GOP size is set to 16, IntraPeriod is set to 16, in the rate control parameter configuration, RateControl is set to 1, indicating that the rate control is turned on.

[0057] (2) The test sequences for verifying the encoding performance of the method of the present application selected four types of official standard test sequences with resolutions of 416*240, 832*480, 1280*720 and 1920*1080. The specific standard test sequences used were BasketballPass, BlowingBubbles, RaceHorses, BQSquare, BasketballDrill, PartyScene, BQMall, FourPeople, Johnny, BasketballDrive, Cactus, ParkScene and BQTerrace;

[0058] (3) In the program of the VVC standard algorithm, for the configuration file encoder_lowdelay_P_main.cfg, the GOP size was set to 8, IntraPeriod was set to 8, and RateControl was set to 1, indicating that the code rate control was closed, and then all the video standard sequences needed to be tested were inputted to obtain the target code rate of each sequence under the condition of LDP configuration and in the case of closing code control, for the configuration file encoder_randomaccess_vtm.cfg, the GOP size was set to 16, IntraPeriod was set to 16, and RateControl was set to 1, indicating that the code rate control was closed, and then all the video standard sequences needed to be tested were inputted to obtain the target code rate of each sequence under the condition of RA configuration and in the case of closing code control;

[0059] (4) In the program of the algorithm of the present application, the parameters in (1) were set for the configuration files encoder_lowdelay_P_main.cfg and encoder_randomaccess_vtm.cfg, and TargetBitrate was set to the target code rate of each sequence in (3), and then all the video standard sequences needed to be tested were inputted to obtain the results of the algorithm of the present application under the conditions of LDP configuration and RA configuration, and the actual code rate, peak signal-to-noise ratio (PSNR), bit consumption per frame and output file data size were recorded and counted;

[0060] (5) In the procedure of VVC standard algorithm, the parameter settings in (1) are made for the configuration files encoder_lowdelay_P_main.cfg and encoder_randomaccess_vtm.cfg, the TargetBitrate is set to the target code rate corresponding to each sequence in (3), and then all the video standard sequences to be tested are input to obtain the results of HEVC algorithm under LDP configuration and RA configuration, and the actual code rate, peak signal-to-noise ratio (PSNR), bit consumption per frame and output file data size are recorded and counted;

[0061] (6) The results of the above two types are processed respectively, wherein the unit of code rate is Kbps; the bit error ΔBR of the present application and VVC is calculated respectively by using formula (16) error , and the smaller the index is, the better; the compression performance of the present application method relative to VVC is calculated by using BD-rate, and if the index is negative, it indicates that the performance is improved;

[0062]

[0063] (7) Table 1 shows the comparison of bit rate error and compression performance between VVC and the present application under LDP configuration. On average, the present application can reduce the bit rate error by 4.2% compared with VVC. Since the 11 video sequences selected are all from video sequences of different rates, it can be fully proved that the present application is superior to VVC in the index of bit rate error. In addition, the present application saves 16.67% BD-rate compared with VVC, and is superior to VVC in compression performance;

[0064] (8) Table 2 shows the comparison of bit rate error between VVC and the present application under RA configuration. On average, the present application can reduce the bit rate error by 7.03% compared with VVC. Since the 11 video sequences selected are all from video sequences of different rates, it can be fully proved that the present application is superior to VVC in the index of bit rate error. In addition, the present application saves 29.88% BD-rate compared with VVC, and is superior to VVC in compression performance;

[0065] (9) Figure 2 The rate-distortion curves of the sequences Cactus (1920x1080) and PartyScene (832x480) based on the present application method and VVC are shown, and the rate-distortion curve of the present application is obviously above that of VVC. In other words, in the case of the same PSNR, the code rate consumed by the method of the present application is less than that consumed by VVC, which fully proves that the method of the present application has certain superiority in encoding performance compared with the VVC standard algorithm;

[0066] (10) Figure 3 The figure shows the fluctuation of the bit consumption of each frame of the sequence SlideEditing (1965kbps) and BQMall (5292kbps) based on the method of the present application compared with VVC. Compared with VVC, the method of the present application can achieve more accurate bit allocation for each frame. The present application reduces the fluctuation degree of the bit allocation of each frame, making the curve more stable, thereby avoiding the phenomenon of bit shortage in the later stage of encoding, and further improving the overall visual effect of the video sequence.

[0067] (11) Figure 4 The figure shows the fluctuation of the PSNR of each frame of the sequence SlideEditing (1965kbps) and BQMall (5292kbps) based on the method of the present application compared with VVC. Compared with VVC, the fluctuation degree of the PSNR of the method of the present application is smaller, and high-quality PSNR can be guaranteed at the end of the video sequence, thereby ensuring the smoothness of the video sequence. Therefore, the present application is obviously superior to VVC in terms of the fluctuation degree of PSNR.

[0068] (12) From Figure 5 It can be seen that the distortion of the upper three frames is significantly greater than that of the lower three frames, and the overall visual smoothness of the lower three frames is superior to that of the upper three frames. This is because the method of the present application is based on the characteristics of the human visual system, and by establishing a joint space-time model, the quality fluctuation of the video is reduced, thereby improving the continuity and subjective quality of the video. In order to more intuitively illustrate the above, Figure 6 The figure shows the contrast of the red rectangular area. Compared with the serious blocking effect of VVC, the reconstructed frame based on the present application has better subjective quality in the above area, which further proves the superiority of the present application in subjective quality and also proves the rationality of the bit allocation process.

[0069] Table 1 Bit error ΔBR of the present application and VVC under LDP configuration error Comparison of the BD-rate saved by the present application compared with VVC

[0070]

[0071] Table 2 Bit error ΔBR of the present application and VVC under RA configuration error Comparison of the BD-rate saved by the present application compared with VVC

[0072]

Claims

1. A VVC rate control optimization method based on HVS characteristics, characterized in that: (1) The gradient information of the current CTU is obtained by using the Sobel operator, which is taken as the spatial information Grad of the CTU CTU ; (2) Using the MV of the current frame reference frame to adaptively obtain the time domain information M of each CTU in the current frame CTU For the special case that the reference frame contains I frames and I frames, use Sobel operator to obtain the gradient information of each CTU of the reference frame, and take the difference between it and the gradient information of each CTU of the current frame as the time domain information M of each CTU CTU Then get the threshold value φ of the time domain information of the current frame time And divide the CTU into time domain sensitive CTU and non-time domain sensitive CTU; (3) The target frame bit number is divided into time domain sensitive bit number and non-time domain sensitive bit number by using the time domain information of time domain sensitive CTU and non-time domain sensitive CTU, and the specific process is as follows: wherein Bit curframe represents the number of bits allocated for the current frame, N sens and N nonsen respectively represent the total number of time-domain sensitive CTUs and non-time-domain sensitive CTUs in the current frame, M sensCTU and M nonsensCTU respectively represent the time-domain information of the time-domain sensitive CTUs and non-time-domain sensitive CTUs, Bit sense and Bit nonsense respectively represent the number of time-domain sensitive bits and non-time-domain sensitive bits. (4) According to the spatial domain information belonging to the time domain sensitive CTU and the non-time domain sensitive CTU, the bit allocation is carried out for all CTUs in the current frame, and the specific process is as follows: where Bit sensCTU and Bit nonsensCTU represent the number of bits allocated to each of the temporal sensitive CTU and the non-temporal sensitive CTU in the current frame, respectively, Bit leftsens and Bit leftnonsens represent the remaining number of temporal sensitive bits and non-temporal sensitive bits, respectively, Grad sensCTU and Grad nonsensCTU represent the spatial information of the temporal sensitive CTU and the non-temporal sensitive CTU, respectively. (5) After the current CTU or frame coding is completed, the related parameters are updated by using the quasi-Newton convergence algorithm, and for the extreme case in the quasi-Newton algorithm, the linear amplification of the first-order convergence value is carried out for parameter updating; (6) Return to (1) until the coding is completed.

Images