A vvc adaptive lambda weighting method based on human eye visual masking effect

Abstract

This invention proposes an adaptive λ-weighting method for VVC based on the human visual masking effect. In the Multifunctional Video Coding (VVC) standard, the determination of the key rate-distortion optimization parameter λ is still based on a relatively fixed mapping relationship with the quantization parameter QP, without fully considering the characteristics of human vision. Therefore, an adaptive λ-weighting method is proposed based on the human visual masking effect. By extracting the texture and motion features of CTUs in video frames and combining the different perceptions of texture complexity and motion intensity by the human eye, the λ factor used for CTU encoding is adaptively weighted to improve the rate-distortion performance of video coding. Results show that, under low-latency P-frame configuration, using peak signal-to-noise ratio (PSNR) and multi-scale structural similarity as video quality evaluation indicators, the average bitrate is reduced by 0.09% and 0.15% respectively while maintaining the same quality, thus improving the rate-distortion performance of the video encoder.

Description

Technical Field

[0001] This invention relates to rate-distortion optimization techniques in the field of video coding, and in particular to a rate-distortion optimization method at the CTU level for Versatile Video Coding (VVC) based on the human visual masking effect. Background Technology

[0002] The Versatile Video Coding (VVC) standard addresses the diverse video needs of its users. Compared to the previous generation High Efficiency Video Coding (HEVC) standard, VVC offers performance improvements exceeding 45%. Rate-Distortion Optimization (RDO) is a crucial method for balancing video compression distortion and bitrate, ensuring coding efficiency. However, like HEVC, VVC still relies on a relatively fixed mapping between the key parameter λ and the quantization parameter QP in its RDO optimization. Furthermore, it uses objective metrics such as Mean Square Error (MSE) to measure distortion, failing to fully satisfy human visual perception. This results in a discrepancy between the objective quality improvement in video compression and the subjective visual experience perceived by the human eye. By combining human visual characteristics to optimize video coding rate-distortion performance, some scholars have proposed using Structure Similarity (SSIM) as a perceptual visual metric to replace traditional objective distortion metrics, which can effectively improve coding gain. Others have proposed considering spatial and temporal perceptual correlations for each Coding Tree Unit (CTU), using these perceptual features to adjust the corresponding Lagrange multipliers to guide the rate-distortion optimization process, using SSIM and other metrics as distortion metrics, and introducing this method into HEVC, which can effectively improve video perceptual coding performance. Still others have proposed an RDO model based on perceptual weighted mean square error, deriving a new Lagrange multiplier. Summary of the Invention

[0003] Since the VVC (Multifunctional Video Coding Standard) still has room for improvement in rate-distortion optimization to enhance human visual perception coding performance, this invention aims to propose an adaptive λ-weighted method for VVC based on the human visual masking effect.

[0004] The basic idea of ​​this invention is to extract the texture and motion feature perceptual factors of the current CTU, adaptively adjust the Lagrange multipliers using these feature factors, and apply this method to the next-generation video coding standard VVC. Multi-scale Structure Similarity (MS-SSIM) and Peak Signal-to-Noise Ratio (PSNR) are used as video quality evaluation metrics to improve the perceptual coding quality of video. This method mainly includes the following steps:

[0005] Step 1: Extract the texture features of CTUs in the video frame. By calculating the variance of the luminance component of each CTU in the current video frame and combining the feature that the human eye has different sensitivity to different texture complexities, construct the texture feature perception factor of CTUs in the current video frame.

[0006] Step 2: Extract the motion features of CTUs in the video frame. By calculating the SAD value of the luminance component of each CTU in the current video frame and the corresponding CTU in the reference frame, and combining the feature that the human eye has different sensitivity to different degrees of motion intensity, construct the motion feature perception factor of CTUs in the current video frame.

[0007] Step 3: Combine the texture and motion feature perception factors of the current frame CTU to construct perception feature factors based on the human visual masking effect;

[0008] Step 4: Use the perceptual feature factors of the current frame CTU to weight the Lagrange multipliers used for encoding the current CTU, and obtain new Lagrange multipliers based on visual perception;

[0009] Step 5: Apply the new Langron multipliers to the current CTU encoding.

[0010] The specific process is as follows:

[0011] (1) Extract the texture features of the CTU in the video frame and construct the texture feature perceptual factor of the CTU in the current video frame. The specific method is as follows:

[0012] ① Calculate the variance of the luminance component of the k-th CTU in the current frame:

[0013]

[0014] In the formula, I(x,y) is the luminance component value of the pixel at position (x,y) in the k-th CTU of the current frame, and L is the size of the CTU;

[0015] ② Calculate the average variance of all CTU luminance components in the current frame, i.e.:

[0016]

[0017] In the formula, N is the mean of the variance values ​​of all CTU luminance values ​​in the current frame. CTU This represents the total number of CTUs in the current frame.

[0018] ③ Construct the texture feature perceptual factor of the current CTU by using the variance of the current CTU luminance component and the average of the variances of all CTU luminance components in the current frame. The construction method is as follows:

[0019]

[0020] In the formula, F tex (k) represents the texture feature perceptual factor of the k-th CTU in the current frame, and Var(k) represents the variance of the luminance component of the k-th CTU. Var is the mean of the variances of all CTU luminance components in the current frame. max Var represents the maximum value of the CTU luminance variance in the current frame. min C1 is the minimum CTU luminance variance value in the current frame. C1 is a constant set to avoid the denominator being 0, and is taken as C1 = 1 / 255. 2 .

[0021] (2) Extract the motion features of the CTU in the video frame and construct the motion feature perception factor of the CTU in the current video frame. The specific method is as follows:

[0022] ① Calculate the SAD value of the k-th CTU in the current frame and the CTU at the same position in the reference frame:

[0023]

[0024] In the formula, SAD(k) is the SAD value of the k-th CTU in the current frame and the CTU at the same position in the reference frame. c(k) (x,y) represents the luminance component value of the pixel at position (x,y) within the k-th CTU. r(k) (x,y) represents the luminance component value of the CTU at the same position in the reference frame, and L represents the size of the CTU.

[0025] ② Calculate the average SAD value of the luminance components of all CTUs in the current frame and the corresponding CTUs in the reference frame, i.e.:

[0026]

[0027] In the formula, N is the mean SAD value of all CTUs in the current frame. CTU This represents the total number of CTUs in the current frame.

[0028] ③ Construct the motion feature perception factor of the current CTU by using the SAD value of the current CTU and the average SAD value of all CTUs in the current frame. The construction method is as follows:

[0029]

[0030] Among them, F mot (k) represents the motion feature sensing factor of the k-th CTU in the current frame, and SAD(k) represents the SAD value of the k-th CTU in the current frame and the corresponding CTU in the reference frame. SAD is the average SAD value of all CTUs in the current frame. max The maximum SAD value of CTU in the current frame. min C2 is the minimum SAD value of CTU in the current frame. C2 is a constant set to avoid the denominator being 0, and is set to C2 = 1 / 255.

[0031] (3) Construct perceptual feature factors based on the human visual masking effect. The construction method is as follows:

[0032]

[0033] Where F(k) is the perceptual feature factor of the k-th CTU in the current frame, w1 and w2 are weight factors, each set to 0.5, and s is the scaling factor, set to s = 0.05. This represents the mean of the non-zero motion feature perception factors in the current frame. The mean of the non-zero texture feature perceptual factors in the current frame:

[0034]

[0035]

[0036] Where, N pmot N represents the number of CTUs in the current frame whose motion feature sensing factor is non-zero. ptex The number of CTUs with non-zero texture feature perceptual factors in the current frame, where α is the adaptive adjustment coefficient:

[0037]

[0038] Where R0 is the proportion of CTUs in the current frame whose texture feature perceptual factor and motion feature perceptual factor are both 0, and R1 is the proportion of the remaining CTUs in the current frame, i.e., R1 = 1 - R0. The range of F(k) is limited as follows:

[0039]

[0040] The constructed perceptual feature factor F(k) is used to weight the parameter λ used for CTU encoding in VVC.

[0041] (4) The Lagrange multiplier λ is weighted using the current CTU's perceptual feature factor F(k) to obtain a visually perceptual Lagrange multiplier, which is used for rate-distortion optimization of CTU coding in non-I-frames. The specific method is as follows:

[0042]

[0043] Using λ hvp Optimize the encoding of CTUs for non-I-frames.

[0044] The advantages and beneficial technical effects of this invention compared with the prior art are as follows:

[0045] (1) The VVC adaptive λ weighting method based on the human visual masking effect proposed in this invention combines the texture and motion characteristics of the video, fully considers the different perceptions of the human visual masking effect on different texture complexities and different motion intensity, and adjusts the coding parameters accordingly. This makes up for the problem that the rate-distortion optimization parameter λ in the existing VVC standard is based on a relatively fixed mapping relationship with the quantization parameter QP, which does not fully meet the human visual sensory quality.

[0046] (2) In constructing the perceptual feature factor of the human eye visual masking effect, the present invention adopts an adaptive adjustment coefficient, which can adaptively adjust the perceptual factor according to the different proportions of visually sensitive CTU and visually insensitive CTU in each video frame, thus avoiding the problem of unsatisfactory rate distortion performance caused by using a fixed value range due to the difference in the proportions of visually sensitive CTU and visually insensitive CTU in the video frame.

[0047] (3) The perceptual feature factor of the human eye visual masking effect proposed in this invention is only used to adjust the encoding parameters of visually insensitive CTUs. For CTUs with both texture feature factor and motion feature factor being 0, i.e. visually sensitive CTUs, the standard encoding parameters are still used to avoid the problem of increased distortion in visually sensitive areas due to parameter changes. Attached Figure Description

[0048] Figure 1 The rate-distortion curve for the sequence BQSquare using YUV-PSNR as the quality evaluation index is shown.

[0049] Figure 2 The rate-distortion curve for the sequence BQSquare using YUV-MS-SSIM as the quality evaluation index is shown. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to the embodiments. It should be noted that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above-described invention, and these improvements and adjustments should still fall within the scope of protection of the present invention.

[0051] (1) The method proposed in this invention is carried out on the VVC standard test code VTM12.0 platform. Under LDP configuration, the configuration file is encoder_lowdelay_P_vtm.cfg, the encoding structure is IPPPPPPP; GOPSize is 8, that is, 1 GOP contains 8 frames; IntraPeriod is -1, that is, only the first frame is an I frame, and the rest are P frames, the number of encoded frames is 100 frames, and the standard video test sequence is encoded with quantization parameter QP of 22, 27, 32 and 37. The bit rate, peak signal-to-noise ratio PSNR and multi-scale structural similarity MS-SSIM of the method are recorded when the video is encoded under different QP.

[0052] (2) The encoded object is the standard VVC test video. Three test sequences are selected from each group from Class B to F for testing. The resolutions of the video sequences in Class B to E are 1920×1080, 832×480, 416×240, and 1280×720, respectively. Class F is a test sequence of screen content with different resolutions.

[0053] (3) Encode the video sequence using the method of the present invention under the above conditions, and record the bit rate, peak signal-to-noise ratio (PSNR), and multi-scale structural similarity (MS-SSIM) when the video is encoded using the method of the present invention under different QPs.

[0054] (4) We use BD-rate to calculate the compression performance of the method of the present invention relative to VVC. A BD-rate of "-" indicates a bit rate saving under the same MS-SSIM or PSNR, and a BD-rate of "+" indicates a bit rate increase under the same MS-SSIM or PSNR. That is, a negative BD-rate indicates improved performance.

[0055] (5) Table 1 shows the BD-rate results of our method relative to VTM12.0 under Low Latency Configuration (LDP), and Table 2 compares the average BD-rate values ​​of each video group under LDP. Compared with the VVC standard, when YUV-PSNR is used as the video quality evaluation index, the BD-rate decreases by an average of 0.09%, indicating that the video sequence can save 0.09% of the bitrate under the same PSNR; when YUV-MS-SSIM is used as the video quality evaluation index, the BD-rate decreases by an average of 0.15%, indicating that the video sequence can save 0.15% of the bitrate under the same MS-SSIM. The comparison of the average BD-rate values ​​of each video sequence group shows that the Class E rate distortion performance is significantly improved, with an average BD-rate decrease of 0.15% and 0.23%, indicating that our method has achieved good results.

[0056] (6) From Figure 1 and Figure 2 As can be seen, the rate-distortion curve of our proposed method is above that of the VVC standard. This indicates that the proposed algorithm has a lower bitrate for the same video quality, and at the same bitrate, it has better video quality. In other words, our proposed method has a better effect in saving bitrate.

[0057] Table 1. BD-rate results of the present invention relative to VTM12.0 under Low Latency Configuration (LDP) (unit: %)

[0058]

[0059] Table 2 Comparison of average BD-rate for each video group under Low Latency Configuration (LDP) (Unit: %)

[0060]

Claims

1. An adaptive λ-weighted VVC method based on the human visual masking effect, which mainly improves the rate-distortion optimization process of the multi-functional video coding standard VVC. The steps are as follows: (1) Extract the texture features of CTUs in the video frame, calculate the variance value Var(k) of the luminance component of each CTU in the current frame and the mean value MVar of the variance values ​​of the luminance components of all CTUs in the current frame, and construct the texture feature perceptual factor Ftex(k) of the kth CTU in the current frame by using the variance value of the luminance component of the current CTU and the mean value of the variance values ​​of the luminance components of all CTUs in the current frame: (1) in, Varmax is the maximum value of the CTU brightness variance in the current frame, Varmin is the minimum value of the CTU brightness variance in the current frame, and C1 is a constant set to avoid the denominator being 0, C1=1 / 255²; (2) Extract the motion features of CTUs in the video frame, calculate the SAD value SAD(k) of the luminance component of each CTU in the current frame and the corresponding CTU in the reference frame, and the mean SAD of the luminance component of all CTUs in the current frame and the corresponding CTU in the reference frame. Then, construct the motion feature perception factor Fmot(k) of the kth CTU in the current frame using the SAD value of the current CTU and the mean SAD value of all CTUs in the current frame. (2) Where SADmax is the maximum value of SAD of CTU in the current frame, SADmin is the minimum value of SAD of CTU in the current frame, and C2 is a constant set to avoid the denominator being 0, C2=1 / 255; (3) Calculate the mean value Fpmot of non-zero motion feature perceptual factor and the mean value Fptex of non-zero texture feature perceptual factor in the current frame, and determine the adaptive adjustment coefficient α based on the proportion R0 of CTUs with both texture feature perceptual factor and motion feature perceptual factor being 0 in the current frame and the proportion R1 of the remaining CTUs in the current frame. (4) Using w1 and w2 as weighting factors, each with a value of 0.5, and s as a scaling factor with a value of s=0.05, the perceptual feature factor F(k) of the k-th CTU in the current frame is constructed by combining the texture feature perceptual factor, motion feature perceptual factor, the mean of the non-zero values, and the adaptive adjustment coefficient, and the range of values ​​for F(k) is limited: (3) (5) Using the constructed perceptual feature factor F(k), the Lagrange multiplier λVVC used for CTU encoding in VVC is multiplicatively weighted to obtain the visual perception-based Lagrange multiplier λhvp: (4) The visual perception-based Lagrange multiplier λhvp is used for rate-distortion optimization of CTU coding in non-I-frames; wherein, the perceptual feature factor is only used to adjust the coding parameters of visually insensitive CTUs where the texture feature perceptual factor and motion feature perceptual factor are not both 0, and for visually sensitive CTUs where both the texture feature perceptual factor and motion feature perceptual factor are 0, the VVC standard coding parameters are used.

2. The VVC adaptive λ-weighted method as described in claim 1, characterized in that, The variance value Var(k) of the luminance component of the k-th CTU in the current frame is calculated as follows: (5) Where I(x,y) is the luminance component value of the pixel at position (x,y) within the k-th CTU of the current frame, and L is the size of the CTU; the mean MVar of the variance of all CTU luminance components in the current frame is calculated as follows: (6) Wherein, NCTU is the total number of CTUs in the current frame.

3. The VVC adaptive λ-weighted method as described in claim 1, characterized in that, The SAD value SAD(k) of the k-th CTU in the current frame and the CTU at the same position in the reference frame is calculated as follows: (7) Where Ic(k)(x,y) is the luminance component value of the pixel at position (x,y) within the k-th CTU, Ir(k)(x,y) is the luminance component value of the pixel at the same position in the reference frame, and L is the size of the CTU; the mean SAD value of the luminance components of all CTUs in the current frame and the corresponding CTUs in the reference frame is calculated as follows: (8) Wherein, NCTU is the total number of CTUs in the current frame.

4. The VVC adaptive λ-weighted method as described in claim 1, characterized in that, The mean value Fpmot of the non-zero motion feature perceptual factor and the mean value Fptex of the non-zero texture feature perceptual factor are calculated as follows: (9) (10) Where Npmot is the number of CTUs with non-zero motion feature perceptual factors in the current frame, and Nptex is the number of CTUs with non-zero texture feature perceptual factors in the current frame.

5. The VVC adaptive λ-weighted method as described in claim 1, characterized in that, The adaptive adjustment coefficient α is calculated as follows: (11) Where R0 is the proportion of CTUs in the current frame whose texture feature perceptual factor and motion feature perceptual factor are both 0, and R1 is the proportion of the remaining CTUs in the current frame, and satisfies the following conditions: .

Images