Video coding optimization method based on double saliency perception features

Abstract

The invention discloses a video coding optimization method based on double saliency perception characteristics, belongs to the technical field of image communication, and aims to solve the problem that code rate compression and perception quality are difficult to balance due to the fact that traditional video coding is not combined with human visual perception characteristics. The method comprises the following steps: firstly, extracting dual saliency perception features at a coding unit level, wherein the dual saliency perception features comprise a saliency intensity feature obtained by a pixel-level continuous saliency map and a saliency region feature obtained by a pixel-level binary saliency mask; and calculating the quantization parameter adjustment amount according to the dual characteristics, determining the final quantization parameter of each coding unit, and realizing the adaptive regulation and control of coding quantization. And finally, constructing a perceptual distortion value fitting human vision, and reconstructing a perceptual rate-distortion objective function so as to optimize the rate-distortion decision process of the coding unit. According to the method, joint regulation and control of coding unit-level dual saliency features are realized, the coding efficiency is effectively improved on the basis of guaranteeing the subjective visual quality of the video, and the method can adapt to mainstream video coding standards such as AVC, HEVC and VVC.

Description

Technical Field

[0001] This invention belongs to the field of image communication technology, specifically a video coding optimization method based on dual saliency sensing features. Background Technology

[0002] With the widespread application of ultra-high-definition and high-frame-rate video, the pressure on video data storage and transmission continues to rise. Improving the compression efficiency of video encoding while taking into account visual perception quality has become a core industry requirement. Mainstream video encoding standards such as AVC, HEVC, and VVC all optimize their encoding based on uniform quantization parameter settings and pixel-level traditional distortion, without fine-tuning the differences in visual characteristics of encoding units, making it difficult to achieve the optimal balance between bitrate compression and perception quality.

[0003] The human visual system is more sensitive to highly salient regions in videos, making coding distortions in these regions easily detectable, while slight distortions in non-salient regions are difficult to identify. This characteristic provides a basis for optimizing perceptual video coding. However, existing perceptual coding technologies either fail to effectively extract and jointly control dual saliency perceptual features (saliency intensity and saliency region) at the coding unit level, or only perform local optimization using a single saliency feature, without deeply integrating dual saliency perceptual features with quantization parameter adjustment and rate-distortion optimization. This results in a lack of targeted allocation of coding resources, leading to coding quality loss in salient visual regions and wasted bitrate in non-saliency regions. Therefore, there is an urgent need for a coding optimization method that extracts dual saliency perceptual features at the coding unit level and can adaptively adjust quantization parameters and reconstruct perceptual rate-distortion based on these features, enabling precise allocation of coding resources and further improving the overall performance of video coding. Summary of the Invention

[0004] This invention addresses the shortcomings of existing video coding standards, such as neglecting human visual perception characteristics, failing to extract and jointly control dual saliency perceptual features at the coding unit level, and neglecting to incorporate dual saliency perceptual distortion in rate-distortion optimization. It proposes an innovative solution: by extracting dual saliency perceptual features—comprising saliency intensity features and saliency region features—at the coding unit level, and performing joint control based on these features, this invention achieves more precise quantization parameter adjustment and bitrate allocation, thereby effectively improving coding efficiency.

[0005] This invention provides a video coding optimization method based on dual saliency-aware features, characterized by the following steps: S1: Extract dual saliency sensing features at the coding unit level from the input video frame sequence. The dual saliency sensing features are coding unit-level saliency intensity features and coding unit-level saliency region features. The coding unit-level saliency intensity features are calculated based on pixel-level continuous saliency maps, and the coding unit-level saliency region features are calculated based on pixel-level binary saliency masks. S2: Based on the dual saliency sensing features, an adaptive quantization parameter adjustment strategy is constructed. Through joint control based on the dual saliency sensing features, the quantization parameter adjustment amount of each coding unit is calculated. The basic quantization parameters preset by the encoder are combined with the quantization parameter adjustment amount to obtain the final quantization parameters of the coding unit, which are used for quantization operations in video encoding. S3: Construct a perceptual distortion value based on the aforementioned dual saliency perceptual features. Through joint regulation based on the dual saliency perceptual features, replace the traditional rate-distortion objective function in the video coding standard method with a perceptual rate-distortion objective function based on the aforementioned perceptual distortion value, thereby optimizing the rate-distortion optimization process of the coding unit.

[0006] Preferably, the pixel-level continuous saliency map in step S1 is generated by a video saliency prediction model. The method for calculating the coding unit-level saliency intensity feature based on this pixel-level continuous saliency map is as follows: for each coding unit, the pixel-level saliency value of each pixel is obtained from the pixel-level continuous saliency map; the average of the saliency values ​​of all pixels within the coding unit is taken to obtain the coding unit-level saliency intensity feature; and the global saliency mean of the current frame is calculated simultaneously. The specific calculation formula is as follows: , , Where S cu S(x,y) represents the saliency intensity feature at the coding unit level, and S(x,y) represents the pixel-level saliency value. avg Let W be the mean global saliency intensity of the current frame, and let W and H be the width and height of the coding unit, respectively. frame H frame These are the width and height of the current frame, respectively.

[0007] Preferably, the pixel-level binary saliency mask mentioned in step S1 is a pixel-level 0-1 binary decision mask generated by the video saliency detection model after performing saliency region detection on the input video frame. The method for calculating the coding unit-level saliency region features based on this pixel-level binary saliency mask is as follows: for each coding unit, the number of salient pixels is counted from the pixel-level binary saliency mask, the proportion of salient pixels within the coding unit is calculated, and this proportion is used as the coding unit-level saliency region feature. The specific calculation formula is as follows: , Where Mcu For salient region features at the coding unit level, N s The number of significant pixels with a pixel-level binary saliency mask value of 1 within the encoding unit, where W and H are the width and height of the encoding unit, respectively; the determination rule for significant pixels is: a pixel with a pixel-level binary saliency mask value of 1 indicates that the corresponding pixel is a significant pixel, and a pixel with a pixel-level binary saliency mask value of 0 indicates that the corresponding pixel is a non-significant pixel.

[0008] Preferably, the salient region features at the coding unit level are linearly normalized. Using the maximum and minimum values ​​of the salient region features of all coding units in the current frame as a benchmark, the salient region features of each coding unit are normalized to the interval 0-1, resulting in the normalized coding unit-level salient region features M'. cu .

[0009] Preferably, in step S2, an adaptive quantization parameter adjustment strategy is constructed based on the dual saliency-aware features. Specifically, the quantization parameter adjustment amount is calculated by using the ratio of the coding unit-level saliency intensity feature to the mean global saliency intensity of the current frame and the normalized saliency region feature, and then combined with the basic quantization parameters to obtain the final quantization parameters. The formula is as follows: , Where ∆QP is the quantization parameter adjustment amount, QP final QP is the final quantization parameter for the coding unit. base Here are the basic quantization parameters for the encoder, k and m are adjustment parameters, and S... cu For the saliency strength feature at the coding unit level, S avg M' is the mean global saliency intensity of the current frame. cu The normalized coding unit-level salient region features are given by tanh, which is the hyperbolic tangent function. If the calculated QP... final If the value is less than 10, take 10; if the value is greater than 51, take 51.

[0010] Preferably, the specific calculation formulas for the perception distortion value and the perception rate distortion objective function in step S3 are as follows: Perceptual distortion value: , Perception rate distortion objective function: , Where D is the traditional distortion term in the standard video coding method; α and β are weight parameters; S cu For the saliency strength feature at the coding unit level, S avg M' is the mean global saliency intensity of the current frame. cu λ represents the normalized salient region features at the coding unit level, λ is the Lagrange multiplier of the encoder, and R is the coding unit code rate.

[0011] Preferably, the video coding standard method refers to the AVC, HEVC, or VVC video coding standards.

[0012] Compared with the prior art, the present invention has at least the following beneficial effects: 1. This invention constructs a visual region importance assessment model that closely resembles the human visual attention mechanism by extracting dual saliency perception features—comprising saliency intensity features and saliency region features—at the coding unit level and jointly regulating them. Unlike methods that rely solely on a single saliency feature, this invention accurately characterizes the overall saliency of a coding unit through saliency intensity features and quantifies the proportion of salient pixels within a coding unit through saliency region features. This dual-dimensional approach enables refined identification of salient regions in videos, providing a more accurate two-dimensional basis for optimal bitrate allocation.

[0013] 2. This invention proposes a nonlinear quantization parameter adjustment strategy based on the saliency intensity ratio and normalized saliency region features. The hyperbolic tangent function (tanh) is used to map the normalized saliency region features, and ΔQP is generated by combining the ratio of saliency intensity features to the global mean. This not only achieves high-quality protection of key visual regions with high saliency intensity and high saliency pixel ratio, but also automatically increases compression intensity in non-saliency regions, achieving highly adaptive allocation of bit resources and avoiding bitrate waste in non-saliency regions.

[0014] 3. The perceptual distortion model constructed in this invention transforms traditional pure pixel-level errors into perceptual distortion error D. p The algorithm is directly embedded into the rate-distortion optimization objective function. Without changing the standard video decoder, the encoder spontaneously tends to choose the encoding mode that maximizes the visual quality benefit to the human eye. Experiments show that, while maintaining good subjective quality, this invention achieves significant bitrate savings compared to traditional video coding standard methods. Attached Figure Description

[0015] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the overall process of the video coding optimization method based on dual saliency sensing features of the present invention; Figure 2 This is a schematic diagram of the extraction process of dual saliency sensing features at the coding unit level in this invention; Figure 3 This is a schematic diagram of the adaptive adjustment process of quantization parameters based on dual saliency sensing features according to the present invention. Figure 4 This is a schematic diagram of the construction process for the perception rate distortion optimization based on dual saliency perception features of the present invention. Detailed Implementation

[0016] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0017] like Figure 1 The diagram shown illustrates the overall process of the video coding optimization method based on dual saliency sensing features according to the present invention. This embodiment provides a video coding optimization method based on dual saliency sensing features, comprising the following steps: S1: Extract dual saliency sensing features at the coding unit level from the input video frame sequence. The dual saliency sensing features are coding unit-level saliency intensity features and coding unit-level saliency region features. The coding unit-level saliency intensity features are calculated based on pixel-level continuous saliency maps, and the coding unit-level saliency region features are calculated based on pixel-level binary saliency masks. S2: Based on the dual saliency sensing features, an adaptive quantization parameter adjustment strategy is constructed. Through joint control based on the dual saliency sensing features, the quantization parameter adjustment amount of each coding unit is calculated. The basic quantization parameters preset by the encoder are combined with the quantization parameter adjustment amount to obtain the final quantization parameters of the coding unit, which are used for quantization operations in video encoding. S3: Construct a perceptual distortion value based on the aforementioned dual saliency perceptual features. Through joint regulation based on the dual saliency perceptual features, replace the traditional rate-distortion objective function in the video coding standard method with a perceptual rate-distortion objective function based on the aforementioned perceptual distortion value, thereby optimizing the rate-distortion optimization process of the coding unit.

[0018] Example 1: Dual saliency-aware feature extraction at the coding unit level.

[0019] This embodiment corresponds to step S1 above, realizing the extraction of dual saliency sensing features at the coding unit level and the normalization processing of saliency region features. The specific process is as follows: Figure 2 As shown, the detailed steps are as follows: S11. Calculation of saliency intensity features.

[0020] For each coding unit (CU) of the input coded frame, a deep learning-based video saliency prediction model is used to process the frame. This invention employs the MSFF-Net video saliency prediction model to generate a frame-level pixel-level continuous saliency map. This map iterates through all pixels (x, y) within the coding unit, according to the formula: , The mean of pixel saliency values ​​within a coding unit is calculated to obtain the coding unit-level saliency intensity feature S. cu Simultaneously, iterate through all pixels of the entire frame, according to the formula: , Calculate the mean global saliency intensity S of the current frame. avg .

[0021] S12. Calculation of salient region features.

[0022] For each coding unit (CU), a deep learning-based video saliency detection model is used to process the frame. This invention employs the DCFNet video saliency detection model to generate a frame-level pixel-level 0-1 binary saliency mask. It then iterates through all pixels within the coding unit and counts the number N salient pixels with a mask value of 1. s According to the formula: , The proportion of salient pixels within a coding unit is calculated to obtain the coding unit-level salient region feature M. cu ; where a pixel-level binary saliency mask of 1 indicates that the corresponding pixel is a salient pixel, and a pixel-level binary saliency mask of 0 indicates that the corresponding pixel is a non-salient pixel.

[0023] S13. Normalization of salient region features.

[0024] The salient region features M of all coding units in the current frame cu Based on the maximum and minimum values, M is used for each coding unit. cu Perform linear normalization to map it to the 0~1 interval, and obtain the normalized coding unit-level salient region feature M'. cu This eliminates the differences in feature dimensions between different coding units, preparing for subsequent joint regulation.

[0025] Example 2: Adaptive quantization parameter adjustment.

[0026] This embodiment corresponds to step S2 above. Based on the dual saliency-aware features extracted in embodiment one, an adaptive quantization parameter adjustment strategy is constructed to achieve differentiated settings of quantization parameters at the encoding unit level. The specific process is as follows: Figure 3 As shown, the detailed steps are as follows: S21. Calculation of quantization parameter adjustment amount.

[0027] After experimental optimization, the parameters were set to k=3.0 and m=1.8, and the saliency intensity feature S at the coding unit level was... cu The mean global saliency intensity S of the current frame avg Normalized coding unit-level salient region features M' cuSubstitute into the formula: , The quantization parameter adjustment amount of the coding unit is calculated. .

[0028] S22. Final quantization parameters are determined.

[0029] Obtain the encoder's preset basic quantization parameter QP base According to the formula: , Calculate the final quantization parameter QP of this coding unit. final And the calculation results are cropped. If the calculated QP final If the value is less than 10, take 10; if the value is greater than 51, take 51, ensuring QP. final ∈[10,51]; The final quantization parameters are passed to the quantization module of the encoder for the encoding quantization processing of the encoding unit.

[0030] Example 3: Optimization of perception rate distortion.

[0031] This embodiment corresponds to step S3 above. Based on the dual saliency perception features extracted in embodiment one, a perception distortion value is constructed and a perception rate-distortion objective function is designed to optimize the rate-distortion optimization process of the coding unit. The specific process is as follows: Figure 4 As shown, the detailed steps are as follows: S31. Generation of perceptual distortion values.

[0032] The weighting parameters α=0.6 and β=0.4 were set; these values ​​were obtained through experimental optimization to ensure that the perceptual distortion value closely matches human visual perception. The saliency intensity feature S at the coding unit level was then used. cu The mean global saliency intensity S of the current frame avg Normalized coding unit-level salient region features M' cu And the traditional distortion term D in the video coding standard method is substituted into the formula. The reconstructed value is the perceptual distortion D that closely matches human visual perception. p .

[0033] S32. Replacement of the objective function for perceptual rate distortion.

[0034] The traditional rate-distortion objective function in the standard coding method is replaced with the perceptual rate-distortion objective function designed in this invention. , where λ is the Lagrange multiplier preset by the encoder and R is the bit rate of the coding unit; the perceptual rate distortion objective function is embedded into the rate distortion optimization module of the encoder as the core judgment basis for coding unit mode selection, prediction and transformation, to complete the perceptual rate distortion optimization of the coding unit, and finally output the encoded video bitstream.

[0035] To verify the encoding optimization performance of the method of this invention, a comparative experiment was conducted using the VVC encoding standard as a benchmark and typical sequences of Class BE from the JVET standard test video set. The experiment covered four resolutions: 1920×1080, 834×420, 1280×720, and 416×240, encompassing typical video scenes such as dynamic, texture-rich, and static content. The experiment used BD-Rate (bitrate saving rate) as the core evaluation indicator; a negative value represents the bitrate saving ratio of this invention relative to the benchmark encoding. Simultaneously, subjective visual scoring using MOS (Modular Opinion) was used to verify that there was no significant decrease in video visual quality after encoding and compression. Specific experimental results are shown in Table 1.

[0036] Table 1 Experimental results of the method of the present invention

[0037] Experimental results show that, while maintaining subjective visual quality (MOS≥4.7) without significant degradation, the present invention achieves an average bitrate saving of 11.1% compared to the VVC baseline coding, verifying the universality and effectiveness of the present invention under different resolutions and video scenarios, and effectively improving video coding efficiency while ensuring visual perception quality.

Claims

1. A video coding optimization method based on dual saliency sensing features, characterized in that, Includes the following steps: S1: Extract dual saliency sensing features at the coding unit level from the input video frame sequence. The dual saliency sensing features are coding unit-level saliency intensity features and coding unit-level saliency region features. The coding unit-level saliency intensity features are calculated based on pixel-level continuous saliency maps, and the coding unit-level saliency region features are calculated based on pixel-level binary saliency masks. S2: Based on the dual saliency sensing features, an adaptive quantization parameter adjustment strategy is constructed. Through joint control based on the dual saliency sensing features, the quantization parameter adjustment amount of each coding unit is calculated. The basic quantization parameters preset by the encoder are combined with the quantization parameter adjustment amount to obtain the final quantization parameters of the coding unit, which are used for quantization operations in video encoding. S3: Construct a perceptual distortion value based on the aforementioned dual saliency perceptual features. Through joint regulation based on the dual saliency perceptual features, replace the traditional rate-distortion objective function in the video coding standard method with a perceptual rate-distortion objective function based on the aforementioned perceptual distortion value, thereby optimizing the rate-distortion optimization process of the coding unit.

2. The video coding optimization method based on dual saliency sensing features as described in claim 1, characterized in that, The pixel-level continuous saliency map mentioned in step S1 is generated by the video saliency prediction model. The method for calculating the coding unit-level saliency intensity feature based on this pixel-level continuous saliency map is as follows: For each coding unit, the pixel-level saliency value of each pixel is obtained from the pixel-level continuous saliency map. The average of the saliency values ​​of all pixels within the coding unit is taken to obtain the coding unit-level saliency intensity feature. At the same time, the global saliency mean of the current frame is calculated. The specific calculation formula is as follows: ， Where S cu S(x,y) represents the saliency intensity feature at the coding unit level, and S(x,y) represents the pixel-level saliency value. avg Let W be the mean global saliency intensity of the current frame, and let W and H be the width and height of the coding unit, respectively. frame H frame These are the width and height of the current frame, respectively.

3. The video coding optimization method based on dual saliency sensing features as described in claim 1, characterized in that, The pixel-level binary saliency mask mentioned in step S1 is a pixel-level 0-1 binary decision mask generated by the video saliency detection model after detecting salient regions in the input video frame. The method for calculating the coding unit-level saliency region features based on this pixel-level binary saliency mask is as follows: for each coding unit, the number of salient pixels is counted from the pixel-level binary saliency mask, and the proportion of salient pixels within the coding unit is calculated. This proportion is used as the coding unit-level saliency region feature. The specific calculation formula is as follows: ， Where M cu For salient region features at the coding unit level, N s The number of significant pixels with a pixel-level binary saliency mask value of 1 within the encoding unit, where W and H are the width and height of the encoding unit, respectively; the determination rule for significant pixels is: a pixel with a pixel-level binary saliency mask value of 1 indicates that the corresponding pixel is a significant pixel, and a pixel with a pixel-level binary saliency mask value of 0 indicates that the corresponding pixel is a non-significant pixel.

4. The video coding optimization method based on dual saliency-aware features as described in claim 1, characterized in that, Linear normalization is performed on the salient region features at the coding unit level. Based on the maximum and minimum values ​​of the salient region features of all coding units in the current frame, the salient region features of each coding unit are normalized to the interval of 0 to 1, resulting in normalized salient region features.

5. The video coding optimization method based on dual saliency sensing features as described in claim 1, characterized in that, In step S2, an adaptive quantization parameter adjustment strategy is constructed based on the aforementioned dual saliency-aware features. Specifically, the quantization parameter adjustment amount is calculated using the ratio of the coding unit-level saliency intensity feature to the mean global saliency intensity of the current frame and the normalized saliency region feature. This is then combined with the basic quantization parameters to obtain the final quantization parameters for the coding unit. The specific calculation formula is as follows: ， Where ∆QP is the quantization parameter adjustment amount, QP final QP is the final quantization parameter for the coding unit. base Here are the basic quantization parameters for the encoder, k and m are adjustment parameters, and S... cu For the saliency strength feature at the coding unit level, S avg M' is the mean global saliency intensity of the current frame. cu The normalized coding unit-level salient region features are given by tanh, which is the hyperbolic tangent function. If the calculated QP... final If the value is less than 10, take 10; if the value is greater than 51, take 51.

6. The video coding optimization method based on dual saliency sensing features as described in claim 1, characterized in that, The specific calculation formulas for the perceptual distortion value and the perceptual rate distortion objective function mentioned in step S3 are as follows: Perceptual distortion value: , Perception rate distortion objective function: , Where D is the traditional distortion term in the standard video coding method; α and β are weight parameters; S cu For the saliency strength feature at the coding unit level, S avg M' is the mean global saliency intensity of the current frame. cu λ represents the normalized salient region features at the coding unit level, λ is the Lagrange multiplier of the encoder, and R is the coding unit code rate.

7. The video coding optimization method based on dual saliency sensing features as described in claim 1, characterized in that, The video coding standard methods mentioned refer to AVC, HEVC, and VVC video coding standards.

Images