Perceptual video coding method combining saliency and just-noticeable distortion

Abstract

The present invention discloses a perceptual video coding method combining saliency and just-perceptible distortion. The method comprises the following steps: obtaining a video sequence to be coded, calculating a saliency value and a just-perceptible distortion threshold for each pixel; calculating its block-level and frame-level saliency factors and just-perceptible distortion factors; modifying the video coding standard method, respectively calculating the quantization parameter offsets of saliency and just-perceptible distortion in the quantization parameter selection link, and integrating the two into an adaptive quantization parameter offset; in the rate-distortion optimization link, integrating just-perceptible distortion into the calculation of distortion, and adjusting the rate-distortion balance through saliency. The proposed method can significantly improve coding efficiency and save coding time.

Description

Technical Field

[0001] The present invention relates to the technical field of image communication methods, and in particular to a perceptual video coding method combining saliency and just perceptible distortion. Background Art

[0002] With the rapid development of video applications and technologies, the amount of video data has grown exponentially, posing a huge challenge to data storage and communication. Existing video coding standards are mainly aimed at removing statistical redundancy in videos, and their compression rates are close to the theoretical upper limit. Perceptual video coding can adjust the coding method based on the characteristics of the human eye to reduce visual redundancy, thereby further improving the compression rate. It is an effective solution to this challenge. Perceptual video coding usually uses two visual characteristics of the human eye: saliency or just noticeable distortion. Saliency-based perceptual video coding allocates more bitrate to salient areas and allows appropriate distortion in non-salient areas to reduce the bitrate while maintaining subjective perceived quality. Perceptual video coding based on just noticeable distortion aims to allow distortion within the just noticeable distortion threshold in the encoded video, thereby reducing the bitrate.

[0003] Existing perceptual video coding has improved coding efficiency to a certain extent, but most methods only consider the single human visual factor and fail to fully build a visual perception model. Although some methods try to combine saliency with just-perceptible distortion characteristics, most of them simply add or multiply the two without fully considering the connotation and application scenarios of the two visual models, and therefore fail to effectively combine the two to improve coding efficiency. How to efficiently combine these two visual characteristics to further reduce visual redundancy is an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above problems in the existing methods, the present invention proposes a perceptual video coding method combining saliency and just perceptible distortion.

[0005] A perceptual video coding method combining saliency and just perceptible distortion, characterized by comprising the following steps:

[0006] S1: Get the video sequence to be encoded, and calculate the saliency value S(i, j) and the just noticeable distortion threshold JND(i, ​​j) of each pixel according to the saliency detection algorithm and the just noticeable distortion algorithm, where i and j are the horizontal and vertical coordinates of the pixel respectively;

[0007] S2: Calculate the block-level and frame-level saliency factors and just-perceptible distortion factors;

[0008] S3: Modify the video encoding standard method as follows to obtain a modified video encoding method:

[0009] In the quantization parameter selection phase, the quantization parameter offsets of significant and just perceptible distortion are calculated respectively, where the significant quantization parameter offset is:

[0010] ,

[0011] in, , Set to 5; when the block-level saliency value is greater than the frame-level saliency value, is negative to reduce distortion; when the block-level saliency value is smaller than the frame-level saliency value, is positive to allow for distortion; the just perceptible distortion parameter offset is:

[0012] ,

[0013] in, , Set to 2; when the just perceptible distortion factor is greater than the frame level just perceptible distortion factor, When it is positive, appropriate distortion is allowed; when the block-level just-perceptible distortion factor is less than the frame-level just-perceptible distortion factor, is negative, reducing distortion; then the two are integrated into the adaptive quantization parameter offset ; The quantization parameters used for encoding are:

[0014] ,

[0015] Among them, QP is the original encoding parameter;

[0016] In the rate-distortion optimization phase, the just perceptible distortion is incorporated into the distortion calculation to obtain the perceptual distortion term D p , and adjust the Lagrange multiplier λ by saliency to balance the rate distortion; the adjusted perceptual rate distortion loss is:

[0017] ,

[0018] Among them, λ p is the adjusted Lagrange multiplier, R is the bit rate used for encoding;

[0019] S4: Encode the video using the modified video encoding method.

[0020] A further technical approach is that the saliency detection algorithm adopts a video saliency detection algorithm based on deep learning, and the just perceptible distortion algorithm adopts a nonlinear additive mask model.

[0021] A further technical method is that the block-level and frame-level saliency factors or just noticeable distortion factors are average values ​​of the saliency or just noticeable distortion of the areas included in the block and the frame.

[0022] A further technical approach is that the block-level significance factor is:

[0023] ,

[0024] Among them, the size of the block is nxn; the frame-level saliency factor is:

[0025] ,

[0026] Among them, width and height are the width and height of the frame respectively.

[0027] A further technical approach is that the block-level just perceptible distortion factor is:

[0028] ,

[0029] The frame-level just perceptible distortion factor is:

[0030] ,

[0031] The calculation of the above factors prepares for the subsequent perceptual encoding process.

[0032] A further technical method is that the video coding standard method refers to AVC, HEVC, VVC video coding standards.

[0033] A further technical method is to integrate the two into an adaptive quantization parameter offset as the superposition effect of the two, as follows:

[0034] ,

[0035] It increases the offset in the same direction by 0.1 times.

[0036] A further technical approach is to use the perceptual distortion term D p Specifically manifested in the sum of squared errors and absolute errors; adjusted squared error and SSE p for:

[0037] ,

[0038] Adjusted absolute error SAD p for:

[0039] ,

[0040] Among them, I(i, j) and I'(i, j) represent the original pixel and the reconstructed pixel respectively. is the just perceptible distortion factor used to filter the distortion, and its calculation formula is:

[0041] ,

[0042] By adjusting the perceptual distortion term D p It can reflect the distortion characteristics that can be just perceived by the human eye, which is smaller than the original distortion item.

[0043] A further technical method is that the adjusted Lagrange multiplier is:

[0044] ,

[0045] Where λ is the original Lagrange multiplier, and a=0.8 and b=0.2 are set.

[0046] The advantages and positive effects of the present invention are:

[0047] The present invention can make full use of the two visual characteristics of human eye saliency and just noticeable distortion, apply them in quantization parameter selection and rate-distortion optimization algorithms, fully model the human eye visual system, thereby effectively removing visual redundancy, reducing encoding time, and improving video encoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0049] Figure 1 is a flowchart of a perceptual video coding method combining saliency and just-perceptible distortion in Embodiment 1 of the present invention;

[0050] Figure 2 is a schematic diagram of adjusting the quantization parameter selection link in the first embodiment of the present invention;

[0051] Figure 3 It is a schematic diagram of adjusting the rate-distortion optimization link in the first embodiment of the present invention;

[0052] Figure 4 and Figure 5 This is a comparison diagram of the encoded video effects in the first embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following is a clear and complete description of the technical methods in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0055] Embodiment 1

[0056] The embodiment of the present invention provides a perceptual video coding method for combining saliency and just perceptible distortion, characterized in that the steps are as follows: Figure 1 As shown, including:

[0057] S1: Get the video sequence to be encoded, and calculate the saliency value S(i, j) and the just noticeable distortion threshold JND(i, ​​j) of each pixel according to the saliency detection algorithm and the just noticeable distortion algorithm, where i and j are the horizontal and vertical coordinates of the pixel respectively;

[0058] S2: Calculate the block-level and frame-level saliency factors and just-perceptible distortion factors;

[0059] S3: Modify the video encoding standard method as follows to obtain a modified video encoding method:

[0060] In the quantization parameter selection phase, the quantization parameter offsets of significant and just perceptible distortion are calculated respectively, where the significant quantization parameter offset is:

[0061] ,

[0062] in, , Set to 5; when the block-level saliency value is greater than the frame-level saliency value, is negative to reduce distortion; when the block-level saliency value is smaller than the frame-level saliency value, is positive to allow for distortion; the just perceptible distortion parameter offset is:

[0063] ,

[0064] in, , Set to 2; when the just perceptible distortion factor is greater than the frame level just perceptible distortion factor, When it is positive, appropriate distortion is allowed; when the block-level just-perceptible distortion factor is less than the frame-level just-perceptible distortion factor, is negative, reducing distortion; then the two are integrated into the adaptive quantization parameter offset ; The quantization parameters used for encoding are:

[0065] ,

[0066] Among them, QP is the original encoding parameter; the algorithm flow is as follows Figure 2 As shown;

[0067] In the rate-distortion optimization phase, the just perceptible distortion is incorporated into the distortion calculation to obtain the perceptual distortion term D p , and adjust the Lagrange multiplier λ by saliency to balance the rate distortion; the adjusted perceptual rate distortion loss is:

[0068] ,

[0069] Among them, λ p is the adjusted Lagrange multiplier, R is the bit rate used for encoding; the algorithm flow is as follows Figure 3 As shown;

[0070] S4: Encode the video using the modified video encoding method.

[0071] The present invention provides a preferred embodiment to perform S1. The saliency detection algorithm adopts a video saliency detection algorithm based on deep learning, and the just perceptible distortion algorithm adopts a nonlinear additive mask model.

[0072] The present invention provides a preferred embodiment to perform S2. The block-level and frame-level saliency factors or just-noticeable distortion factors are average values ​​of saliency or just-noticeable distortion of the areas included in the block and the frame.

[0073] The block-level significance factor is:

[0074] ,

[0075] Among them, the size of the block is nxn; the frame-level saliency factor is:

[0076] ,

[0077] Among them, width and height are the width and height of the frame respectively.

[0078] The block-level just-noticeable distortion factor is:

[0079] ,

[0080] The frame-level just perceptible distortion factor is:

[0081] ,

[0082] The calculation of the above factors prepares for the subsequent perceptual encoding process.

[0083] The present invention provides a preferred embodiment to perform S3. The video coding standard method refers to AVC, HEVC, VVC video coding standards.

[0084] Integrating the two into an adaptive quantization parameter offset is the superposition effect of the two, as follows:

[0085] ,

[0086] It increases the offset in the same direction by 0.1 times.

[0087] Perceptual distortion term D p Specifically manifested in the sum of squared errors and absolute errors; adjusted squared error and SSE p for:

[0088] ,

[0089] Adjusted absolute error SAD p for:

[0090] ,

[0091] Among them, I(i, j) and I'(i, j) represent the original pixel and the reconstructed pixel respectively. is the just perceptible distortion factor used to filter the distortion, and its calculation formula is:

[0092] ,

[0093] By adjusting the perceptual distortion term D p It can reflect the distortion characteristics that can be just perceived by the human eye, which is smaller than the original distortion item.

[0094] The adjusted Lagrange multiplier is:

[0095] ,

[0096] Where λ is the original Lagrange multiplier, and a=0.8 and b=0.2 are set.

[0097] In order to verify the effectiveness of the above embodiment 1, the performance of the method of the present invention and the VVC encoding method are compared on the standard test video sequences Class B~E, and the commonly used BD-MOS and BD-Rate are selected for evaluation. The experimental results are shown in Table 1, the encoding time comparison is shown in Table 2, and the qualitative experiment is shown in Table 2. Figure 4 and Figure 5 shown.

[0098] Table 1 Subjective evaluation of videos encoded by this model

[0099]

[0100] It can be seen from Table 1 that, in this embodiment, while maintaining the subjective visual quality, the proposed method can save 22.87% of the bit rate compared with VVC.

[0101] Table 2 Encoding time of this method

[0102]

[0103] It can be seen from Table 2 that this embodiment can improve the encoding efficiency and shorten the encoding time.

[0104] Depend on Figure 4 and Figure 5 It can be seen that this embodiment can improve the encoding quality of the salient area and is more in line with the characteristics of the human eye.

[0105] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A perceptual video coding method combining saliency and just perceptible distortion, characterized in that: The steps include: S1: Get the video sequence to be encoded, and calculate the saliency value S(i, j) and the just noticeable distortion threshold JND(i, ​​j) of each pixel according to the saliency detection algorithm and the just noticeable distortion algorithm, where i and j are the horizontal and vertical coordinates of the pixel respectively; S2: Calculate the block-level and frame-level saliency factors and just-perceptible distortion factors; S3: Modify the video encoding standard method as follows to obtain a modified video encoding method: In the quantization parameter selection phase, the quantization parameter offsets of significant and just perceptible distortion are calculated respectively, where the significant quantization parameter offset is: ， in, and are block-level and frame-level saliency factors, respectively. , Set to 5; when the block-level saliency value is greater than the frame-level saliency value, is negative to reduce distortion; when the block-level saliency value is smaller than the frame-level saliency value, is positive to allow for distortion; the just perceptible distortion parameter offset is: ， in, and are the just perceptible distortion factors at the block level and frame level, , Set to 2; when the block-level just noticeable distortion factor is greater than the frame-level just noticeable distortion factor, When it is positive, distortion is allowed; when the block-level just-perceptible distortion factor is less than the frame-level just-perceptible distortion factor, is negative, reducing distortion; then the two are integrated into the adaptive quantization parameter offset ; The quantization parameters used for encoding are: ， Among them, QP is the original encoding parameter; In the rate-distortion optimization phase, the just perceptible distortion is incorporated into the distortion calculation to obtain the perceptual distortion term D p , and adjust the Lagrange multiplier λ by saliency to balance the rate distortion; the adjusted perceptual rate distortion loss is: ， Among them, λ p is the adjusted Lagrange multiplier, R is the bit rate used for encoding; S4: Encode the video using the modified video encoding method.

2. The perceptual video coding method for joint saliency and just perceptible distortion as claimed in claim 1, characterized in that The saliency detection algorithm adopts a video saliency detection algorithm based on deep learning, and the just perceptible distortion algorithm adopts a nonlinear additive mask model.

3. The perceptual video coding method for joint saliency and just perceptible distortion as claimed in claim 1, characterized in that The block-level and frame-level significance factors and just-perceptible distortion factors are average values ​​over the region where the item is located.

4. The perceptual video coding method for joint saliency and just perceptible distortion as claimed in claim 3, characterized in that The block-level significance factor is: ， Among them, the size of the block is nxn; the frame-level saliency factor is: ， Among them, width and height are the width and height of the frame respectively.

5. The perceptual video coding method for joint saliency and just perceptible distortion as claimed in claim 3, characterized in that The block-level just perceptible distortion factor is: ， The frame-level just perceptible distortion factor is: ， The calculation of the above factors prepares for the subsequent perceptual encoding process.

6. The perceptual video coding method for joint saliency and just perceptible distortion as claimed in claim 1, characterized in that The video coding standard method refers to AVC, HEVC, VVC video coding standards.

7. The perceptual video coding method for joint saliency and just perceptible distortion as claimed in claim 1, characterized in that The integration of the two into the adaptive quantization parameter offset is the superposition effect of the two, as follows: ， It increases the offset in the same direction by 0.1 times.

8. The perceptual video coding method for joint saliency and just perceptible distortion as claimed in claim 1, characterized in that The perceptual distortion term D p Specifically manifested in the sum of squared errors and absolute errors; adjusted squared error SSE p SAD p They are: ， ， Among them, I(i, j) and I'(i, j) represent the original pixel and the reconstructed pixel respectively. is the just perceptible distortion factor used to filter the distortion, and its calculation formula is: ， By adjusting the perceptual distortion term D p It can reflect the distortion characteristics that can be just perceived by the human eye, which is smaller than the original distortion item.

9. The perceptual video coding method for joint saliency and just perceptible distortion as claimed in claim 1, characterized in that The adjusted Lagrange multiplier is: ， Where λ is the original Lagrange multiplier, set a=0.8, b=0.2.

Images