An improved method based on HEVC sample adaptive compensation

Abstract

The present invention relates to an improved method based on HEVC sample adaptive compensation, and belongs to the technical field of video coding and decoding. The method comprises: preprocessing the input video signal to extract key frames and non-key frames; performing block-based motion estimation on the key frames to obtain the motion offset of each block; using the motion offset to perform motion compensation on the non-key frames to generate compensated frames; performing sample adaptive compensation on the compensated frames to improve the quality of the compensated frames; comparing the compensated frames with the original frames to generate residual signals; transforming and quantizing the residual signals to generate compressed video data; and entropy encoding the compressed video data to generate the final compressed video stream. The block effect problem caused by SAO can be improved more effectively, and the subjective block effect problem can be avoided as much as possible.

Description

Technical Field

[0001] The present invention belongs to the technical field of video coding and decoding, and in particular relates to an improved method based on HEVC sample adaptive compensation. Background Art

[0002] In the boundary area where the pixel value of the image changes dramatically, a ripple-like distortion phenomenon usually occurs after encoding-decoding reconstruction. This distortion is called the ringing effect. The root cause of the ringing effect is that the high-frequency components of the boundary area are lost during the encoding process. In order to reduce the loss of high-frequency components without reducing the encoding efficiency, HEVC introduces SAO (sample adaptive offset) technology to compensate for the ringing area from the pixel domain.

[0003] like Figure 2 As shown in the figure, the horizontal axis represents the pixel coordinates, the vertical axis represents the pixel value, the dotted line represents the actual value, and the solid line represents the result after encoding. It can be seen that after encoding, the edge should have a large fluctuation. The SAO algorithm adds an offset value to the sample point to make the solid line closer to the dotted line, such as adding a negative offset to the sample point at the peak and a positive offset to the sample point at the trough.

[0004] SAO in HEVC includes BO (Band offset) and EO (Edge offset). By analyzing the relationship between the data after DB and the original YUV, the data after DB is operated so that the reconstructed data is as close to the original YUV as possible, so as to improve the PSNR. One of the most basic ideas is to interpolate the reconstructed data of DB with each pixel in the same position in the original YUV, and pass this interpolation to the decoding end, so that the original YUV can be completely restored. However, if an offset is transmitted for each pixel, the bit rate will be very high, affecting the compression effect. Therefore, SAO in HEVC is divided into boundary compensation mode EO and strip compensation mode BO in units of CTU. Boundary compensation (EO) classifies the current pixel by comparing the size of the current pixel with that of the adjacent pixels. In this mode, SAO first needs to select a gradient mode for the CTB. In order to balance complexity and coding performance, EO uses 4 one-dimensional modes: horizontal / vertical / 45 degrees / 135 degrees, such as Figure 3 As shown, in the figure, c represents the current pixel value, and a and b represent two adjacent pixel values;

[0005] The encoder will select one of the four EO types according to the RDO criterion. When a gradient mode is selected for a CTU, all pixel values ​​in the CTU are Figure 4The conditions can be divided into 5 categories; if the current pixel value does not belong to type 1-4, SAO is turned off. For types 1-4, SAO assigns an offset to each type, and this offset will be added to the reconstructed pixel, that is, SAO will assign an offset to each EO type according to the classification. Therefore, each CTU only needs 4 offsets, which can reduce the bit rate consumption.

[0006] Strip compensation (BO) divides the pixel value intensity level into several strips, makes a histogram statistics for each pixel in the CTU, classifies it into each strip, calculates the average value of the pixel value in each strip as the offset of the current strip, and the pixels in each strip have the same compensation value. When performing compensation, the corresponding strip compensation value is selected according to the strip where the reconstructed pixel point is located. The SAO algorithm divides the valid pixel value range (0-255) into 32 strips on average, and selects 4 consecutive strips through RDO. When the pixel value in the CTU is in these 4 selected strips, the compensation value of the strip is used to compensate the pixel.

[0007] SAO uses the difference between the CTU reconstructed pixels and the original pixels to calculate the offset, and uses the offset to compensate the final encoded pixels in order to make the encoded pixel values ​​closer to the original pixel values. However, in actual encoding scenarios, when the SAO algorithm is turned on, although the PSNR of the video will be better, subjectively speaking, the CTU with SAO will be more prone to block effects. Summary of the invention

[0008] In order to solve the above problems existing in the prior art, the present invention provides an improved method based on HEVC sample adaptive compensation;

[0009] The purpose of the present invention can be achieved through the following technical solutions:

[0010] An improved method based on HEVC sample adaptive compensation, comprising:

[0011] The input video signal is preprocessed to extract key frames and non-key frames; block-based motion estimation is performed on the key frames to obtain motion offsets of each block; motion compensation is performed on the non-key frames using the motion offsets to generate compensated frames; sample point adaptive compensation is performed on the compensated frames to improve the quality of the compensated frames; the compensated frames are compared with the original frames to generate residual signals; the residual signals are transformed and quantized to generate compressed video data; and the compressed video data is entropy encoded to generate a final compressed video stream.

[0012] Specifically, the preprocessing method is to form a mean square error curve by calculating the mean square error between adjacent frames, and extract the extreme points on the mean square error curve. If the difference between the extreme point and the two frames before and after is the average of the differences between all frames, then the video frame corresponding to the extreme point is extracted to form a key frame.

[0013] Specifically, the motion estimation method is: classifying the motion intensity of each coding tree unit according to the temporal and spatial characteristics of the key frame image, including a non-motion area, a slight motion area, and a severe motion area.

[0014] Specifically, the motion compensation method is:

[0015] The pixel values ​​of the original image and the pixel values ​​after the deblocking filter are obtained in the coding tree unit, and the difference and sum of the pixel values ​​of the original image and the pixel values ​​after the deblocking filter in the coding tree unit are calculated, and then divided by the statistics of the number of pixels in the corresponding edge offset direction or strip in the coding tree unit to obtain the initial offset of the sample adaptive offset. The calculation formula is:

[0016]

[0017] Among them, offset0 is the initial offset, n is the statistics of the number of pixels in the edge offset direction or in the strip, Pixel ori is the original image pixel value, Pixel rec is the pixel value after deblocking filter;

[0018] The motion level of each coding unit block is calculated using the absolute difference sum obtained by inter-frame pre-search of each coding unit in the coding tree unit and the absolute difference sum obtained by intra-frame pre-search. The coding unit is set to a size of 16×16. The motion levels of the coding unit blocks are accumulated and statistically distributed. The motion level of the corresponding coding tree unit is determined according to the distribution result. The initial offset allocation reference coefficient is compensated and adjusted according to the motion level of the coding tree unit.

[0019] Specifically, the edge offset direction is determined according to a gradient mode of edge compensation of sample adaptive offset in the High Efficiency Video Coding standard.

[0020] Specifically, the statistics of the number of pixels in the stripe are obtained according to the histogram statistics of the stripe distribution of the sample adaptive offset in the high efficiency video coding standard.

[0021] Specifically, the inter-frame pre-search and the intra-frame pre-search are implemented by comparing the pixel value differences between adjacent frames. In the inter-frame pre-search process, the encoded frames around the current coding unit are first selected as reference frames, and then the area most similar to the current coding unit is searched in the reference frame. By calculating the absolute difference between different areas and the current coding unit, the area with the smallest absolute difference is selected as the best matching block. The intra-frame pre-search process is the same as the inter-frame pre-search, but the reference area is limited to the current frame. The best matching block is found by comparing the absolute difference between the current coding unit and the surrounding pixel blocks.

[0022] Specifically, the sample adaptive compensation is adjusted based on the motion level of the current coding unit.

[0023] Specifically, the residual signal is processed using transform coding technology, and the local characteristics are dynamically adjusted according to the image content through an adaptive transform matrix, and the transform coefficients need to be quantized. The step size of the quantization process is selected based on the complexity of the image content and the bit rate of the encoding target.

[0024] Specifically, the transformation and quantization method uses a gradient descent method to iteratively optimize the adaptive transformation matrix to minimize the quantization error.

[0025] The beneficial effects of the present invention are:

[0026] The above-mentioned improved method can effectively improve the efficiency and quality of video compression. First, by preprocessing to extract key frames and non-key frames, unnecessary data processing can be reduced and coding efficiency can be improved. Secondly, block-based motion estimation can accurately capture the motion information between video frames, thereby providing an accurate reference for motion compensation, and calculate the reference coefficient according to the setting of the motion level to correct the offset of SAO, effectively minimizing its impact on non-motion areas while retaining the optimization of SAO on motion areas. During the motion compensation process, sample adaptive compensation further optimizes the quality of the compensated frame, making the compensated video closer to the original video. It can effectively improve the block effect problem caused by SAO and avoid the subjective block effect problem as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0028] Figure 1 A schematic flow chart of an improved method based on HEVC sample adaptive compensation according to the present invention;

[0029] Figure 2 A schematic diagram of the ringing effect in the present invention;

[0030] Figure 3 It is a schematic diagram of a one-dimensional gradient mode for boundary compensation in the present invention;

[0031] Figure 4 This is a schematic diagram of the classification of EO types in the present invention;

[0032] Figure 5 A schematic diagram of the conditions for determining the motion level of a CTU in the present invention;

[0033] Figure 6 It is a schematic diagram of the structure of the encoder in the present invention;

[0034] Figure 7 Schematic diagram of input and output of the SAO module in the present invention. DETAILED DESCRIPTION

[0035] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation method, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0036] See also Figure 1-7 , an improved method based on HEVC sample adaptive compensation, comprising:

[0037] The input video signal is preprocessed to extract key frames and non-key frames; block-based motion estimation is performed on the key frames to obtain motion offsets of each block; motion compensation is performed on the non-key frames using the motion offsets to generate compensated frames; sample point adaptive compensation is performed on the compensated frames to improve the quality of the compensated frames; the compensated frames are compared with the original frames to generate residual signals; the residual signals are transformed and quantized to generate compressed video data; and the compressed video data is entropy encoded to generate a final compressed video stream.

[0038] Specifically, the preprocessing method is to form a mean square error curve by calculating the mean square error between adjacent frames, and extract the extreme points on the mean square error curve. If the difference between the extreme point and the two frames before and after is the average of the differences between all frames, then the video frame corresponding to the extreme point is extracted to form a key frame.

[0039] Specifically, the motion estimation method is: classifying the motion intensity of each coding tree unit according to the temporal and spatial characteristics of the key frame image, including a non-motion area, a slight motion area, and a severe motion area.

[0040] Specifically, the motion compensation method is:

[0041] The pixel values ​​of the original image and the pixel values ​​after the deblocking filter are obtained in the coding tree unit, and the difference and sum of the pixel values ​​of the original image and the pixel values ​​after the deblocking filter in the coding tree unit are calculated, and then divided by the statistics of the number of pixels in the corresponding edge offset direction or strip in the coding tree unit to obtain the initial offset of the sample adaptive offset. The calculation formula is:

[0042]

[0043] Among them, offset0 is the initial offset, n is the statistics of the number of pixels in the edge offset direction or in the strip, Pixel ori is the original image pixel value, Pixel rec is the pixel value after deblocking filter;

[0044] The motion level of each coding unit block is calculated using the absolute difference sum obtained by inter-frame pre-search of each coding unit in the coding tree unit and the absolute difference sum obtained by intra-frame pre-search. The coding unit is set to a size of 16×16. The motion levels of the coding unit blocks are accumulated and statistically distributed. The motion level of the corresponding coding tree unit is determined according to the distribution result. The initial offset allocation reference coefficient is compensated and adjusted according to the motion level of the coding tree unit.

[0045] In this embodiment, in the monitoring scene, the difference between the reconstructed pixels and the original pixel values ​​of a large number of background areas is not large, so the area that needs SAO compensation generally exists in the area with large movement. Therefore, if the moving area can be found, a smaller offset is used for the non-moving area, so that the influence of SAO on the non-moving area and the slightly moving area can be minimized; the motion intensity of each CTU is graded by some temporal and spatial features of the original image, and divided into three areas of level 0 (non-moving), level 1 (slightly moving) and level 2 (violent moving). For the non-moving area, the compensation value of SAO is α*offset (α is a small coefficient), for the slightly moving area, the compensation value of SAO is set to μ*offset (μ is a coefficient between α and β), and for the violent moving area, β*offset (β is a large coefficient) is used. Specifically, the parameters of the present invention are α=0.05, μ=0.35, β=1. This set of parameters can effectively minimize the influence of SAO on the non-moving area while retaining the optimization of SAO on the moving area. In order to more accurately identify the motion area, the 16X16 size unit is used to calculate the features in the time domain and the spatial domain, and the number of motion levels of each 16X16 block in the CTU is accumulated, such as Figure 5 Make judgment as shown.

[0046] In this embodiment, the sample adaptive compensation SAO module is located after the encoder Deblocking module, such as Figure 6 The operation object is a reconstructed frame, with a CTU as the basic unit. Through the calculation of PSNR (Peak Signal-to-Noise Ratio), it can be known that the sum of the squares of the differences between the reconstructed data and the original data is the factor that determines the PSNR. SAO compensates the pixels after deblocking by analyzing the relationship between the pixels after deblocking and the original pixels, making them as close to the original pixels as possible, so as to achieve the purpose of improving PSNR. Figure 7 As shown, the SAO module inputs the original pixel value and the pixel value after deblocking, and outputs the calculated SAO offset, which is passed to the entropy module (which compresses the encoded data) for encoding.

[0047] Specifically, the edge offset direction is determined according to a gradient mode of edge compensation of sample adaptive offset in the High Efficiency Video Coding standard.

[0048] Specifically, the statistics of the number of pixels in the stripe are obtained according to the histogram statistics of the stripe distribution of the sample adaptive offset in the high efficiency video coding standard.

[0049] Specifically, the inter-frame pre-search and the intra-frame pre-search are implemented by comparing the pixel value differences between adjacent frames. In the inter-frame pre-search process, the encoded frames around the current coding unit are first selected as reference frames, and then the area most similar to the current coding unit is searched in the reference frame. By calculating the absolute difference between different areas and the current coding unit, the area with the smallest absolute difference is selected as the best matching block. The intra-frame pre-search process is the same as the inter-frame pre-search, but the reference area is limited to the current frame. The best matching block is found by comparing the absolute difference between the current coding unit and the surrounding pixel blocks.

[0050] Specifically, the sample adaptive compensation is adjusted based on the motion level of the current coding unit.

[0051] Specifically, the residual signal is processed using transform coding technology, and the local characteristics are dynamically adjusted according to the image content through an adaptive transform matrix, and the transform coefficients need to be quantized. The step size of the quantization process is selected based on the complexity of the image content and the bit rate of the encoding target.

[0052] Specifically, the transformation and quantization method uses a gradient descent method to iteratively optimize the adaptive transformation matrix to minimize the quantization error.

[0053] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0054] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0055] The program code included on the computer readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages-such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0056] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. An improved method based on HEVC sample adaptive compensation, characterized in that: include: Preprocessing the input video signal to extract key frames and non-key frames; Performing block-based motion estimation on the key frame to obtain motion offsets of each block; Performing motion compensation on the non-key frame using the motion offset to generate a compensated frame; Performing sample adaptive compensation on the compensation frame; Comparing the compensated frame with the original frame to generate a residual signal; The residual signal is transformed and quantized to generate compressed video data; and the compressed video data is entropy encoded to generate a final compressed video stream.

2. The method according to claim 1, characterized in that The preprocessing method is to form a mean square error curve by calculating the mean square error between adjacent frames, extract the extreme points on the mean square error curve, and if the difference between the extreme point and the two frames before and after is the average of the differences between all frames, then the video frame corresponding to the extreme point is extracted to form a key frame.

3. The method according to claim 1, characterized in that: The motion estimation method is: classifying the motion intensity of each coding tree unit according to the temporal and spatial characteristics of the key frame image, including a non-motion area, a slight motion area and a severe motion area.

4. The method according to claim 3, characterized in that The motion compensation method is: The pixel values ​​of the original image and the pixel values ​​after the deblocking filter are obtained in the coding tree unit, and the difference and sum of the pixel values ​​of the original image and the pixel values ​​after the deblocking filter in the coding tree unit are calculated, and then divided by the statistics of the number of pixels in the corresponding edge offset direction or strip in the coding tree unit to obtain the initial offset of the sample adaptive offset. The calculation formula is: Among them, offset0 is the initial offset, n is the statistics of the number of pixels in the edge offset direction or in the strip, Pixel ori is the original image pixel value, Pixel rec is the pixel value after deblocking filter; The motion level of each coding unit block is calculated using the absolute difference sum obtained by inter-frame pre-search of each coding unit in the coding tree unit and the absolute difference sum obtained by intra-frame pre-search. The coding unit is set to a size of 16×16. The motion levels of the coding unit blocks are accumulated and statistically distributed. The motion level of the corresponding coding tree unit is determined according to the distribution result. The initial offset allocation reference coefficient is compensated and adjusted according to the motion level of the coding tree unit.

5. The method according to claim 1, characterized in that The edge offset direction is determined according to a gradient mode of edge compensation of sample adaptive offset in the High Efficiency Video Coding standard.

6. The method according to claim 1, characterized in that The statistics of the number of pixels in the stripe are obtained based on the histogram statistics of the stripe distribution of the sample adaptive offset in the high efficiency video coding standard.

7. The method according to claim 1, characterized in that The inter-frame pre-search and the intra-frame pre-search are implemented by comparing the pixel value differences between adjacent frames. In the inter-frame pre-search process, the encoded frames around the current coding unit are first selected as reference frames, and then the area most similar to the current coding unit is searched in the reference frame. By calculating the absolute difference between different areas and the current coding unit, the area with the smallest absolute difference is selected as the best matching block. The intra-frame pre-search process is the same as the inter-frame pre-search, but the reference area is limited to the current frame. The best matching block is found by comparing the absolute difference between the current coding unit and the surrounding pixel blocks.

8. The method according to claim 7, characterized in that The sample adaptive compensation is adjusted based on the motion level of the current coding unit.

9. The method according to claim 1, characterized in that: The residual signal is processed using transform coding technology, and the local characteristics are dynamically adjusted according to the image content through an adaptive transform matrix, and the transform coefficients need to be quantized. The step size of the quantization process is selected based on the complexity of the image content and the bit rate of the encoding target.

10. The method according to claim 9, characterized in that The transformation and quantization method uses a gradient descent method to iteratively optimize the adaptive transformation matrix to minimize the quantization error.

Images