An encoding method based on a region of interest and LCU features

Abstract

The application relates to an encoding method based on a region of interest and LCU features, comprising the following steps: S1, acquiring a video sequence and corresponding video frames; S2, calculating average texture complexity and texture complexity; S3, calculating average distortion degree and distortion degree, and average structure feature and structure feature; S4, calculating the ratio of the average distortion degree and the average structure feature, and the ratio of the distortion degree and the structure feature; S5, calculating a related parameter weighted combination; S6, marking the region of interest and the region of non-interest; S7, calculating the importance degree of a current encoding LCU to an encoding frame; S8, completing the updating of a QP parameter for all encoding frames; and S9, encoding the video sequence based on the updated QP parameter. Compared with the prior art, the application has the advantages of good rate-distortion performance.

Description

Technical Field

[0001] This invention relates to the technical field of video coding, and in particular to a coding method based on region of interest (ROI) and LCU features. Background Technology

[0002] In video coding technology, the bitrate control module is a key module that ensures the quality of video transmission under a given target bitrate. It selects a series of coding parameters under a given channel bandwidth resource, so that the actual bitrate of the encoding fluctuates around the set bitrate, making the best use of bandwidth resources and ensuring the quality of the encoded video, while reducing the distortion of the reconstructed video.

[0003] High-efficiency video coding bitrate allocation strategies do not consider visual redundancy, and the LCU layer image content features are considered in a single way, resulting in poor rate-distortion performance. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing an encoding method based on region of interest and LCU features. The method uses the Scharr operator to calculate the gradient of the LCU to be encoded and represents it as the texture complexity of the encoding region. The SATD value of the LCU pixel residual is calculated to reflect the distortion degree of the current LCU, and the structural strength of the LCU is calculated to reflect the structural information of the current LCU. By combining the above-mentioned LCU feature information with the region of interest information, a new strategy for adjusting the guiding quantization parameter QP is formed.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] An encoding method based on region of interest (ROI) and LCU features includes the following steps:

[0007] S1. Obtain the video sequence and the corresponding video frames;

[0008] S2. Calculate the normalized average texture complexity corresponding to the current encoded frame and the normalized texture complexity corresponding to the current encoded LCU;

[0009] S3. Calculate the average distortion level corresponding to the current encoded frame, the distortion level corresponding to the current encoded LCU, the average structural feature corresponding to the current encoded frame, and the structural feature corresponding to the current encoded LCU.

[0010] S4. Calculate the ratio of average distortion to average structural features corresponding to the normalized current coded frame, and at the same time, calculate the ratio of distortion to structural features corresponding to the normalized current coded LCU.

[0011] S5. Based on the normalized average texture complexity and the corresponding ratio of the current encoded frame, obtain the weighted combination of relevant parameters corresponding to the current encoded frame. Based on the normalized texture complexity and the corresponding ratio of the current encoded LCU, obtain the weighted combination of relevant parameters corresponding to the current encoded LCU.

[0012] S6. If the current encoded LCU is in the region of interest of the video frame, it is marked as an LCU of interest; otherwise, it is marked as an LCU of no interest. The region of interest of the video frame is determined according to the YOLOX object detection network.

[0013] S7. Subtract the weighted combination of relevant parameters corresponding to the current encoded LCU from the weighted combination of relevant parameters corresponding to the current encoded frame to obtain the importance of the current encoded LCU to the encoded frame.

[0014] S8. Calculate the QP parameter based on the current coding LCU marking result in S6 and the importance of the coding frame in S7, select a new coding LCU, return to S2, and continue until the QP parameter is updated for all coding LCUs in the current coding frame. Then select a new coding frame and return to S2, and continue until the QP parameter is updated for all coding frames.

[0015] S9. Encode the video sequence based on the updated QP parameters.

[0016] Furthermore, the process of calculating texture complexity is as follows:

[0017] The Scharr operator is used, and its edge detection filter size is set to 3x3. The gradient values ​​of the pixel in the X and Y directions are calculated to obtain the texture intensity value of the current pixel. The texture complexity of the encoded LCU is obtained by synthesizing the texture intensity values ​​of all pixels in the encoded LCU.

[0018] The average texture complexity of an encoded frame is obtained by averaging the texture complexity of all encoded LCUs within that frame.

[0019] Furthermore, the texture intensity value of the current pixel is:

[0020]

[0021] Among them, G s G represents the texture intensity value of the current pixel. x G represents the gradient value of a pixel in the X direction. y This represents the gradient value in the Y direction of the pixel.

[0022] The gradient value of the pixel in the X direction is:

[0023]

[0024] The gradient value of the pixel in the Y direction is:

[0025]

[0026] Where P is a 3×3 pixel matrix centered on the current encoded pixel.

[0027] Furthermore, the degree of distortion of the encoded LCU is represented by the SATD value of the LCU pixel residual, and the expression for the SATD value is:

[0028]

[0029] Where M is the size of the square matrix, and H is the normalized Hadamard matrix of size M×M;

[0030] The average distortion level of a coded frame is obtained by averaging the distortion levels of all coded LCUs within that frame.

[0031] Furthermore, the structural features corresponding to the LCU encoding are:

[0032]

[0033] Among them, SS m Here, m represents the structural feature value of the encoded LCU, indicating the LCU sequence number. and Let represent the partial derivative, q represent the current encoded pixel, F represent the input signal, and MB represent the current encoded LCU;

[0034] The average structural features corresponding to the coded frame are obtained by averaging the structural features of all coded LCUs within the coded frame.

[0035] Furthermore, the weighted combination of the relevant parameters corresponding to the encoded frame is as follows:

[0036] ω avg =a·G avg +(1-a)·SH avg

[0037] Where, ω avg The weighted combination of relevant parameters corresponding to the encoded frame, where 'a' is the weighting coefficient, taking a value between 0 and 1, and G... avg SH represents the normalized average texture complexity corresponding to the encoded frame. avg It is the ratio of the average distortion to the average structural features corresponding to the current coded frame in the normalized form.

[0038] Furthermore, the weighted combination of relevant parameters corresponding to the LCU encoding is as follows:

[0039] ω curlcu =a·G curlcu+(1-a)·SH curlcu

[0040] Where, ω curlcu This is a weighted combination of relevant parameters corresponding to the LCU encoding, where 'a' is a weighting coefficient with a value between 0 and 1, and G... curlcu To encode the normalized texture complexity corresponding to LCU, SH curlcu It is the ratio of the distortion level to the structural features corresponding to the normalized current encoded LCU.

[0041] Furthermore, the weighting coefficient 'a' is set to 0.8.

[0042] Furthermore, the current importance of the encoded LCU to the encoded frame is as follows:

[0043] ω cur =ω curlcu -ω avg

[0044] Where, ω cur ω represents the importance of the current encoding LCU to the encoded frame. avg ω is a weighted combination of the relevant parameters corresponding to the encoded frame. curlcu Weighted combination of relevant parameters for encoding LCU.

[0045] Furthermore, the optimization formula for parameter QP is:

[0046]

[0047]

[0048]

[0049] Among them, QP est ω is the QP value predicted by the encoder initially. cur ROI0 represents the importance of the current encoded LCU to the encoded frame, indicating that the current encoded LCU is an uninterested LCU. cur0 ω represents the importance of the uninterested LCU to the coded frame, ROI1 represents the currently coded LCU as the LCU of interest, and ω represents the LCU of interest. cur1 This indicates the level of importance of the LCU (Low-Value Unit) to the encoded frame.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) This invention utilizes texture complexity, pixel residual SATD and structural strength information to construct a new LCU weight representation formula to guide the adjustment of quantization parameter QP. Compared with the prior art, it has better RD performance and subjective visual quality.

[0052] (2) The present invention obtains more accurate region of interest information in each frame and combines it with LCU features to obtain better BD-Rate gain based on SSIM. Attached Figure Description

[0053] Figure 1 This is a flowchart of the present invention;

[0054] Figure 2 This is a flowchart illustrating the region of interest in this invention. Detailed Implementation

[0055] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0056] Definitions:

[0057] LCU: Maximum coding unit.

[0058] Scharr operator: An image edge detection algorithm used to calculate the gradient values ​​of pixels in the X and Y directions.

[0059] SATD: Sum of Absolute Variations.

[0060] SSIM: Structural Similarity.

[0061] YOLOX: A type of object detection network.

[0062] QP: Quantization parameter.

[0063] RD: The relationship between bitrate and distortion under specific coding parameters in video coding, used to reflect the rate-distortion performance of the encoder.

[0064] Hadamard: The Hadamard transform is a linear transform that converts a signal from the time domain to the frequency domain.

[0065] R-λ: A bitrate control model in video coding.

[0066] This invention proposes an encoding method based on Region of Interest (ROI) and LCU features. The flowchart of the method is as follows: Figure 1 As shown. The method includes the following steps:

[0067] S1. Obtain the video sequence and the corresponding video frames;

[0068] S2. Calculate the normalized average texture complexity corresponding to the current encoded frame and the normalized texture complexity corresponding to the current encoded LCU;

[0069] S3. Calculate the average distortion level corresponding to the current encoded frame, the distortion level corresponding to the current encoded LCU, the average structural feature corresponding to the current encoded frame, and the structural feature corresponding to the current encoded LCU.

[0070] S4. Calculate the ratio of average distortion to average structural features corresponding to the normalized current coded frame, and at the same time, calculate the ratio of distortion to structural features corresponding to the normalized current coded LCU.

[0071] S5. Based on the normalized average texture complexity and the corresponding ratio of the current encoded frame, obtain the weighted combination of relevant parameters corresponding to the current encoded frame. Based on the normalized texture complexity and the corresponding ratio of the current encoded LCU, obtain the weighted combination of relevant parameters corresponding to the current encoded LCU.

[0072] S6. If the current encoded LCU is in the region of interest of the video frame, it is marked as an LCU of interest; otherwise, it is marked as an LCU of no interest. The region of interest of the video frame is determined according to the YOLOX object detection network.

[0073] S7. Subtract the weighted combination of relevant parameters corresponding to the current encoded LCU from the weighted combination of relevant parameters corresponding to the current encoded frame to obtain the importance of the current encoded LCU to the encoded frame.

[0074] S8. Calculate the QP parameter based on the current coding LCU marking result in S6 and the importance of the coding frame in S7, select a new coding LCU, return to S2, and continue until the QP parameter is updated for all coding LCUs in the current coding frame. Then select a new coding frame and return to S2, and continue until the QP parameter is updated for all coding frames.

[0075] S9. Encode the video sequence based on the updated QP parameters.

[0076] In S6, the YOLOX object detection network offers a good balance between speed and accuracy. During video encoding, video frames are input into the object detection network to segment regions of interest (ROIs) and non-ROIs, and these are then labeled with their corresponding Logical Units (LCUs). The process of segmenting ROIs using the YOLOX object detection network is as follows: Figure 2 As shown.

[0077] In S2, texture is used as one of the features. Texture can reflect the homogeneity in the image. The gradient of the LCU to be encoded is calculated using the Scharr operator and expressed as the texture complexity of the encoded region.

[0078] The process of calculating texture complexity is as follows:

[0079] The Scharr operator's edge detection filter size is 3x3. The weight coefficients in the filter can be amplified to increase the difference between pixel values. The gradient values ​​in the X and Y directions of the pixel are calculated using formulas (1) and (2).

[0080]

[0081]

[0082] After calculating the gradient values ​​in the X and Y directions of the current pixel, the texture intensity value of the current pixel is calculated using formula (3):

[0083]

[0084] In S3, the degree of distortion is used as one of the features. The Hadamard transform is usually used to calculate the SATD (sum of absolute transformed difference) of the residual, which can reflect the energy distribution of the current coding region. The degree of distortion of the current LCU is reflected by calculating the SATD value of the LCU pixel residual.

[0085] The calculation process for the SATD value of pixel residuals is as follows:

[0086] The SATD value of the LCU pixel residual is calculated to reflect the current distortion level of the LCU and is used as part of the QP adjustment strategy. SATD is expressed by formula (4):

[0087]

[0088] M is the size of the square matrix, and H is the normalized Hadamard matrix of size M.

[0089] In S3, structural features are used as one of the features. The ordered information in the image is related to the skeleton and edges of the object. By using the information of the edge structure, the approximate content of the video can be quickly obtained, and the structural strength of the LCU is calculated to reflect the structural information of the current LCU.

[0090] The calculation process for structural strength is as follows:

[0091] Through the combined effect of texture and structural information, people can obtain the complete content of the video. The expression for structural information is shown in equation (5):

[0092]

[0093] Where F is the input signal. and Let represent the partial derivative, and q represent the current encoded pixel. The structural information not only considers the temporal relationship between encoded pixels but also calculates the relationship with pixels in adjacent frames, which better reflects the structural information of each LCU.

[0094] In summary, S2 and S3, quantization parameters are adjusted in S4 and S5. To address the shortcomings of quantization parameter adjustment in the R-λ model-based rate control algorithm, this invention comprehensively considers the texture features, distortion level, and structural features of the image coding region, and proposes a calculation formula for LCU coding weights. First, the average texture complexity of the current coding frame is calculated according to equation (3), and after normalization, it is expressed as G. avg Calculate the normalized texture complexity G of the current encoded LCU. curlcu The average distortion H of the current encoded frame is calculated using equations (4) and (5), respectively. avg With average structural characteristics S avg Then calculate S avg With H avg The ratio is normalized using SH. avg This means calculating and normalizing the ratio of the structural features to the distortion level of the current encoded LCU, using SH... curlcu The weighted combination of the coded frame-related parameters is given by equation (6), and the weighted combination of the coded LCU-related parameters is given by equation (7):

[0095] ω avg =a·G avg +(1-a)·SH avg (6)

[0096] ω curlcu =a·G curlcu +(1-a)·SH curlcu (7)

[0097] In the above formula, 'a' is a weighting coefficient, and its value should be between 0 and 1. After multiple experiments and statistical analysis of coding performance, the value of 'a' was determined to be 0.8.

[0098] In S6 and S7, before adjusting the quantization parameters, the region of interest (ROI) in the encoded frame is divided using a region of interest network. The LCUs placed within the ROI are then marked as ROI1, while the LCUs not in the ROI are marked as ROI0.

[0099] In S8 and S9, equation (8) is used to reflect the importance of the current encoded LCU to the encoded frame, and is marked as ω based on whether it is in the region of interest. cur0 and ω cur1 As shown in equation (9), the quantization parameter QP is adjusted according to equation (10).

[0100] ω cur =ω curlcu -ω avg (8)

[0101]

[0102]

[0103] This invention addresses the problem of decreased visual quality when watching videos due to efficient video coding bitrate allocation strategies neglecting visual redundancy and considering only a single image content feature in the LCU layer. It proposes a bitrate allocation algorithm based on Region of Interest (ROI) and LCU features. A pre-trained object detection network divides the video frame content into ROIs and other regions, labeling the corresponding LCUs. The Scharr operator reflects the texture complexity of the current coding region, the sum of absolute errors (SATD) between the coding and prediction regions reflects the distortion level of the current region, and structural strength reflects the structural regularity of the current region. A new weight calculation formula is constructed to guide the adjustment of quantization parameters. Experimental results show that the proposed algorithm achieves an average gain of 0.64% in the Y component of SSIM under low-latency P configuration, improving the subjective quality of the video.

[0104] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A coding method based on region of interest and LCU features, characterized in that, Includes the following steps: S1. Obtain the video sequence and the corresponding video frames; S2. Calculate the normalized average texture complexity corresponding to the current encoded frame and the normalized texture complexity corresponding to the current encoded LCU; S3. Calculate the average distortion level corresponding to the current encoded frame, the distortion level corresponding to the current encoded LCU, the average structural feature corresponding to the current encoded frame, and the structural feature corresponding to the current encoded LCU. S4. Calculate the ratio of average distortion to average structural features corresponding to the normalized current coded frame, and at the same time, calculate the ratio of distortion to structural features corresponding to the normalized current coded LCU. S5. Based on the normalized average texture complexity corresponding to the current encoded frame and the ratio of the normalized average distortion and average structural features corresponding to the current encoded frame, obtain the weighted combination of relevant parameters corresponding to the current encoded frame. Based on the normalized texture complexity and the corresponding ratio corresponding to the current encoded LCU, obtain the weighted combination of relevant parameters corresponding to the current encoded LCU. S6. If the current encoded LCU is in the region of interest of the video frame, it is marked as an LCU of interest; otherwise, it is marked as an LCU of no interest. The region of interest of the video frame is determined according to the YOLOX object detection network. S7. Subtract the weighted combination of relevant parameters corresponding to the current encoded LCU from the weighted combination of relevant parameters corresponding to the current encoded frame to obtain the importance of the current encoded LCU to the encoded frame. S8. Calculate the QP parameter based on the current coding LCU marking result in S6 and the importance of the coding frame in S7, select a new coding LCU, return to S2, and continue until the QP parameter is updated for all coding LCUs in the current coding frame. Then select a new coding frame and return to S2, and continue until the QP parameter is updated for all coding frames. S9. Encode the video sequence based on the updated QP parameters.

2. The encoding method based on region of interest and LCU features according to claim 1, characterized in that, The process of calculating texture complexity is as follows: The Scharr operator is used, and its edge detection filter size is set to 3x3. The gradient values ​​of the pixel in the X and Y directions are calculated to obtain the texture intensity value of the current pixel. The texture complexity of the encoded LCU is obtained by synthesizing the texture intensity values ​​of all pixels in the encoded LCU. The average texture complexity of an encoded frame is obtained by averaging the texture complexity of all encoded LCUs within that frame.

3. The encoding method based on region of interest and LCU features according to claim 2, characterized in that, The texture intensity value of the current pixel is: Among them, G s G represents the texture intensity value of the current pixel. x G represents the gradient value of a pixel in the X direction. y This represents the gradient value of the pixel in the Y direction. The gradient value of the pixel in the X direction is: The gradient value of the pixel in the Y direction is: Where P is a 3×3 pixel matrix centered on the current encoded pixel.

4. The encoding method based on region of interest and LCU features according to claim 1, characterized in that, The degree of distortion in the encoded LCU is represented by the SATD value of the LCU pixel residual, and the expression for the SATD value is: Where M is the size of the square matrix, and H is the normalized Hadamard matrix of size M×M; The average distortion level of a coded frame is obtained by averaging the distortion levels of all coded LCUs within that frame.

5. The encoding method based on region of interest and LCU features according to claim 1, characterized in that, The structural features corresponding to the LCU encoding are: Among them, SS m Here, m represents the structural feature value of the encoded LCU, indicating the LCU sequence number. and Let represent the partial derivative, q represent the current encoded pixel, F represent the input signal, and MB represent the current encoded LCU; The average structural features corresponding to the coded frame are obtained by averaging the structural features of all coded LCUs within the coded frame.

6. The encoding method based on region of interest and LCU features according to claim 1, characterized in that, The weighted combination of the relevant parameters corresponding to the encoded frame is: oh avg =a·G avg +(1-a)·SH avg Where, ω avg The weighted combination of relevant parameters corresponding to the encoded frame, where 'a' is the weighting coefficient, taking a value between 0 and 1, and G... avg SH represents the normalized average texture complexity corresponding to the encoded frame. avg It is the ratio of the average distortion to the average structural features corresponding to the current coded frame in the normalized form.

7. The encoding method based on region of interest and LCU features according to claim 6, characterized in that, The weighted combination of relevant parameters corresponding to the LCU encoding is as follows: oh curlcu =a·G curlcu +(1-a)·SH curlcu Where, ω curlcu This is a weighted combination of relevant parameters corresponding to the LCU encoding, where 'a' is a weighting coefficient with a value between 0 and 1, and G... curlcu To encode the normalized texture complexity corresponding to LCU, SH curlcu It is the ratio of the distortion level to the structural features corresponding to the normalized current encoded LCU.

8. The encoding method based on region of interest and LCU features according to claim 7, characterized in that, The weighting coefficient 'a' is set to 0.

8.

9. The encoding method based on region of interest and LCU features according to claim 7, characterized in that, The current importance of the LCU to the encoded frame is as follows: oh cur =ω curlcu -oh avg Where, ω cur ω represents the importance of the current encoding LCU to the encoded frame. avg ω is a weighted combination of the relevant parameters corresponding to the encoded frame. curlcu Weighted combination of relevant parameters for encoding LCU.

10. The encoding method based on region of interest and LCU features according to claim 9, characterized in that, The optimization formula for parameter QP is: Among them, QP est ω is the QP value predicted by the encoder initially. cur ROI0 represents the importance of the current encoded LCU to the encoded frame, indicating that the current encoded LCU is an uninterested LCU. cur0 ω represents the importance of the uninterested LCU to the coded frame, ROI1 represents the currently coded LCU as the LCU of interest, and ω represents the LCU of interest. cur1 This indicates the level of importance of the LCU (Low-Value Unit) to the encoded frame.

Images