A low-latency panoramic video coding method considering temporal correlation and entropy balance

Abstract

The present invention belongs to the field of panoramic video coding technology, and specifically relates to a low-latency panoramic video coding method that considers time domain correlation and entropy balance. The present invention analyzes the inter-frame reference relationship under the low-latency coding structure and establishes a corresponding spherical domain distortion propagation chain, and uses the distortion impact factor of the current coding unit to adjust the coding parameters to achieve the goal of optimizing the coding. At the same time, the present invention assigns corresponding weights to different areas of the sphere according to the entropy balance principle to adjust the distortion impact size of the area. The present invention can greatly improve the coding rate distortion performance and significantly reduce the coding time.

Description

Technical Field

[0001] The present invention belongs to the technical field of panoramic video encoding, and in particular relates to a low-delay panoramic video encoding method that takes time domain correlation and entropy balance into consideration. Background Art

[0002] Panoramic video is a video that is shot in all directions at 360 degrees using a camera array or a 3D camera. When watching the video, the user can adjust the video up, down, left, and right at will. Panoramic video has the characteristics of high frame rate, high resolution (at least 4K), and wide field of view. The storage and transmission of panoramic video consumes a lot of resources. The low-latency hierarchical coding structure adopts a cyclic GOP (Group Of Pictures) structure. A GOP is divided into 3 layers. The coded frames of each layer use the same quantization parameters. Each frame has multiple reference frames, which are the previous frame in the time domain and the 3 adjacent key frames that have been encoded. In H.266 / VVC, the GOP size is increased to 8, and the playback order POC (Picture Order Count) of the coded frames is the same as the encoding order EOC (Encoding Order Count).

[0003] Panoramic video images need to be projected onto a two-dimensional plane and then encoded using a traditional encoder. ERP projection is the most commonly used projection method and is also the default projection method of 360Lib. This projection will produce stretching deformation in high-latitude areas of the spherical surface and produce oversampling through interpolation filling.

[0004] The encoding process in H.266 / VVC is:

[0005] 1) Coding unit division

[0006] 2) Prediction mode selection, assuming that each coding unit is independent of each other

[0007]

[0008] Where λ i is the Lagrange multiplier corresponding to the i-th coding unit, P i is the encoding parameter (such as quantization parameter, encoding mode, etc.), J i is the rate-distortion cost of the i-th coding unit, and the encoder passes the minimum rate-distortion cost J i To select the coding mode, the coding mode here mainly refers to the intra-frame and inter-frame prediction mode.

[0009] 3) Transformation: Hadamard transformation of the prediction residual

[0010] 3) Quantization: Quantize the transformed residual and introduce quantization distortion

[0011] QSTEP =2 (QP-4) / 6

[0012] Q STEP is the quantization step size, and its relationship with the quantization parameter is as shown above.

[0013] The existing 360Lib-based VTM encoder does not consider the oversampling problem caused by panoramic video projection distortion. At the same time, in order to simplify the problem, the VTM encoder does not consider the temporal correlation between coding units. Moreover, the impact of spherical domain distortion is not only related to the coding reference frame structure, but also to the spherical projection distortion, so there is still a lot of room for improvement in rate-distortion performance. Summary of the invention

[0014] In view of the above problems, the present invention provides a low-latency panoramic video encoding method that considers time domain correlation and entropy balance. The present invention analyzes the inter-frame reference relationship under the low-latency coding structure and establishes a corresponding spherical domain distortion propagation chain, and uses the distortion impact factor of the current coding unit to adjust the coding parameters to achieve the goal of optimizing the coding. At the same time, the present invention assigns corresponding weights to different areas of the sphere according to the entropy balance principle to adjust the distortion impact of the area.

[0015] The technical solution of the present invention is:

[0016] A low-delay panoramic video encoding method considering temporal correlation and entropy balance comprises the following steps:

[0017] S1, input the sequence to be encoded to the encoder;

[0018] S2. Determine whether the current frame is the first frame. If so, execute:

[0019] Calculate the latitude value corresponding to the CTU row and calculate the area stretching ratio of the sphere and its projection plane;

[0020] Calculate the frame-level Lagrange multiplier based on the frame-level QP;

[0021] After encoding the current frame, input the next picture and repeat S2;

[0022] Otherwise, proceed to step S3;

[0023] S3, calculating the mean value and variance value of the pixels of the current coding frame;

[0024] S4, judging whether the difference between the pixel variance value of the current frame and the previous frame is less than the first threshold, and whether the frame-level temporal correlation is greater than the second threshold, if so, proceeding to S5, otherwise proceeding to S6;

[0025] S5, the encoding quantization parameter of the current frame increases by 1;

[0026] S6, encode CTU in sequence;

[0027] S7, determine whether the current frame number is greater than 3, if so, go to step S9, otherwise go to S8;

[0028] S8, adjust the Lagrange multiplier based on the R-λ model and enter S11;

[0029] S9, adjusting the CTU Lagrange multiplier based on the R-λ model and the temporal correlation of the previous frame;

[0030] S10, calculating the quantization parameter of the current CTU and encoding the CTU;

[0031] S11, determine whether it is the last CTU, if so, encode the current frame and go to step S12, otherwise, return to step S6;

[0032] S12, obtaining the coding distortion and motion compensation prediction error of each CTU saved in the current frame, and calculating the time domain distortion influence factor of each CTU in the current frame;

[0033] S13, determine whether it is the last frame, if so, enter S14, otherwise input the next picture and return to S2;

[0034] S14. The current panoramic video sequence encoding is completed.

[0035] Furthermore, in step S2, the method for calculating the area stretching ratio is: define the radius of the panoramic video sphere as r, then the area of ​​the spherical ring zone with a latitude of θ is:

[0036] S s (θ)=2π·r 2 ·cosθ·sindθ

[0037] After projecting the 2D plane, the area of ​​the stretched region is:

[0038]

[0039] The area stretching ratio is:

[0040]

[0041] Furthermore, the specific method of step 10 is:

[0042]

[0043] Among them, λ θi is the Lagrange multiplier adjusted by the R-λ model, and k is the time domain correlation.

[0044] The beneficial effects of the present invention are as follows: according to the problem that the spherical image of the panoramic video undergoes different degrees of geometric deformation in different latitude regions during the projection process, pixel redundancy is caused by interpolation filling, and then the entropy distribution is destroyed, and the encoding parameters are adjusted according to the bit rate control model, which can greatly improve the encoding performance and the coding rate-distortion performance, while significantly reducing the encoding time. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of panoramic video projection.

[0046] Figure 2 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be described in detail below with reference to the accompanying drawings.

[0048] The encoder selects a set of optimal coding parameters and a coding mode with the lowest rate-distortion cost for the input video through rate-distortion optimization (RDO) technology, with the goal of reducing coding distortion as much as possible under certain bit rate constraints or reducing coding bits as much as possible under certain coding distortion constraints. Figure 2 shown.

[0049] For the first frame of the input image, it is necessary to calculate the latitude value corresponding to the CTU row and calculate the area stretching ratio of the sphere and its projection plane. The specific method is:

[0050] Define r as the radius of the sphere, and the area of ​​the spherical ring zone at latitude θ can be calculated as

[0051] S s (θ)=2π·r 2 ·cosθ·sindθ

[0052] Corresponding to the projected 2D plane, the area of ​​the stretched region is

[0053]

[0054] The area ratio before and after projection is

[0055]

[0056] It can be seen that the ratio decreases with increasing latitude. In the equatorial region, that is, at latitude θ of 0, the area ratio is 1, and there is no stretching. Assuming that the entropy of the equatorial region does not change before and after the projection, the closer to the pole, the greater the entropy change.

[0057] At the same time, the first frame is encoded by calculating the frame-level Lagrange multiplier based on the frame-level QP.

[0058] For the second and subsequent coded frame images, first calculate the pixel mean and variance of the current coded frame:

[0059]

[0060]

[0061] Among them, P and D represent the mean and variance respectively, P i,j Represents the pixel value of each pixel, with a resolution of n*m.

[0062] According to the difference between the pixel variance of the current coded frame and the previous frame, if the difference is less than the threshold value of 50, then determine whether the frame-level temporal correlation is greater than 0.8. If the condition is met, the coding quantization parameter of the previous frame is increased by 1, and then the CTU is encoded;

[0063] If the difference between the pixel variance of the current coded frame and the previous frame is greater than the threshold of 50, and the number of frames is less than 3, the Lagrange multiplier is adjusted through the R-λ model in the bit rate control:

[0064] The rate-distortion model adopted by the VVC standard reference code VTM is

[0065] D=c·R -k

[0066] Where c and k are model parameters. D and R are coding distortion and coding bits, respectively, expressed by mean square error (MSE) and bpp (Bit Per Pixel), where MSE is obtained by the mean square error of the corresponding pixels of the original frame and the reconstructed frame. According to the R-λ model,

[0067]

[0068] λ i is the Lagrange multiplier corresponding to the i-th coding unit, and is the slope of the rate-distortion curve. The relationship between the Lagrange multiplier λi and the code rate Ri

[0069]

[0070] The parameters in the above formula are automatically updated after encoding a frame or a coding tree unit CTU. A large number of experiments have shown that the parameter 1 / α i and -1 / β i They are floating point numbers greater than 1 and negative numbers less than 0. Since the entropy distribution model does not require accurate bit rate allocation, and in order to reduce complexity, the parameter 1 / α i and -1 / β i The values ​​are 1 and -1 respectively. The above formula is simplified to

[0071] λ i =Ri -1

[0072] Then the Lagrange multiplier is adjusted by the R-λ model in the rate control

[0073]

[0074]

[0075] The latitude value corresponding to θ is as follows Figure 1 As shown, 0 represents the equatorial region.

[0076] Otherwise, the CTU Lagrange multiplier is adjusted according to the R-λ model and the temporal correlation of the previous frame; the definition of temporal correlation is: in video coding, during the inter-frame prediction process, the coding quality of the current coded frame is affected by the quality of the reference frame, that is, the distortion of the reference frame will affect the distortion of the subsequent coded frames, thereby forming a correlation in the temporal domain.

[0077] Here, the ratio of the reconstruction distortion of the coding tree unit and the motion compensation prediction error distortion is used to measure the temporal correlation in video coding, that is:

[0078]

[0079] Distortion of the coded block D cur The motion compensation prediction error with the coding block For the current coding block, its distortion cannot be obtained before the actual encoding. In order to achieve zero delay, consider using the coding tree unit information of the corresponding position of the previous frame to approximately replace the time domain dependency of the current block. The reason for this is that the image characteristics of two adjacent frames are generally similar. At the same time, the difference in pixel mean and variance between the current frame and the previous frame is used to determine whether the image has scene switching, thereby deciding whether to use the distortion influencing factor. It should be noted that because the first frame uses intra-frame coding and does not refer to relevant information in the time domain, it is not adjusted. The frame-level time domain correlation K is calculated using the same idea. frame .

[0080] The method of adjusting the Lagrangian multiplier of each coding tree unit according to the time domain correlation k and the R-λ model is to obtain a new Lagrangian multiplier λ by dividing the original Lagrangian multiplier by the adjusted weight n , and then achieve time domain rate distortion optimization:

[0081]

[0082] where λ ERP(θi)is the Lagrange multiplier adjusted by the R-λ model. At the same time, for coding tree units with strong temporal correlation, the larger the k, the smaller the adjusted Lagrange multiplier, which means that its distortion will be reduced, which is conducive to achieving higher rate-distortion performance for subsequent frames; conversely, coding tree units with weak temporal dependence will be relatively poorly coded. The coding quantization parameter is calculated by the following formula:

[0083] QP θi =4.5499·log(λ θi )-0.0956+0.5

[0084] 0.5 is rounded off, and the remaining data are model parameters.

[0085] In the scheme of the present invention, for the low-latency coding structure, due to the image stretching distortion after spherical projection, the distorted area is filled with redundant pixels by interpolation, thereby destroying the original entropy distribution. Under the assumption that the spherical coding bits are evenly distributed, the coding parameters are calculated according to the R-λ model to achieve the goal of entropy balance.

[0086] The present invention uses statistical features such as the mean and variance of image pixels to determine whether the panoramic video image scene has changed. If the image scene has not switched, the reconstruction distortion information and motion compensation prediction error information of the coded block of the previous frame are used to estimate the distortion propagation influence factor of the current frame to be coded, and used to adjust the coding parameters to achieve time domain rate distortion optimization; otherwise, the encoder's own algorithm is used for coding. This is the key to improving rate distortion performance.

[0087] The following experimental results illustrate the effectiveness of the scheme of the present invention. The panoramic video bit rate control algorithm is integrated into the H.266 / VVC reference software VTM14.0 based on 360lib. The encoder is configured as LOW-DELAY. The experimental conditions comply with the standard panoramic video sequence recommended by the international coding standards organization JCT-VC as the test video, including all 14 video sequences in three categories: 8K, 6K and 4K. The comparison indicators are BD-Rate and encoding time. When the BD-Rate is a negative value, it means that the bit rate is reduced at the same reconstruction quality, and the technology has a gain. When the BD-Rate is a positive value, it means that the bit rate is increased at the same reconstruction quality, and the technology has a loss. Here, the encoding quality of the VVC encoder at a given QP is used as the comparison benchmark.

[0088] Table 1 compares the rate-distortion performance of the algorithm of the present invention and the benchmark. It can be seen that the average BD-RATE of the present invention is saved by about 6.6%.

[0089] Table 1 Comparison of rate-distortion performance with 360lib vtm14.0 benchmark

[0090]

[0091]

[0092] Table 2 compares the encoding time of the algorithm of the present invention and the benchmark, and it can be seen that the present invention saves 14.62% of the time.

[0093] Table 2 Comparison of encoding time

[0094]

[0095]

[0096]

Claims

1. A low-delay panoramic video encoding method considering temporal correlation and entropy balance, characterized in that: The following steps are involved: S1, input the sequence to be encoded to the encoder; S2. Determine whether the current frame is the first frame. If so, execute: Calculate the latitude value corresponding to the CTU row and calculate the area stretching ratio of the sphere and its projection plane; Calculate the frame-level Lagrange multiplier based on the frame-level QP; After encoding the current frame, input the next picture and repeat S2; Otherwise, proceed to step S3; S3, calculating the mean value and variance value of the pixels of the current coding frame; S4, judging whether the difference between the pixel variance value of the current frame and the previous frame is less than the first threshold, and whether the frame-level temporal correlation is greater than the second threshold, if so, proceeding to S5, otherwise proceeding to S6; S5, the encoding quantization parameter of the current frame increases by 1; S6, encode CTU in sequence; S7, determine whether the current frame number is greater than 3, if so, go to step S9, otherwise go to S8; S8, adjust the Lagrange multiplier based on the R-λ model, and enter S10; S9, adjusting the Lagrange multiplier based on the R-λ model and the time domain correlation of the previous frame; S10, calculating the quantization parameter of the current CTU and encoding the CTU; S11, determine whether it is the last CTU, if so, encode the current frame and go to step S12, otherwise, return to step S6; S12, obtaining the coding distortion and motion compensation prediction error of each CTU saved in the current frame, and calculating the time domain distortion influence factor of each CTU in the current frame; S13, determine whether it is the last frame, if so, enter S14, otherwise input the next picture and return to S2; S14. The current panoramic video sequence encoding is completed.

2. According to claim 1, a low-delay panoramic video encoding method considering temporal correlation and entropy balance is characterized in that: In step S2, the method for calculating the area stretching ratio is: define the radius of the panoramic video sphere as r, then the area of ​​the spherical ring zone with a latitude of θ is: S s (θ)=2π·r 2 ·cosθ·sinθ After projecting the 2D plane, the area of ​​the stretched region is: The area stretching ratio is:

3. The low-delay panoramic video encoding method considering temporal correlation and entropy balance according to claim 1, characterized in that: In step S8, the Lagrange multiplier is adjusted based on the R-λ model as follows: D=c·R -k Among them, c and k are model parameters, D and R are coding distortion and coding bits, respectively, which are expressed by mean square error MSE and bpp, respectively. MSE is obtained by the mean square error of the corresponding pixels of the original frame and the reconstructed frame. According to the R-λ model, λ i is the Lagrange multiplier corresponding to the i-th coding unit, is the slope of the rate-distortion curve; the relationship between the Lagrange multiplier λi and the code rate Ri The parameters in the above formula are automatically updated after encoding a frame or a coding tree unit CTU; since the entropy distribution model does not require accurate bit rate allocation, let the parameter 1 / α i and -1 / β i Take the values ​​of 1 and -1 respectively, simplifying the above formula to l i =R i -1 Then the Lagrange multiplier is adjusted by the R-λ model in the rate control Where θ is the corresponding latitude value.

4. The low-delay panoramic video encoding method considering temporal correlation and entropy balance according to claim 3, characterized in that: The specific method of step 9 is: in, is the Lagrange multiplier adjusted by the R-λ model, and k is the time domain correlation factor.

Images