A low-complexity panoramic video coding method considering temporal distortion propagation

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a low-complexity panoramic video encoding method that considers the propagation of time-domain distortion. The present invention provides a panoramic video encoding method that considers the propagation of time-domain distortion. The present invention analyzes the inter-frame reference relationship under the random access coding structure and establishes a corresponding distortion propagation chain, and uses the distortion influence factor of the current coding unit to adjust the coding parameters to achieve the goal of optimizing the coding. According to the problem of oversampling caused by different degrees of geometric deformation of the spherical image of the panoramic video in different latitudes during the projection process, the present invention adjusts the coding parameters according to the area change ratio of different latitudes. It can greatly improve the coding performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a low-complexity panoramic video encoding method considering the propagation of time-domain distortion. Background Art

[0002] Panoramic video is a video that is shot in all directions at 360 degrees using a camera array or a 3D camera. When watching the video, the user can adjust the video up, down, left, and right at will. Panoramic video has the characteristics of high frame rate, high resolution (at least 4K), and wide field of view. The storage and transmission of panoramic video consumes a lot of resources. The random access coding structure (RA) adopts a cyclic GOP structure. The coded frames at the same position in the GOP use the same reference frame and quantization parameters. In H.266 / VVC, the GOP size is increased to 16, and the playback order POC (Picture Order Count) of the coded frames is not equal to the encoding order EOC (Encoding Order Count). At this time, the 16 coded images in the GOP are allocated to 5 time domain layers TL. The QPoffset of each TL is set in a way related to the quantization parameter QP of the I frame. The following problems exist when encoding panoramic videos of RA structure using traditional encoders. First, the effect of pixel redundancy caused by panoramic image projection distortion on encoding performance is not considered. Second, due to the high encoding complexity, the temporal dependency between frames is not considered for encoding. At the same time, the complex reference relationship under the RA structure also makes the encoding performance have great room for improvement. In view of the above problems, the present invention invents a low-complexity panoramic video encoding method that considers the propagation of temporal distortion. First, the encoding parameters are adjusted according to the area ratio of the spherical image and the projected image to reduce the encoding performance loss caused by pixel redundancy; at the same time, according to the continuity characteristics of the video image, the frame pixel variance value change is used to determine whether a scene switch occurs; if no scene switch occurs, the image continuity characteristics are used to adjust the encoding parameters of the referenced encoding unit according to the special reference relationship under the RA structure, thereby greatly improving the encoding performance. The invention does not need to pre-store the unencoded frame information, but only uses the distortion information of the reference frame to calculate its distortion influence factor. The complexity is extremely low, and the encoding performance can be effectively improved.

[0003] The encoding process in H.266 / VVC is:

[0004] 1) Coding unit division

[0005] 2) Prediction mode selection, assuming that each coding unit is independent of each other

[0006]

[0007] Where λ iis the Lagrange multiplier corresponding to the i-th coding unit, P i is the encoding parameter (such as quantization parameter, encoding mode, etc.), J i is the rate-distortion cost of the i-th coding unit, and the encoder passes the minimum rate-distortion cost J i To select the coding mode, the coding mode here mainly refers to the intra-frame and inter-frame prediction mode.

[0008] 3) Transformation: Hadamard transformation of the prediction residual

[0009] 3) Quantization: Quantize the transformed residual and introduce quantization distortion

[0010] Q STEP =2 (QP-4) / 6

[0011] Q STEP is the quantization step size, and its relationship with the quantization parameter is as shown above.

[0012] The existing VTM encoder based on 360Lib does not consider the oversampling problem caused by the panoramic video projection distortion. At the same time, in order to simplify the problem, the VTM encoder does not consider the temporal correlation between coding units, and the rate-distortion performance still needs to be improved. Summary of the invention

[0013] In view of the above problems, the present invention provides a panoramic video encoding method considering the propagation of time domain distortion. The present invention first determines whether a scene switch occurs by comparing the pixel variance value changes between the current frame and the previous encoded frame (not necessarily the previous played frame). In most cases where a scene switch does not occur, based on the image continuity characteristics, the inter-frame reference relationship under the random access coding structure is analyzed and the corresponding distortion propagation chain is established. The distortion influence factor of the reference coding unit is used to adjust the coding parameters of the current coding unit to achieve the goal of optimizing the coding.

[0014] The technical solution of the present invention is:

[0015] A panoramic video encoding method considering time domain distortion propagation comprises the following steps:

[0016] S1, input the sequence to be encoded to the encoder;

[0017] S2. Determine whether the current frame is the first frame. If so, execute:

[0018] Calculate the latitude value corresponding to the CTU row and calculate the area stretching ratio of the sphere and its projection plane;

[0019] Calculate the frame-level Lagrange multiplier based on the frame-level QP;

[0020] After encoding the current frame, repeat step S2;

[0021] Otherwise, proceed to step S3;

[0022] S3, calculating the mean value and variance value of the pixels of the current coding frame;

[0023] S4, encode CTU in sequence;

[0024] S5, determining whether the difference between the pixel variance value of the current frame and the previous frame is greater than a threshold value, if so, proceeding to step S6, otherwise proceeding to step S7;

[0025] S6, adjusting the Lagrange multiplier of the current CTU according to the area stretching ratio obtained in step S2, and proceeding to step S17;

[0026] S7, determine whether the current frame number is greater than 3, if so, go to S8, otherwise go to S6;

[0027] S8, determine whether the current frame number modulo 16 is equal to 0, if so, go to S12, otherwise go to S9;

[0028] S9, determine whether the current frame number modulo 8 is equal to 0, if so, go to S13, otherwise go to S10;

[0029] S10, determine whether the current frame number modulo 4 is equal to 0, if so, go to S14, otherwise go to S11;

[0030] S11, determine whether the current frame number modulo 2 is equal to 0, if so, go to S15, otherwise go to S16;

[0031] S12, adjusting the Lagrange multiplier of the CTU according to the time domain distortion influence factor of the reference frame obtained by subtracting sixteen from the current frame number and the area stretching ratio obtained in S2;

[0032] S13, adjusting the Lagrange multiplier of the CTU according to the time domain distortion influence factors of the two reference frames whose current frame number is plus or minus eight and the area stretching ratio obtained in S2;

[0033] S14, adjusting the Lagrange multiplier of the CTU according to the time domain distortion influence factors of the two reference frames of the current frame number plus or minus four and the area stretching ratio obtained in S2;

[0034] S15, adjusting the Lagrange multiplier of the CTU according to the time domain distortion influence factors of the two reference frames of the current frame number plus or minus two and the area stretching ratio obtained in S2;

[0035] S16, adjusting the Lagrange multiplier of the CTU according to the time domain distortion influence factors of the two reference frames whose current frame number is plus or minus one and the area stretching ratio obtained in S2;

[0036] S17, calculating a quantization parameter QP of the current CTU according to the Lagrange multiplier of the current CTU calculated in S6 or S12 or S13 or S14 or S15 or S16, and encoding the CTU;

[0037] S18, determine whether it is the last CTU, if so, complete encoding of the current frame and proceed to step S11, otherwise, return to step S4;

[0038] S19, determine whether the current frame is an odd frame, if so, enter S21, otherwise, enter S20

[0039] S20, obtaining a reconstruction error and a motion compensation prediction error of each CTU saved in the current frame to calculate a time domain distortion influence factor of each CTU in the current frame;

[0040] S21, determine whether it is the last frame, if so, enter S22, otherwise enter S2;

[0041] S22: The current panoramic video sequence encoding is completed.

[0042] Furthermore, in step S2, the method for calculating the area stretching ratio is: define the radius of the panoramic video sphere as r, then the area of ​​the spherical ring zone with a latitude of θ is:

[0043] S s (θ)=2π·r 2 ·cosθ·sindθ

[0044] After projecting the 2D plane, the area of ​​the stretched region is:

[0045]

[0046] The area stretching ratio is:

[0047]

[0048] Furthermore, step S3 is specifically as follows:

[0049]

[0050]

[0051] Among them, P and D represent the mean and variance respectively, P i,j Represents the pixel value of each pixel, with a resolution of n*m.

[0052] Furthermore, the distortion impact factor calculated in steps S6 and S12 to S16 is:

[0053]

[0054] Among them, κ i is the distortion impact factor in the i-th reference frame, l is the total number of reference frames, and k is the distortion impact factor of the final current coding unit.

[0055] The beneficial effects of the present invention are as follows: the present invention adjusts the encoding parameters according to the area change ratio of different latitudes based on the problem that the spherical image of the panoramic video undergoes different degrees of geometric deformation in different latitudes during the projection process, resulting in oversampling. At the same time, a very low-complexity method is used to calculate the distortion influence factor and use it for encoding, which can greatly improve the encoding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a reference schematic diagram of the reference frame in the present invention.

[0057] Figure 2 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be described in detail below with reference to the accompanying drawings.

[0059] Figure 1 The reference dependency relationship between the coded frames in a group of images GOP (size 16) under the RA coding structure is shown. It can be seen that the coded frames in a GOP are divided into 5 layers, where the first layer is an odd frame, the frame number modulo 2 is equal to 1, and the frames in this layer will not be referenced, so there is no need to calculate the distortion impact factor of each coding unit in the first layer; the fifth layer is a key frame layer, the coded frame in this layer refers to the previous key frame, and the frame number modulo 16 in this layer is equal to 0; the frame number modulo 8 in the fourth layer is equal to 0, and the coded frame in this layer refers to the adjacent key frame in the fifth layer, that is, the frame with a frame number plus or minus 8; the third layer is a frame with a frame number modulo 4 equal to 0, and the coded frame in this layer mainly refers to the adjacent frame with a frame number plus or minus 4; the second layer is a frame with a frame number modulo 2 equal to 0, and the coded frame in this layer refers to the adjacent frame with a frame number plus or minus 2. Therefore, this reference relationship determines the source of the distortion impact factor used by the coding unit. It should be pointed out that in actual coding, the modulo operation needs to be performed in order of the layers to avoid the problem of unclear program logic.

[0060] The encoder selects a set of optimal coding parameters and a coding mode with the lowest rate-distortion cost for the input video through rate-distortion optimization (RDO) technology, with the goal of reducing coding distortion as much as possible under certain bit rate constraints or reducing coding bits as much as possible under certain coding distortion constraints. Figure 2 shown.

[0061] For the first frame of the input image, unlike the traditional method, the method of the present invention needs to calculate the latitude value corresponding to the CTU row and calculate the area stretching ratio of the sphere and its projection plane. The specific method is:

[0062] Define r as the radius of the sphere, and the area of ​​the spherical ring zone at latitude θ can be calculated as

[0063] S s (θ)=2π·r 2 ·cosθ·sindθ

[0064] Corresponding to the projected 2D plane, the area of ​​the stretched region is

[0065]

[0066] The area ratio before and after projection is

[0067]

[0068] It can be seen that the ratio decreases with increasing latitude. In the equatorial region, that is, at latitude θ of 0, the area ratio is 1, and there is no stretching. Assuming that the entropy of the equatorial region does not change before and after the projection, the closer to the pole, the greater the entropy change.

[0069] At the same time, the first frame is encoded by calculating the frame-level Lagrange multiplier based on the frame-level QP.

[0070] For the second and subsequent coded frame images, first calculate the pixel mean and variance of the current coded frame:

[0071]

[0072]

[0073] Among them, P and D represent the mean and variance respectively, P i,j Represents the pixel value of each pixel, with a resolution of n*m.

[0074] According to the difference between the pixel variance of the current coded frame and the previous frame, if the difference is greater than a threshold of 50, the Lagrange multiplier is adjusted by selecting an area stretching ratio.

[0075] For the third frame and subsequent images, since the time domain distortion factor has been obtained, the Lagrange multiplier is adjusted based on the time domain distortion factor and the area stretching ratio. Specifically, the new Lagrange multiplier λ is obtained by dividing the original Lagrange multiplier by the adjusted weight. n , thereby achieving time-domain rate-distortion optimization.

[0076]

[0077] where λ Pis the frame-level Lagrangian multiplier. Obviously, the larger the latitude value θ is, the smaller k(θ) is, and the larger the adjusted Lagrangian multiplier is, and vice versa. At the same time, for coding tree units with strong temporal dependence, the larger k is, the smaller the adjusted Lagrangian multiplier is, which means that its distortion will be reduced, which is conducive to achieving higher rate-distortion performance for subsequent frames. On the contrary, coding tree units with weak temporal dependence will be relatively poorly coded.

[0078] Then the quantization parameter QP is calculated and the CTU is encoded.

[0079] After one frame of image encoding is completed, the bit rate control parameters and the actual number of bits used for the current frame and each CTU are updated, and the reconstruction error and motion compensation prediction error of each CTU saved in the current frame are obtained to calculate the time domain distortion impact factor of each CTU in the current frame. If the number of frames in the current encoding frame is an odd number, it will not be referenced by subsequent frames and the distortion impact factor does not need to be calculated. Otherwise, it is calculated according to the subsequent formula. Specifically, the ratio of the reconstruction distortion of the coding tree unit and the motion compensation prediction error distortion is used to measure the time domain dependency in video encoding, that is,

[0080]

[0081] Distortion of the coded block D cur The motion compensation prediction error with the coding block For the current coding block, its distortion cannot be obtained before the actual coding. In order to achieve zero delay, consider using the coding tree unit information of the corresponding position of the previous frame to approximately replace the time domain dependency of the current block. The reason for this is that the image characteristics of two adjacent frames are generally similar. At the same time, the difference in the pixel mean and variance between the current frame and the previous frame is used to determine whether the image has scene switching, thereby deciding whether to use the distortion influence factor.

[0082] It will be automatically updated after encoding a frame or a coding tree unit CTU. The parameter update formula is:

[0083] and

[0084] Where distortion and bit rate D i , R i It can be determined after encoding a frame or a CTU, and the frame-level λ i The Lagrange multiplier of the nearest neighbor coded frame at the same level in the same GOP is used instead, and the λ at the CTU level i The Lagrange multiplier corresponding to the CTU of the nearest coded frame at the same level as the coded frame is used instead, and then used to update the parameter c i , k i .

[0085] The following experimental results illustrate the effectiveness of the solution of the present invention. The panoramic video bit rate control algorithm is integrated into the H.266 / VVC reference software VTM14.0 based on 360lib. The encoder is configured as RANDOM ACCESS. The experimental conditions comply with the standard panoramic video sequence recommended by the international coding standards organization JCT-VC as the test video, including all 14 video sequences in three categories: 8K, 6K and 4K. The comparison indicators are BD-Rate and encoding time. When the BD-Rate is a negative value, it means that the bit rate is reduced at the same reconstruction quality, and the technology has a gain. When the BD-Rate is a positive value, it means that the bit rate is increased at the same reconstruction quality, and the technology has a loss. Here, the encoding quality of the VVC encoder at a given QP is used as the comparison benchmark.

[0086] Table 1 compares the rate-distortion performance of the algorithm of the present invention and the benchmark. It can be seen that the average BD-RATE of the present invention is saved by about 4.2%.

[0087] Table 1 Comparison of rate-distortion performance with 360lib vtm14.0 benchmark

[0088]

[0089]

[0090] Table 2 compares the encoding time of the algorithm of the present invention and the benchmark, and it can be seen that the present invention saves 16.2% of the time.

[0091] Table 2 Comparison of encoding time

[0092]

[0093]

Claims

1. A low-complexity panoramic video coding method considering temporal distortion propagation, characterized in that: The following steps are involved: S1, input the sequence to be encoded to the encoder; S2. Determine whether the current frame is the first frame. If so, execute: Calculate the latitude value corresponding to the CTU row and calculate the area stretching ratio of the sphere and its projection plane; Calculate the frame-level Lagrange multiplier based on the frame-level QP; After encoding the current frame, repeat step S2; Otherwise, proceed to step S3; S3, calculating the mean value and variance value of the pixels of the current coding frame; S4, encode CTU in sequence; S5, determining whether the difference between the pixel variance value of the current frame and the previous frame is greater than a threshold value, if so, proceeding to step S6, otherwise proceeding to step S7; S6, adjusting the Lagrange multiplier of the current CTU according to the area stretching ratio obtained in step S2, and proceeding to step S17; S7, determine whether the current frame number is greater than 3, if so, go to S8, otherwise go to S6; S8, determine whether the current frame number modulo 16 is equal to 0, if so, go to S12, otherwise go to S9; S9, determine whether the current frame number modulo 8 is equal to 0, if so, go to S13, otherwise go to S10; S10, determine whether the current frame number modulo 4 is equal to 0, if so, go to S14, otherwise go to S11; S11, determine whether the current frame number modulo 2 is equal to 0, if so, go to S15, otherwise go to S16; S12, adjusting the Lagrange multiplier of the CTU according to the time domain distortion influence factor of the reference frame obtained by subtracting sixteen from the current frame number and the area stretching ratio obtained in S2; S13, adjusting the Lagrange multiplier of the CTU according to the time domain distortion influence factors of the two reference frames whose current frame number is plus or minus eight and the area stretching ratio obtained in S2; S14, adjusting the Lagrange multiplier of the CTU according to the time domain distortion influence factors of the two reference frames of the current frame number plus or minus four and the area stretching ratio obtained in S2; S15, adjusting the Lagrange multiplier of the CTU according to the time domain distortion influence factors of the two reference frames of the current frame number plus or minus two and the area stretching ratio obtained in S2; S16, adjusting the Lagrange multiplier of the CTU according to the time domain distortion influence factors of the two reference frames whose current frame number is plus or minus one and the area stretching ratio obtained in S2; S17, calculating a quantization parameter QP of the current CTU according to the Lagrange multiplier of the current CTU calculated in S6 or S12 or S13 or S14 or S15 or S16, and encoding the CTU; S18, determine whether it is the last CTU, if so, encode the current frame and go to step S11, otherwise, return to step S4; S19, determine whether the current frame is an odd frame, if so, enter S21, otherwise, enter S20 S20, obtaining a reconstruction error and a motion compensation prediction error of each CTU saved in the current frame to calculate a time domain distortion influence factor of each CTU in the current frame; S21, determine whether it is the last frame, if so, enter S22, otherwise enter S2; S22: The current panoramic video sequence encoding is completed.

2. The low-complexity panoramic video encoding method considering temporal distortion propagation according to claim 1, characterized in that: In step S2, the method for calculating the area stretching ratio is: define the radius of the panoramic video sphere as r, then the area of ​​the spherical ring zone with a latitude of θ is: S s (θ)=2π·r 2 ·cosθ·sinθ After projecting the 2D plane, the area of ​​the stretched region is: The area stretching ratio is:

3. The low-complexity panoramic video encoding method considering temporal distortion propagation according to claim 2, characterized in that: Step S3 is specifically as follows: Among them, P and D represent the mean and variance respectively, P i,j Represents the pixel value of each pixel, with a resolution of n*m.

4. The low-complexity panoramic video encoding method considering temporal distortion propagation according to claim 3, characterized in that: The method for calculating the distortion impact factor is: Among them, k i is the distortion impact factor in the i-th reference frame, l is the total number of reference frames, and κ is the distortion impact factor of the final current coding unit.

Images