A zero-delay panoramic video bitrate control method considering time domain distortion propagation

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a zero-delay panoramic video rate control method considering the propagation of time domain distortion. The method of the present invention is mainly to optimize the target bit allocation and global rate distortion optimization coding, including rate control at the coding tree unit (CTU) level and time domain global rate distortion optimization. CTU-level rate control mainly includes optimizing target bit allocation and updating rate control parameters; time domain global rate distortion optimization mainly uses the reconstruction error and motion compensation prediction error information of the encoded frame to estimate the time domain dependency of each CTU in the current encoded frame and adjust the rate distortion optimization parameters of the CTU accordingly, and at the same time, according to the process of projecting the panoramic spherical graphic onto the 2D plane (ERP projection), the characteristics of oversampling and increasing pixel redundancy will be generated at the two poles, and the coding parameters are adjusted according to the projection area change ratio. The present invention can effectively shorten the encoding time and improve the video encoding quality under the condition of a given bandwidth.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a zero-delay panoramic video bit rate control method considering the propagation of time domain distortion. Background Art

[0002] Panoramic video is a video that is shot in all directions at 360 degrees using a camera array or a 3D camera. When watching the video, users can adjust the video up, down, left, and right at will. Panoramic video has the characteristics of high frame rate, high resolution (at least 4K), and wide field of view. Storing and transmitting panoramic video consumes a lot of resources. Bit rate control is to optimize the allocation of bits layer by layer according to the target bit rate, and dynamically adjust the encoding parameters of the encoder according to a certain encoding strategy, so that the encoder output bit stream meets the bandwidth limitation of the transmission channel and the storage space requirements of the storage device, and makes the actual output bit rate consistent with the target bit rate as much as possible. It is widely used in real-time video communication systems. Traditional panoramic video coding has the following problems. First, it will generate pixel redundancy during the projection process, which affects the coding performance; at the same time, the traditional encoder does not use the temporal correlation between the inter-frame coding units for encoding, and the coding performance still has a lot of room for improvement; the main reason why the traditional encoder does not consider the influencing factor of the inter-frame temporal distortion is that its computational complexity is too high, which is not conducive to real-time communication. In view of the above problems, the present invention proposes a zero-delay panoramic video bit rate control method that considers the propagation of time domain distortion. The present invention first adjusts the encoding parameters according to the ratio of the spherical area and the projection area of ​​the panoramic video to reduce the encoding performance loss caused by pixel redundancy; at the same time, it determines whether the image scene has changed by calculating the change in the mean and variance values ​​of the encoded frame pixels; in the case where no scene change occurs, according to the continuity of the video image, the reconstruction error and motion compensation prediction error of the previous encoded frame in the time domain are used to calculate the time domain distortion influence factor, and the factor is used to encode the current frame. The present invention does not need to pre-store unencoded frames, and uses the encoded frame information to calculate the time domain distortion influence factor, so the calculation amount is small, the encoding complexity is extremely low, and the rate distortion performance is greatly improved, so the present invention is easy to promote.

[0003] Current rate control methods include:

[0004] The H.266 / VVC rate control method mainly includes the following steps:

[0005] Step 1: Bit allocation

[0006] VTM adopts a hierarchical bit allocation structure, and allocates bits at the GOP layer, frame layer, and CTU layer. The lower you go, the more accurate the bit rate allocation is, and the easier it is to correct errors. VTM evenly distributes the target bit rate at each level. In order to gradually eliminate the bit rate control error generated by the encoded unit, it uses a sliding window to smooth the bit rate fluctuations to prevent the bit rate from fluctuating too much and affecting the video quality. The hierarchical bit rate allocation is achieved through the following formula, namely:

[0007]

[0008]

[0009]

[0010]

[0011] Among them, R tar is the target bit rate, FR is the frame rate, R pic is the average number of bits per frame in the entire sequence, T gop 、T pic 、T ctu are the target number of bits allocated to the GOP to be encoded, the coding frame, and the coding unit, respectively; SW is the sliding window for smooth bit allocation; N coded , R coded 、N gop , ω pic ,ω ctu , ω pic , ω ctu They are the number of encoded frames, the bits consumed in the encoded video sequence, the number of frames in a GOP, the number of encoded frames in a GOP, the weight of encoded frames, the weight of coding units, the sum of the weights of uncoded frames, and the sum of the weights of uncoded CTUs in a frame.

[0012] Step 2: Calculate the Lagrange multiplier according to the R-λ model

[0013]

[0014] Lagrange multiplier λ i is the slope of the rate-distortion curve, D i , R i are the distortion and coded bits of the i-th coding unit, respectively, and the Lagrange multiplier λ i With code rate R i The relationship between

[0015]

[0016] Step 3: Update encoding parameters

[0017] The parameters in the above formula are automatically updated after encoding a frame or a coding tree unit CTU. The parameter update formula is

[0018]

[0019] Where D i , R i It can be determined after encoding a frame or a CTU, and the frame-level λ i The Lagrange multiplier of the nearest neighbor coded frame at the same level in the same GOP is used instead, and the λ at the CTU level i The Lagrange multiplier corresponding to the CTU of the nearest coded frame at the same level as the coded frame is used instead, and then used to update the parameter c i , k i .

[0020] Step 4: Calculate the quantization parameter QP used in the encoding process

[0021] QP i =4.2005·lnλ i +13.7122

[0022] It can be seen that the quantization parameter in the rate control algorithm and the logarithm of the Lagrange multiplier can be fitted with a linear function relationship.

[0023] The above traditional rate control methods have the following problems: the R-Lambda rate control model in VVC can achieve high control accuracy, but it does not consider the pixel redundancy problem caused by the projection of panoramic video. At the same time, it does not use the temporal correlation between frames for rate-distortion optimization coding, and the coding quality has a lot of room for improvement. Summary of the invention

[0024] In view of the above problems, the present invention provides a zero-delay panoramic video rate control method that takes into account the propagation of time domain distortion. The method of the present invention is mainly to optimize the target bit allocation and global rate distortion optimization coding, including rate control at the coding tree unit (CTU) level and time domain global rate distortion optimization. CTU-level rate control mainly includes optimizing target bit allocation and updating rate control parameters; time domain global rate distortion optimization mainly uses the reconstruction error and motion compensation prediction error information of the encoded frame to estimate the time domain dependency of each CTU in the current encoded frame and adjust the rate distortion optimization parameters of the CTU accordingly. At the same time, according to the process of projecting the panoramic spherical graphic onto the 2D plane (ERP projection), the characteristics of oversampling and increasing pixel redundancy will be generated at the two poles, and the encoding parameters are adjusted according to the ratio of the projection area change.

[0025] The technical solution of the present invention is:

[0026] A zero-delay panoramic video bit rate control method considering time domain distortion propagation includes the following steps:

[0027] S1, input the target bit rate and the sequence to be encoded to the encoder;

[0028] S2. Determine whether the current frame is the first frame. If so, execute:

[0029] Calculate the latitude value corresponding to the CTU row and calculate the area stretching ratio of the sphere and its projection plane;

[0030] Initialize the parameters of the rate control unit;

[0031] Calculate the target number of bits for the current frame;

[0032] Calculate the frame-level Lagrange multiplier of the current frame according to the target bit number;

[0033] Calculate frame-level QP based on frame-level Lagrange multipliers and perform intra-frame coding;

[0034] After encoding the current frame, repeat step S2;

[0035] Otherwise, proceed to step S3;

[0036] S3, calculating the frame pixel mean and variance value;

[0037] S4, determine whether the current frame number is greater than 2 times the GOP size, if so, go to S5; otherwise, go to S6;

[0038] S5, calculating the global Lagrange multiplier according to the iterative algorithm and optimizing the bit allocation of each CTU in the frame, and proceeding to step S7;

[0039] S6. Use the encoder's built-in bit allocation algorithm to allocate bits to each CTU in the frame;

[0040] S7, calculating frame-level Lagrange multipliers and QP;

[0041] S8, encode CTU in sequence;

[0042] S9, determining whether the difference between the pixel variance value of the current frame and the previous frame is greater than a threshold value, if so, proceeding to step S10, otherwise proceeding to step S11;

[0043] S10, adjusting the Lagrange multiplier of the current CTU according to the area stretching ratio obtained in step S2, and proceeding to step S13;

[0044] S11, determine whether the current frame number is greater than 3, if so, go to S13, otherwise go to S10;

[0045] S12, adjusting the Lagrange multiplier of each CTU according to the time domain distortion impact factor of the previous frame and the area ratio obtained in S2;

[0046] S13, calculating a quantization parameter QP of the current CTU according to the Lagrange multiplier of the current CTU calculated in S10 or S12 and encoding the CTU;

[0047] S14, determine whether it is the last CTU, if so, then encode the current frame and go to step S15, otherwise, return to step S8;

[0048] S15, updating the bit rate control parameters of the current frame and each CTU and the number of bits actually used;

[0049] S16, obtaining the reconstruction error and motion compensation prediction error of each CTU saved in the current frame to calculate the time domain distortion influence factor of each CTU in the current frame;

[0050] S17, determine whether it is the last frame, if so, enter S18, otherwise enter S2

[0051] S18. The current panoramic video sequence encoding is completed.

[0052] Furthermore, in step S2, the radius of the panoramic video sphere is defined as r, and the area of ​​the spherical ring zone at latitude θ is:

[0053] S s (θ)=2π·r 2 ·cosθ·sindθ

[0054] After projecting the 2D plane, the area of ​​the stretched region is:

[0055]

[0056] The area stretching ratio is:

[0057]

[0058] Furthermore, step S3 is specifically as follows:

[0059]

[0060]

[0061] Among them, P and D represent the mean and variance respectively, P i,j Represents the pixel value of each pixel, with a resolution of n*m.

[0062] Furthermore, step S5 is specifically as follows:

[0063]

[0064] Among them, R and r are the target bits at the frame level and CTU level respectively, and λ g is the global Lagrange multiplier, α i =c i ·k i , λ i is the slope of the rate-distortion curve, D i , R i are the distortion and coding bits of the i-th coding unit respectively, and M is the number of CTUs.

[0065] Furthermore, the Lagrange multiplier adjustment method in step 10 is:

[0066]

[0067] Among them, λ P is the frame-level Lagrange multiplier, λ n is the adjusted Lagrange multiplier.

[0068] Furthermore, the Lagrange multiplier adjustment method in step 12 is:

[0069]

[0070] Wherein, k is the time domain distortion influencing factor.

[0071] The beneficial effects of the present invention are as follows: the present invention can reduce the coding complexity and make the bit rate control error very small without caching subsequent frames, can effectively shorten the coding time and improve the video coding quality under the condition of a given bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0073] The present invention will be described in detail below with reference to the accompanying drawings.

[0074] In video bit rate control, the encoder selects a set of optimal coding parameters and the coding mode with the lowest rate distortion cost for the input video through rate distortion optimization (RDO) technology. Its goal is to reduce the coding distortion as much as possible under certain bit rate constraints or to reduce the coding bits as much as possible under certain coding distortion constraints. The zero-delay method proposed in the present invention means that the encoder is not allowed to obtain the information of subsequent frames in advance, that is, after obtaining the frame to be encoded, the analysis data must be encoded immediately without caching. The specific implementation method is as follows Figure 1 shown.

[0075] For the first frame of the input image, since the time domain distortion factor has not yet been obtained inside the encoder, it is necessary to initialize the internal parameters of the encoder based on the given target bit rate. The initialization method is consistent with the VVC bit rate control method. The target number of bits for each coding level is:

[0076]

[0077]

[0078]

[0079]

[0080] Among them, R tar is the target bit rate, FR is the frame rate, R pic is the average number of bits per frame in the entire sequence, T gop 、T pic 、T ctu are the target number of bits allocated to the GOP to be encoded, the coding frame, and the coding unit, respectively; SW is the sliding window for smooth bit allocation; N coded , R coded 、N gop , ω pic ,ω ctu , ω pic , ω ctu They are the number of encoded frames, the bits consumed in the encoded video sequence, the number of frames in a GOP, the number of encoded frames in a GOP, the weight of encoded frames, the weight of coding units, the sum of the weights of uncoded frames, and the sum of the weights of uncoded CTUs in a frame.

[0081] Different from the traditional method, the method of the present invention also needs to calculate the latitude value corresponding to the CTU row and calculate the area stretching ratio of the sphere and its projection plane. The specific method is:

[0082] Define r as the radius of the sphere, and we can calculate the area of ​​the spherical ring zone at latitude θ as

[0083] S s (Ω)=2π·r 2 ·cosΩ·sindθ

[0084] Corresponding to the projected 2D plane, the area of ​​the stretched region is

[0085]

[0086] The area ratio before and after projection is

[0087]

[0088] It can be seen that the ratio decreases with increasing latitude. In the equatorial region, that is, at latitude θ of 0, the area ratio is 1, and there is no stretching. Assuming that the entropy of the equatorial region does not change before and after the projection, the closer to the pole, the greater the entropy change.

[0089] Starting from the second group of pictures GOP, the optimal Lagrangian multiplier in the bit allocation formula is approximated by an iterative algorithm, and the bits are allocated using the optimal Lagrangian multiplier, specifically:

[0090]

[0091] Where R and r are the target bits at the frame level and CTU level respectively, and λ g is the global Lagrange multiplier, α i =c i ·k i , There are M CTUs in one frame.

[0092] For the second frame and subsequent images, the Lagrange multiplier can be adjusted by selecting the area stretch ratio. For the third frame and subsequent images, since the time domain distortion factor has been obtained, the Lagrange multiplier is adjusted based on the time domain distortion factor and the area stretch ratio. Specifically, the new Lagrange multiplier λ is obtained by dividing the original Lagrange multiplier by the adjusted weight. n , thereby achieving time-domain rate-distortion optimization.

[0093]

[0094] where λ P is the frame-level Lagrangian multiplier. Obviously, the larger the latitude value θ is, the smaller k(Ω) is, and the larger the adjusted Lagrangian multiplier is, and vice versa. At the same time, for coding tree units with strong temporal dependence, the larger k is, the smaller the adjusted Lagrangian multiplier is, which means that its distortion will be reduced, which is conducive to achieving higher rate-distortion performance for subsequent frames. On the contrary, coding tree units with weak temporal dependence will be relatively poorly coded.

[0095] The condition for selecting the Lagrange multiplier adjustment method is to determine whether the difference between the current frame pixel variance value and the previous frame is greater than the threshold value of 50. Because the premise of temporal distortion propagation is the continuity of the video image, if the image scene switches, the propagation chain will be disconnected, so the change in the image variance value is used to determine whether the scene switches.

[0096]

[0097]

[0098] Among them, P and D represent the mean and variance respectively, P i，i Represents the pixel value of each pixel, with a resolution of n*m.

[0099] The Lagrange multipliers and QP are calculated as:

[0100] First, calculate the average target bit number per pixel of the current frame image (T bpp ), the calculation formula is

[0101] T bpp =T pic / N pixels

[0102] N pixels Refers to the number of pixels in a picture. Then calculate the frame-level Lagrangian parameter λ of the current image.

[0103] For I frames: a rate control method based on intra-frame complexity is used. According to experiments, the intra-frame complexity is defined as but It has the following relationship with the Lagrange multiplier:

[0104]

[0105] Where MAD is the mean absolute error of the pixel and BPP is the average target bit of the pixel. The initial values ​​of the α and β parameters are empirically set to 9.9416 and -1.367.

[0106] For P frames: the calculation method is λ = αR β , the initial value of α is set to 1058, and the value of β is set to -1.327

[0107] The corresponding frame-level QP is calculated using the following relationship:

[0108] QP i =4.2005·lnλ i +13.7122

[0109] After a frame of image encoding is completed, the bit rate control parameters and the actual number of bits used for the current frame and each CTU are updated, and the reconstruction error and motion compensation prediction error of each CTU saved in the current frame are obtained to calculate the temporal distortion impact factor of each CTU in the current frame. Specifically, the ratio of the reconstruction distortion of the coding tree unit and the motion compensation prediction error distortion is used to measure the temporal dependency in video encoding, that is,

[0110]

[0111] Distortion of the coded block D cur The motion compensation prediction error with the coding block For the current coding block, its distortion cannot be obtained before the actual coding. In order to achieve zero delay, consider using the coding tree unit information of the corresponding position of the previous frame to approximately replace the time domain dependency of the current block. The reason for this is that the image characteristics of two adjacent frames are generally similar. At the same time, the difference in the pixel mean and variance between the current frame and the previous frame is used to determine whether the image has scene switching, thereby deciding whether to use the distortion influence factor.

[0112] It will be automatically updated after encoding a frame or a coding tree unit CTU. The parameter update formula is:

[0113]

[0114] Where distortion and bit rate D i , R i It can be determined after encoding a frame or a CTU, and the frame-level λ i The Lagrange multiplier of the nearest neighbor coded frame at the same level in the same GOP is used instead, and the λ at the CTU level i The Lagrange multiplier corresponding to the CTU of the nearest coded frame at the same level as the coded frame is used instead, and then used to update the parameter c i , k i .

[0115] The following experimental results illustrate the effectiveness of the solution of the present invention. The panoramic video rate control algorithm is integrated into the H.266 / VVC reference software VTM14.0 based on 360lib. The encoder is configured as Lowdelay-P. The experimental conditions comply with the standard panoramic video sequence recommended by the international coding standards organization JCT-VC as the test video, including all 14 video sequences in three categories: 8K, 6K and 4K. The comparison indicators are BD-Rate, rate control accuracy and encoding time. When the BD-Rate is a negative value, it means that the rate is reduced at the same reconstruction quality, and the technology has a gain. When the BD-Rate is a positive value, it means that the rate is increased at the same reconstruction quality, and the technology has a loss. Here, the encoding quality of the VVC encoder at a given QP and the rate control algorithm of the VVC are used as the comparison benchmarks.

[0116] Table 1 shows the performance of this scheme and the rate control methods of VTM14.0 and 360lib on BD-Rate. Compared with the rate control method of vtm, the rate is greatly saved, and this method has a rate saving of 8.7%.

[0117] Table 1 Comparison of BD-rate encoding with VTM14.0 bitrate control algorithm

[0118]

[0119] Table 2 shows the comparison between this solution and the rate-distortion optimization algorithm of VTM14.0 on BD-rate, with an average bit rate saving of about 4.3%.

[0120] Table 2 BD-rate comparison with VTM14.0 rate-distortion optimization algorithm

[0121]

[0122] Table 3 shows the size of the encoding rate control error of this scheme and the VTM14.0 and 360lib rate control schemes. The bit error rate is calculated by dividing the absolute error between the actual encoding output rate and the set target rate by the target rate. The larger the deviation, the higher the bit error rate and the lower the rate control accuracy. This scheme ensures an extremely low rate control error of 0.0891%.

[0123] Table 3 Comparison of rate control errors

[0124]

[0125]

[0126]

[0127] Table 4 shows the performance of this scheme in terms of encoding time. The total encoding time for all sequences is reduced from 3972 hours to 3963 hours.

[0128] Table 4 Comparison of encoding time

[0129]

[0130]

[0131]

Claims

1. A zero-delay panoramic video bit rate control method considering time domain distortion propagation, characterized in that: The following steps are involved: S1, input the target bit rate and the sequence to be encoded to the encoder; S2. Determine whether the current frame is the first frame. If so, execute: Calculate the latitude value corresponding to the CTU row and calculate the area stretching ratio of the sphere and its projection plane; Initialize the parameters of the rate control unit; Calculate the target number of bits for the current frame; Calculate the frame-level Lagrange multiplier of the current frame according to the target bit number; Calculate frame-level QP based on frame-level Lagrange multipliers and perform intra-frame coding; After encoding the current frame, repeat step S2; Otherwise, proceed to step S3; S3, calculating the frame pixel mean and variance value; S4, determine whether the current frame number is greater than 2 times the GOP size, if so, go to S5; otherwise, go to S6; S5, calculating the global Lagrange multiplier according to the iterative algorithm and optimizing the bit allocation of each CTU in the frame, and proceeding to step S7; S6. Use the encoder's built-in bit allocation algorithm to allocate bits to each CTU in the frame; S7, calculating frame-level Lagrange multipliers and QP; S8, encode CTU in sequence; S9, determining whether the difference between the pixel variance value of the current frame and the previous frame is greater than a threshold value, if so, proceeding to step S10, otherwise proceeding to step S11; S10, adjusting the Lagrange multiplier of the current CTU according to the area stretching ratio obtained in step S2, and proceeding to step S13; S11, determine whether the current frame number is greater than 3, if so, go to S13, otherwise go to S10; S12, adjusting the Lagrange multiplier of each CTU according to the time domain distortion impact factor of the previous frame and the area ratio obtained in S2; S13, calculating a quantization parameter QP of the current CTU according to the Lagrange multiplier of the current CTU calculated in S10 or S12 and encoding the CTU; S14, determine whether it is the last CTU, if so, then encode the current frame and go to step S15, otherwise, return to step S8; S15, updating the bit rate control parameters of the current frame and each CTU and the number of bits actually used; S16, obtaining the reconstruction error and motion compensation prediction error of each CTU saved in the current frame to calculate the time domain distortion influence factor of each CTU in the current frame; S17, determine whether it is the last frame, if so, go to S18, otherwise go to S2; S18. The current panoramic video sequence encoding is completed.

2. According to claim 1, a zero-delay panoramic video bit rate control method considering time domain distortion propagation is characterized in that: In step S2, the method for calculating the area stretching ratio is: define the radius of the panoramic video sphere as r, then the area of ​​the spherical ring zone with a latitude of θ is: S s (θ)=2π·r 2 ·cosθ·sinθ After projecting the 2D plane, the area of ​​the stretched region is: The area stretching ratio is:

3. The zero-delay panoramic video bit rate control method considering time domain distortion propagation according to claim 2, characterized in that: Step S5 is specifically as follows: Among them, R and r are the target bits at the frame level and CTU level respectively, and λ g is the global Lagrange multiplier, α i =c i ·k i , λ i is the slope of the rate-distortion curve, D i , R i are the distortion and coding bits of the i-th coding unit respectively, and M is the number of CTUs.

4. The zero-delay panoramic video bit rate control method considering time domain distortion propagation according to claim 3, characterized in that: The Lagrange multiplier adjustment method in step 10 is: Among them, λ P is the frame-level Lagrange multiplier, λ n is the adjusted Lagrange multiplier.

5. The zero-delay panoramic video bit rate control method considering time domain distortion propagation according to claim 3, characterized in that: The Lagrange multiplier adjustment method in step 12 is: Among them, k is the time domain distortion influencing factor.

Images