Video content perception characteristic-based three-dimensional wavelet scalable video code rate control method and system

Abstract

The invention belongs to the technical field of video coding and decoding, and relates to a three-dimensional wavelet scalable video code rate control method and system based on video content perception characteristics. According to the method, the influence of a wavelet filter type, a sub-band coupling phenomenon and a time domain sub-band content perception characteristic on reconstruction error transmission is fully considered, and a self-adaptive Lagrange multiplier selection algorithm suitable for a wavelet domain is provided; then, establishing an R-lambda model based on video content perception characteristics; and finally, based on the provided code rate control model, adaptively adjusting coding parameters to achieve the target of code rate control. Experimental results show that compared with a code rate control method adopted in a current classical three-dimensional wavelet video coding and decoding scheme, the method provided by the invention has the advantages that on the premise that the complexity is not obviously increased, the rate distortion performance of a coder and decoder is obviously improved while accurate code rate control is realized, the continuity of video quality is better maintained, and the video coding and decoding efficiency is improved. And more stable reconstructed video quality is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of video coding and decoding, and relates to bit rate control of three-dimensional wavelet scalable video, and in particular to a three-dimensional wavelet scalable video bit rate control method and system based on video content perception characteristics. Background Art

[0002] The main purpose of rate control is to obtain the highest possible reconstructed video quality under the given target rate conditions to meet the requirements of channel transmission bandwidth and storage device storage space. Rate control is the key guarantee for stable and reliable video transmission. Researchers can design appropriate rate control strategies for different application scenarios. Reasonable and effective rate control technology, on the one hand, can improve channel utilization while ensuring video quality; on the other hand, it is of great significance to break through bandwidth limitations and reduce storage costs. One of the core issues of rate control is to estimate the rate-distortion function of the encoded video sequence.

[0003] Rate control technology has been evolving with the development of video coding technology. Each generation of video coding standards recommends its corresponding rate control algorithm to be integrated into the test model, such as TM5 of MPEG-2, TMN8 of H.263, VM8 of MEPG-4, JM of H.264 / AVC and HM of HEVC. In the earlier H.261 and MEPG-1 video coding standards, only buffer state feedback was used to adjust the coding quantization step size to achieve rate control. In order to accurately control the output bit rate, the rate control algorithms recommended by H.263, MPEG-4, H.264 / AVC, HEVC and video coding standard AVS all use rate-distortion models. At present, rate control models are mainly divided into three categories: RQ model, R-ρ model and R-λ model:

[0004] (1) For the RQ model, the rate control method using the RQ model needs to obtain the quantization parameter QP, but in the 3D wavelet scalable video coding system, the quantization parameter QP is not available. It is implicitly present in the bit plane interception stage in an embedded manner rather than an explicit expression value. Therefore, it is very difficult to establish an RQ rate control model. Furthermore, in the 3D wavelet-based scalable video coding system, all motion vector information is retained and not compressed. Only the residual information is coded and compressed. Bits are allocated according to the sub-bit planes generated by coding, thereby achieving compression of low-frequency and high-frequency sub-bands. However, quantization is only effective for residual information, so it is difficult to accurately establish a model to describe the relationship between non-residual information (such as motion vector information, coding mode information, etc.) and QP; therefore, the adjustment of the quantization parameter QP has little significance for the bit overhead on non-residual information. In addition, the quantization parameter QP can only select integers. When QP increases by 6, the quantization step Qstep doubles; and QP can only select some discrete values, which also restricts the development of the accuracy of achieving the target bit rate through the quantization parameter QP.

[0005] (2) For the R-ρ model, the rate control method of the R-ρ model has good coding performance under the condition of using a fixed size transformation. However, when the R-ρ model is applied to a 3D wavelet video coding framework that supports multiple transformation sizes, the coding performance decreases. Therefore, the rate control algorithm based on the R-ρ model is difficult to apply to the 3D wavelet video coding framework.

[0006] (3) For the R-λ model, the Lagrange multiplier λ is the key factor in controlling the bit rate. The parameter λ can be calculated through adaptive bit rate allocation and parameter update. Since the λ selection algorithm directly affects the rate distortion performance and coding efficiency of the video coding and decoding system, it is necessary to select the optimal λ for encoding in order to achieve a more accurate bit rate control effect. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a three-dimensional wavelet scalable video rate control method and system based on video content perception characteristics, which can achieve precise rate control without significantly increasing the complexity, and also significantly improve the rate-distortion performance of the codec, while better maintaining the continuity of video quality and obtaining a smoother reconstructed video quality.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] On the one hand, the present invention provides a 3D wavelet scalable video bit rate control method based on video content perception characteristics, which specifically includes the following steps:

[0010] Step 1: In the motion compensation time domain filtering stage, based on the influence of wavelet filter type, subband coupling phenomenon and time domain subband content perception characteristics on reconstruction error transmission, an adaptive Lagrange multiplier selection algorithm for wavelet domain is constructed, and the optimal adaptive Lagrange multiplier λ for wavelet domain is selected. i_SSIM ; Wherein, the adaptive Lagrange multiplier selection algorithm is used to adaptively select the Lagrange multiplier for each time domain subband;

[0011] Step 2: Establish an R-λ model based on video content perception characteristics;

[0012] Step 3: Based on the R-λ model, the bit rate control of the 3D wavelet scalable video is implemented by using an adaptive bit rate control technology.

[0013] Furthermore, the specific implementation process of step 1 is as follows:

[0014] Step 1.1, based on the wavelet basis coefficients selected for time domain wavelet decomposition and the time domain subband coupling phenomenon, a distortion relationship model between the time domain subband frame and the reconstructed video frame is constructed, that is, the following formula (7);

[0015] Step 1.2, calculate the time domain subband weight factor according to the mutual information value of the brightness component of adjacent subband frames, the gradient value of each subband frame and its texture consistency. The calculation process is shown in the following formula (8);

[0016] Step 1.3: Adaptively select the current optimal wavelet domain adaptive Lagrange multiplier based on the structural similarity index of the video subband frame.

[0017] Furthermore, the method of adaptively selecting the structural similarity index based on the video subband frame in step 1.3 obtains the rate-distortion optimization objective function J SSIM (F i )for:

[0018]

[0019] In formula (17), is the weight distortion of the high frequency sub-band; is the weight distortion of the low-frequency subband; i_SSIM is the SSIM-based Lagrange multiplier of the i-th frame; is the high frequency sub-band SSIM distortion; is the low frequency sub-band SSIM distortion.

[0020] Furthermore, i_SSIM The calculation formula is as follows:

[0021]

[0022]

[0023] In formula (9), is an empirical parameter for different video sequences, with a value of 6.2; MAD is the mean absolute error between the original sub-band frame and the reconstructed sub-band frame; η is a parameter related to the temporal sub-band activity factor; R T is the target bit rate; ω i is the weight factor of the i-th time domain subband frame; in formula (16), MSE = (MAD) 2 ; is the variance of the residual pixel value before wavelet transform; c2 is a positive constant; λ i is the Lagrange multiplier of the i-th frame.

[0024] Furthermore, i The acquisition process is:

[0025]

[0026] In formula (6), d i is the distortion of the i-th time domain subband frame, r i is the bit rate of the i-th time domain subband frame; β q are parameters related to wavelet transform; is the variance of the residual pixel value before wavelet transform; Q is the total number of wavelet coefficients; k′ and η are parameters related to the temporal subband activity factor, which can be adaptively adjusted according to different video content and different temporal subband activities;

[0027] Among them, β q It is the parameter of different wavelet decomposition levels after wavelet transform for different frame modes. Its specific values ​​are as follows: q (q=1,2,3)

[0028]

[0029] in, Then formula (6) is simplified to:

[0030]

[0031] Then the Lagrange multiplier λ of the i-th frame is i The calculation formula is:

[0032]

[0033] In formula (8), is an empirical parameter for different video sequences, with a value of 6.2; MAD is the mean absolute error between the original sub-band frame and the reconstructed sub-band frame; η is a parameter related to the temporal sub-band activity factor; R T is the target bit rate; ω i is the weight factor of the i-th time domain subband frame.

[0034] Furthermore, in step 2, the R-λ model is:

[0035]

[0036] In formula (18), λ i_SSIM is the SSIM-based Lagrange multiplier of the i-th frame; is the variance of the residual pixel value before wavelet transform; c2 is a positive constant; MSE is the mean square error between the original sub-band frame and the reconstructed sub-band frame; R T is the target bit rate; is an empirical parameter for different video sequences, with a value of 6.2; η is a parameter related to the temporal subband activity factor; ω i is the weight factor of the i-th time domain subband frame.

[0037] Furthermore, step 3, based on the R-λ model, implementing bit rate control of the 3D wavelet scalable video by adaptive bit rate control technology, specifically includes:

[0038] Step 3.1, GOP level bit allocation, target bit of each GOP level GOP The calculation formula is:

[0039]

[0040]

[0041] In formula (19), Bit avg_frame is the average number of bits per video frame, calculated by formula (20); N coded is the number of frames that have been encoded; N GOP is the number of video frames contained in GOP; SW is the sliding window size, SW ≥ N GOP ;Bit coded is the total number of bits consumed by all encoded frames; in formula (20), R tar is the target bitrate; framerate is the frame rate;

[0042] Step 3.2: Frame-level bit allocation, frame-level target bit Bit frame It is calculated according to formula (21), that is, the remaining bits in the GOP are distributed to the remaining frames according to the weights of different video frames;

[0043] Bit frame =(Bit GOP -Coded GOP )×ω frame (twenty one)

[0044] In formula (21), Bit GOP is the target number of bits at the GOP level; Coded GOP is the number of bits consumed in encoding the current GOP; frame is the weight coefficient of the current video frame, calculated by formula (22);

[0045]

[0046] In formula (22), E frame is the current frame energy; E GOP is the current GOP energy; W is the video frame width; H is the video frame height; I(x,y) is the pixel value at (x,y);

[0047] Step 3.3: basic unit level bit allocation, define the basic unit as different time domain subbands after MCTF time domain decomposition, basic unit level target bit Bit sub According to formula (23), we can get:

[0048] Bit sub =(Bit frame -Bit header -Coded frame )×ω sub (twenty three)

[0049] In formula (23), Bit frame The target bit number of the current frame; Bit header The number of bits consumed for encoding header information; Coded frame is the number of bits consumed in encoding the current frame; ω sub is the weight coefficient of the time domain subband;

[0050] Step 3.4: Use the adaptive bit rate control model to implement bit rate control of 3D wavelet scalable video.

[0051] Furthermore, in step 3.4, the adaptive rate control model is calculated by formula (24):

[0052] λ=α·2 βR (twenty four)

[0053] In formula (24), α and β are model parameters related to the characteristics of the video sequence and can be obtained by adaptive updating, that is:

[0054] α new =α old +2δ·(lnλ real -lnλ theory )×α old (25)

[0055] β new =β old +2δ·(lnλ real -lnλ theory )·Rln2 (26)

[0056] In formula (25), α old is the α value before updating; α new is the updated α value; λ real is the λ value actually used in encoding; λ theory is the theoretical value of λ obtained by the R-λ model; δ is the disturbance parameter, and δ∈(0,1); in formula (26), R is the actual number of coded bits; β old is the β value before updating; β new is the updated β value; λ real is the λ value actually used in encoding; λ theory is the theoretical value of λ obtained by the R-λ model; δ is the disturbance parameter, and δ∈(0,1).

[0057] On the other hand, the present invention also provides a system based on part or all of the above three-dimensional wavelet scalable video rate control method, comprising an encoder, a bitstream extractor, and a decoder connected in sequence; at the encoder end, the original video is first analyzed by the MCTF analysis module, and the time domain subband weight is obtained according to the obtained time domain subband, and an adaptive λ selection module is performed; the time domain subband passes through the space domain analysis module to form a three-dimensional wavelet time-space subband, and then the bitstream information is organized into a fully embedded coding bitstream through a quantization module, a 3D-EZBC coding module and a motion vector lossless coding module; the bitstream extractor processes the received fully embedded coding bitstream according to the process of "constructing R-λ model → bit allocation → bitrate control" to obtain a scalable coding bitstream; the decoder decodes the received scalable coding bitstream to generate a reconstructed video.

[0058] Furthermore, in the system, the proportion of unconnected pixels r is used. u Adaptively adjust the MCTF decomposition level, specifically:

[0059] Definition m is the proportion of multi-connected pixels, r u is the proportion of unconnected pixels, r c is the proportion of connected pixels; m≥0, r u ≥0, r c ≥0, and r m +r c +r u =1; where r u The definition is as follows:

[0060] r u =Uncon_num / Pixel_num (3)

[0061] In formula (3), Uncon_num is the number of unconnected pixels; Pixel_num is the number of all pixels in each frame; when r u When the given threshold T=0.5 is exceeded, the MCTF process is terminated.

[0062] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0063] 1) In the motion compensation time domain filtering stage, the present invention fully considers the influence of wavelet filter type, subband coupling phenomenon and time domain subband content perception characteristics on the reconstruction error transmission. First, according to the wavelet basis coefficients selected for time domain wavelet decomposition and the time domain subband coupling phenomenon, a distortion relationship model between the time domain subband frame and the reconstructed video frame is constructed; secondly, the time domain subband weight factor is calculated according to the mutual information value of the brightness component of adjacent subband frames, the gradient value of each subband frame and its texture consistency; finally, based on the structural similarity index (Structural Similarity Index, SSIM) of the video subband frame, the Lagrange multiplier is adaptively selected. The algorithm can adaptively select Lagrange multipliers for each time domain subband content perception, which can not only accurately obtain the error transmission model of MCTF, but also obtain better video reconstruction quality.

[0064] 2) The inventors of the present application have discovered that in a three-dimensional wavelet video coding and decoding system, there is a more robust relationship between the bit rate R and the Lagrange multiplier λ. When establishing the R-λ model, the activity factor of the video subband frame is first obtained by calculating the color histogram and directional gradient histogram of the time domain subband, and then the R-λ model based on the perceptual characteristics of the video content is established. Finally, based on the proposed R-λ model, the encoding parameters are adaptively adjusted to achieve the goal of controlling the bit rate.

[0065] In summary, compared with the rate control method used in the current classic 3D wavelet video coding and decoding scheme, the rate control method provided by the present invention can achieve precise rate control without significantly increasing the complexity, while also significantly improving the rate-distortion performance of the codec, while better maintaining the continuity of video quality and obtaining a smoother reconstructed video quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings are incorporated in and constitute a part of this specification and, together with the description, serve to explain the principles of the present invention.

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0068] Figure 1 A flowchart of a three-dimensional wavelet scalable video bit rate control method based on video content perception characteristics provided by the present invention;

[0069] Figure 2 A system composition connection block diagram of the three-dimensional wavelet-based scalable video bit rate control method provided by the present invention;

[0070] Figure 3 A processing flow chart of the code stream extractor provided by the present invention;

[0071] Figure 4 A comparison chart of rate-distortion performance of different coding schemes provided in Example 2 at different target bit rates;

[0072] Figure 5 A comparison result diagram of average SSIM (MSSIM) of test video sequences with different resolutions provided in Example 2 on different 3D wavelet video coding and decoding systems;

[0073] Figure 6 A comparison diagram of the accuracy of rate control in different 3D wavelet video codecs provided in Example 2;

[0074] Figure 7 The average PSNR standard deviation of different 3D wavelet test video sequences provided in Example 2. DETAILED DESCRIPTION

[0075] Here, exemplary embodiments will be described in detail, and the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples consistent with some aspects of the present invention described in detail in the appended claims.

[0076] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0077] Explanation of relevant technical terms:

[0078] MCTF: Motion Compensated Temporal Filtering. Rate control technology consists of two steps: the first step is to allocate the appropriate number of bits to each level of coding unit, usually including the picture group GOP level, frame level and basic unit BU level; the second step is to try to achieve the number of bits pre-allocated for each level. Rate control technology allows the encoder to select appropriate coding parameters from some discrete legal coding parameter sets to achieve the target bit rate. 3D-EZBC: Three-dimensional embedded zero block coding. SSIM: Structural Similarity Index. SAD: Sum of Absolute Differences. SAD distortion is a common image and video quality assessment indicator, mainly used to measure the degree of difference between two images or video frames. MAD: Mean absolute error. MSE: Mean Square Error. LMS: Least Mean Square. NRMSE: Normalized Root Mean Square Error. PSNR: Peak Signal to Noise Ratio, peak signal-to-noise ratio.

[0079] The rate-distortion RD curve is a convex function, and λ is the slope of the rate-distortion RD curve. Assuming that D and R in the rate-distortion function are differentiable, λ can be calculated by minimizing the Lagrange cost function, namely, the following equations (1) and (2):

[0080]

[0081]

[0082] In the rate-distortion optimization process of video coding, the Lagrange multiplier λ is the key coding parameter to balance the bit rate and distortion. It determines the selection of coding mode in the coding process and is responsible for the coordination between the video coding bit rate and coding distortion. It plays a vital role in the encoder. λ, bit rate and distortion affect each other. The rate control method based on the R-λ model takes the Lagrange multiplier λ as the main factor to control the bit rate, and calculates the parameter λ through adaptive bit rate allocation and parameter update.

[0083] Example 1

[0084] See also Figure 1 As shown, this embodiment provides a 3D wavelet scalable video bit rate control method based on video content perception characteristics, which specifically includes the following steps:

[0085] Step 1: In the motion compensation time domain filtering stage, based on the influence of wavelet filter type, subband coupling phenomenon and time domain subband content perception characteristics on reconstruction error transmission, an adaptive Lagrange multiplier λ selection algorithm for wavelet domain is constructed. The specific process is as follows:

[0086] (1) Adaptive adjustment of the time domain wavelet decomposition level:

[0087] After motion estimation, pixels are divided into four types: connected pixels, unconnected pixels, multi-connected pixels, and intra pixels. For most video sequences, only a few pixels in the current frame have no matching pixels in the reference frame, and these pixels are called intra pixels. The present invention omits the analysis of this type of pixels (intra pixels).

[0088] Let r m is the proportion of multi-connected pixels, r u is the proportion of unconnected pixels, r c is the proportion of connected pixels, where r m ≥0, r u ≥0, r c ≥0, and r m +r c +r u =1. For different types of video frames, the connection types of pixels after motion estimation are various, and the proportions of different types of pixels in low-frequency frames are also different. In addition, in the same GOP, the proportions of different types of pixels in low-frequency frames are also different in different MCTF temporal decomposition levels. As the decomposition level increases, the proportion of connected pixels in low-frequency frames decreases, while the proportions of multi-connected pixels and unconnected pixels increase.

[0089] In the MCTF time domain decomposition process, when unconnected pixels appear in low-frequency frames, motion estimation cannot find pixels that match them, which will cause periodic fluctuations in the quality of the reconstructed video. In particular, when the bidirectional prediction matching results produce too many unconnected pixels, the time domain analysis cannot be continued. Therefore, we use the proportion of unconnected pixels r u Adaptively adjust the MCTF decomposition level. u When the given threshold T=0.5 is exceeded, we will terminate the MCTF process. The given threshold T is related to the level of time domain decomposition. u The definition of is shown in formula (3):

[0090] r u =Uncon_num / Pixel_num (3)

[0091] In formula (3), Uncon_num is the number of unconnected pixels; Pixel_num is the number of all pixels in each frame.

[0092] (2) Adaptive Lagrange multiplier selection

[0093] λ can be regarded as a trade-off factor between distortion and bit rate. The larger the λ is, the larger the proportion of R in the cost function J is, which means that the compression rate requirement is increased when selecting the mode, and the image quality is ignored to a certain extent; the smaller the λ is, the smaller the proportion of R in the cost function J is, which means that the encoder needs to sacrifice a certain number of bits in order to improve the image quality. Therefore, we can allocate more bits to the time domain subband containing more detailed information by reducing the λ value. Since the distortion of the video frame is a linear combination of the distortions of all time domain subbands, and the wavelet coefficients of each time domain subband obey the Gaussian distribution, the subband distortion after wavelet transform is:

[0094]

[0095] In formula (4), d i is the distortion of the i-th time domain subband frame, r i is the bit rate of the i-th time domain subband frame; is the variance of the qth wavelet coefficient; Q is the total number of wavelet coefficients; k′ and η are parameters related to the time domain subband activity factor. However, in practical applications, it is not easy to calculate the variance of wavelet coefficients; therefore, we can make the following approximation:

[0096]

[0097] In formula (5), β q are the parameters related to wavelet transform: is the variance of the residual pixel value before wavelet transform; Substituting formula (5) into formula (4), we get formula (6):

[0098]

[0099]

[0100] In formula (6), It can be estimated by MAD approximation, and formula (6) can be expressed as:

[0101]

[0102] In formula (7), Therefore, the Lagrange multiplier λ of the i-th frame is i Obtained by the following formula:

[0103]

[0104] Based on formula (8), the Lagrange multiplier λ of the i-th frame is i It can be approximated by the following formula:

[0105]

[0106] In formula (9), In order to adapt the empirical parameters to different video sequences, the value here is 6.2.

[0107] In order to obtain the best subjective perceptual quality under bit rate constraints, it is necessary to select a rate-distortion optimized coding mode guided by subjective distortion. The present invention considers introducing the structural similarity index SSIM distortion into the adaptive MCTF framework in 3D wavelet video coding to select the best coding mode, thereby achieving the best subjective rate-distortion performance.

[0108] Define the Lagrange cost function J based on SSIM SSIM for:

[0109] J SSIM =D SSIM +λ SSIM ·R (10)

[0110] In formula (10), D SSIM is the SSIM distortion, and its calculation formula is (11); R is the coding rate; λ SSIM It is the Lagrange multiplier for weighing the coding rate R and distortion D in SSIM-based rate-distortion optimization.

[0111]

[0112] In the SSIM-based rate-distortion optimization framework, SSIM distortion is used instead of SAD distortion to obtain better subjective quality, but the difficulty lies in λ i_SSIM How to determine. Since different frames and different macroblocks have different rate-distortion characteristics, if we can adaptively adjust λ according to the saliency of image content at the macroblock level, i_SSIM The bit rate and SSIM distortion can be better balanced, further improving the subjective rate-distortion performance.

[0113] In order to minimize the Lagrange cost function of the coding mode based on SSIM distortion, the formula (10) is derived and the derivative is set to zero, then:

[0114]

[0115] Assuming that the distortion of each time domain subband frame is independent and identically distributed and obeys the additive Gaussian white noise distribution with a mean of 0, the SSIM can be approximated as:

[0116]

[0117] Since the relationship between MAD and MSE can be approximately: MSE = (MAD) 2 , therefore, the SSIM distortion of the i-th frame is d i_SSIM It can be approximated as:

[0118]

[0119]

[0120] Combining formula (15) and formula (9), we can know

[0121]

[0122] In formula (16), λ i_SSIM is the SSIM-based Lagrange multiplier for the i-th frame, is the variance of the original subband frame; c2 is a positive constant.

[0123] In summary, based on the adaptive Lagrange multiplier λ selection algorithm, its rate-distortion optimization objective function is shown in formula (17):

[0124]

[0125] In formula (17), is the weight distortion of the high frequency sub-band; is the weight distortion of the low-frequency subband; i_SSIM is the SSIM-based Lagrange multiplier of the i-th frame; is the high frequency sub-band SSIM distortion; is the low frequency sub-band SSIM distortion.

[0126] Step 2: Establish an R-λ model based on the video content perception characteristics. The specific process is as follows:

[0127] The adaptive R-λ model of the i-th subband frame is defined as:

[0128]

[0129] In formula (18), λ i_SSIM is the SSIM-based Lagrange multiplier of the i-th frame; is the variance of the original sub-band frame; c2 is a positive constant; MSE is the mean square error between the original sub-band frame and the reconstructed sub-band frame; R Tis the target bitrate.

[0130] Step 3: Based on the R-λ model, the bit rate control of the 3D wavelet scalable video is realized by using an adaptive bit rate control technology. The specific process is as follows:

[0131] Step 3.1, GOP level bit allocation:

[0132] In order to better transmit the video in the network, the GOP level should implement variable bit rate control, namely VBR control mode. The number of bits allocated to each GOP level can be adaptively adjusted according to the network status. In the bit rate control algorithm proposed in this invention, the target bit of GOP level is calculated using the following formula (19): GOP .

[0133]

[0134] In formula (19), Bit avg_frame is the average number of bits per video frame, calculated by the following formula (20); N coded is the number of frames that have been encoded; N GOP is the number of video frames contained in GOP; SW is the sliding window size, SW ≥ N GOP ;Bit coded The total number of bits consumed for all encoded frames.

[0135]

[0136] In formula (20), R tar is the target bit rate; frame rate is the frame rate.

[0137] Step 3.2: Frame-level bit allocation: Frame-level target bit Bit frame It is calculated according to formula (21), that is, the remaining bits in the GOP are distributed to the remaining frames according to the weights of different video frames:

[0138] Bit frame =(Bit GOP -Coded GOP )×ω frame (twenty one)

[0139] In formula (21), Bit GOP is the target number of bits at the GOP level; Coded GOP is the number of bits consumed in encoding the current GOP; frame is the weight coefficient of the current video frame, obtained by formula (22):

[0140]

[0141] In formula (22), E frame is the current frame energy; E GOP is the current GOP energy; W is the video frame width; H is the video frame height; I(x,y) is the pixel value at (x,y).

[0142] Step 3.3: Basic unit level bit allocation:

[0143] The target bit allocation at the basic unit level is to allocate the remaining bits of the current video frame to the remaining basic units of the current frame in proportion. Here, the basic unit refers to the different time domain subbands after MCTF time domain decomposition. sub Calculated according to formula (23).

[0144] Bit sub =(Bit frame -Bit header -Coded frame )×ω sub (twenty three)

[0145] In formula (23), Bit frame The target bit number of the current frame; Bit header The number of bits consumed by the encoding header information; Coded frame is the number of bits consumed in encoding the current frame; ω sub is the weight coefficient of the time domain subband.

[0146] Step 3.4, bit rate control: Determine appropriate coding parameters based on the established R-λ model, and simplify the R-λ model shown in formula (18) to:

[0147] λ=α·2 βR (twenty four)

[0148] In formula (24), α and β are model parameters related to the characteristics of the video sequence and can be obtained by adaptive updating, that is:

[0149] α new =α old +2δ·(lnλ real -lnλ theory )×α old (25)

[0150] β new =β old +2δ·(lnλ real -lnλ theory )·Rln2 (26)

[0151] In formula (25) and formula (26), R is the actual number of coded bits; αold is the α value before updating; α new is the updated α value; β old is the β value before updating; β new is the updated β value; λ real is the λ value actually used in encoding; λ theory is the theoretical value of λ obtained by the R-λ model. δ is the disturbance parameter, and δ∈(0,1). It is worth noting that δ in formula (25) and formula (26) can take different values.

[0152] In order to verify the rationality and accuracy of the adaptive update parameters provided by the present invention, the following verification is carried out:

[0153] Taking the logarithm of formula (24) yields:

[0154] lnλ=lnα+ln2 βR (27)

[0155] Therefore, we have:

[0156]

[0157] The actual lambda used real The value and theoretically obtained λ theory The squared error between values ​​is:

[0158] e 2 =(lnλ real -lnλ theory ) 2 (29)

[0159]

[0160] The LMS method is used for one iteration, and formula (30) yields:

[0161]

[0162] From formula (25), we can get:

[0163] lnα new = lnα old +2δ·(lnλ real -lnλ theory ) (33)

[0164] therefore,

[0165]

[0166] Formula (34) is Taylor expanded and high-order terms are ignored, then:

[0167]

[0168] Similarly, we can get:

[0169]

[0170] It can be seen from formula (35) and formula (36) that they are consistent with formula (25) and formula (26) respectively, so the adaptive update model parameters obtained are reasonable.

[0171] Example 2

[0172] On the basis of Example 1, in order to evaluate the performance of the rate control method based on the video content perception characteristics and the 3D wavelet scalable video codec framework proposed in the present invention, 14 standard test videos with YUV format, 8-bit bit depth and 4:2:0 sampling rate were selected for the experiment, including five types of resolutions: CIF (352×288), 4CIF (704×576), 720p (1280×720), 1080p (1920×1080) and 2K (2560×1600). For test videos of different resolutions, the extracted target bitrates are also different (CIF and 4CIF: 256, 384, 512, 640, 768, 896, 1024 kbps; 720p: 384, 512, 640, 768, 896, 1024, 1536 kbps; 1080p and 2K: 2048, 3072, 4096, 5120, 6144, 7168, 10240 kbps).

[0173] It should be noted that the algorithm implementation and model verification mentioned in this embodiment are all carried out on an Intel 4-core (i5-2400@3.10GHz CPU) and 8G memory desktop computer, using the program environment of Matlab R2012b and VC++6.0. During the experiment, GOP selected 16 frames (GOP=16), MCTF time domain decomposition was 4 levels, motion estimation used a full search with 1 / 4 pixel accuracy, the horizontal and vertical search ranges were [-16,15], the coding block size range was 4×4 to 128×128, and other coding parameters were consistent with the default configuration file. The classic three-dimensional wavelet scalable video codec systems MC-EZBC, RPI-MC-EZBC, RWTH-MC-EZBC and ENH-MC-EZBC can obtain better coding performance in the field of three-dimensional wavelet video coding. This embodiment will use the rate control scheme of the above-mentioned codec system as a comparative experiment, and the performance evaluation indicators are as follows:

[0174] (1) PSNR indicator

[0175] Since the human eye is most sensitive to the brightness component, we only consider the brightness component of the video sequence in the experiment. This paper uses the Dolby number (decibel, dB) of the peak signal-to-noise ratio (PSNR) as an indicator for objectively evaluating video quality. As a commonly used video objective quality evaluation indicator, the peak signal-to-noise ratio is defined as: given an original image f(x, y) of size W×H and a distorted image g(x, y), the peak signal-to-noise ratio (PSNR) of the image f(x, y) is calculated by formula (37):

[0176]

[0177] In formula (37), MSE is the mean square error; f max is the maximum grayscale value of the original input image f(x,y). For example, in a commonly used 8-bit grayscale image, f max The value of is 255.

[0178] (2) SSIM indicator

[0179] PSNR has long been favored by people because of its clear physical meaning and high computational efficiency. For the same video content, the higher the PSNR value, the better the video quality. However, the evaluation results of PSNR are often inconsistent with the results of human subjective evaluation. Studies have shown that only calculating the difference at the pixel level cannot accurately measure the visual quality of the image. The structural similarity index SSIM estimates the quality of the distorted image by comparing the similarity of the brightness information, contrast information and structural information of the original image and the distorted image. The calculation formula is shown in the following formula (38).

[0180]

[0181] In formula (38), c1 and c2 are positive constants, which are used to ensure that the entire fraction can obtain a stable calculation result when the denominator approaches zero; μ f is the mean of the original image; μ g is the mean of the distorted image; is the variance of the original image; is the variance of the distorted image; σ f,g is the covariance between the original image and the distorted image. The value range of SSIM is [0,1]. The larger the SSIM value, the smaller the image distortion and the better the image quality. In most practices and applications, it is feasible, convenient and practical to use PSNR and SSIM to measure the objective quality of the image. Therefore, this embodiment uses PSNR and SSIM as objective evaluation indicators of the decoded video quality.

[0182] In the experiment, we use MSSIM (Mean SSIM) as the evaluation index to measure the subjective quality of the reconstructed video, which can be calculated by the following formula (39).

[0183]

[0184] In formula (39), K is the total number of video frames; SSIM i is the SSIM value of the i-th frame, obtained by formula (38).

[0185] (3)PSNR STD index

[0186] In encoders with MCTF structures, there are periodic fluctuations in the quality of reconstructed video (PSNR). The average PSNR alone is not comprehensive as a video quality evaluation indicator. Video quality fluctuation should also be used as another indicator to measure the performance of 3D wavelet video encoders. Therefore, we use PSNR to measure the quality of reconstructed video, and the standard deviation of PSNR (PSNR Standard Deviation, PSNR STD ) is used to measure the volatility of the reconstructed video quality, which is defined as follows:

[0187]

[0188] In formula (40), PSNR i is the PSNR value of the i-th reconstructed video frame; K is the total number of video frames. The higher the PSNR value, the better the rate-distortion performance; at the same time, PSNR STD The smaller the value, the smaller the fluctuation of the reconstructed video quality.

[0189] (4) NRMSE indicator

[0190] We use the standard root mean square error NRMSE to measure the accuracy of bit rate control, and the calculation formula is shown in the following formula (41).

[0191]

[0192] In formula (41), is the actual encoding bit rate of the i-th frame; is the target bit rate of the i-th frame; is the average value of the actual encoding bit rate; K is the total number of video frames. The smaller the NRMSE value, the more accurate the bit rate control, and vice versa.

[0193] (5)R 2 index

[0194] To evaluate the accuracy of the algorithm model, we used R 2Quantitatively measure the deviation between the estimated value of λ and the theoretical value, R 2 The calculation of is as follows:

[0195]

[0196] In formula (42), X i is the theoretical value of the i-th data; is the estimated value of the i-th data; is the average value of all data. When R 2 Reaches a maximum value of 1. R 2 The closer it is to 1, the closer the estimated value is to the theoretical value, and the more accurate the established model is.

[0197] After comparative analysis, the corresponding experimental results and their analysis are as follows:

[0198] ① Discussion on the accuracy analysis of R-λ model

[0199] In our experiments, we test videos at different MCTF decomposition levels. i_SSIM R 2 The values ​​are shown in Table 1. As can be seen from Table 1, R 2 The value is almost close to 1. Therefore, it can be concluded that the λ adaptive selection model can fit different coding rates well, and the proposed rate control method and its mathematical model are reasonable.

[0200] Table 1 R of video sequences at different MCTF decomposition levels 2 Table 1. The R 2 values ​​of each MCTF level for test video sequences.

[0201]

[0202]

[0203] ② Rate-distortion performance analysis and discussion

[0204] In order to verify the rate-distortion performance of 3D wavelet scalable video codec after adopting the video content-aware bit rate control method under the rate-distortion optimization criterion, our coding scheme (Ours) will be compared with the most representative 3D wavelet motion compensated subband coding schemes (ENH-MC-EZBC, RWTH-MC-EZBC, RPI-MC-EZBC and MC-EZBC). Let "Scheme 1", "Scheme 2", "Scheme 3" and "Scheme 4" be the comparison of our coding scheme with ENH-MC-EZBC, RWTH-MC-EZBC, RPI-MC-EZBC and MC-EZBC schemes respectively. Figure 4 The rate-distortion performance comparison of different coding schemes at different target bit rates is given. Figure 4 It can be seen that our solution can improve the average PSNR gain by 0.79-2.58dB compared with the other four solutions. Figure 5 The average SSIM (MSSIM) comparison results of the test video sequences with different resolutions on different 3D wavelet video coding and decoding systems are given. Figure 5 It can be seen that the rate control algorithm we proposed can better reflect the perceptual quality of the reconstructed video at different target bit rates.

[0205] ③ Analysis and discussion on bit rate control accuracy

[0206] Figure 6 A comparison of the accuracy of rate control in different 3D wavelet video codecs is given. Figure 6 The experimental results show that the λ domain rate control scheme we proposed can obtain more accurate rate control results. Especially for video sequences containing rich texture information and scene switching, the rate control effect is more significant. For example, compared with ENH-MC-EZBC, our rate control scheme is better and can achieve the purpose of more accurate rate control. The rate control method we proposed has an average NRMSE of 0.89%-2.18%. The average NRMSE of rate control in the ENH-MC-EZBC codec is 2.93%-4.89%. Compared with the rate control schemes in RWTH-MC-EZBC, RPI-MC-EZBC and MC-EZBC, the rate control method proposed in this embodiment can also obtain better rate control accuracy.

[0207] ④Discussion on video quality fluctuation analysis

[0208] Figure 7 The comparison results of the fluctuation of video quality reconstructed by different 3D wavelet video codecs are given. Figure 7Experimental results show that the proposed rate control method based on video content perception in the λ domain has a smaller PSNR standard deviation than other 3D wavelet video codecs, and can achieve a more stable reconstructed video quality. STD The value is 1.5. Compared with ENH-MC-EZBC, the PSNR standard deviation of our proposed algorithm can be reduced by 2.08. At the same time, the stability of the reconstructed video quality using the rate control algorithm proposed in this paper is also better than the reconstructed video quality obtained by the rate control methods in RWTH-MC-EZBC, RPI-MC-EZBC and MC-EZBC.

[0209] In summary, the present embodiment provides a rate control method for a 3D wavelet scalable video coding system. In this method, firstly, a Lagrange multiplier is adaptively selected for each time domain subband frame; then, an R-λ model is constructed based on the proposed adaptive Lagrange multiplier selection algorithm; finally, a λ domain rate control method based on video content perception characteristics is proposed, which effectively improves the rate control accuracy by adaptively adjusting the encoding parameters.

[0210] Moreover, based on the present invention, the deep learning framework can be used to carry out targeted design and training according to different application scenarios, and autonomously learn the features from the bottom layer to the high layer from a large number of images, showing excellent performance in artificial intelligence applications, and has a wide range of application value. In future research work, we can make full use of the advantages of wavelet transform and deep neural network, use adaptive wavelet neural network as the encoding core, and use the deep learning framework to design the video encoding rate control algorithm with the highest encoding efficiency to improve the encoding performance of the video, and effectively reduce the amount of video data while improving the video frame quality. In addition, more technologies inspired by biology and neurology can be introduced, such as attention, concept formation, metacognition and memory, which have good application prospects.

[0211] Example 3

[0212] Based on the embodiment, this embodiment also provides a system for a three-dimensional wavelet scalable video bit rate control method, combined with Figure 2As shown in the figure, the system is based on the ENH-MC-EZBC codec system platform and includes three parts: encoder, bitstream extractor and decoder. Its working principle is as follows: at the encoder end, the original video sequence is first analyzed by the MCTF analysis module, and the time domain subband weight is obtained according to the obtained time domain subband, and adaptive λ selection is performed; then, the time domain subband is analyzed in the spatial domain to form a three-dimensional wavelet time-space subband, and then quantized, 3D-EZBC encoded bitstream and motion vector lossless encoded bitstream information are organized together into a fully embeddable encoded bitstream; at the bitstream extractor end, an R-λ model is constructed, and bit allocation and bit rate control schemes are performed to obtain an efficient and flexible scalable encoded bitstream; at the decoder end, the inverse operation of the encoder end is performed to generate a reconstructed decoded video sequence.

[0213] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention.

[0214] It should be understood that the present invention is not limited to what has been described above and that various modifications and changes may be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A 3D wavelet scalable video rate control method based on video content perception characteristics, characterized in that: The specific steps include: Step 1: In the motion compensation time domain filtering stage, based on the influence of wavelet filter type, subband coupling phenomenon and time domain subband content perception characteristics on reconstruction error transmission, an adaptive Lagrange multiplier selection algorithm for wavelet domain is constructed, and the optimal adaptive Lagrange multiplier λ for wavelet domain is selected. i_SSIM ; Wherein, the adaptive Lagrange multiplier selection algorithm is used to adaptively select the Lagrange multiplier for each time domain subband; Step 2: Establish an R-λ model based on video content perception characteristics; Step 3: Based on the R-λ model, the bit rate control of the 3D wavelet scalable video is implemented by using an adaptive bit rate control technology.

2. The method for scalable video bit rate control based on three-dimensional wavelet based video content perception characteristics according to claim 1, characterized in that: The specific implementation process of step 1 is as follows: Step 1.1, construct a distortion relationship model between the time domain sub-band frame and the reconstructed video frame according to the wavelet basis coefficients selected for the time domain wavelet decomposition and the time domain sub-band coupling phenomenon; Step 1.2, calculating the time domain subband weight factor according to the mutual information value of the brightness component of adjacent subband frames, the gradient value of each subband frame and its texture consistency; Step 1.3: Adaptively select the current optimal wavelet domain adaptive Lagrange multiplier based on the structural similarity index of the video subband frame.

3. The method for scalable video bit rate control based on three-dimensional wavelet based video content perception characteristics according to claim 2, characterized in that: In step 1.3, the method of adaptively selecting the structural similarity index based on the video subband frame is used to obtain the rate-distortion optimization objective function J SSIM (F i )for: In formula (17), is the weight distortion of the high frequency sub-band; is the weight distortion of the low-frequency subband; i_SSIM is the SSIM-based Lagrange multiplier of the i-th frame; is the high frequency sub-band SSIM distortion; is the low frequency sub-band SSIM distortion.

4. The method for scalable video bit rate control based on three-dimensional wavelet based video content perception characteristics according to claim 3, characterized in that: λ i_SSIM The calculation formula is as follows: In formula (9), is an empirical parameter for different video sequences, with a value of 6.2; MAD is the mean absolute error between the original sub-band frame and the reconstructed sub-band frame; η is a parameter related to the temporal sub-band activity factor; R T is the target bit rate; ω i is the weight factor of the i-th time domain subband frame; In formula (16), MSE=(MAD) 2 ; is the variance of the residual pixel value before wavelet transform; c2 is a positive constant; λ i is the Lagrange multiplier of the i-th frame.

5. The method for scalable video bit rate control based on three-dimensional wavelet based video content perception characteristics according to claim 4, characterized in that: λ i The acquisition process is: In formula (6), d i is the distortion of the i-th time domain subband frame, r i is the bit rate of the i-th time domain subband frame; β q are parameters related to wavelet transform; is the variance of the residual pixel value before wavelet transform; Q is the total number of wavelet coefficients; k′ and η are parameters related to the time domain subband activity factor; where, Then formula (6) is simplified to: Then the Lagrange multiplier λ of the i-th frame is i The calculation formula is: In formula (8), is an empirical parameter for different video sequences, with a value of 6.2; MAD is the mean absolute error between the original sub-band frame and the reconstructed sub-band frame; η is a parameter related to the temporal sub-band activity factor; R T is the target bit rate; ω i is the weight factor of the i-th time domain subband frame.

6. The method for scalable video bit rate control based on three-dimensional wavelet based video content perception characteristics according to claim 1, characterized in that: In step 2, the R-λ model is: In formula (18), λ i_SSIM is the SSIM-based Lagrange multiplier of the i-th frame; is the variance of the residual pixel value before wavelet transform; c2 is a positive constant; MSE is the mean square error between the original sub-band frame and the reconstructed sub-band frame; R T is the target bit rate; is an empirical parameter for different video sequences, with a value of 6.2; η is a parameter related to the temporal subband activity factor; ω i is the weight factor of the i-th time domain subband frame.

7. The method for scalable video bit rate control based on three-dimensional wavelet based video content perception characteristics according to claim 1, characterized in that: Step 3: Based on the R-λ model, the bit rate control of the 3D wavelet scalable video is implemented by using an adaptive bit rate control technology, which specifically includes: Step 3.1, GOP level bit allocation, target bit of each GOP level GOP The calculation formula is: In formula (19), Bit avg_frame is the average number of bits per video frame, calculated by formula (20); N coded is the number of frames that have been encoded; N GOP is the number of video frames contained in GOP; SW is the sliding window size, SW ≥ N GOP ;Bit coded is the total number of bits consumed by all encoded frames; in formula (20), R tar is the target bit rate; frame rate is the frame rate; Step 3.2: Frame-level bit allocation, frame-level target bit Bit frame It is calculated according to formula (21), that is, the remaining bits in the GOP are distributed to the remaining frames according to the weights of different video frames; Bit frame =(Bit GOP -Coded GOP )×ω frame (21) In formula (21), Bit GOP is the target number of bits at the GOP level; Coded GOP is the number of bits consumed in encoding the current GOP; frame is the weight coefficient of the current video frame, calculated by formula (22); In formula (22), E frame is the current frame energy; E GOP is the current GOP energy; W is the video frame width; H is the video frame height; I(x,y) is the pixel value at (x,y); Step 3.3: basic unit level bit allocation, define the basic unit as different time domain subbands after MCTF time domain decomposition, basic unit level target bit Bit sub According to formula (23), we can get: Bit sub =(Bit frame -Bit header -Coded frame )×ω sub (23) In formula (23), Bit frame The target bit number of the current frame; Bit header The number of bits consumed by the encoding header information; Coded frame is the number of bits consumed in encoding the current frame; ω sub is the weight coefficient of the time domain subband; Step 3.4: Use the adaptive bit rate control model to implement bit rate control of 3D wavelet scalable video.

8. The method for scalable video bit rate control based on three-dimensional wavelet based video content perception characteristics according to claim 7, characterized in that: In step 3.4, the adaptive rate control model is calculated by formula (24): λ=a·2 βR (24) In formula (24), α and β are model parameters related to the characteristics of the video sequence and can be obtained by adaptive updating, that is: a new =a old +2δ·(lnλ real -lnλ theory )×a old (25) b new =b old +2δ·(lnλ real -lnλ theory )·Rln2 (26) In formula (25), R is the actual number of coded bits; α old is the α value before updating; α new is the updated α value; λ real is the λ value actually used in encoding; λ theory is the theoretical value of λ obtained by the R-λ model; δ is the disturbance parameter, and δ∈(0,1); In formula (26), R is the actual number of coded bits; β old is the β value before updating; β new is the updated β value; λ real is the λ value actually used in encoding; λ theory is the theoretical value of λ obtained by the R-λ model; δ is the disturbance parameter, and δ∈(0,1).

9. A system based on the three-dimensional wavelet scalable video rate control method according to any one of claims 1 to 8, comprising an encoder, a bit stream extractor, and a decoder connected in sequence, characterized in that: At the encoder end, the original video is first analyzed by the MCTF analysis module, and the time domain subband weight is obtained according to the obtained time domain subband, and the adaptive λ selection module is performed; the time domain subband is formed into a three-dimensional wavelet time-space subband after passing through the space domain analysis module, and then the bit stream information is organized into a fully embeddable coding bit stream through the quantization module, the 3D-EZBC encoding module and the motion vector lossless encoding module; The bit stream extractor processes the received fully embeddable coding bit stream according to the process of "building R-λ model→bit allocation→bit rate control" to obtain a scalable coding bit stream; The decoder decodes the received scalable coded bitstream to generate a reconstructed video.

10. The system of the three-dimensional wavelet scalable video bit rate control method according to claim 9, characterized in that: The proportion of unconnected pixels r u Adaptively adjust the MCTF decomposition level, specifically: Definition m is the proportion of multi-connected pixels, r u is the proportion of unconnected pixels, r c is the proportion of connected pixels; m ≥0, r u ≥0, r c ≥0, and r m +r c +r u =1; where r u The definition is as follows: r u =Uncon_num / Pixel_num (3) In formula (3), Uncon_num is the number of unconnected pixels; Pixel_num is the number of all pixels in each frame; When u When the given threshold T=0.5 is exceeded, the MCTF process is terminated.

Images