Scale-spatial stream and hierarchical hyper-prior-based end-to-end video coding method research

Abstract

The invention discloses an end-to-end video coding method based on a scale-spatial stream and hierarchical hyper-prior, and the method employs a hierarchical hyper-prior coding and decoding network to introduce hyper-prior information to assist the generation of entropy model parameters, thereby enabling an entropy model to obtain more accurate estimation probability distribution, and improving the compression efficiency. And meanwhile, the motion information of the video is better estimated in combination with a scale-spatial stream, and the video coding performance is improved by estimating a more accurate entropy model and motion information. Experimental results show that when SSIM and MS-SSIM are respectively adopted as evaluation standards, compared with an HEVC (High Efficiency Video Coding) standard, the coding method provided by the invention has the advantage that the average code rates are respectively saved by 16.40% and 15.63% under the same distortion condition for video sequences with different resolutions. Compared with a traditional video compression method, the quality of the compressed low-resolution video can be effectively improved, and a good visual effect can be obtained through the method provided by the invention.

Description

Technical Field

[0001] The present invention will give full play to the advantages of deep learning methods, use the hierarchical hyper-prior codec network to introduce hyper-prior information to assist in the generation of entropy model parameters, so that the entropy model can obtain a more accurate estimated probability distribution and improve compression efficiency. At the same time, the scale-space flow is combined to better estimate the motion information of the video, and an end-to-end video coding method based on scale-space flow and hierarchical hyper-prior is proposed, which improves the video coding performance by estimating more accurate entropy models and motion information. Background Art

[0002] Learning-based methods have shown excellent performance in processing various tasks. Combining deep learning methods with video encoding and decoding, by introducing external information in the network training process, can effectively reduce the uncertainty of the source and reduce the source entropy, thereby further improving the video encoding performance. After years of development, although the traditional video coding method has achieved good coding performance, it has become increasingly difficult to improve it due to the increasing complexity of its coding framework. Although the end-to-end video coding method based on learning started late, due to the powerful performance shown by deep learning, the end-to-end video coding method based on learning has made certain progress. The present invention uses a residual coding framework to introduce deep learning technology into the video encoding and decoding process, replaces each module in the traditional video coding framework with a neural network, and further improves the video coding performance by training it end-to-end.

[0003] In the learning-based video coding method, a common approach is to obtain motion information based on optical flow estimation, and use bilinear warping for motion compensation to obtain the prediction of the current frame, then subtract it from the current frame to obtain the residual, and finally encode and decode the residual and sum it with the predicted value to obtain the reconstructed frame. However, the aperture problem often occurs in the process of optical flow estimation. If the motion of the stripes is observed from the hole, the motion of the three groups of stripes is consistent, all moving from the lower right to the upper left. However, the actual motion of the three groups of stripes is from the lower right to the upper left, from bottom to top, and from right to left. This shows that the true motion of the stripes cannot be known only through the hole, that is, the overall motion trajectory of the object cannot be known by observing the local part of the object. In optical flow estimation, the estimated object is a single pixel. Due to the existence of the aperture problem, it is often impossible to accurately estimate the actual motion of the object, which will lead to deviations in optical flow estimation. In addition, for dense optical flow estimation, each pixel must be estimated and warped, which will cause large errors in the occluded area, ultimately affecting the coding performance. This section expands the optical flow estimation by adding a scale flow on the basis of the optical flow (spatial flow). By introducing the scale-spatial flow and combining it with the scale-spatial flow to expand the bilinear warping into the scale-spatial warping, the model can blur the original video frame and then estimate it in places where the motion trajectory is difficult to estimate, thereby reducing the estimation error.

[0004] In the learning-based video coding method, the entropy model based on the fully decomposed probability distribution cannot capture the statistical dependence between the hidden layer feature components, resulting in its inability to fit the actual probability distribution well. The hierarchical hyper-prior probability distribution model improves the entropy model by introducing hyper-prior information, thereby obtaining a more accurate estimated probability distribution and improving video coding performance. In addition, due to the existence of the aperture problem, the traditional optical flow estimation cannot accurately estimate the motion trajectory of objects in some areas, while the scale-space flow estimation will blur the original video frame before estimating it in places where the motion trajectory is difficult to estimate, thereby reducing the estimation error. Combining the advantages of the above methods, this section will introduce an end-to-end video coding method based on scale-space flow and hierarchical hyper-prior. Summary of the invention

[0005] The present invention will give full play to the advantages of deep learning methods, use the hierarchical hyper-prior codec network to introduce hyper-prior information to assist in the generation of entropy model parameters, so that the entropy model can obtain a more accurate estimated probability distribution and improve compression efficiency. At the same time, the scale-space flow is combined to better estimate the motion information of the video, and an end-to-end video coding method based on scale-space flow and hierarchical hyper-prior is proposed. By estimating more accurate entropy models and motion information, the video coding performance is improved. The method mainly includes three steps, namely:

[0006] Step 1: The video codec network based on the hierarchical super prior entropy model consists of a main codec network and a hierarchical super prior codec network. The main codec network first encodes the input frame through an automatic encoder to obtain its hidden features. The hidden features are quantized and then entropy encoded into a bitstream for transmission. At the decoding end, the received bitstream is first entropy decoded to obtain the quantized hidden features, and then an automatic decoder is used to obtain the reconstructed frame. The entropy encoding process needs to rely on the estimated prior probability distribution of the quantized hidden features obtained by the hierarchical super prior codec network, that is, the entropy model.

[0007] Step 2: This section expands the optical flow estimation by adding a scale flow on the basis of the optical flow (spatial flow). By introducing the scale-spatial flow and combining the scale-spatial flow to expand the bilinear warping into the scale-spatial warping, the model can blur the original video frame before estimating the motion trajectory in places where it is difficult to estimate, thereby reducing the estimation error.

[0008] Step 3: The entropy model based on the fully decomposed probability distribution cannot capture the statistical dependence between the hidden layer feature components, resulting in its inability to fit the actual probability distribution well. The hierarchical hyper-prior probability distribution model improves the entropy model by introducing hyper-prior information, thereby obtaining a more accurate estimated probability distribution and improving video coding performance. In addition, due to the existence of the aperture problem, the traditional optical flow estimation cannot accurately estimate the motion trajectory of objects in some areas, while the scale-space flow estimation will blur the original video frame before estimating it in places where the motion trajectory is difficult to estimate, thereby reducing the estimation error.

[0009] The specific process is as follows:

[0010] (1) The end-to-end video coding method based on scale-space flow and hierarchical hyper-prior designed by the present invention is composed of a full decomposition probability distribution entropy model and a video coding and decoding network based on a hierarchical hyper-prior entropy model. Figure 1 ,2,3. The variational autoencoder model based on full decomposition probability distribution of the present invention is as follows Figure 1 As shown in the figure, the input video frame x is first analyzed and transformed by the inference module to obtain the hidden layer feature y, where φ g To analyze the transformation parameters, uniform noise U is then added to y to approximate the quantization process to obtain On the decoding side, The reconstructed video frame is obtained by generating a module In the test phase, the process of adding uniform noise to the hidden features y is switched back to the quantization Q and entropy encoding / decoding process. In the variational autoencoder model, in order to estimate the hidden features The true posterior probability Usually a parametric variational density function is used To fit the true posterior probability The goal of the inference module is to minimize and The mathematical expectation of the KL divergence is used to obtain the most accurate estimated probability distribution, thereby improving the rate-distortion performance of video coding. and The mathematical expectation of the KL divergence can be expressed as:

[0011]

[0012] The first term in formula (1) represents the process of analyzing and transforming the input video frame and adding uniform noise to it in the inference module:

[0013]

[0014] Among them, g a represents the analytical transformation process, φ g Represents the analysis transformation parameter, U represents the y i The uniform distribution is centered on . Since the width of the uniform distribution is always equal to 1, The probability density function of is always 1, and the result of taking its logarithm is always 0, so the calculation and The mathematical expectation of the KL divergence is that the first term of formula (1) is zero and can be ignored.

[0015] Assuming that the probability distribution in the second term of formula (1) follows a Gaussian distribution, then The distribution of x conditional on can be expressed as:

[0016]

[0017] Among them, g s represents the synthesis transformation process, θ g represents the synthesis transformation parameters, is the reconstructed video frame, x is the original video frame. The result of taking its logarithm is the original video frame x and the reconstructed video frame Therefore, the second term in formula (1) can be regarded as the distortion term in the rate-distortion function, and minimizing this term is equivalent to minimizing the distortion between the original video frame and the reconstructed video frame.

[0018] The probability distribution in the third term of formula (1) is the estimated probability distribution of the hidden layer features, that is, the entropy model. Therefore, the third term of formula (1) is the hidden layer features. The marginal density function and estimated probability distribution The cross entropy between them is the encoding hidden layer feature Therefore, the third term in formula (1) can be regarded as the code rate term in the rate-distortion function. Minimizing this term is equivalent to obtaining the hidden layer feature A more accurate estimated probability distribution of makes it more consistent with its true marginal distribution, thereby minimizing the number of bits required to encode the video frame. Given by formula (4):

[0019]

[0020] In the fully decomposable probability distribution model, the entropy model is modeled as a fully decomposable probability density function, which can be expressed as:

[0021]

[0022] Among them, ψ (i) Represents a univariate distribution The distribution of the entropy model is determined by the parameter ψ (i) * represents convolution, which convolves each univariate probability density function with a standard uniform density function, thereby adding uniform noise to the hidden feature y, which can make the estimated probability distribution of the hidden feature better fit its true marginal distribution. The joint full decomposition probability density function obtained from the univariate probability density function is the final estimated probability distribution.

[0023] Variational autoencoder models based on hierarchical hyper-prior probability distributions, such as Figure 2 As shown, in the inference module, the input video frame x undergoes the analysis transformation g a Get the hidden layer feature y, where φ g Represents the analytical transformation parameters of x, and then adds uniform noise U to y to approximate the quantization process to obtain At the same time, the hidden layer feature y is transformed by analysis a ′ gets the low-level super prior z1, and then z1 is obtained by adding uniform noise U On the other hand, after analyzing the transformation h a ″ and add uniform noise to obtain high-level hyper-prior The analytical transformation parameters of y and z1 are respectively given by φ h ′ and φ h In the generation module, by using high-level hyper-prior information Perform synthetic transformation h s To estimate Entropy model parameters Then guide the lower-level hyper-prior The encoding process is then used to encode the low-level hyper-prior Perform synthetic transformation h s ' to estimate Entropy model parameters To guide The entropy coding process, the final hidden layer feature By synthesizing the transformation g s Get the reconstructed video frame in and The synthetic transformation parameters are respectively given by θ h ′、θ h ″ and θ g In the test phase, the process of adding uniform noise to the hidden layer features y and the hierarchical super-prior information z1 and z2 is replaced by the quantization Q and entropy encoding / decoding process.

[0024] The true posterior probability and the fitted true posterior probability Taking the minimum value of the mathematical expectation of the KL divergence between can obtain the most accurate estimated probability distribution of the hidden layer features, thereby improving the rate-distortion performance of video coding. For the hierarchical hyper-prior probability distribution model, the hidden layer features The true posterior probability and the fitted true posterior probability The mathematical expectation of the KL divergence between can be expressed as:

[0025]

[0026] The first term in formula (6) represents the process of analyzing and transforming the input video frame and adding uniform noise to it in the inference module:

[0027]

[0028] Among them, the input video frame x is analyzed and transformed g a Get the hidden layer feature y, and then analyze and transform the hidden layer feature y a Get the super prior information z, where φ g and φ h represents the transformation parameter, and U represents the uniform distribution. Similar to the full decomposition probability distribution model, the first term in formula (6) is always zero after taking the logarithm and can be ignored.

[0029] The second term in formula (6) follows formula (3). From the analysis in the previous section, it can be seen that this term can be regarded as the distortion term in the rate-distortion function. The second term in the mathematical expectation of minimizing the KL divergence can be equivalent to minimizing the distortion between the original video frame and the reconstructed video frame.

[0030] The third term in formula (6) represents the estimated probability distribution of the hidden layer features, that is, the entropy model. The entropy model after introducing hierarchical hyper-prior information can be expressed as:

[0031]

[0032] In the formula, each hidden feature component is modeled as a Gaussian distribution with a mean of 0 and a standard deviation of σ. Each Gaussian distribution is then convolved with a standard uniform distribution to approximate the quantization process. Finally, the joint probability density function of all components is calculated to obtain the estimated probability distribution of the hidden feature. Super prior information After synthetic transformation h s Get, θ h is the transformation parameter.

[0033] The last two terms in formula (6) represent the estimated probability distribution of the super prior information. For the super prior information z, since the low-level super prior information can be obtained under the guidance of the high-level super prior information, the estimated probability distribution of the low-level super prior is modeled as a distribution similar to formula (8), which can be expressed as:

[0034]

[0035] in, represents low-level hyper-prior information, Represents high-level hyper-prior information.

[0036] For high-level hyper-prior information No new priors are introduced, and its probability distribution is modeled as a fully decomposable density function, which can be expressed as:

[0037]

[0038] in, Represents high-level super-prior information The estimated probability distribution of each single component of (i) Represents a single component distribution , convolve each single component distribution with the standard uniform distribution to approximate the quantization process, and finally calculate the joint probability density function of all components to obtain the estimated probability distribution of the high-level hyper-prior.

[0039] In formula (6), the second term is the same as the corresponding term in the full decomposition probability distribution model and can be regarded as the distortion term of the rate-distortion function. The last three terms in formula (6) represent the estimated probability distribution of the hidden features and their corresponding super-prior information. Minimizing them is equivalent to obtaining the most accurate estimated probability distribution of the hidden features, and then the minimum cross entropy of the hidden features can be obtained. In the process of entropy coding, the hidden features can be encoded with a smaller number of bits. Therefore, the last three terms in the formula can be regarded as the bit rate terms in the rate-distortion function. The video encoding and decoding network based on the hierarchical super-prior entropy model is as follows: Figure 3As shown in the figure, the parameter transformation process in the reasoning module and the generation module is replaced by the corresponding neural network module. The input video frame x first passes through the main coding network to obtain the hidden layer feature y. The main coding network has 4 layers. The combination of convolutional layer and Relu activation layer constitutes the first 3 layers of the network. The last layer of the network consists of a convolutional layer. Subsequently, the hidden layer feature y passes through the low-level super-prior coding network to obtain the low-level super-prior information z1. The low-level super-prior information z1 then passes through the high-level super-prior coding network to obtain the high-level super-prior information z2. The super-prior coding network is composed of convolutional layers and Relu layers alternately. After quantization and entropy coding / decoding, the super-prior information z2 is passed into the high-level super-prior decoding network to estimate the parameters of the entropy model of the low-level super-prior information z1, thereby guiding the entropy coding process of the low-level super-prior information z1. After the low-level super prior information z1 is quantized, it is entropy encoded / decoded according to the probability distribution estimated by the high-level super prior information z2. Then, the entropy model parameters of the hidden feature y are estimated through the low-level super prior decoding network. The super prior decoding network is composed of deconvolution layers and Relu layers alternately. After the hidden feature y is quantized, it is entropy encoded / decoded under the guidance of the estimated entropy model parameters. Finally, the reconstructed video frame is obtained through the main decoding network. The main decoding network consists of 4 layers. Each of the first 3 layers consists of a deconvolution layer + Relu layer, and the last layer consists of a deconvolution layer. The relevant configuration of the hierarchical super prior entropy model encoding and decoding network is shown in Table 1.

[0040] (2) The scale-spatial stream encoding and decoding network designed by the present invention is constructed. Figure 4 As shown, for a video frame x with a width of W and a height of H, respectively, f = (f x ,f y ) represents the estimated optical flow, then the process of bilinear warping of x based on the optical flow f can be expressed as:

[0041] x′[x,y]=x[x+f x [x,y],y+f y [x,y]] (11)

[0042] Among them, x[x,y] means sampling the video frame x at the coordinate (x,y) using the bilinear interpolation method, and f x [x,y] and f y [x,y] represents the estimated optical flow in the x-direction and y-direction at the coordinate (x,y), respectively.

[0043] For the scale-space stream, firstly, an image block X containing x and its blurred images of different degrees is constructed based on the original video frame x, which can be expressed as:

[0044] X=[x,x*G(σ0),x*G(2σ0),···,x*G(2 M-1 σ0)] (12)

[0045] Wherein, G(σ) represents a Gaussian distribution with a standard deviation of σ, * represents the process of convolution, and x*G(σ) represents Gaussian blurring of the original video frame x. X represents a set of images gradually blurred on the basis of the original video frame x, and its dimension is W×H×(M+1), and M in the present invention is 5.

[0046] The image block X adds a dimension to the single video frame x, and can be sampled at the coordinates (x, y, z) using trilinear interpolation. Correspondingly, the scale-space flow extends the scale flow based on the optical flow and can be expressed as g = (g x ,g y ,g z ), for a video frame x, the process of scale-space warping based on scale-space flow can be expressed as:

[0047] x′[x,y]=x[x+g x [x,y],y+g y [x,y],g z [x,y]] (13)

[0048] Among them, g x [x,y] and g y [x,y], respectively represents the estimated optical flow in the x direction and y direction at the coordinate (x,y), g z [x, y] represents the estimated scale flow in the image block X. The original video frame and the video frame to be estimated are used as the input of the network. The hidden layer features of the scale-spatial flow are obtained through the scale-spatial flow encoding network composed of convolutional layers and Relu layers. At the decoding end, the hidden layer features are sequentially passed through the scale-spatial flow decoding network composed of deconvolutional layers and Relu layers to obtain the scale-spatial flow g at the output end.

[0049] (3) The end-to-end video coding framework based on scale-spatial stream and hierarchical hyper-prior is shown in the attached figure. Figure 5 As shown in Figure 1, the network includes an I-frame codec module and an inter-frame codec module. The specific structures of the I-frame codec network, the scale-spatial stream codec network, and the residual codec network are given by the video codec network based on the hierarchical super prior entropy model in Section 2.2. In the I-frame codec module, the network uses an independent I-frame codec network to encode / decode the I frame to obtain the reconstructed frame of the I frame. In the inter-frame encoding and decoding module, the network first reconstructs the previous frame With the current input frame x i As the input of the scale-space stream encoding and decoding network, the scale-space stream g is obtained i , then the network is based on the scale-space flow g iThe reconstructed frame of the previous frame is warped using scale-space flow to perform motion compensation to obtain the predicted frame of the current frame. The network then takes the current input frame x i Predicted frame with current frame The residual r i The residual is encoded / decoded as the input of the residual codec network, and finally the predicted frame of the current frame is The decoded residual Sum to get the reconstructed frame of the current frame

[0050] In the I-frame encoding and decoding network, the I-frame input frame x0 passes through the main encoding network to obtain the hidden layer feature y0 of the I-frame, and then y0 is quantized and subjected to hierarchical hyper-prior entropy modeling to obtain the estimated entropy model parameters, thereby guiding the entropy encoding of y0. Then the decoded bitstream passes through the main decoding network to obtain the reconstructed frame of the I-frame. The encoding and decoding process of I frame can be expressed as:

[0051] y0=g a (x0,φ g ) (14)

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] Among them, formula (2-16) to formula (2-18) represent the main encoding / decoding process of the I frame encoding and decoding network, g a (.) represents the main encoding network, g s (.) represents the main decoding network, (φ g ,θ g ) represents the network parameters, Q(.) represents quantization. During the training phase, quantization is achieved by adding uniform noise. Formula (2-19) to Formula (2-21) represent the hierarchical hyper-prior entropy modeling process of the I-frame encoding and decoding network, h a (.) represents the super prior encoding network, h s (.) represents the super prior decoding network, (φ h ,θ h ) represents the network parameters, z0′ represents the low-level hyper-prior information, z0″ represents the high-level hyper-prior information, represents the estimated entropy parameter, The mean is 0 and the standard deviation is Gaussian distribution, * indicates convolution, Represents uniform distribution, and the estimated entropy model parameters are obtained based on the hyper-prior information Then we can get the estimated probability distribution of hidden layer features to guide its entropy coding process.

[0058] In the inter-frame encoding and decoding module, the scale-spatial stream encoding and decoding network is first used to jointly estimate and encode / decode the scale-spatial stream. The specific process can be expressed as:

[0059]

[0060]

[0061]

[0062] After jointly estimating and encoding the scale-space stream, its hidden layer feature y is obtained i , then entropy coding is performed under the guidance of the probability distribution estimated by the hierarchical super-prior codec network. After decoding, the coded bitstream is passed through the main decoding network to obtain the scale-space stream g i . The hidden layer features y of the scale-space flow i The process of hierarchical super-prior encoding and decoding is similar to that of I frame, which can be expressed as:

[0063]

[0064]

[0065] Among them, H a (.) represents the hierarchical hyper-prior encoding process, H s (.) represents the hierarchical hyper-prior decoding process, z g Hidden layer features y representing scale-space flow i The hierarchical hyper-prior information of the scale-space stream is i The entropy model expressed by formula (2-21) can be used to obtain its estimated probability distribution.

[0066] In the scale-spatial stream codec network, the scale-spatial stream g of the current video frame and the previous reconstructed video frame is obtained. i Then, based on the scale-space flow g i The predicted frame of the current frame is obtained by warping the previous reconstructed frame, which can be expressed as:

[0067]

[0068] in, represents the reconstructed frame of the previous frame, represents the predicted frame of the current frame, and w(.) represents the warping function. By warping the reconstructed video frame based on the scale-space flow, the areas where the motion is difficult to grasp can be blurred, thereby reducing the estimation error and improving the quality of the predicted frame.

[0069] Finally, the network converts the current video frame x i With the current predicted video frame The residual r i The residual codec network is used to encode and transmit the data, and the decoded residual With the current predicted video frame Get the reconstructed video frame of the current frame The specific process of residual encoding and decoding is similar to the above-mentioned I frame encoding and decoding and scale-spatial stream encoding and decoding process, which will not be repeated here. The residual acquisition process and the reconstruction process of the current reconstructed frame can be expressed as:

[0070]

[0071]

[0072] Among them, R(.) represents the residual acquisition process, C(.) represents the reconstruction process of the video frame, Represents the reconstructed frame of the current frame.

[0073] The advantages and beneficial technical effects of the present invention compared with the prior art are as follows:

[0074] (1) The video encoding and decoding network based on the hierarchical hyper-prior entropy model introduces hierarchical hyper-prior information to improve the entropy model in the video encoding process. The hierarchical hyper-prior information is used to assist the generation of entropy model parameters, so that the entropy model can obtain a more accurate estimated probability distribution of hidden layer features and improve compression efficiency.

[0075] (2) The scale-spatial stream codec network adds a scale stream based on the optical flow, which can better estimate the motion information of the video and thus improve the video coding performance.

[0076] (3) The learning-based end-to-end video coding method of the present invention can effectively improve the coding performance. Compared with the HEVC standard, the proposed video coding method reduces the BDBR by an average of 6.14% under the PSNR evaluation index, and reduces the BDBR by an average of 16.4% and 15.63% under the SSIM and MS-SSIM evaluation indicators, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A network framework for an end-to-end video coding method based on scale-spatial flow and hierarchical hyper-prior.

[0078] Figure 2It is a variational autoencoder model based on fully decomposed probability distribution.

[0079] Figure 3 It is a variational autoencoder model based on hierarchical hyper-prior probability distribution.

[0080] Figure 4 A video encoding and decoding network based on a hierarchical hyper-prior entropy model.

[0081] Figure 5 It is the scale-space stream encoding and decoding network.

[0082] Figure 6 Rate-distortion comparison of the HEVC standard, the method proposed by LuG et al., and the proposed method. DETAILED DESCRIPTION

[0083] The present invention is further described in detail below in conjunction with the embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical personnel familiar with the relevant field may make some non-essential improvements and adjustments to the present invention according to the above invention content for specific implementation, which should still fall within the scope of protection of the present invention.

[0084] (1) In this test environment, the HEVC standard test sequence is encoded / decoded by the inter-frame coding mode of the HEVC open source video encoder X265 and the network model proposed in the present invention, and the corresponding bit rate and distortion under different evaluation indicators are recorded and the encoding results are compared. The X265 version used in the experiment is 3.5, and the test sequences used include four different resolutions of 416×240, 832×480, 1280×720 and 1920×1080. The experimental environment used in the present invention is configured with a CPU of Intel(R) Xeon(R) Platinum 8255C CPU@2.50GHz and a GPU of Nvidia GeForce RTX 3090;

[0085] (2) When comparing the rate-distortion performance of different coding methods, the present invention uses Peak Signal to Noise Ratio (PSNR)

[66] , Structural Similarity Index (SSIM)

[67] and Multi-scale SSIM Index (MS-SSIM)

[68] Evaluate the quality of encoded video and use objective evaluation criteria Delta Bit Rate(BDBR)

[69] To measure the bit rate saving of the proposed coding method under different evaluation indicators.

[0086] (3) The invention first compares the proposed method with the HEVC standard based on the objective evaluation index PSNR. Table 2.2 compares the PSNR of the proposed method with the HEVC standard at different bit rates and the BDBR results of different video sequences based on the PSNR index.

[0087] (4) The encoded objects are downsampled standard HEVC test videos, whose names and resolutions are: BasketballPass(416×240), BlowingBubbles(416×240), BQSquare(416×240), RaceHorses(416×240), BasketballDrill(832×480), BQMall(832×480), PartyScene(832×480), FourPeople(1280×720), Johnny(1280×720), KristenAndSara(1280×720), ParkScene(1920×1080), Kimono1(1920×1080).

[0088] The comparison method of the present invention is:

[0089] Method 1: The method proposed by Lu G et al., reference "Lu G, Ouyang W, Xu D, et al. DVC: An end-to-end deep video compression framework [C]. IEEE Conference on Computer Vision and Pattern Recognition. 2019: 10998-11007."

[0090] Method 2: HEVC standard video sequence.

[0091] Table 1 PSNR results of encoding video sequences with different resolutions using the HEVC standard and the method of the present invention

[0092]

[0093]

[0094] Table 2 SSIM / MS-SSIM results of video sequences with different resolutions encoded using the HEVC standard and the method of the present invention

[0095]

[0096]

[0097] Table 3 Comparison of encoding results of HEVC standard, document

[14] and the video encoding method of the present invention

[0098]

Claims

1. The overall framework of the end-to-end video coding method based on scale-spatial flow and hierarchical hyper-prior is characterized by The following steps are involved: Step 1: The parameter transformation process in the inference module and the generation module is replaced by the corresponding neural network module. The input video frame first passes through the main coding network to obtain the hidden layer features. The main coding network has 4 layers. The combination of convolutional layer and Relu activation layer constitutes the first 3 layers of the network. The last layer of the network consists of a convolutional layer. Subsequently, the hidden layer features pass through the low-level super-prior coding network to obtain the low-level super-prior information. The low-level super-prior information passes through the high-level super-prior coding network to obtain the high-level super-prior information. The super-prior coding network is composed of convolutional layers and Relu layers alternately. The high-level super-prior information is quantized and entropy encoded / decoded and then passed to the high-level super-prior decoding network to estimate The parameters of the entropy model of the low-level super-prior information are calculated, thereby guiding the entropy coding process of the low-level super-prior information. The low-level super-prior information is quantized and then entropy encoded / decoded according to the probability distribution estimated by the high-level super-prior information. Then, the entropy model parameters of the hidden layer features are estimated through the low-level super-prior decoding network. The super-prior decoding network is composed of deconvolution layers and Relu layers alternately. The hidden layer features are quantized and entropy encoded / decoded under the guidance of the estimated entropy model parameters. Finally, the reconstructed video frame is obtained through the main decoding network. The main decoding network consists of 4 layers, each of the first 3 layers consists of a deconvolution layer + a Relu layer, and the last layer consists of a deconvolution layer. Step 2: For the video frame, firstly, a series of Gaussian blurs of different degrees are performed on it to construct an image block containing the original image and blurred images of different degrees. Then, in the scale-spatial flow estimation, the scale flow is estimated according to the degree of change of the optical flow in different areas of the image, and the corresponding blurred image is selected. Finally, the video frame is warped according to the estimated optical flow in the blurred image to perform motion compensation on the video frame. The original video frame and the video frame to be estimated are used as the input of the network together. The hidden layer features of the scale-spatial flow are obtained through the scale-spatial flow encoding network composed of convolutional layers and Relu layers. At the decoding end, the hidden layer features are sequentially passed through the scale-spatial flow decoding network composed of deconvolutional layers and Relu layers to obtain the scale-spatial flow at the output end. The scale-spatial flow network realizes the joint estimation and encoding of the scale-spatial flow, so that the reconstructed frame of the previous frame can be warped based on the scale-spatial flow to achieve better predicted frame quality; Step 3: The network includes an I-frame encoding and decoding module and an inter-frame encoding and decoding module. In the I-frame encoding and decoding module, the network uses an independent I-frame encoding and decoding network to encode / decode the I-frame to obtain the reconstructed frame of the I-frame. In the inter-frame encoding and decoding module, the network first uses the reconstructed frame of the previous frame and the current input frame as the input of the scale-spatial stream encoding and decoding network to obtain the scale-spatial stream. Then the network uses scale-spatial stream warping to perform motion compensation on the reconstructed frame of the previous frame based on the scale-spatial stream to obtain the predicted frame of the current frame. Then the network uses the residual obtained by the current input frame and the predicted frame of the current frame as the input of the residual encoding and decoding network to encode / decode the residual. Finally, the predicted frame of the current frame is summed with the decoded residual to obtain the reconstructed frame of the current frame.

Images