A VVC inter-frame fast coding method based on reference blocks
Through the VVC inter-frame fast encoding method based on reference blocks, the division probability network model is used to predict the division probability of the encoding unit, and the problems of low inter-frame encoding efficiency and high computational complexity in the prior art are solved, and more efficient video encoding is achieved .
Patent Information
- Application Number
- CN202310299186.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-24
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-03-24
AI Technical Summary
In the existing VVC video encoding technology, inter-frame coding efficiency is low and computational complexity is high. Deep learning methods cannot effectively learn inter-frame motion information, and the model lacks transferability.
Using the VVC inter-frame fast encoding method based on reference blocks, by constructing a division probability network model, using deep learning to predict the division probability of the encoding unit, select the optimal division mode, and reduce the encoding time complexity.
It improves inter-frame encoding efficiency, reduces the complexity of video encoding time, and solves the problem that traditional deep learning cannot effectively learn inter-frame encoding mode information.
Smart Images

Figure CN116489386B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of VVC video coding, and in particular relates to a VVC inter-frame fast coding method based on reference blocks. Background Art
[0002] With the rapid development of digitalization and network technology, the application fields of video are constantly expanding, such as video communication, video surveillance and video entertainment, but the high bandwidth and large storage resources required for video transmission have become the main constraints. In order to improve the efficiency and quality of video coding, it is necessary to continuously research and develop new video coding standards to serve the needs of a large number of video applications in a more efficient, stable and high-quality way. At present, the research on reducing the coding complexity of the Versital Video Coding (VVC) standard mainly includes three aspects: 1. Traditional fast coding algorithms based on statistics; 2. Fast algorithms based on machine learning; 3. Deep learning methods that use neural networks to automatically learn and reason.
[0003] Although the above existing methods reduce the coding complexity and improve the coding efficiency, there are still many shortcomings, such as: 1. The fast CU coding method based on statistical analysis can improve the computational efficiency, but there is an incomplete analysis problem, such as the relationship between the motion degree and the division of the inter-frame coding CU; 2. The machine learning method requires a large amount of training data to effectively infer. The trained model is only for specific problems or applications, and is not transferable. The training and reasoning of machine learning usually require a lot of calculations and reasoning, which greatly increases the coding time and coding complexity; 3. The texture information of the inter-frame coding or coded reconstruction image is learned by the deep learning method, which improves the efficiency of image learning, but lacks research on inter-frame coding, and the use of deep learning only uses CU texture, rate distortion cost and data label as training data, which cannot fully learn the motion information between frames. Therefore, how to improve the efficiency of inter-frame coding and reduce the computational complexity of inter-frame coding is a problem that needs to be solved urgently. Summary of the invention
[0004] In order to solve the above problems in the prior art, the present invention proposes a VVC inter-frame fast coding method based on reference blocks, which includes:
[0005] S1: Obtain current coding unit information, where the coding unit information includes depth and mode information of the current coding unit;
[0006] S2: Determine whether to perform network prediction division according to the current coding information. If network prediction division is performed, execute step S3. If network prediction division is not performed, execute the original division process of the encoder.
[0007] S3: construct a partition probability network model and train the partition probability network model;
[0008] S4: input the current coding unit into the trained partition probability network model to obtain a partition probability prediction value of the current coding unit;
[0009] S5: Select a prediction mode according to the partition probability prediction value, and determine whether the prediction mode is a partition mode. If it is a partition mode, return to S4; if it is not a partition mode, execute S6;
[0010] S6: Use the current coding unit CU to perform MergeSkip mode encoding on the video to be encoded, and calculate the RDCost value of the MergeSkip mode encoding;
[0011] S7: Determine whether to adopt the Skip mode to skip coding prediction according to the RDCost value, otherwise traverse all default candidate prediction modes.
[0012] Preferably, the process of training the partition probability network model includes: the partition probability network model is composed of a block convolution layer, a conditional residual module, a sub-network structure and a fully connected layer; the process of training the partition probability network model includes:
[0013] Step 1: Get all coding units, filter the CU types, and aggregate the filtered CU types to obtain a training data set;
[0014] Step 2: Optimize the redundant CU types in the training data set;
[0015] Step 3: Input the optimized training data set into the block convolution layer of the partitioned probability network model to learn the different features of the original data, reference data, and co-located data;
[0016] Step 4: Input the learned features into the conditional residual module for deep feature extraction;
[0017] Step 5: Input the extracted deep features into the sub-network structure to obtain the division results of different data;
[0018] Step 6: Input the segmentation result into the fully connected layer to obtain the segmentation probability prediction value;
[0019] Step 7: Calculate the loss function of the model based on the predicted value of the partition probability, and continuously adjust the parameters of the model. When the loss function is minimized, the model training is completed.
[0020] Preferably, the process of determining the partitioning mode according to the partitioning probability prediction value includes: inputting the partitioning probability prediction value into the softmax function, converting the predicted value of each category into a probability value between 0 and 1, wherein the maximum probability value is compared with the set threshold interval [τ1, τ2], when the maximum probability value is less than τ1 in the threshold interval, the encoder default partitioning mode is used to divide the CU, when the maximum probability value is greater than τ2 in the threshold interval, the predicted coding mode is used to divide the CU, otherwise the optimization mode is used to divide the CU.
[0021] Furthermore, the process of dividing the CU includes:
[0022] The CU is divided by the encoder default division mode, including: the encoder divides a CU by default, wherein the maximum number of divisions is 5, including quadrifugal division QT, vertical binary division BV, horizontal binary division BH, vertical trifugal division TV, and horizontal trifugal division TH;
[0023] Using the predicted coding mode to divide the CU includes: clearing the division mode stack; dividing the data according to the network predicted division mode, where the network predicted division mode includes no division NS, quad division QT, horizontal division HS, and vertical division VS;
[0024] The process of dividing the CU using the optimization mode includes: clearing the division mode stack, determining the current division mode, if the current division mode is horizontal division HS, then performing horizontal binary division and horizontal trifurcation division on the divided CU again, if the current division mode is vertical division VS, then performing vertical binary division and vertical trifurcation division on the divided CU again, if the current division mode is neither horizontal division HS nor vertical division VS, then using the predicted coding mode to divide the CU.
[0025] Beneficial effects of the present invention:
[0026] The present invention uses reference blocks through deep learning according to the motion characteristics between frames to solve the problem that traditional deep learning cannot effectively learn inter-frame coding mode information and the prediction is inaccurate. The present invention uses a probabilistic network model to process the current coding unit information, predicts the division probability prediction value of the current coding unit, and selects the optimal division mode according to the current division probability prediction value, thereby reducing the time complexity of video encoding. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 1. It is a structural diagram of different CU types in an embodiment of the present invention;
[0028] Figure 2 Graph showing the degree of division under different time domain layers in an embodiment of the present invention;
[0029] Figure 3 A structural diagram of a partitioning probability network model in an embodiment of the present invention;
[0030] Figure 4 This is a CU partition prediction flow chart in an embodiment of the present invention;
[0031] Figure 5 The present invention is a flowchart of a VVC inter-frame fast coding method based on reference blocks in an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] A specific implementation of a VVC inter-frame fast coding method based on reference blocks, such as Figure 5 As shown, the method includes:
[0034] S1: Obtain current coding unit information, where the coding unit information includes depth and mode information of the current coding unit;
[0035] S2: Determine whether to perform network prediction division according to the current coding information. If network prediction division is performed, execute step S3. If network prediction division is not performed, execute the original division process of the encoder.
[0036] S3: construct a partition probability network model and train the partition probability network model;
[0037] S4: input the current coding unit into the trained partition probability network model to obtain a partition probability prediction value of the current coding unit;
[0038] S5: Select a prediction mode according to the partition probability prediction value, and determine whether the prediction mode is a partition mode. If it is a partition mode, return to S4; if it is not a partition mode, execute S6;
[0039] S6: Use the current coding unit CU to perform MergeSkip mode encoding on the video to be encoded, and calculate the RDCost value of the MergeSkip mode encoding;
[0040] S7: Determine whether to adopt the Skip mode to skip coding prediction according to the RDCost value, otherwise traverse all default candidate prediction modes.
[0041] In the division module of the Versital Video Coding (VVC) standard, the division method of the coding unit (CU) has been increased from the four-fork division of the previous generation video coding standard High Efficiency Video Coding (HEVC) to a multi-fork division method. There are up to six division methods, namely, no division (NS), four-fork division (QT), horizontal binary division (BH), vertical binary division (BV), horizontal trifurcated division (TH) and vertical trifurcated division (TV). The encoder determines which division method to use based on a recursive rate-distortion optimization (RDO) process, which achieves the best CU division method by weighing the compression ratio and image quality. This recursive process can be summarized as a top-down and bottom-up process. Specifically: a 128x128 standard brightness block is divided into six division methods. Each sub-block obtained by division will also have multiple corresponding division methods (determined by the aspect ratio and block size). The sub-blocks will also use the corresponding division method one by one until the sub-block reaches the minimum specified block size (such as 4x4) or is determined to be unable to be divided further during the division. At this time, the top-down process is completed. The subsequent process is a bottom-up process. Specifically, after no more division, the coding mode of the divided CU coding block will be predicted, and then the rate-distortion cost (RDCost) of the current CU coding will be obtained. The parent block determines the RDCost of the division method according to the sum of the RDCost of all the sub-block coding costs and the coding bit consumption generated by its corresponding division method. After the RDCost of each division method of the CU is calculated, the RDCost value with the lowest cost can be compared, and the corresponding division method is the optimal division method of the current parent block, and the coding method of all sub-blocks is also the optimal method (if the whole process is described as a tree, the optimal coding method only appears on the leaf nodes of the tree), and the parent block may have its corresponding parent block, which is compared with its brother block as a sub-block to obtain the optimal division method of the parent block, until no parent block reaches the top 128x128 block. It can be seen from the whole process that the depth and width of this recursion are quite large, and the cost of eliminating the local optimal division method increases with the decrease of the division depth.
[0042] In this embodiment, the acquired data set includes: Figure 1As shown in the figure, regarding the selection of CU type, various division methods make the shape of the coding block CU different. There are up to 28 different types of CU in the frame, and the minimum aspect ratio is 1 and the maximum is 16. Due to the limitation of its division, the distribution of the division methods of different CU types is also somewhat different. According to experimental statistics, there are a total of 28 distributions of different block sizes, as shown in the following table. Let the width and height be represented by 2^k, where k = {2, 3, 4, 5, 6, 7}, and the ratio of width and height is 2^τ, where τ
[0043] ={0,1,2,4,8,16}. When τ>=4 in the aspect ratio, the probability of its division mode being NS is 88.16% on average, and the average probability of CU division mode being NS for k<4 is 85.6. It can be seen that CUs with smaller division heights and larger aspect ratios tend to be undivided and the probability is very high. In the encoding process, the possible division mode is directly used to skip other division modes with lower probability, thus avoiding the recursive process of local optimal division, which can greatly save encoding time and generate less encoding cost.
[0044] Table 1 CU types under different aspect ratios
[0045] Aspect ratio τ CU Type 1 128x128, 64x64, 32x32, 16x16, 8x8, 4x4 2 128x64, 64x128, 64x32, 32x64, 32x16, 16x32, 8x16, 16x8, 8x4, 4x8 4 64x16, 16x64, 32x8, 8x32, 4x16, 16x4 8 64x8, 8x64, 32x4, 4x32 16 64x4, 4x64
[0046] Table 2: The proportion of different CU types
[0047]
[0048] According to the analysis of the above table, if accurate division prediction can be performed at the CU stage with low division depth and large width and height, the local optimal division process can be reduced. The present invention designs a convolutional neural network to predict the division method of inter-frame CU, which can effectively reduce the encoding time complexity while ensuring the average encoding bit rate unchanged. The selection basis for which type of CU division needs to be predicted is the CU with width and height in k>=4 and ratio τ<=4 or min(width, height)>=8 and τ={4,8}; except for the CU with τ!=1, after transposition, the CU with the same width and height can be found and regarded as the same category. Let the category be WxH, which means width is W and height is H, with a total of 9 categories (refer to Table 3). In actual coding division, the difference between horizontal division and vertical division is quite obvious. The division between trifurcated horizontal and binary horizontal or binary vertical and trifurcated vertical is quite similar. Therefore, the network structure classification categories designed in the present invention are converted from the traditional six divisions NS, QT, BH, BV, TH, and TV into the optimized four divisions NS, QT, HS, and VS, merging the binary and trifurcated division methods in the horizontal and vertical directions.
[0049] Table 3 CU categories of network input
[0050]
[0051] The input parameters of the network model of the present invention in the convolution layer are the pixel blocks of the current CU and in order to enable the model to learn and infer the CU motion characteristics between frames, the model input, in addition to the original pixel blocks and partition data labels of the CU, also adds the co-located blocks and corresponding reference blocks of the current CU in the reference frame. The matching degree of the reference block determines the accuracy of the division method to a certain extent. After the convolution layer is completed, some encoding parameters generated by the current CU in the encoding are appended to the fully connected layer by flattening, and further generalized learning is performed in the fully connected layer. These encoding parameters are composed of: partition depth parameter, quantization value QP, time domain layer Id, CU width and height, and RDCost of the current block in Merge mode. In order to obtain these training data, the data set uses the video sequence of the DVC data set, and the resolution categories include: 3840x2160, 1920x1080, 960x544, 480x272, 3840x2176, 832x480, 416x240. All sequences are encoded using the official VVC standard encoder VTM, version 10.2, and the quantization parameters QP = {22, 27, 32, 37} are used in the Random Access encoding mode configuration. The encoding information of the division process is saved during the encoding process, and finally the labels of different blocks and the corresponding training data are obtained. For the authenticity of the experiment, the video sequences selected for the training set and the validation set are different. A total of 100 video sequences are selected as the training set and 20 video sequences are selected as the validation set. Generate training sets and validation sets corresponding to different types of CUs, and then train the model offline.
[0052] In this embodiment, if Figure 3 As shown in the figure, the main network structure of the model consists of a block convolution layer, a conditional residual module, a sub-network structure and a fully connected layer. Different network models are trained for different types of CUs. Considering the complexity of the coding block, the block convolution layer is used to learn the different features of the original block, the reference block and the co-located block respectively. In order to avoid the problems of gradient explosion and gradient disappearance during the convolution process, the learned local features are fused after the block convolution. The local features of each block are further learned through the conditional residual module block to improve the network's expression ability. At the same time, corresponding sub-networks are designed for different CU types. The sub-networks designed according to the shape and size of the block improve the accuracy of the prediction. Finally, all neurons in the convolution layer are connected to all neurons in the current layer, thereby realizing the extraction of global features and classification of patterns. In order to better adapt to the different features of the data set, the activation function in the network adopts PreLU, which can adjust the slope of the negative input according to the learning parameters, so that the network can better handle different features in the data set. Its mathematical form is:
[0053] f(x)=max(0,x)+α·min(0,x)
[0054] Among them, α is a learnable parameter used to control the slope of negative inputs. When α = 0, PReLU is equivalent to the standard ReLU activation function.
[0055] In this embodiment, the process of training the partition probability network model includes: the partition probability network model is composed of a block convolution layer, a conditional residual module, a sub-network structure and a fully connected layer; the process of training the partition probability network model includes:
[0056] Step 1: Get all coding units, filter the CU types, and aggregate the filtered CU types to obtain a training data set;
[0057] Step 2: Optimize the redundant CU types in the training data set;
[0058] Step 3: Input the optimized training data set into the block convolution layer of the partitioned probability network model to learn the different features of the original data, reference data, and co-located data;
[0059] Step 4: Input the learned features into the conditional residual module for deep feature extraction;
[0060] Step 5: Input the extracted deep features into the sub-network structure to obtain the division results of different data;
[0061] Step 6: Input the segmentation result into the fully connected layer to obtain the segmentation probability prediction value;
[0062] Step 7: Calculate the loss function of the model based on the predicted value of the partition probability, and continuously adjust the parameters of the model. When the loss function is minimized, the model training is completed.
[0063] In this embodiment, the process of screening the CU type includes: obtaining all CU types, determining the width W, height H and width-to-height ratio τ of each type of CU; setting screening conditions according to the width W, height H and width-to-height ratio τ of the CU, and screening the CU type according to the screening conditions; wherein the screening conditions include that any side of the width and height of the CU is greater than 16, and when one side is 16 or 32, the other side cannot be 128; or min(W,H)=8 and τ={4,8} of the CU.
[0064] Optimizing redundant CU types in a data set includes: CU types in the data set include non-partition NS, quadrifugal partition QT, horizontal binary partition BH, vertical binary partition BV, horizontal trifugal partition TH and vertical trifugal partition TV; horizontal binary partition BH and horizontal trifugal partition TH are merged to obtain horizontal partition HS, and vertical binary partition BV and vertical trifugal partition TV are merged to obtain vertical partition VS; optimized partition types include NS, QT, HS and VS.
[0065] The process of the sub-network structure processing the input data includes: the input channel type of the sub-network structure is (BATCH_SIZE, 16, H, W), wherein the sub-network structure is composed of a convolutional layer and a fully connected layer, and the size and step size of the convolution kernel of the first convolutional network layer of each sub-network are (4*τ, 4*τ) and (4*τ, 4*τ), respectively, which are used to normalize the output feature shape; during the training of the sub-network, the network is trained by adding additional coding information; wherein the additional coding information includes: quantization parameter QP, partition depth Depth, time domain layer Id and RDCost generated by coding; specifically:
[0066] For the subnetwork with min(W,H)=8, it consists of two convolutional layers. The convolution kernel size and stride of the second convolutional layer are (2, 2) and (2, 2) respectively. The output dimension of the convolutional layer is (BATCH_SIZE, 32, 1, 1). The output data is flattened to obtain data with (BATCH_SIZE, 32) dimension.
[0067] For the subnetwork with min(W,H)=16, it consists of three convolutional layers. The convolution kernel size and step size of the second and third layers are (2,2) and (2,2). The output dimension of the convolutional layer is (BATCH_SIZE,64,1,1). The output data is flattened to obtain data with (BATCH_SIZE,64) dimension.
[0068] For the sub-network with min(W,H)=32, it consists of three convolutional layers. The convolution kernel size and step size of the second and third convolutional layers of the sub-network with min(W,H)=32 are (4, 4) and (4, 4) respectively; the dimension of the convolutional layer output of this sub-network is (BATCH_SIZE, 128, 1, 1); the output data is flattened to obtain data with (BATCH_SIZE, 128) dimension;
[0069] The subnetwork with min(W,H)=64 consists of three convolutional layers. The convolution kernel size and step size of the second and third convolutional layers of this subnetwork are (2, 2) and (2, 2) respectively. The dimension of the convolutional layer output of this subnetwork is (BATCH_SIZE, 32, 2, 2). The output data is flattened to obtain data of (BATCH_SIZE, 128) dimension.
[0070] The process of the fully connected layer processing the division results includes: the division results are (BATCH_SIZE, 32) dimensional data, (BATCH_SIZE, 64) dimensional data and (BATCH_SIZE, 128) dimensional data obtained according to different sub-networks; the corresponding encoding information is added to the obtained dimensional data and used as the input of the fully connected layer to obtain the division probability prediction value of the current CU type.
[0071] The loss function of the network model proposed in the present invention is defined as follows: for a CU with a width of w and a height of h, all possible partitions are M = {NO, QT, HS, VS}, and for a partition mode m∈M, it represents the label of the corresponding partition mode. Since the CU type in each min-batch during training is the same, assuming the size of the min-batch is N, its cross entropy loss function is:
[0072]
[0073] in represents the loss, y n,m represents the true label of CU, It represents the prediction label of CU, n represents the number of CUs in a min-batch, and the overall description is the loss of the partition mode m of the n-th CU partition prediction in a min-batch.
[0074] Since the proportion of different partition modes in CU is very unbalanced, the original cross entropy loss function cannot adapt to this situation. Therefore, the present invention proposes to impose different penalties on different partition modes. The improved loss function for the original cross entropy loss function formula is defined as:
[0075]
[0076] Where L represents the loss, p m Indicates the proportion of partition mode m in this type of CU, and satisfies the sum of the proportions of all partition modes ∑ m∈M p m =1, α∈[0,1] is a variable parameter used to adjust the penalty of the partitioning mode.
[0077] In this embodiment, it is necessary to predict the divided CU; specifically, since the prediction of the network may have a certain deviation, and this deviation needs to be manually corrected, the present invention sets a pair of range thresholds [τ1, τ2], according to the predicted value P m In the range of [τ1,τ2], different strategies are adopted: default mode, optimized mode, and trusted mode. The default mode means that the original candidate partitioning mode of the encoder is adopted; when the prediction result is a horizontal mode and the network prediction result is in the optimized mode, the current CU will adopt horizontal binary and horizontal trifurcation as candidate prediction modes. For the prediction result is a vertical mode and the network prediction result is in the optimized mode, the current CU will adopt vertical binary and vertical trifurcation as candidate prediction modes; for the trusted mode, it means that the accuracy of the current CU in the network prediction is high, and the partitioning mode of network prediction can be adopted. It can be summarized as:
[0078]
[0079] A specific implementation method of CU division is as follows: Figure 4 As shown, the method includes: inputting the current coding unit information into the partition prediction network to obtain the probability prediction value, inputting the partition probability prediction value into the softmax function, converting the predicted value of each category into a probability value between 0 and 1, wherein the maximum probability value is compared with the set threshold interval [τ1, τ2], when the maximum probability value is less than τ1 in the threshold interval, the encoder default partition mode is used to partition the CU, when the maximum probability value is greater than τ2 in the threshold interval, the predicted coding mode is used to partition the CU, otherwise the optimization mode is used to partition the CU. The degree of partition is as follows Figure 2 shown.
[0080] Optimized, the range of τ1 and τ2 selected in the encoder test process is [0.5, 0.75]; when the maximum probability p<0.5, the encoder default candidate mode is used to divide the CU; when p>0.75, the predicted coding mode is directly used to divide the CU, otherwise the optimized mode is used to divide the CU. Using the encoder default division mode to divide the CU includes: the encoder divides a CU by default, where the maximum number of divisions is 5, including four-fork division QT, vertical binary division BV, horizontal binary division BH, vertical trifurcated division TV and horizontal trifurcated division TH. Using the predicted coding mode to divide the CU includes: clearing the division mode stack; dividing the data according to the network predicted division mode, where the network predicted division mode includes no division NS, four-fork division QT, horizontal division HS, and vertical division VS. The process of dividing the CU using the optimization mode includes: clearing the division mode stack, determining the current division mode, if the current division mode is horizontal division HS, then performing horizontal binary division and horizontal trifurcation division on the divided CU again, if the current division mode is vertical division VS, then performing vertical binary division and vertical trifurcation division on the divided CU again, if the current division mode is neither horizontal division HS nor vertical division VS, then using the predicted coding mode to divide the CU.
[0081] The standard for which coding mode to use in VVC is to decide based on the minimum rate-distortion cost, and the process of obtaining this minimum rate-distortion value is called rate-distortion optimization. The size of the rate-distortion cost is related to the coding prediction distortion and the coding bit cost. Usually, in some coding modes, the image distortion is small, but the bit rate is large; some modes have large image distortion, but the bit rate is small. Therefore, the process of rate-distortion optimization is to optimize the current coding bit rate R. curr Less than the encoding rate R max The coding distortion D is smaller, and this process can be expressed as:
[0082] min{D}stR curr <R max
[0083] Among them, R max Indicates the maximum encoding rate, R curr represents the current encoding bit rate, and D represents the encoding distortion.
[0084] Let M k is the kth coding mode among the coding modes to be tried, D k (M k ) is the coding rate distortion of the k-th coding mode, R k (M k) is the coding bit spent in the kth coding mode, and the best coding mode that obtains the minimum coding RD rate distortion value among all coding modes can be expressed as follows:
[0085] Since each coding mode is independent of each other, M k The optimal solution can be described as:
[0086]
[0087] Here, λ represents the Lagrangian factor.
[0088] The kth prediction mode M among all candidate prediction modes k The RD rate distortion value J k is represented as follows:
[0089] J k =D k (M k )+λR k (M k )
[0090] After comparing the rate distortion in each mode, the RD rate distortion value of the optimal prediction mode can be expressed as follows:
[0091]
[0092] Since the SKIP mode accounts for a large proportion of all modes and the encoding mode is relatively simple, the required encoding time and bit rate are less than other complex encoding methods, while the encoding quality is guaranteed. Therefore, the prior probability can be obtained based on the rate distortion between different modes, and then the distribution range of the rate distortion value of the SKIP mode can be used to determine whether the current block can adopt the SKIP mode.
[0093] The probability value of whether the best mode of the current RDCost is the SKIP mode can be obtained by the Bayesian decision algorithm as follows:
[0094]
[0095] Among them, p(m s |c) indicates the probability value of whether the best mode of the current RDCost is the SKIP mode, m s Indicates that c is the rate distortion value RDCost, p(m s ) represents the prior probability of the SKIP mode, and p(c) represents the total probability of RDCost in all modes.
[0096] p(c)=p(c|m s )·p(m s )+p(c|m o)·p(m o )
[0097] Among them, p(c|m o ) represents the conditional probability of RDCost when the current best mode is other modes, p(m o ) represents the prior probability of other modes, m o Indicates other modes.
[0098] By combining the above two equations, we can get
[0099]
[0100] Where m s and m o Represent SKIP mode and other modes respectively, c is the rate distortion value RDCost, p(m s ) represents the prior probability of the SKIP mode, p(m o ) represents the prior probability of other modes, p(c|m s ) represents the conditional probability of RDCost when the current best mode is SKIP. p(c|m o ) represents the conditional probability of RDCost when the current best mode is other modes. p(c) represents the total probability of RDCost in all modes.
[0101] When p(m s |c) When the probability value is very large and close to 1, the RDCost value is usually small, which also indicates that the best mode of CU is SKIP mode at this time. As RDCost increases, p(m s |c) is getting smaller and smaller and approaches zero. At this time, the best mode of CU is more likely to be other coding modes. However, within a certain distribution interval, the best coding mode of RDCost may be other modes or SKIP mode. In order to determine the specific coding mode, by setting the rate distortion threshold λ, when c<λ, we can consider the current best coding mode to be m s , when c ≥ λ, the possibility of other coding modes is more consistent with the current coding mode.
[0102] In order to find the value of the threshold λ when the SKIP mode is the optimal mode, let p(λ|m s ) is the conditional probability of λ when using SKIP mode, p(λ|m o ) is the conditional probability of λ when using other modes, then:
[0103]
[0104]
[0105] where f(c|m s ) is the conditional probability density function of c when using the SKIP mode, f(c|m o ) is the conditional probability density function of c when using other modes. Combining the above two equations, we can get the following and equation () to get the conditional probability of the threshold λ.
[0106]
[0107] By adjusting p(m s |c), apply this algorithm to the encoding process of the encoder, count the bit rate consumption of each sequence, determine the value of λ based on the statistical results, and finally determine the value of λ to be 0.8 based on multiple experimental tests.
[0108] The network structure designed by the present invention is embedded in the VTM encoder to deploy the designed algorithm flow, and the CTC standard test sequence specified by the coding standard setter is used to select some sequences from category A to category E to test the algorithm effect. The experimental results show that the algorithm design proposed by the present invention has greatly improved the encoding time complexity when the bit rate fluctuation is small. The experimental results are as follows. DB represents the bit rate spent on encoding, PSNR represents the quality of the encoded video, and T represents time.
[0109] Table 4 Algorithm deployment experimental results
[0110]
[0111] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A VVC inter-frame fast coding method based on reference blocks, characterized in that: include: S1: Obtain current coding unit information, where the current coding unit information includes depth and mode information of the current coding unit; S2: Determine whether to perform network prediction division according to the current coding unit information. If network prediction division is performed, execute step S3. If network prediction division is not performed, execute the original division process of the encoder. S3: construct a partition probability network model and train the partition probability network model; The partition probability network model consists of a block convolution layer, a conditional residual module, a sub-network structure, and a fully connected layer. The process of training the partition probability network model includes: Step 1: Get all coding units, filter the CU types, and collect the filtered CU types to obtain a training data set; the process of filtering the CU types includes: getting all CU types, determining the width W, height H, and width-to-height ratio τ of each type of CU; setting filtering conditions according to the width W, height H, and width-to-height ratio τ of the CU, and filtering the CU types according to the filtering conditions; wherein the filtering conditions include that any side of the width and height of the CU is greater than 16, and when one side is 16 or 32, the other side cannot be 128; or min(W,H)=8 and τ={4,8} of the CU; Step 2: Optimize the CU types after screening in the training data set; the specific process includes: the CU types in the data set include non-division NS, quadrifugal division QT, horizontal binary division BH, vertical binary division BV, horizontal trifugal division TH and vertical trifugal division TV; merge the horizontal binary division BH and the horizontal trifugal division TH to obtain the horizontal division HS, merge the vertical binary division BV and the vertical trifugal division TV to obtain the vertical division VS; the optimized division types include NS, QT, HS and VS; Step 3: Input the optimized training data set into the block convolution layer of the partitioned probability network model to learn the different features of the original data, reference data, and co-located data; Step 4: Input the learned features into the conditional residual module for deep feature extraction; Step 5: Input the extracted deep features into the sub-network structure to obtain the division results of different data; The process of the sub-network structure processing the input data includes: the input channel type of the sub-network structure is (BATCH_SIZE, 16, H, W), wherein the sub-network structure is composed of a convolutional layer and a fully connected layer, and the size and step size of the convolution kernel of the first convolutional network layer of each sub-network are (4*τ, 4*τ) and (4*τ, 4*τ), respectively, which are used to normalize the output feature shape; in the process of training the sub-network, the network is trained by adding additional coding information; wherein the additional coding information includes: quantization parameter QP, partition depth Depth, time domain layer Id and RDCost generated by coding; For the subnetwork with min(W,H)=8, it consists of two convolutional layers. The convolution kernel size and stride of the second convolutional layer are (2, 2) and (2, 2) respectively. The output dimension of the convolutional layer is (BATCH_SIZE, 32, 1, 1). The output data is flattened to obtain data with (BATCH_SIZE, 32) dimension. For the subnetwork with min(W,H)=16, it consists of three convolutional layers. The convolution kernel size and step size of the second and third layers are (2,2) and (2,2). The output dimension of the convolutional layer is (BATCH_SIZE,64,1,1). The output data is flattened to obtain data with (BATCH_SIZE,64) dimension. For the sub-network with min(W,H)=32, it consists of three convolutional layers. The convolution kernel size and step size of the second and third convolutional layers of the sub-network with min(W,H)=32 are (4, 4) and (4, 4) respectively; the dimension of the convolutional layer output of this sub-network is (BATCH_SIZE, 128, 1, 1); the output data is flattened to obtain data with (BATCH_SIZE, 128) dimension; The subnetwork with min(W,H)=64 consists of three convolutional layers. The convolution kernel size and step size of the second and third convolutional layers of this subnetwork are (2,2) and (2,2) respectively. The dimension of the convolutional layer output of this subnetwork is (BATCH_SIZE,32,2,2). The output data is flattened to obtain data with a dimension of (BATCH_SIZE,128). Step 6: Input the segmentation result into the fully connected layer to obtain the segmentation probability prediction value; Step 7: Calculate the loss function of the model according to the predicted value of the partition probability, and continuously adjust the parameters of the model. When the loss function is minimized, the model training is completed. S4: input the current coding unit into the trained partition probability network model to obtain a partition probability prediction value of the current coding unit; S5: Select a prediction mode according to the partition probability prediction value, and determine whether the prediction mode is a partition mode. If it is a partition mode, return to S4; if it is not a partition mode, execute S6; S6: Use the current coding unit CU to perform MergeSkip mode encoding on the video to be encoded, and calculate the RDCost value of the MergeSkip mode encoding; S7: Determine whether to adopt the Skip mode to skip coding prediction according to the RDCost value, otherwise traverse all default candidate prediction modes.
2. A VVC inter-frame fast coding method based on reference blocks according to claim 1, characterized in that: The process of the fully connected layer processing the division results includes: the division results are (BATCH_SIZE, 32) dimensional data, (BATCH_SIZE, 64) dimensional data and (BATCH_SIZE, 128) dimensional data obtained according to different sub-networks; the corresponding encoding information is added to the obtained dimensional data and used as the input of the fully connected layer to obtain the division probability prediction value of the current CU type.
3. The VVC inter-frame fast coding method based on reference blocks according to claim 1, characterized in that: The loss function of the partition probability network model during training is: Among them, p m Indicates the proportion of partition mode m in this type of CU; α represents a variable parameter used to adjust the penalty of the partition mode; m represents the partition mode; M represents the set and sum of all partition modes, y n,m Indicates the actual partition label of CU; Represents the predicted label of CU; N represents the number of predicted categories; n represents the nth category among N categories.
4. The VVC inter-frame fast coding method based on reference blocks according to claim 1, characterized in that: The process of determining the partition mode according to the partition probability prediction value includes: inputting the partition probability prediction value into the softmax function, converting the predicted value of each category into a probability value between 0 and 1, wherein the maximum probability value is compared with the set threshold interval [τ1, τ2], when the maximum probability value is less than τ1 in the threshold interval, the encoder default partition mode is used to partition the CU, when the maximum probability value is greater than τ2 in the threshold interval, the predicted coding mode is used to partition the CU, otherwise the optimization mode is used to partition the CU; The process of dividing the CU includes: dividing the CU using the encoder default division mode includes: the encoder divides a CU by default, wherein the maximum number of divisions is 5, including quadrifugal division QT, vertical binary division BV, horizontal binary division BH, vertical trifugal division TV and horizontal trifugal division TH; Using the predicted coding mode to divide the CU includes: clearing the division mode stack; dividing the data according to the network predicted division mode, where the network predicted division mode includes no division NS, quad division QT, horizontal division HS, and vertical division VS; The process of dividing the CU using the optimization mode includes: clearing the division mode stack, determining the current division mode, if the current division mode is horizontal division HS, then performing horizontal binary division and horizontal trifurcation division on the divided CU again, if the current division mode is vertical division VS, then performing vertical binary division and vertical trifurcation division on the divided CU again, if the current division mode is neither horizontal division HS nor vertical division VS, then using the predicted coding mode to divide the CU.
Citation Information
Patent Citations
Rapid prediction method and device for H.266 / VVC screen content intra-frame CU division
CN114710667A
