Spatial SHVC fast coding method based on CU division prediction
By using a spatial SHVC fast coding method based on CU partitioning prediction, feature maps are generated using enhancement layer frames and QP information. Combined with neural networks and binary classification networks to determine CU partitioning, the problem of slow coding speed in existing technologies is solved, and efficient video coding is achieved.
Patent Information
- Application Number
- CN202511796755.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-02
- Publication Date
- 2026-03-03
AI Technical Summary
Existing video coding methods rely on manually designed features and heuristic rules, making it difficult to capture high-level representations of different content, and directly obtaining inter-layer reference frames reduces coding speed.
The spatial SHVC fast encoding method based on CU partitioning prediction uses enhanced layer frames and QP information as input to the regression model. It combines a 3-layer neural network and attention weights to calculate entropy and cosine similarity, generate feature maps, and determine CU partitioning through a binary classification network, thus avoiding the direct acquisition of inter-layer reference frames.
Encoding speed is increased by more than 69% with 1.5x and 2x spatial scalability, while BDBR only increases by 0.28% and 0.47%, significantly improving encoding efficiency.
Smart Images
Figure CN121603671A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video coding, and in particular to a fast spatial SHVC coding method based on CU partitioning prediction. Background Technology
[0002] In heuristic methods, computationally intensive RDOs for CU segmentation can be accelerated by leveraging intermediate features to skip unlikely depths or terminate the search process early. Subsequent research on complexity reduction can be broadly categorized into three types: texture-based methods utilizing gradient and entropy metrics, correlation-based methods leveraging spatial and temporal similarity, and techniques based on residual geometry for analyzing transform coefficients and reconstruction errors. These traditional fast video coding methods typically rely on manually designed features and heuristic rules, making it difficult to capture high-level representations that generalize diverse content.
[0003] In addition, existing technologies rely on inter-layer reference frames for CU partitioning, but directly obtaining inter-layer reference frames reduces coding speed. Summary of the Invention
[0004] To address the problems existing in the prior art and improve coding speed, this invention proposes a fast spatial SHVC coding method based on CU partitioning prediction. This method involves separating base layer frames and enhancement layer frames from the original coded frames of a video sequence, specifically including the following steps:
[0005] The enhanced layer frames and QP information are used as input to the regression model to predict inter-layer reference frames;
[0006] The original frame and the inter-layer reference frame are cut into coding units of a specified size. The cosine similarity between coding units in the brightness channel of adjacent original frames and the cosine similarity between the corresponding inter-layer reference coding units are calculated as inputs. The three-layer neural network outputs two attention weights. The attention weights are expanded in the spatial dimension to make them the same size as the original coding units and the predicted inter-layer reference coding units.
[0007] The two sets of weights obtained are the original coding unit and the inter-layer reference coding unit, respectively, and then weighted to obtain two feature maps;
[0008] The original coding unit is divided into multiple non-overlapping blocks using a non-overlapping sliding window. The probability distribution of each block is calculated by estimating the probability density of 32 bins, and then the entropy value of each block is calculated. The calculated entropy matrix is reshaped to the same resolution as the original coding frame to obtain one entropy feature map.
[0009] The entropy feature map is expanded to the size of the coding unit and then concatenated with the attention-weighted feature map. Then, the basic features are extracted through cascaded convolutional layers, normalization layers and ReLU activation function layers.
[0010] The quantization parameters of the base layer and the enhancement layer are respectively processed by the embedding layer to obtain an 8-dimensional embedding vector, which is then expanded into 8 feature maps of the same size as the current original encoded frame;
[0011] The basic features are concatenated with eight feature maps, and the concatenated feature maps are weighted using a channel attention module to obtain the final features.
[0012] The extracted final features are compressed into a 128-dimensional vector, and then concatenated with two cosine similarity scalars and the state of whether the corresponding CUs of the previous frame are divided as external features. These are then fed into a fully connected feature vector and input into a binary classification network. The binary classification network determines whether the data needs to be divided into encoding units of a specified size.
[0013] Compared with the prior art, the present invention improves the average encoding speed by more than 69% with 1.5 times spatial scalability, while the BDBR only increases by 0.28; the present invention improves the average encoding speed by more than 65% with 2 times spatial scalability, while the BDBR only increases by 0.47. Attached Figure Description
[0014] Figure 1 This is a flowchart of a spatial SHVC fast coding method based on CU partitioning prediction according to the present invention. Detailed Implementation
[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0016] This invention proposes a fast spatial SHVC coding method based on CU partitioning prediction, which includes separating base layer frames and enhancement layer frames from the original coded frames of a video sequence, specifically including the following steps:
[0017] The enhanced layer frames and QP information are used as input to the regression model to predict inter-layer reference frames;
[0018] The original frame and the interlayer reference frame are cut into coding units of a specified size, such as 64×64 / 32×32 and 16×16 coding units. The cosine similarity between coding units in the brightness channel of adjacent original frames and the cosine similarity between the corresponding interlayer reference coding units are calculated as inputs. The three-layer neural network outputs two attention weights. The attention weights are expanded in spatial dimension to make them the same size as the original coding units and the predicted interlayer reference coding units.
[0019] The two sets of weights obtained are the original coding unit and the inter-layer reference coding unit, respectively, and then weighted to obtain two feature maps;
[0020] The original coding unit is divided into multiple non-overlapping blocks using a non-overlapping sliding window. The probability distribution of each block is calculated by estimating the probability density of 32 bins, and then the entropy value of each block is calculated. The calculated entropy matrix is reshaped to the same resolution as the original coding frame to obtain one entropy feature map.
[0021] The entropy feature map is expanded to the size of the coding unit and then concatenated with the attention-weighted feature map. Then, the basic features are extracted through cascaded convolutional layers, normalization layers and ReLU activation function layers.
[0022] The quantization parameters of the base layer and the enhancement layer are respectively processed by the embedding layer to obtain an 8-dimensional embedding vector, which is then expanded into 8 feature maps of the same size as the current original encoded frame;
[0023] The basic features are concatenated with eight feature maps, and the concatenated feature maps are weighted using a channel attention module to obtain the final features.
[0024] The extracted final features are compressed into a 128-dimensional vector, and then concatenated with two cosine similarity scalars and the state of whether the corresponding CUs of the previous frame are divided as external features. These are then fed into a fully connected feature vector and input into a binary classification network. The binary classification network determines whether the data needs to be divided into encoding units of a specified size.
[0025] In this embodiment, a large-scale CTU partitioning database is established to facilitate fast encoding of SSHVC based on deep learning. This database is constructed from raw YUV video sequences with resolutions of 1920×1080 and 1280×720, including standard test sequences such as Blue-sky, Ducks, Park_Joy, Pedestrian, Tractor, Town, and Station2. Each 64×64 CTU is recursively partitioned into CUs from 64×64 (depth 0) to 8×8 (depth 3) using a quadtree structure. The partitioning state (split or not split) of each CU is directly derived from the depth information generated by the actual SSHVC reference encoder.
[0026] This embodiment takes into account the impact of ILR mode on CU segmentation, extracts the reconstructed frames of BL for ILR frame regression prediction, aligns each ILR frame with the original frame in EL, and segments it into 64x64 CTUs. Finally, this embodiment extracts approximately 1 million 64×64 CUs and 3 million 32×32 CUs. Each CU in the original frame has a spatially aligned corresponding CU in the ILR frame. Samples of each CU size are randomly divided into a training set (95%) and a validation set (5%).
[0027] To ensure robustness, data generation includes multiple QP configurations: the base layer (BL) has QP values of {22, 26, 30, 34}, and the enhancement layer (EL) has QP values of {24, 28, 32, 36}. Therefore, the complete database comprises 12 distinct subsets, each corresponding to a unique combination of four QP pairs consisting of base layer QPs and enhancement layer QPs (the four QP pairs include (22, 24), (26, 28), (30, 32), and (34, 36)) and three supported CU sizes (i.e., the three different CU sizes of 64×64, 32×32, and 16×16).
[0028] In this invention, instead of directly obtaining the interlayer reference frame, the enhanced layer frame and QP information are used as input to the regression model to predict the interlayer reference frame. The specific process includes:
[0029] QP information is encoded through the Embedding layer, and after linear transformation and activation function processing, a QP feature map matching the size of the input feature map is generated through the expand space expansion operation.
[0030] The QP feature map is concatenated with the original EL frame along the channel dimension to form an input feature containing spatial-quantization joint information. This input feature is efficiently initialized by a 3×3 convolution combined with an instance normalization layer.
[0031] The initial feature process is processed in two separate paths. In the main path, the original resolution is maintained and the features are refined by using regular convolution to preserve high-frequency details. In the downsampling path, the feature scale is compressed by convolution with a stride of 2, and then the resolution is restored by the upsampling module (which includes feature expansion convolution and pixelShuffle operation).
[0032] After the dual-path outputs are concatenated along the channel dimension, feature fusion and channel compression are achieved through 1×1 convolution.
[0033] The output layer generates a residual image using a non-activated 3×3 convolution, and adds it to the original EL frame through skip connections to obtain the predicted inter-layer reference frame;
[0034] While maintaining lightweight design, the above prediction process effectively improves the reconstruction quality of ILR frames through QP-aware feature fusion and dual-path multi-scale processing. The entire forward process only requires a single convolution operation to complete 2x upsampling, which is significantly better than traditional interpolation methods.
[0035] After obtaining the predicted inter-layer reference frames, the original coded frames and inter-layer reference frames are segmented into coding units of a specified size. When calculating the cosine similarity between coding units in the luminance channel of adjacent original coded frames and the cosine similarity between corresponding inter-layer reference coding units, it is first necessary to normalize the pixel values of each coding unit, i.e.:
[0036]
[0037] in, This represents the normalized pixel value of the i-th pixel in the n-th coding unit; P is the flattened vector of the pixel in the n-th coding unit, which can be converted into a one-dimensional vector using the flatten function in Python; It is the arithmetic mean of vector P; It is the standard deviation of vector P; It is a very small constant to prevent division by zero.
[0038] The calculation of cosine similarity between any two n×n coding units includes:
[0039]
[0040] in, This represents the cosine similarity between two n×n coding units; , These represent the normalized values of the i-th pixel in the two n×n coding units, respectively.
[0041] The cosine similarity between coding units in the luminance channel of the previous frame is obtained using the above method. And the cosine similarity between the coding units of the current original coding frame and its corresponding inter-layer reference frame. .
[0042] Using cosine similarity Similarity to cosine Two weights are generated, which are used to weight the original coded frame and the inter-layer reference frame, respectively. Specifically, they include:
[0043] Cosine similarity is calculated using a 3-layer neural network. The process is performed to obtain the first attention weight. The first attention weight is expanded in terms of spatial dimension to make it the same as the size of the original coding unit. The first attention weight is used to weight the corresponding coding unit process. After weighting each coding unit of the original coding in turn, the weighting of the original coding frame is completed.
[0044] Cosine similarity is calculated using a 3-layer neural network. The process is performed to obtain the second attention weight. The second attention weight is expanded in the spatial dimension to make it the same as the coding unit size of the inter-layer reference frame. The coding unit process of the corresponding inter-layer reference frame is weighted using the second attention weight. After weighting each coding unit of the inter-layer reference frame in turn, the weighting of the inter-layer reference frame is completed.
[0045] The process of obtaining the first attention weight and the second attention weight includes:
[0046]
[0047] Where w is the attention weight of the output; For the sigmoid function; , , For a trainable weight matrix, Represents the set of real numbers; For ReLU functions; The input to the attention weights is the cosine similarity. or cosine similarity ; , , is a trainable bias vector.
[0048] As an optional implementation, the original coding unit is divided into multiple non-overlapping blocks using a non-overlapping sliding window. The probability distribution of each block is calculated using probability density estimation of 32 bins, and then the entropy value of each block is calculated. The calculated entropy matrix is then reshaped to the same resolution as the original coding frame to obtain one entropy feature map. Specifically, the following steps are included:
[0049] 101. Divide the CU into multiple non-overlapping image blocks. In this embodiment, the CU will be divided into 4 sub-blocks. For each image block, a pixel process normalization operation will be performed using maximum-minimum value normalization, i.e.:
[0050] ;
[0051] in, for The normalized value; This represents the pixel value of the j-th pixel in the i-th image patch. This represents the minimum pixel value in the i-th image patch. This represents the maximum pixel value in the i-th image block; It is a constant, typically taking the value of This is used to prevent the denominator from being 0.
[0052] 102. Based on the 32 bins set, calculate the probability density estimate for each normalized pixel in each bin, i.e.:
[0053]
[0054] , ={0, 1, ..., K-1}
[0055] in, This represents the probability density estimate of the j-th pixel in the i-th image patch under the k-th bin; This is the k-th bins parameter.
[0056] 103. Calculate the probability distribution of an image patch based on the probability density estimate of all pixels in all bins, i.e.:
[0057]
[0058] in, This represents the probability distribution of the i-th image patch in the k-th bin; It is the total number of pixels in each image block.
[0059] 104. The entropy value of each image patch is represented as:
[0060]
[0061] in, This represents the entropy value of the i-th image patch.
[0062] As an optional implementation, the process of calculating the basic features based on the entropy feature map and the attention-weighted feature extraction includes:
[0063] The resulting entropy feature map is expanded to the size of the original encoded frame, and then concatenated with the two attention-weighted feature maps along the channel dimension.
[0064] The concatenated feature maps are preprocessed using cascaded convolutional layers, normalization layers, and ReLU activation function layers.
[0065] Each QP value corresponding to the base layer frame and the enhancement layer frame is processed by the embedding layer process. Then, the two resulting embedding vectors are expanded to the size of the original encoded frame, and all feature maps are concatenated together in the channel dimension to obtain the QP feature.
[0066] After concatenating the QP features with the preprocessed features along the channel dimension, spatial information is compressed using global average pooling, i.e.:
[0067]
[0068] Where z is the feature map obtained by compressing spatial information through global average pooling; This represents the feature map obtained by concatenating the QP features and the preprocessed features along the channel dimension. The value of the pixel in the c-th channel, i-th row, j-th column; H and W are the dimensions of the feature map, i.e., the height and width of the feature map;
[0069] The channel weights are learned through two fully connected layers (including dimensionality reduction and restoration), namely:
[0070]
[0071] Where s is the channel weight; , The weight matrix is a learnable matrix. , It is a learnable bias vector; For feature map The number of channels, B is the number of samples that are simultaneously input into the model for forward and backward propagation during one iteration;
[0072] The concatenated feature map is processed using a channel attention module. The final features are obtained by weighting the values.
[0073] The final features obtained need to be compressed into a 128-dimensional vector. The compression process specifically includes:
[0074] For the encoding unit of the Y channel of a 64×64 image frame, three convolutional layers with 3×3 kernels and stride of 2 are used to downsample and achieve 8 times feature compression. Then, global average pooling is used to compress it into a 128-dimensional vector.
[0075] For the encoding unit of the Y channel of a 32×32 image frame, two convolutional layers with 3×3 kernels and stride of 2 are used to downsample and achieve 4 times feature compression. Then, global average pooling is used to compress it into a 128-dimensional vector.
[0076] For the encoding unit of the Y channel of a 16×16 image frame, a convolutional layer with a kernel of 3×3 and a stride of 2 is used to downsample and achieve 2x feature compression. Then, global average pooling is used to compress it into a 128-dimensional vector.
[0077] Finally, the 128-dimensional basic features, the cosine similarity between the current original coded frame and the coding units in the luminance channel of the previous frame, the cosine similarity between the coding units in the corresponding inter-layer reference frames, and the state of whether the corresponding CUs in the previous frame have been divided are used as inputs to the classifier. The classifier outputs the probability of whether a division has occurred. In this embodiment, the classifier uses a binary classification network, which consists of three cascaded fully connected layers. The outputs of the first two fully connected layers are activated using the ReLU activation function before being input to the next layer. The output of the last fully connected layer directly yields the binary classification probability.
[0078] As an optional implementation, the bipartite network is trained using a disclosure loss function, defined as follows:
[0079]
[0080] in, Here, N is the loss function; N is the number of samples in one training iteration. This represents the true classification of the i-th sample in the j-th category, i.e., the distribution when the i-th CU is in the j-th category; The category weight for the j-th category is used to balance the dataset and is achieved through inverse frequency weighting, meaning the weight is inversely proportional to the number of samples in the category. This represents the predicted classification of the i-th sample in the j-th category. In this invention, the number of categories is 1, that is, j=1 means that classification is required, and j=2 means that classification is not required.
[0081] Table 1 Overall performance comparison at 1.5x
[0082]
[0083] Table 2 Overall performance comparison at 2x
[0084]
[0085] Tables 1 and 2 present the results at scalability ratios of 1.5x and 2.0x, respectively. For the 1.5x configuration (Table 1), with QPs of the base layer (BL) and enhancement layer (EL) set to {22, 26, 30, 34} and {24, 28, 32, 36}, respectively, the proposed algorithm achieves a 69.49% time gain with only a 0.28% increase in BDBR, outperforming PAPS (BDBR 0.38%, TS 65.85%) and FIICA (BDBR -0.18%, TS 45.75%). For the 2.0x configuration (Table 2), compared to PAPS (BDBR 0.68%, TS 66.67%) and FIICA (BDBR 0.40%, TS 43.25%), our method again demonstrates superior performance, with an average BDBR of 0.47% and a TS of 65.14%.
[0086] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A spatial SHVC fast coding method based on CU partitioning prediction, comprising separating base layer frames and enhancement layer frames from the original coded frames of a video sequence, characterized in that, Specifically, the following steps are included: The enhanced layer frames and QP information are used as input to the regression model to predict inter-layer reference frames; The original coded frame and the inter-layer reference frame are cut into coded units of a specified size. The cosine similarity between the coded units of the current original coded frame and the previous frame in the luminance channel and the cosine similarity between the coded units of the corresponding inter-layer reference frame are calculated as inputs. The three-layer neural network outputs two attention weights. The attention weights are expanded in the spatial dimension to make them the same size as the coded units of the original coded frame and the inter-layer reference frame. The two sets of weights obtained are the original coding unit and the inter-layer reference coding unit, respectively, and then weighted to obtain two feature maps; The original coding unit is divided into multiple non-overlapping blocks using a non-overlapping sliding window. The probability distribution of each block is calculated by estimating the probability density of 32 bins, and then the entropy value of each block is calculated. The calculated entropy matrix is reshaped to the same resolution as the original coding frame to obtain one entropy feature map. The entropy feature map is expanded to the size of the original encoded frame and then concatenated with the attention-weighted feature map. Then, the basic features are extracted by cascading convolutional layers, normalization layers and ReLU activation function layers. The quantization parameters of the base layer and the enhancement layer are respectively processed by the embedding layer to obtain an 8-dimensional embedding vector, which is then expanded into 8 feature maps of the same size as the current original encoded frame; The basic features are concatenated with eight feature maps, and the concatenated feature maps are weighted using a channel attention module to obtain the final features. The extracted final features are compressed into a 128-dimensional vector, and then concatenated with two cosine similarity scalars and the state of whether the corresponding CUs of the previous frame are divided as external features. These are then fed into a fully connected feature vector and input into a binary classification network. The binary classification network determines whether the data needs to be divided into encoding units of a specified size.
2. The spatial SHVC fast coding method based on CU partitioning prediction according to claim 1, characterized in that, The cosine similarity calculation for two n×n coding units includes: in, This represents the cosine similarity between two n×n coding units; , These represent the normalized values of the i-th pixel in the two n×n coding units, respectively.
3. The spatial SHVC fast coding method based on CU partitioning prediction according to claim 1, characterized in that, The process of obtaining attention weights includes: Where w is the attention weight of the output; For the sigmoid function; , , For a trainable weight matrix, Represents the set of real numbers; For ReLU functions; The input to the attention weight is the cosine similarity between coding units in the luminance channel of adjacent original frames or the cosine similarity between a coding unit in the luminance channel of an original frame and its corresponding inter-layer reference coding unit. , , is a trainable bias vector.
4. The spatial SHVC fast coding method based on CU partitioning prediction according to claim 1, characterized in that, The probability distribution of each block is calculated through probability density estimation, including: Divide the CU into multiple non-overlapping image blocks, and perform pixel process normalization on each image block; Based on the 32 bins set, calculate the probability density estimate of each normalized pixel in each bin; The probability distribution of an image patch is calculated based on the probability density estimate of all pixels in all bins.
5. A spatial SHVC fast coding method based on CU partitioning prediction according to claim 1 or 3, characterized in that, The entropy calculation for each image patch includes: in, This represents the entropy value of the i-th image patch; Estimate the probability density of the i-th image patch in the k-th bin; It is a constant.
6. The spatial SHVC fast coding method based on CU partitioning prediction according to claim 1, characterized in that, The process of extracting basic features includes: The resulting entropy feature map is expanded to the size of the original encoded frame, and then concatenated with the two attention-weighted feature maps along the channel dimension. The concatenated feature maps are preprocessed using cascaded convolutional layers, normalization layers, and ReLU activation function layers. The QP values corresponding to the base layer frame and the enhancement layer frame are processed by the embedding layer process, and then the two embedding vectors are expanded to the size of the original coding frame. The two images are then concatenated together in the channel dimension to obtain the QP feature. After concatenating the QP features and preprocessed features along the channel dimension, spatial information is compressed using global average pooling. Channel weights are then learned from the feature map after spatial compression using two fully connected layers. The basic features are obtained by weighting the feature map concatenated with the QP features and preprocessed features along the channel dimension using the channel weights.
7. The spatial SHVC fast coding method based on CU partitioning prediction according to claim 1, characterized in that, In the parallel CU processing structure, the feature extraction channel is selected according to the size of the encoding unit, where: For the encoding unit of the Y channel of a 64×64 image frame, three convolutional layers with 3×3 kernels and stride of 2 are used to downsample and achieve 8 times feature compression. Then, global average pooling is used to compress it into a 128-dimensional vector. For the encoding unit of the Y channel of a 32×32 image frame, two convolutional layers with 3×3 kernels and stride of 2 are used to downsample and achieve 4 times feature compression. Then, global average pooling is used to compress it into a 128-dimensional vector. For the encoding unit of the Y channel of a 16×16 image frame, a convolutional layer with a kernel of 3×3 and a stride of 2 is used to downsample and achieve 2x feature compression. Then, global average pooling is used to compress it into a 128-dimensional vector.
8. The spatial SHVC fast coding method based on CU partitioning prediction according to claim 1, characterized in that, The binary classification network consists of three cascaded fully connected layers. The outputs of the first two fully connected layers are activated by the ReLU function before being input into the next layer. The output of the last fully connected layer directly yields the binary classification probability.
9. The spatial SHVC fast coding method based on CU partitioning prediction according to claim 1, characterized in that, The network is trained using cross-loss, and the loss function is defined as follows: in, Here, N is the loss function; N is the number of samples in one training iteration. This represents the true classification of the i-th sample in the j-th category; The category weight for the j-th category; This represents the predicted classification of the i-th sample in the j-th category.
10. A fast spatial SHVC coding method based on CU partitioning prediction, characterized in that, A threshold is set. If the probability of the binary classification network outputting a classification is greater than the threshold, then a classification is performed. Otherwise, the RDO value is calculated step by step to select the decision scheme with the minimum loss.