VVC-SCC intra-frame coding method and device based on multi-stage irregular coding unit division

Abstract

The present invention discloses a VVC-SCC intra-frame coding method and device based on multi-stage irregular coding unit division, which relates to the field of video coding, including: designing and training a coding unit division prediction model based on a multi-stage convolutional neural network; secondly, predicting the input video frame through the trained coding unit division prediction model, and outputting the prediction probability of the mode type of all CUs in the current frame; finally, setting different thresholds for different mode categories, guiding the VVC-SCC encoder to encode according to the prediction probability and the corresponding threshold, thereby skipping unnecessary mode category traversal. The present invention solves the problems of long encoding time and high encoder calculation complexity of the existing VVC-SCC intra-frame coding method.

Description

Technical Field

[0001] The present invention relates to the field of video coding, and in particular to a VVC-SCC intra-frame coding method and device based on multi-stage irregular coding unit division. Background Art

[0002] With the rapid development of computer science and technology, screen content video, as a new form of video representation, has attracted increasing attention. Compared with natural content video, screen content video has sharp edges, contains a large number of flat areas, and repeats the same pattern text. If the traditional video coding standard is used to process screen content video, the compression effect is often poor, and problems such as text distortion and image blur will occur. Therefore, the Joint Video Coding Group has developed a new generation of screen content coding standard based on the High Efficiency Video Coding - Screen Content Coding (HEVC-SCC): Versatile Video Coding - Screen Content Coding (VVC-SCC). This standard improves the compression effect of screen content video by using the intra-frame block copy mode and palette mode of HEVC-SCC, but the computational complexity also increases sharply.

[0003] Since VVC-SCC uses a quadtree nested multi-tree partitioning structure, it is necessary to start from the top-down division of the 128×128 coding unit until it is divided into the smallest 4×4 coding unit, and then compare the division cost from bottom to top. The division is determined by comparing the sum of the rate-distortion cost of the coding unit before the division and the rate-distortion cost of the sub-coding unit. If the sum of the rate-distortion cost of the sub-coding unit is less than the rate-distortion cost of the coding unit before the division, the coding unit needs to be divided during encoding, otherwise it is not divided. In order to reduce the computational complexity of the VVC-SCC encoder. Traditional fast algorithms rely on manually extracted features to accelerate VVC-SCC, but the number of features is limited and only applicable to specific sequences. With the breakthrough of machine learning, especially deep learning technology, deep learning technology can be used to improve traditional encoders, thereby improving the speed and efficiency of video encoding. However, the existing convolutional neural network structure has limitations in learning ability. It is necessary to call the coding unit partition prediction model multiple times to predict the coding unit one by one, instead of directly outputting the prediction result of the current frame, which cannot effectively maintain the balance of RDC. Therefore, based on the considerations above, reducing the computational complexity of screen content video coding unit division without affecting subjective quality is a key challenge for the VVC-SCC acceleration algorithm. Summary of the invention

[0004] The purpose of the present application is to propose a VVC-SCC intra-frame coding method and device based on multi-stage irregular coding unit division in response to the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a VVC-SCC intra-frame coding method based on multi-stage irregular coding unit division, comprising the following steps:

[0006] Acquire a screen content video, process the screen content video to obtain a plurality of coding tree units, and divide each coding tree unit into a plurality of current coding units of a first size;

[0007] A coding unit partition prediction model based on a multi-stage convolutional neural network is constructed and trained to obtain a trained coding unit partition prediction model, wherein the coding unit partition prediction model includes five stages, each of which includes a residual module and a sub-network;

[0008] A VVC-SCC encoder is used to encode a current coding unit of a first size, a current quantization parameter is determined and normalized during the encoding process to obtain a normalized current quantization parameter, the current coding unit of the first size and the normalized current quantization parameter are input into a trained coding unit division prediction model, and a prediction result of the current stage is obtained through current stage prediction. The prediction result of the current stage includes prediction probabilities of different mode categories corresponding to the current coding unit input at the current stage during the encoding process, and the mode categories include a non-division mode and a division mode of dividing the current coding unit input at the current stage into coding units with a reduced size of one level; different thresholds are set for different mode categories, and whether to divide the current coding unit into a division mode corresponding to the current coding unit input at the current stage is determined according to the prediction probability of each mode category and its corresponding threshold. The mode category is placed in the rate-distortion optimization list. If the mode category is placed in the rate-distortion optimization list, the rate-distortion cost of the mode category in the rate-distortion optimization list is calculated by the VVC-SCC encoder, and the mode category with the smallest rate-distortion cost is selected as the optimal mode category of the current coding unit. The current coding unit input in the current stage is divided by the optimal mode category, and the divided coding units are obtained and input into the next stage with the normalized current quantization parameter. The prediction result of the next stage is obtained through the prediction of the next stage, and the prediction result of the next stage is used as the prediction result of the current stage to repeat the above steps after obtaining the prediction result of the current stage; if the mode category is not placed in the rate-distortion optimization list, the rate-distortion cost calculation of the mode category is directly skipped.

[0009] Preferably, determining whether to place the mode category in the rate-distortion optimization list according to the prediction probability of each mode category and its corresponding threshold value specifically includes:

[0010] In response to determining that the prediction probability of the mth mode category in the prediction result of the nth current coding unit at the current stage satisfies the following formula:

[0011] ;

[0012] in, represents the prediction probability of the mode category m in the prediction result of the nth current coding unit at the current stage, represents the prediction probability of the mode category i in the prediction result of the nth current coding unit at the current stage, , Represents the threshold value corresponding to mode category m, and max represents the maximum value. It is determined that the mode category m in the prediction result of the nth current coding unit at the current stage is placed in the rate-distortion optimization list. Otherwise, it is determined not to place the mode category m in the prediction result of the nth current coding unit at the current stage in the rate-distortion optimization list.

[0013] Preferably, the division mode includes a quadtree division mode, a horizontal binary tree division mode, a horizontal ternary tree division mode, a vertical binary tree division mode and / or a vertical ternary tree division mode. The division mode in the first stage of the coding unit division prediction model is the quadtree division mode, and the division modes in the second and third stages of the coding unit division prediction model are the quadtree division mode, the horizontal binary tree division mode, the horizontal ternary tree division mode, the vertical binary tree division and the vertical ternary tree division mode; the division modes in the fourth and fifth stages of the coding unit division prediction model are the horizontal binary tree division mode, the horizontal ternary tree division mode, the vertical binary tree division and the vertical ternary tree division mode.

[0014] Preferably, the threshold corresponding to the non-division mode is 1, the threshold corresponding to the quadtree division mode is 0.1, the threshold corresponding to the horizontal binary tree division mode is 0.7, the threshold corresponding to the horizontal ternary tree division mode is 0.05, the threshold corresponding to the vertical binary tree division mode is 0.05, and the threshold corresponding to the vertical ternary tree division mode is 0.1.

[0015] Preferably, in the coding unit division prediction model, the first stage includes a first residual module and a first subnetwork, the first subnetwork includes a first feature extraction layer, a first convolutional layer, a first fully connected layer and a second fully connected layer connected in sequence, the first feature extraction layer is a first asymmetric convolutional network, and the first asymmetric convolutional network includes a convolutional layer with a convolution kernel size of 9×5, a convolutional layer with a convolution kernel size of 7×7 and a convolutional layer with a convolution kernel size of 5×9 connected in sequence; the second stage includes a second residual module and a second subnetwork, and the second subnetwork The method comprises a second feature extraction layer, a second convolution layer, a third convolution layer, a third fully connected layer and a fourth fully connected layer connected in sequence, the second feature extraction layer is a second asymmetric convolution network, and the second asymmetric convolution network comprises a convolution layer with a convolution kernel size of 7×3, a convolution layer with a convolution kernel size of 5×5 and a convolution layer with a convolution kernel size of 3×7 connected in sequence; the third stage comprises a third residual module and a third sub-network, and the third sub-network comprises a third feature extraction layer, a fourth convolution layer, a fifth convolution layer, a fifth fully connected layer and a sixth convolution layer connected in sequence The third feature extraction layer is an adaptive convolution layer with a convolution kernel size of 4hs×4ws; the fourth stage includes the fourth residual module and the fourth sub-network, the fifth stage includes the fifth residual module and the fifth sub-network, the fourth sub-network and the fifth sub-network include the fourth feature extraction layer, the sixth convolution layer, the seventh fully connected layer and the eighth fully connected layer connected in sequence, and the fourth feature extraction layer is an adaptive convolution layer with a convolution kernel size of 2hs×2ws; where hs and ws are the height unit and width unit of the adaptive convolution layer, respectively. If When the previous coding unit is rectangular and the width of the current coding unit is greater than the height, ws=(w / h), hs=1, where w and h are the width and height of the current coding unit; otherwise ws=1, hs=(h / w), when the current coding unit is square, ws=1, hs=1; the normalized current quantization parameter is input to the first fully connected layer, the second fully connected layer, the third fully connected layer, the fourth fully connected layer, the fifth fully connected layer, the sixth fully connected layer, the seventh fully connected layer and the eighth fully connected layer respectively, and the normalized current quantization parameter is expressed as:

[0016] ;

[0017] in, Indicates the current quantization parameter, is the normalized current quantization parameter;

[0018] The first residual module, the second residual module, the third residual module, the fourth residual module, and the fifth residual module all adopt a residual structure, and the residual structure includes a seventh convolutional layer and an eighth convolutional layer connected in sequence, and the input features of the seventh convolutional layer are residually connected with the output features of the eighth convolutional layer.

[0019] Preferably, the loss function used in the training process of the coding unit partition prediction model is as follows:

[0020] ;

[0021] Among them, L represents the loss function, N represents the batch training size, and M represents the set of pattern categories. and represents the true label and predicted label of the nth current coding unit when the mode category is m, is the rate-distortion cost of the nth current coding unit in the case of mode category m, is the weight coefficient.

[0022] In a second aspect, the present invention provides a VVC-SCC intra-frame coding device based on multi-stage irregular coding unit division, comprising:

[0023] A data processing module is configured to obtain a screen content video, process the screen content video to obtain a plurality of coding tree units, and divide each coding tree unit into a plurality of current coding units of a first size;

[0024] A model building module is configured to build and train a coding unit partition prediction model based on a multi-stage convolutional neural network to obtain a trained coding unit partition prediction model, wherein the coding unit partition prediction model includes five stages, each of which includes a residual module and a sub-network;

[0025] The encoding module is configured to use a VVC-SCC encoder to encode a current coding unit of a first size, determine a current quantization parameter and normalize it during the encoding process to obtain a normalized current quantization parameter, input the current coding unit of the first size and the normalized current quantization parameter into a trained coding unit division prediction model, obtain a prediction result of the current stage through current stage prediction, and the prediction result of the current stage includes prediction probabilities of different mode categories corresponding to the current coding unit input at the current stage during the encoding process, and the mode categories include a non-division mode and a division mode of dividing the current coding unit input at the current stage into coding units with a reduced size of one level; different thresholds are set for different mode categories, and the prediction probability of each mode category and its corresponding threshold are determined according to the prediction probability of each mode category. The value determines whether to place the mode category in the rate-distortion optimization list. If the mode category is placed in the rate-distortion optimization list, the VVC-SCC encoder performs rate-distortion cost calculation on the mode category in the rate-distortion optimization list, and selects the mode category with the smallest rate-distortion cost as the optimal mode category of the current coding unit. The current coding unit input in the current stage is divided by the optimal mode category, and the divided coding units are obtained and input into the next stage with the normalized current quantization parameter. The prediction result of the next stage is obtained through the prediction of the next stage, and the prediction result of the next stage is used as the prediction result of the current stage to repeat the above steps after obtaining the prediction result of the current stage; if the mode category is not placed in the rate-distortion optimization list, the rate-distortion cost calculation of the mode category is skipped directly.

[0026] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0027] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0028] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) The VVC-SCC intra-frame coding method based on multi-stage irregular coding unit division proposed in the present invention adopts a coding unit division prediction model based on a multi-stage convolutional neural network, which is composed of five sub-networks corresponding to the five stages of coding unit division in VVC-SCC. Designing different network structures for coding units in different stages can better learn different features, improve training effects, and improve prediction accuracy.

[0031] (2) The VVC-SCC intra-frame coding method based on multi-stage irregular coding unit division proposed in the present invention reduces the error rate of label prediction by setting different thresholds for different division modes, achieving a good trade-off between computational complexity and time saving. It can save coding time and accelerate the VVC-SCC coding process without affecting the subjective quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A schematic diagram of a flow chart of a VVC-SCC intra-frame coding method based on multi-stage irregular coding unit division according to an embodiment of the present application;

[0034] Figure 2 A network diagram of a coding unit division prediction model of a VVC-SCC intra-frame coding method based on multi-stage irregular coding unit division according to an embodiment of the present application;

[0035] Figure 3 A schematic diagram of a VVC-SCC intra-frame coding device based on multi-stage irregular coding unit division according to an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Figure 1A VVC-SCC intra-frame coding method based on multi-stage irregular coding unit division provided by an embodiment of the present application is shown, comprising the following steps:

[0039] S1, obtaining a screen content video, processing the screen content video to obtain a plurality of coding tree units, and dividing each coding tree unit into a plurality of current coding units of a first size.

[0040] Specifically, the input screen content video is processed. The specific processing method includes obtaining a video frame of the screen content video, converting the video frame into a grayscale image, extracting brightness information, and dividing the video frame into coding tree units (CTUs) of size 128×128, and further dividing them into current coding units (CUs) of size 64×64 after extracting the brightness information.

[0041] Furthermore, the embodiments of the present application need to produce training data, first collect a screen content video data set, and the training data includes the current coding unit obtained by the above steps for each screen content video in the screen content video data set and the real labels of each mode category of the CU division in VVC-SCC. The screen content video data set is encoded on the standard test platform VTM7.0 of VVC-SCC in the All-Intra configuration mode, and the real labels of the intra-frame CU divisions of the four quantization level parameters of 22, 27, 32, and 37 are obtained as the real labels of each mode category.

[0042] S2, constructing and training a coding unit partition prediction model based on a multi-stage convolutional neural network to obtain a trained coding unit partition prediction model, the coding unit partition prediction model includes five stages, each stage includes a residual module and a sub-network.

[0043] In a specific embodiment, in the coding unit division prediction model, the first stage includes a first residual module and a first sub-network, the first sub-network includes a first feature extraction layer, a first convolutional layer, a first fully connected layer and a second fully connected layer connected in sequence, the first feature extraction layer is a first asymmetric convolutional network, and the first asymmetric convolutional network includes a convolutional layer with a convolution kernel size of 9×5, a convolutional layer with a convolution kernel size of 7×7 and a convolutional layer with a convolution kernel size of 5×9 connected in sequence; the second stage includes a second residual module and a second sub-network, the second The subnetwork includes a second feature extraction layer, a second convolutional layer, a third convolutional layer, a third fully connected layer and a fourth fully connected layer connected in sequence, the second feature extraction layer is a second asymmetric convolutional network, and the second asymmetric convolutional network includes a convolutional layer with a convolution kernel size of 7×3, a convolutional layer with a convolution kernel size of 5×5 and a convolutional layer with a convolution kernel size of 3×7 connected in sequence; the third stage includes a third residual module and a third subnetwork, and the third subnetwork includes a third feature extraction layer, a fourth convolutional layer, a fifth convolutional layer, a fifth fully connected layer and The sixth fully connected layer, the third feature extraction layer is an adaptive convolution layer with a convolution kernel size of 4hs×4ws; the fourth stage includes the fourth residual module and the fourth sub-network, the fifth stage includes the fifth residual module and the fifth sub-network, the fourth sub-network and the fifth sub-network include the fourth feature extraction layer, the sixth convolution layer, the seventh fully connected layer and the eighth fully connected layer connected in sequence, and the fourth feature extraction layer is an adaptive convolution layer with a convolution kernel size of 2hs×2ws; where hs and ws are the height unit and width unit of the adaptive convolution layer, respectively. If When the current coding unit is rectangular and the width of the current coding unit is greater than the height, ws=(w / h), hs=1, where w and h are the width and height of the current coding unit; otherwise ws=1, hs=(h / w), when the current coding unit is square, ws=1, hs=1; the normalized current quantization parameter is input to the first fully connected layer, the second fully connected layer, the third fully connected layer, the fourth fully connected layer, the fifth fully connected layer, the sixth fully connected layer, the seventh fully connected layer and the eighth fully connected layer respectively, and the normalized current quantization parameter is expressed as:

[0044] ;

[0045] in, Indicates the current quantization parameter, is the normalized current quantization parameter;

[0046] The first residual module, the second residual module, the third residual module, the fourth residual module, and the fifth residual module all adopt a residual structure, and the residual structure includes a seventh convolutional layer and an eighth convolutional layer connected in sequence, and the input features of the seventh convolutional layer are residually connected with the output features of the eighth convolutional layer.

[0047] In a specific embodiment, the loss function used in the training process of the coding unit partition prediction model is as follows:

[0048] ;

[0049] Among them, L represents the loss function, N represents the batch training size, and M represents the set of pattern categories. and represents the true label and predicted label of the nth current coding unit when the mode category is m, is the rate-distortion cost of the nth current coding unit in the case of mode category m, is the weight coefficient.

[0050] Specifically, refer to Figure 2The multi-stage convolutional neural network (MSP-CNN) has a total of five subnetworks, each of which outputs the mode category of the CU of the current stage size level. There are at least two and at most six mode categories in different stages. After the multi-stage convolutional neural network, the position, width and height information of all CUs in the current frame and the corresponding prediction probability are output at each stage. If the mode category needs to be divided, it is divided according to the position, width and height information and mode category of all CUs in the current frame to obtain the divided coding unit. The first stage in the multi-stage convolutional neural network corresponds to the CU of the first size, the second stage in the multi-stage convolutional neural network corresponds to the CU of the second size, the third stage in the multi-stage convolutional neural network corresponds to the CU of the third size, the fourth stage in the multi-stage convolutional neural network corresponds to the CU of the fourth size, and the fifth stage in the multi-stage convolutional neural network corresponds to the CU of the fifth size. In one embodiment, the first size is 64×64, the second size is 32×32, the first size is 16×16, the first size is 8×8, and the first size is 4×4. After extracting the Y channel information of each stage, the CU first passes through the corresponding residual module to deepen its feature information, and then inputs it into the sub-network of the corresponding stage according to its division stage. Different sub-networks correspond to CU divisions at different stages. The multi-stage convolutional neural network consists of five sub-networks: AKQT Net, AKQT-MTT Net, QT-MTT Net, MTT Net1, and MTT Net2. Each sub-network contains a feature extraction layer, a convolution layer, and a fully connected layer. The feature extraction layers of AKQT Net and AKQT-MTT Net use asymmetric convolution networks to extract vertical and horizontal feature maps, and then splice the feature maps into the convolution layer, and then further deepen the features through the convolution layer, and finally output the prediction probabilities of all mode categories of the current coding unit through the fully connected layer; the feature extraction layers of the three sub-networks of QT-MTT Net, MTT Net1, and MTT Net2 all use adaptive convolution kernels for feature extraction, and input the extracted features into the convolution layer, and then output the prediction probabilities of all mode categories of the current coding unit through the fully connected layer.

[0051] In one embodiment, the convolution kernel size of the first convolution layer and the second convolution layer is 4×4, the convolution kernel size of the third convolution layer, the fourth convolution layer, the fifth convolution layer and the sixth convolution layer is 2×2, the convolution kernel size of the seventh convolution layer and the eighth convolution layer is 3×3, and all convolution layers have ReLU activation function; the first fully connected layer, the third fully connected layer, the fifth fully connected layer and the seventh fully connected layer have ReLU activation function, the second fully connected layer, the fourth fully connected layer, the sixth fully connected layer and the eighth fully connected layer have Softmax activation function, and the neurons in the second fully connected layer, the fourth fully connected layer, the sixth fully connected layer and the eighth fully connected layer are 2, 6, 6, 5 and 5 respectively, corresponding to the first stage, the second stage, the third stage, the fourth stage and the fifth stage, respectively. The output mode categories are 2, 6, 6, 5 and 5. In other embodiments, the above parameters can be adjusted according to actual conditions.

[0052] The quantization parameter (QP) has a significant impact on the division of the CU. When the QP value is high, the texture detail information contained in the coded frame is reduced, so the CU is not inclined to be further divided. On the contrary, when the QP value is low, the coded frame retains more texture detail information, and the CU tends to be further divided. Based on this feature, the QP is first normalized to obtain the normalized quantization parameter , and then the normalized quantization parameter As an external feature, it acts on all fully connected layers. This improvement enables the multi-stage convolutional neural network to adaptively adjust the CU partitioning strategy according to the QP value, and optimizes the partitioning prediction performance under different QP settings. This not only improves the adaptability to different encoding qualities, but also enhances its performance under variable bitrate conditions.

[0053] When constructing the above coding unit partition prediction model based on the multi-stage convolutional neural network, the coding unit partition prediction model is trained using training data, and the above loss function is used in the training process to obtain the trained coding unit partition prediction model. In one embodiment, the value of the weight coefficient in the loss function formula is 0.3.

[0054] S3, using a VVC-SCC encoder to encode a current coding unit of a first size, determining and normalizing a current quantization parameter during the encoding process to obtain a normalized current quantization parameter, inputting the current coding unit of the first size and the normalized current quantization parameter into a trained coding unit division prediction model, obtaining a prediction result of the current stage through current stage prediction, the prediction result of the current stage including prediction probabilities of different mode categories corresponding to the current coding unit input at the current stage during the encoding process, the mode categories including a non-division mode and a division mode of dividing the current coding unit input at the current stage into coding units of a reduced size by one level; setting different thresholds for different mode categories, determining the prediction probability of each mode category according to its corresponding threshold Whether to place the mode category in the rate-distortion optimization list. If the mode category is placed in the rate-distortion optimization list, the VVC-SCC encoder calculates the rate-distortion cost of the mode category in the rate-distortion optimization list, and selects the mode category with the smallest rate-distortion cost as the optimal mode category of the current coding unit. The current coding unit input in the current stage is divided by the optimal mode category, and the divided coding units are obtained and input into the next stage with the normalized current quantization parameter. The prediction result of the next stage is obtained through the prediction of the next stage, and the prediction result of the next stage is used as the prediction result of the current stage to repeat the above steps after obtaining the prediction result of the current stage; if the mode category is not placed in the rate-distortion optimization list, the rate-distortion cost calculation of the mode category is skipped directly.

[0055] In a specific embodiment, determining whether to place a mode category in the rate-distortion optimization list according to the prediction probability of each mode category and its corresponding threshold value specifically includes:

[0056] In response to determining that the prediction probability of the mth mode category in the prediction result of the nth current coding unit at the current stage satisfies the following formula:

[0057] ;

[0058] in, represents the prediction probability of the mode category m in the prediction result of the nth current coding unit at the current stage, represents the prediction probability of the mode category i in the prediction result of the nth current coding unit at the current stage, , Represents the threshold value corresponding to mode category m, and max represents the maximum value. It is determined that the mode category m in the prediction result of the nth current coding unit at the current stage is placed in the rate-distortion optimization list. Otherwise, it is determined not to place the mode category m in the prediction result of the nth current coding unit at the current stage in the rate-distortion optimization list.

[0059] In a specific embodiment, the division mode includes a quadtree division mode, a horizontal binary tree division mode, a horizontal ternary tree division mode, a vertical binary tree division mode and / or a vertical ternary tree division mode. The division mode in the first stage of the coding unit division prediction model is the quadtree division mode, and the division modes in the second and third stages of the coding unit division prediction model are the quadtree division mode, the horizontal binary tree division mode, the horizontal ternary tree division mode, the vertical binary tree division and the vertical ternary tree division mode; the division modes in the fourth and fifth stages of the coding unit division prediction model are the horizontal binary tree division mode, the horizontal ternary tree division mode, the vertical binary tree division and the vertical ternary tree division mode.

[0060] In a specific embodiment, the threshold corresponding to the non-division mode is 1, the threshold corresponding to the quadtree division mode is 0.1, the threshold corresponding to the horizontal binary tree division mode is 0.7, the threshold corresponding to the horizontal ternary tree division mode is 0.05, the threshold corresponding to the vertical binary tree division mode is 0.05, and the threshold corresponding to the vertical ternary tree division mode is 0.1.

[0061] Specifically, different thresholds are set for different mode categories for encoding. During the encoding process of the VVC-SCC encoder, the trained coding unit division prediction model is called to obtain the prediction probabilities of all mode categories. If the prediction probability of mode category m in the prediction result of the current stage satisfies the When the formula is met, the rate-distortion value of mode category m is calculated, otherwise the calculation of the rate-distortion value of mode category m is skipped, thereby accelerating the encoding process of VVC-SCC.

[0062] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a VVC-SCC intra-frame coding device based on multi-stage irregular coding unit division, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0063] The embodiment of the present application provides a VVC-SCC intra-frame coding device based on multi-stage irregular coding unit division, including:

[0064] The data processing module 1 is configured to obtain a screen content video, process the screen content video to obtain a plurality of coding tree units, and divide each coding tree unit into a plurality of current coding units of a first size;

[0065] A model building module 2 is configured to build and train a coding unit partition prediction model based on a multi-stage convolutional neural network to obtain a trained coding unit partition prediction model, wherein the coding unit partition prediction model includes five stages, each of which includes a residual module and a sub-network;

[0066] The encoding module 3 is configured to use a VVC-SCC encoder to encode a current coding unit of a first size, determine a current quantization parameter and normalize it during the encoding process to obtain a normalized current quantization parameter, input the current coding unit of the first size and the normalized current quantization parameter into a trained coding unit division prediction model, obtain a prediction result of the current stage through current stage prediction, and the prediction result of the current stage includes prediction probabilities of different mode categories corresponding to the current coding unit input at the current stage during the encoding process, and the mode categories include a non-division mode and a division mode of dividing the current coding unit input at the current stage into coding units with a reduced size of one level; different thresholds are set for different mode categories, and the prediction probability of each mode category and its corresponding threshold value are calculated according to the prediction probability of each mode category. The value determines whether to place the mode category in the rate-distortion optimization list. If the mode category is placed in the rate-distortion optimization list, the VVC-SCC encoder performs rate-distortion cost calculation on the mode category in the rate-distortion optimization list, and selects the mode category with the smallest rate-distortion cost as the optimal mode category of the current coding unit. The current coding unit input in the current stage is divided by the optimal mode category, and the divided coding units are obtained and input into the next stage with the normalized current quantization parameter. The prediction result of the next stage is obtained through the prediction of the next stage, and the prediction result of the next stage is used as the prediction result of the current stage to repeat the above steps after obtaining the prediction result of the current stage; if the mode category is not placed in the rate-distortion optimization list, the rate-distortion cost calculation of the mode category is skipped directly.

[0067] Figure 4 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 4 As shown, the electronic device of this embodiment includes: a processor 401 and a memory 402; wherein the memory 402 is used to store computer-executable instructions; the processor 401 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0068] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0069] When the memory 402 is independently provided, the electronic device further includes a bus 403 for connecting the memory 402 and the processor 401 .

[0070] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 401 executes the computer execution instructions, the above method is implemented.

[0071] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 401, the above method is implemented.

[0072] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0073] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0074] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0075] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 401 to perform some steps of the methods of various embodiments of the present application.

[0076] It should be understood that the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 401 may be any conventional processor 401, etc. The steps of the method disclosed in the invention may be directly embodied in the hardware processor 401 for execution, or may be executed by a combination of hardware and software modules in the processor 401.

[0077] The memory 402 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0078] The bus 403 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 403 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 403 in the drawings of the present application is not limited to only one bus 403 or one type of bus 403.

[0079] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0080] An exemplary storage medium is coupled to the processor 401, so that the processor 401 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 401. The processor 401 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 401 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0081] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A VVC-SCC intra-frame coding method based on multi-stage irregular coding unit division, characterized in that: The following steps are involved: Acquire a screen content video, process the screen content video to obtain a plurality of coding tree units, and divide each coding tree unit into a plurality of current coding units of a first size; A coding unit division prediction model based on a multi-stage convolutional neural network is constructed and trained to obtain a trained coding unit division prediction model, wherein the coding unit division prediction model includes five stages, each stage includes a residual module and a sub-network, and different network structures are designed for coding units in different stages; The current coding unit of the first size is encoded by using a VVC-SCC encoder, and a current quantization parameter is determined and normalized during the encoding process to obtain a normalized current quantization parameter, and the current coding unit of the first size and the normalized current quantization parameter are input into the trained coding unit division prediction model, and a prediction result of the current stage is obtained through current stage prediction, wherein the prediction result of the current stage includes prediction probabilities of different mode categories corresponding to the current coding unit input at the current stage during the encoding process, and the mode category includes a non-division mode and a division mode of dividing the current coding unit input at the current stage into coding units with a reduced size of one level; different thresholds are set for different mode categories, and whether to place the mode category in the rate-distortion optimization list is determined according to the prediction probability of each mode category and its corresponding threshold value. Specifically, the method includes: In response to determining that the prediction probability of the mode category m in the prediction result of the nth current coding unit at the current stage satisfies the following formula: in, represents the prediction probability of the mode category m in the prediction result of the nth current coding unit at the current stage, represents the prediction probability of the mode category i in the prediction result of the nth current coding unit at the current stage, i∈M, M represents the mode category set, Thr m represents the threshold value corresponding to mode category m, and max represents the maximum value. Then, it is determined that the mode category m in the prediction result of the nth current coding unit in the current stage is placed in the rate-distortion optimization list. Otherwise, it is determined not to place the mode category m in the prediction result of the nth current coding unit in the current stage in the rate-distortion optimization list. If the mode category is placed in the rate-distortion optimization list, the VVC-SCC encoder performs rate-distortion cost calculation on the mode category in the rate-distortion optimization list, and selects the mode category with the smallest rate-distortion cost as the optimal mode category of the current coding unit. The current coding unit input in the current stage is divided by the optimal mode category to obtain the divided coding units and input them into the next stage with the normalized current quantization parameter. After the next stage prediction, the prediction result of the next stage is obtained. The prediction result of the next stage is used as the prediction result of the current stage to repeat the above steps after obtaining the prediction result of the current stage. If the mode category is not placed in the rate-distortion optimization list, the rate-distortion cost calculation of the mode category is directly skipped.

2. The VVC-SCC intra-frame coding method based on multi-stage irregular coding unit division according to claim 1, characterized in that: The division modes include quadtree division mode, horizontal binary tree division mode, horizontal ternary tree division mode, vertical binary tree division mode and / or vertical ternary tree division mode. The division mode in the first stage of the coding unit division prediction model is the quadtree division mode. The division modes in the second and third stages of the coding unit division prediction model are the quadtree division mode, the horizontal binary tree division mode, the horizontal ternary tree division mode, the vertical binary tree division and the vertical ternary tree division mode. The division modes in the fourth and fifth stages of the coding unit division prediction model are the horizontal binary tree division mode, the horizontal ternary tree division mode, the vertical binary tree division and the vertical ternary tree division mode.

3. The VVC-SCC intra-frame coding method based on multi-stage irregular coding unit division according to claim 2, characterized in that: The threshold corresponding to the non-division mode is 1, the threshold corresponding to the quadtree division mode is 0.1, the threshold corresponding to the horizontal binary tree division mode is 0.7, the threshold corresponding to the horizontal ternary tree division mode is 0.05, the threshold corresponding to the vertical binary tree division mode is 0.05, and the threshold corresponding to the vertical ternary tree division mode is 0.

1.

4. The VVC-SCC intra-frame coding method based on multi-stage irregular coding unit division according to claim 1, characterized in that: In the coding unit partition prediction model, the first stage includes a first residual module and a first subnetwork, the first subnetwork includes a first feature extraction layer, a first convolution layer, a first fully connected layer, and a second fully connected layer connected in sequence, the first feature extraction layer is a first asymmetric convolution network, and the first asymmetric convolution network includes a convolution layer with a convolution kernel size of 9×5, a convolution layer with a convolution kernel size of 7×7, and a convolution layer with a convolution kernel size of 5×9 connected in sequence; the second stage includes a second residual module and a second subnetwork, the second subnetwork includes a second feature extraction layer, a second convolution layer, a third convolution layer, a third fully connected layer, and a fourth fully connected layer connected in sequence, the second feature extraction layer is a second asymmetric convolution network, and the second asymmetric convolution network includes a convolution layer with a convolution kernel size of 7×3, a convolution kernel size of 5×5, and a convolution kernel size of 3×7 connected in sequence; the third stage includes a third residual module and a third subnetwork, and the third subnetwork includes a third feature extraction layer, a fourth convolution layer, a fifth convolution layer, a fifth fully connected layer connected in sequence and the sixth fully connected layer, the third feature extraction layer is an adaptive convolution layer with a convolution kernel size of 4hs×4ws; the fourth stage includes a fourth residual module and a fourth sub-network, the fifth stage includes a fifth residual module and a fifth sub-network, the fourth sub-network and the fifth sub-network include a fourth feature extraction layer, a sixth convolution layer, a seventh fully connected layer and an eighth fully connected layer connected in sequence, and the fourth feature extraction layer is an adaptive convolution layer with a convolution kernel size of 2hs×2ws; wherein hs and ws are the height unit and width unit of the adaptive convolution layer, respectively, if When the current coding unit is rectangular and the width of the current coding unit is greater than the height, ws=(w / h), hs=1, where w and h are the width and height of the current coding unit; otherwise ws=1, hs=(h / w), when the current coding unit is square, ws=1, hs=1; the normalized current quantization parameter is respectively input to the first fully connected layer, the second fully connected layer, the third fully connected layer, the fourth fully connected layer, the fifth fully connected layer, the sixth fully connected layer, the seventh fully connected layer and the eighth fully connected layer, and the normalized current quantization parameter is expressed as: Wherein, q represents the current quantization parameter, is the normalized current quantization parameter; The first residual module, the second residual module, the third residual module, the fourth residual module, and the fifth residual module all adopt a residual structure, and the residual structure includes a seventh convolutional layer and an eighth convolutional layer connected in sequence, and the input features of the seventh convolutional layer are residually connected with the output features of the eighth convolutional layer.

5. The VVC-SCC intra-frame coding method based on multi-stage irregular coding unit division according to claim 1, characterized in that: The loss function used in the training process of the coding unit partition prediction model is as follows: Among them, L represents the loss function, N represents the batch training size, M represents the pattern category set, and y n,m and represents the true label and predicted label of the nth current coding unit when the mode category is m, R n,m is the rate-distortion cost of the nth current coding unit when the mode category is m, and α is the weight coefficient.

6. A VVC-SCC intra-frame coding device based on multi-stage irregular coding unit division, characterized in that: include: A data processing module is configured to obtain a screen content video, process the screen content video to obtain a plurality of coding tree units, and divide each coding tree unit into a plurality of current coding units of a first size; A model building module is configured to build and train a coding unit division prediction model based on a multi-stage convolutional neural network to obtain a trained coding unit division prediction model, wherein the coding unit division prediction model includes five stages, each stage includes a residual module and a sub-network, and different network structures are designed for coding units in different stages; The encoding module is configured to use a VVC-SCC encoder to encode the current coding unit of the first size, determine and normalize the current quantization parameter during the encoding process to obtain a normalized current quantization parameter, input the current coding unit of the first size and the normalized current quantization parameter into the trained coding unit division prediction model, obtain the prediction result of the current stage through current stage prediction, and the prediction result of the current stage includes the prediction probabilities of different mode categories corresponding to the current coding unit input at the current stage during the encoding process, and the mode category includes a non-division mode and a division mode of dividing the current coding unit input at the current stage into coding units with a reduced size of one level; different thresholds are set for different mode categories, and whether to place the mode category in the rate-distortion optimization list is determined according to the prediction probability of each mode category and its corresponding threshold, specifically including: In response to determining that the prediction probability of the mode category m in the prediction result of the nth current coding unit at the current stage satisfies the following formula: in, represents the prediction probability of the mode category m in the prediction result of the nth current coding unit at the current stage, represents the prediction probability of the mode category i in the prediction result of the nth current coding unit at the current stage, i∈M, M represents the mode category set, Thr m represents the threshold value corresponding to mode category m, and max represents the maximum value. Then, it is determined that the mode category m in the prediction result of the nth current coding unit in the current stage is placed in the rate-distortion optimization list. Otherwise, it is determined not to place the mode category m in the prediction result of the nth current coding unit in the current stage in the rate-distortion optimization list. If the mode category is placed in the rate-distortion optimization list, the VVC-SCC encoder performs rate-distortion cost calculation on the mode category in the rate-distortion optimization list, and selects the mode category with the smallest rate-distortion cost as the optimal mode category of the current coding unit. The current coding unit input in the current stage is divided by the optimal mode category to obtain the divided coding units and input them into the next stage with the normalized current quantization parameter. After the next stage prediction, the prediction result of the next stage is obtained. The prediction result of the next stage is used as the prediction result of the current stage to repeat the above steps after obtaining the prediction result of the current stage. If the mode category is not placed in the rate-distortion optimization list, the rate-distortion cost calculation of the mode category is directly skipped.

7. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images