Fast h.266VVC screen content intra-frame CU division method and apparatus based on multi-task learning

Abstract

The present invention relates to the field of video coding. Disclosed are a fast H.266VVC screen content intra-frame CU division method and apparatus based on multi-task learning. The method comprises: acquiring a screen content video, and directly dividing a 128×128-sized CTU into 64×64-sized CUs; constructing a multi-task learning network model, wherein the multi-task learning network model comprises a backbone network, a first sub-network and a second sub-network, the backbone network being used for extracting CU features; inputting the CU features into the first sub-network and the second sub-network to obtain a CU division type and a coding mode, and comprehensively determining a prediction result in view of the coding mode and a prediction probability thereof, and the division type of an adjacent CU; inputting the 64×64-sized CUs into the trained multi-task learning network model, so as to obtain a first prediction result; and if the first prediction result indicates division, further dividing each 64×64-sized CU into four 32×32-sized CUs, and inputting the 32×32-sized CUs into the trained multi-task learning network model, so as to obtain a second prediction result. The problem of the complexity of H.266VVC screen content intra-frame coding being high is solved.

Description

Method and device for fast CU segmentation within H.266 VVC screen content frame based on multi-task learning Technical Field

[0001] The present invention relates to the field of video coding, and in particular to a method and device for fast CU segmentation within an H.266 VVC screen content frame based on multi-task learning. Background Art

[0002] The rapid development of multimedia communication technologies and video terminal devices has placed higher demands on screen video coding technology. Because H.265 / HEVC-SCC no longer meets the performance requirements for ultra-high-definition screen video compression, the Moving Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG) established JVET to develop the next-generation video coding standard H.266VVC. Early versions of H.266VVC incorporated coding technology for screen content video.

[0003] Compared with H.265 / HEVC-SCC, H.266VVC achieves higher coding efficiency. H.266VVC adds four CU division methods, including horizontal binary tree, horizontal ternary tree, vertical binary tree, and vertical ternary tree. The CU faces 6 choices. The standard encoder needs to execute all 5781 possibilities once and record their costs. Finally, it compares which combination has the lowest cost as the final division result. In addition, H.266VVC also introduces new coding technologies for screen content videos, such as IBC and PLT coding modes. These two coding modes also affect the CU division method. The flexible CU division method and special coding mode in H.266VVC significantly improve the coding performance, but at the same time increase the computational complexity.

[0004] Therefore, how to effectively reduce the complexity of screen content encoding while maintaining the H.266 VVC encoding performance has become an urgent problem that H.266 VVC needs to solve.

[0005] Summary of the Invention

[0006] In response to the technical problem of high complexity of intra-frame coding of H.266 VVC screen content mentioned above, the purpose of the embodiments of this application is to propose a method and device for fast CU division within H.266 VVC screen content frames based on multi-task learning to solve the technical problems mentioned in the above background technology section, and use coding mode information to assist in deciding the CU division type, so as to effectively reduce the encoder computational complexity without affecting the coding efficiency.

[0007] In a first aspect, the present invention provides a method for fast CU segmentation within an H.266 VVC screen content frame based on multi-task learning, comprising the following steps:

[0008] Obtain the screen content video, encode it using a standard encoder, and directly divide the 128×128 coding tree unit (CTU) into 64×64 coding units (CU);

[0009] Constructing a multi-task learning network model and training it to obtain a trained multi-task learning network model, the multi-task learning network model including a backbone network, a first sub-network, and a second sub-network, the first sub-network and the second sub-network being connected to the backbone network respectively, the backbone network being used to extract CU features, inputting the CU features into the first sub-network, predicting the CU partition type and its corresponding prediction probability, inputting the CU features into the second sub-network, predicting the coding mode and its corresponding prediction probability, using the CU partition type as a prediction result or comprehensively determining the prediction result based on the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU;

[0010] During the encoding process of the standard encoder, a trained multi-task learning network model is called, a CU of size 64×64 is input into the trained multi-task learning network model, a first prediction result is obtained, and CU division is performed according to the first prediction result; in response to determining that the first prediction result is division, the CU of size 64×64 is divided into four CUs of size 32×32, the CU of size 32×32 is input into the trained multi-task learning network model, a second prediction result is obtained, and CU division is performed according to the second prediction result;

[0011] CU division is performed according to the first prediction result, specifically including:

[0012] In response to determining that the CU partition type of the first prediction result is not partitioned, terminating the rate-distortion optimization search process;

[0013] In response to determining that the CU partition type of the first prediction result is split, the 64×64 CU is divided into four 32×32 CUs;

[0014] CU division is performed according to the second prediction result, specifically including:

[0015] In response to determining that the CU partition type of the second prediction result is not partitioned, terminating the rate-distortion optimization search process;

[0016] In response to determining that the CU partition type of the second prediction result is quadtree partitioning, four CUs of 16×16 size are obtained;

[0017] In response to determining that the CU partition type of the second prediction result is horizontal binary tree partitioning, two CUs of 16×32 size are obtained;

[0018] In response to determining that the CU partition type of the second prediction result is vertical binary tree partitioning, two CUs of 32×16 size are obtained;

[0019] In response to determining that the CU partition type of the second prediction result is horizontal ternary tree partitioning, two CUs of 8×32 size and one CU of 16×32 size are obtained;

[0020] In response to determining that the CU partition type of the second prediction result is vertical ternary tree partition, two CUs of 32×8 size and one CU of 32×16 size are obtained.

[0021] Preferably, the backbone network includes a first convolutional layer, a second convolutional layer, a first pooling layer, a third convolutional layer, a fourth convolutional layer, and a second pooling layer connected in sequence. The convolution kernel size of the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are all 3×3, the step size is 1, the padding is 1, and the number of channels are 64, 64, 128, and 128 respectively.

[0022] Preferably, the first subnetwork includes a fifth convolutional layer, a sixth convolutional layer and three first fully connected layers connected in sequence, the convolution kernel size of the fifth convolutional layer and the sixth convolutional layer are both 1×1, the step size is 1, the padding is 1, the number of channels is 256 and 256 respectively, the number of neurons in the three first fully connected layers is 16384, 512, 2 or 6 respectively, and the dropout ratio is 0.3.

[0023] Preferably, the second subnetwork includes a seventh convolutional layer, an eighth convolutional layer and three second fully connected layers connected in sequence. The convolution kernel size of the seventh convolutional layer and the eighth convolutional layer are both 1×1, the step size is 1, the padding is 1, the number of channels are 256 and 256 respectively, the number of neurons in the three second fully connected layers are 16384, 512, and 4 respectively, and the dropout ratio is 0.25.

[0024] Preferably, the CU partition type is used as the prediction result or the prediction result is comprehensively determined according to the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU, specifically including:

[0025] In response to determining that there is no conflict between the CU partition type and the coding mode, taking the CU partition type as the prediction result;

[0026] In response to determining that there is a contradiction between the CU partition type and the coding mode, a joint judgment is performed based on the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU to determine the prediction result, specifically including:

[0027] In response to determining that the CU partition type is not partitioned and the coding mode is not allocated, a judgment is made based on the prediction probability corresponding to the coding mode to determine whether the prediction probability of the coding mode is greater than the threshold and greater than the prediction probability corresponding to the CU partition type, and the CUs to the left and above this CU are both partitioned. If so, the CU partition type with the largest prediction probability is selected as the prediction result, otherwise the CU partition type in the prediction result is not partitioned;

[0028] In response to determining that the CU partition type is partition and the encoding mode is other than the non-allocation mode, it is determined whether the prediction probability corresponding to the CU partition type is greater than the threshold and greater than the prediction probability corresponding to the encoding mode. If so, the CU partition type in the prediction result is partition; otherwise, the CU partition type in the prediction result is partition.

[0029] Preferably, the loss function used in the training process of the multi-task learning network model is as follows:

[0030] Among them, α represents the weight of the main task CU division, β represents the weight of the auxiliary task coding mode, and w1 represents the CU division type q CU The proportion of CU partition type q CU The corresponding values ​​are labels 0, 1 or 0, 1, 2, 3, 4, 5, p for CUs of different sizes. CU Indicates CU partition type q CU Corresponding prediction probability; w2 represents the coding mode q M The proportion of coding mode q M The corresponding value is the CU coding mode label 0, 1, 2, 3, p M Indicates the encoding mode q M The corresponding prediction probability, N represents the number of batches of training samples.

[0031] In a second aspect, the present invention provides a device for fast intra-frame CU segmentation of H.266 VVC screen content based on multi-task learning, comprising:

[0032] The encoding module is configured to obtain a screen content video, encode the screen content video using a standard encoder, and directly divide the 128×128 coding tree unit CTU into a 64×64 coding unit CU;

[0033] A model construction module is configured to construct and train a multi-task learning network model to obtain a trained multi-task learning network model, wherein the multi-task learning network model includes a backbone network, a first sub-network, and a second sub-network, wherein the first sub-network and the second sub-network are respectively connected to the backbone network, the backbone network is used to extract CU features, input the CU features into the first sub-network, predict the CU partition type and its corresponding prediction probability, input the CU features into the second sub-network, predict the coding mode and its corresponding prediction probability, use the CU partition type as a prediction result, or comprehensively determine the prediction result based on the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU;

[0034] The prediction module is configured to call a trained multi-task learning network model during the encoding process of the standard encoder, input a 64×64 CU into the trained multi-task learning network model, obtain a first prediction result, and perform CU division according to the first prediction result; in response to determining that the first prediction result is a division, divide the 64×64 CU into four 32×32 CUs, input the 32×32 CU into the trained multi-task learning network model, obtain a second prediction result, and perform CU division according to the second prediction result.

[0035] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0036] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) The proposed method for fast intra-frame CU segmentation in H.266 VVC screen content based on multi-task learning predicts CU segmentation through a multi-task learning network model. By mining the correlation between coding modes and CU segmentation and using coding modes to supervise CU segmentation, the prediction accuracy is effectively improved, some unnecessary cost calculations can be skipped, and coding complexity is greatly reduced without affecting coding efficiency and video quality.

[0039] (2) The multi-task learning network model of the H.266VVC screen content intra-frame fast CU segmentation method based on multi-task learning proposed in the present invention extracts CU features through the backbone network, and then uses the first sub-network and the second sub-network to predict the CU segmentation type and encoding method respectively. When the two results of the CU segmentation type and the encoding method conflict, the final CU segmentation type is determined by combining the prediction probability and the segmentation type of the adjacent CU to ensure the accuracy of the prediction result.

[0040] (3) The multi-task learning network model of the H.266 VVC screen content intra-frame CU fast segmentation method based on multi-task learning proposed in the present invention adopts pooling layers and 1×1 convolutions, which has a fast operation time and is easy to deploy on portable devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] FIG1 is a diagram of an exemplary device architecture in which an embodiment of the present application may be applied;

[0043] FIG2 is a flowchart of a method for fast intra-frame CU segmentation of H.266 VVC screen content based on multi-task learning according to an embodiment of the present application;

[0044] FIG3 is a schematic diagram of the structure of a multi-task learning network model of a method for fast intra-CU segmentation of H.266 VVC screen content frames based on multi-task learning according to an embodiment of the present application;

[0045] FIG4 is a schematic diagram of the encoding process of the H.266 VVC screen content intra-frame CU fast partitioning method based on multi-task learning according to an embodiment of the present application;

[0046] FIG5 is a schematic diagram of a device for fast intra-frame CU segmentation of H.266 VVC screen content based on multi-task learning according to an embodiment of the present application;

[0047] FIG6 is a schematic diagram of the structure of a computer device suitable for implementing an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0049] FIG1 shows an exemplary device architecture 100 to which an H.266 VVC screen content intra-frame CU fast partitioning method based on multi-task learning or an H.266 VVC screen content intra-frame CU fast partitioning device based on multi-task learning can be applied.

[0050] As shown in Figure 1, an apparatus architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 is used as a medium for providing communication links between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0051] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications, such as data processing applications and file processing applications, can be installed on terminal devices 101, 102, 103.

[0052] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitations are given here.

[0053] The server 105 may be a server that provides various services, such as a background data processing server that processes files or data uploaded by the terminal devices 101, 102, and 103. The background data processing server may process the acquired files or data and generate processing results.

[0054] It should be noted that the method for fast CU division within the H.266 VVC screen content frame based on multi-task learning provided in the embodiment of the present application can be executed by the server 105, or by the terminal devices 101, 102, and 103. Accordingly, the device for fast CU division within the H.266 VVC screen content frame based on multi-task learning can be set in the server 105, or in the terminal devices 101, 102, and 103.

[0055] It should be understood that the number of terminal devices, networks, and servers in FIG1 is merely illustrative. Any number of terminal devices, networks, and servers may be provided as needed. If the processed data does not need to be acquired remotely, the above-described apparatus architecture may not include a network, but only servers or terminal devices.

[0056] FIG2 shows a method for fast CU segmentation within an H.266 VVC screen content frame based on multi-task learning provided by an embodiment of the present application, comprising the following steps:

[0057] S1, obtain the screen content video, use a standard encoder to encode the screen content video, and directly divide the 128×128 coding tree unit CTU into 64×64 coding units CU.

[0058] Specifically, during the encoding process of the standard encoder, a 128×128 CTU is first directly divided into 64×64 CUs. A neural network-based partitioning method is then used to predict the partitioning type of the subsequent 64×64 CUs and 32×32 CUs, significantly reducing encoding complexity. The specific neural network structure is described later.

[0059] S2. Construct a multi-task learning network model and train it to obtain a trained multi-task learning network model. The multi-task learning network model includes a backbone network, a first sub-network, and a second sub-network. The first sub-network and the second sub-network are respectively connected to the backbone network. The backbone network is used to extract CU features, input the CU features into the first sub-network, predict the CU partition type and its corresponding prediction probability, input the CU features into the second sub-network, predict the coding mode and its corresponding prediction probability, use the CU partition type as the prediction result, or comprehensively determine the prediction result based on the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU.

[0060] In a specific embodiment, the backbone network includes a first convolutional layer, a second convolutional layer, a first pooling layer, a third convolutional layer, a fourth convolutional layer, and a second pooling layer connected in sequence. The convolution kernel size of the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are all 3×3, the step size is 1, the padding is 1, and the number of channels is 64, 64, 128, and 128 respectively.

[0061] In a specific embodiment, the first sub-network includes a fifth convolutional layer, a sixth convolutional layer and three first fully connected layers connected in sequence. The convolution kernel size of the fifth convolutional layer and the sixth convolutional layer is 1×1, the stride is 1, the padding is 1, the number of channels is 256 and 256 respectively, the number of neurons in the three first fully connected layers is 16384, 512, 2 or 6 respectively, and the dropout ratio is 0.3.

[0062] In a specific embodiment, the second sub-network includes a seventh convolutional layer, an eighth convolutional layer and three second fully connected layers connected in sequence. The convolution kernel size of the seventh convolutional layer and the eighth convolutional layer is 1×1, the step size is 1, the padding is 1, the number of channels is 256 and 256 respectively, the number of neurons in the three second fully connected layers is 16384, 512, and 4 respectively, and the dropout ratio is 0.25.

[0063] Specifically, referring to Figure 3, the multi-task learning network model includes a backbone network and two sub-networks: a first sub-network and a second sub-network. The first sub-network is used to predict the CU partition type and its corresponding prediction probability, serving as the primary task; the second sub-network is used to predict the coding mode and its corresponding prediction probability, serving as an auxiliary task. The coding mode predicted by the auxiliary task can supervise the CU partition type prediction results, improving the accuracy of the prediction results.

[0064] In a specific embodiment, the loss function used in the training process of the multi-task learning network model is shown in the following formula:

[0065] Among them, α represents the weight of the main task CU division, β represents the weight of the auxiliary task coding mode, and w1 represents the CU division type q CU The proportion of CU partition type q CU The corresponding values ​​are labels 0, 1 or 0, 1, 2, 3, 4, 5, p for CUs of different sizes. CU Indicates CU partition type q CU Corresponding prediction probability; w2 represents the coding mode q M The proportion of coding mode q M The corresponding value is the CU coding mode label 0, 1, 2, 3, p M Indicates the encoding mode q MThe corresponding prediction probability, N represents the number of batches of training samples.

[0066] Specifically, the training process of the multi-task learning network model is as follows:

[0067] (1) Obtaining true labels. Collect screen content videos, encode them using a standard encoder, and count the CU partition type and coding mode information. Different CU partition types and coding modes are assigned labels. 64×64 CUs have two partition labels: 0 for no partition, 1 for quadtree partition, and four coding mode labels: 0 for no allocation mode, 1 for intra, 2 for inter-blockchain (IBC), and 3 for polynomial (PLT). 32×32 CUs have six partition labels: 0 for no partition, 1 for quadtree partition, 2 for horizontal binary tree partition, 3 for vertical binary tree partition, 4 for horizontal ternary tree partition, and 5 for vertical ternary tree partition. Four coding mode labels: 0 for no allocation mode, 1 for intra, 2 for inter-blockchain (IBC), and 3 for polynomial (PLT). 64×64 and 32×32 CUs are used as training, validation, and test sets, and are randomly assigned in a ratio of 8:1:1.

[0068] (2) Considering the imbalance in the CU partition ratio and mode selection ratio, a weighted loss function is designed for each sub-network. The weight α of the main task CU partition and the weight β of the auxiliary task encoding mode change with the accuracy of the validation set during training. During the entire training process, the main task CU partition needs to be trained to near convergence first, and then the auxiliary task mode selection needs to converge, and finally the main task CU partition will be converged.

[0069] During training, the Adam algorithm was used as the optimization function, with a total of 20,000 iterations. The initial learning rate was 0.0001. From 0 to 10,000 iterations, the learning rate was decreased by 10% every 1,000 iterations. From 10,001 to 20,000 iterations, the learning rate was decreased by 10% every 500 iterations. The batch size for both the training and validation sets was 256. When the CU partitioning accuracy reached approximately 60%, the weight of the auxiliary task loss function was increased. When the auxiliary task accuracy reached approximately 70%, its weight was adjusted again, while the weight of the main task was increased. During training, the learning rate was decreased by 10% every 1,000 iterations.

[0070] In a specific embodiment, the CU partition type is used as a prediction result or the prediction result is determined comprehensively based on the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU, specifically including:

[0071] In response to determining that there is no conflict between the CU partition type and the coding mode, taking the CU partition type as the prediction result;

[0072] In response to determining that there is a contradiction between the CU partition type and the coding mode, a joint judgment is performed based on the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU to determine the prediction result, specifically including:

[0073] In response to determining that the CU partition type is not partitioned and the coding mode is not allocated, a judgment is made based on the prediction probability corresponding to the coding mode to determine whether the prediction probability of the coding mode is greater than the threshold and greater than the prediction probability corresponding to the CU partition type, and the CUs to the left and above this CU are both partitioned. If so, the CU partition type with the largest prediction probability is selected as the prediction result, otherwise the CU partition type in the prediction result is not partitioned;

[0074] In response to determining that the CU partition type is partition and the encoding mode is other than the non-allocation mode, it is determined whether the prediction probability corresponding to the CU partition type is greater than the threshold and greater than the prediction probability corresponding to the encoding mode. If so, the CU partition type in the prediction result is partition; otherwise, the CU partition type in the prediction result is partition.

[0075] Specifically, under normal circumstances, the coding mode corresponding to the CU partition type is the non-allocation mode, and the coding mode corresponding to the CU partition type is the allocation Intra, IBC, and PLT modes. When the coding mode and the CU partition type are contradictory, it is necessary to make a joint judgment based on the predicted probability and the partition type of the adjacent CU. In one embodiment, the threshold is set to 0.8. The first contradictory situation is: when the CU partition type predicted by the first sub-network is non-division, but the coding mode predicted by the second sub-network is non-allocation mode, which is contrary to the actual coding situation, it is necessary to make a judgment based on the predicted probability P corresponding to the coding mode. mode Make a judgment, when the predicted probability P mode Greater than 0.8 and greater than the predicted probability P corresponding to the CU partition type split , and when the CU to the left and above this CU are both divided, the prediction result of determining that the CU is not divided is invalid, and the CU division type corresponding to the maximum prediction probability is selected as the prediction result. The second contradictory situation is: when the CU division type predicted by the first sub-network is division, and the coding mode predicted by the second sub-network is one of Intra, IBC, and PLT, which is contrary to the actual coding situation, the prediction probability P corresponding to the CU division type is required. split Greater than 0.8 and greater than the predicted probability P corresponding to the coding mode mode , can determine that the CU partition type of the prediction result is partition.

[0076] S3, during the encoding process of the standard encoder, calling the trained multi-task learning network model, inputting the 64×64 CU into the trained multi-task learning network model, obtaining a first prediction result, and performing CU division according to the first prediction result; in response to determining that the first prediction result is a division, dividing the 64×64 CU into 4 32×32 CUs, inputting the 32×32 CU into the trained multi-task learning network model, obtaining a second prediction result, and performing CU division according to the second prediction result.

[0077] In a specific embodiment, performing CU division according to the first prediction result specifically includes:

[0078] In response to determining that the CU partition type of the first prediction result is not partitioned, terminating the rate-distortion optimization search process;

[0079] In response to determining that the CU partition type of the first prediction result is split, the 64×64 CU is divided into four 32×32 CUs;

[0080] CU division is performed according to the second prediction result, specifically including:

[0081] In response to determining that the CU partition type of the second prediction result is not partitioned, terminating the rate-distortion optimization search process;

[0082] In response to determining that the CU partition type of the second prediction result is quadtree partitioning, four CUs of 16×16 size are obtained;

[0083] In response to determining that the CU partition type of the second prediction result is horizontal binary tree partitioning, two CUs of 16×32 size are obtained;

[0084] In response to determining that the CU partition type of the second prediction result is vertical binary tree partitioning, two CUs of 32×16 size are obtained;

[0085] In response to determining that the CU partition type of the second prediction result is horizontal ternary tree partitioning, two CUs of 8×32 size and one CU of 16×32 size are obtained;

[0086] In response to determining that the CU partition type of the second prediction result is vertical ternary tree partition, two CUs of 32×8 size and one CU of 32×16 size are obtained.

[0087] Specifically, referring to Figure 4, during the encoding process, the trained multi-task learning network model is called and a 64×64 CU is input into the trained multi-task learning network model for prediction. The predicted probabilities of its CU partition type and coding mode are obtained. The predicted probabilities and the partitioning methods of adjacent CUs are combined to obtain a first prediction result, and CU partitioning is performed based on the CU partition type in the first prediction result. Specifically, if the CU partition type of the first prediction result is not partitioned, the rate-distortion optimization search process is terminated; if the CU partition type of the first prediction result is partitioned, the 64×64 CU is partitioned into four 32×32 CUs.

[0088] Furthermore, in response to confirming that the first prediction result is not divided, the encoding process is exited; in response to confirming that the first prediction result is divided, the 64×64 CU is divided into 4 32×32 CUs, and the trained multi-task learning network model is input for prediction, and the second prediction result is obtained by comprehensively considering the prediction probability and the division method of the adjacent CUs, and CU division is performed according to the CU division type in the second prediction result. Specifically, if the CU partition type of the second prediction result is not divided, the rate-distortion optimization search process is terminated; if the CU partition type of the second prediction result is quadtree partitioning, 4 CUs of 16×16 size are obtained; if the CU partition type of the second prediction result is horizontal binary tree partitioning, 2 CUs of 16×32 size are obtained; if the CU partition type of the second prediction result is vertical binary tree partitioning, 2 CUs of 32×16 size are obtained; if the CU partition type of the second prediction result is horizontal ternary tree partitioning, 2 CUs of 8×32 size and 1 CU of 16×32 size are obtained; if the CU partition type of the second prediction result is vertical ternary tree partitioning, 2 CUs of 32×8 size and 1 CU of 32×16 size are obtained.

[0089] The present invention predicts the division mode of the two CUs based on a multi-task learning network model, skips some unnecessary cost calculations, and greatly reduces the coding complexity of the screen content in VVC without affecting the coding efficiency.

[0090] The above steps S1-S3 do not merely represent the order of the steps, but are symbolic representations of the steps.

[0091] Further referring to Figure 5, as an implementation of the methods shown in the above figures, the present application provides an embodiment of a device for fast CU segmentation within an H.266VVC screen content frame based on multi-task learning. The device embodiment corresponds to the method embodiment shown in Figure 2, and the device can be specifically applied to various electronic devices.

[0092] The present embodiment provides a device for quickly dividing CUs within an H.266 VVC screen content frame based on multi-task learning, and applies a method for quickly dividing CUs within an H.266 VVC screen content frame based on multi-task learning in the aforementioned embodiment, including:

[0093] The encoding module 1 is configured to obtain a screen content video, encode the screen content video using a standard encoder, and directly divide the 128×128 coding tree unit CTU into a 64×64 coding unit CU;

[0094] Model construction module 2 is configured to construct and train a multi-task learning network model to obtain a trained multi-task learning network model, where the multi-task learning network model includes a backbone network, a first sub-network, and a second sub-network, where the first sub-network and the second sub-network are respectively connected to the backbone network, the backbone network is used to extract CU features, input the CU features into the first sub-network, predict the CU partition type and its corresponding prediction probability, input the CU features into the second sub-network, predict the coding mode and its corresponding prediction probability, use the CU partition type as a prediction result, or comprehensively determine the prediction result based on the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU;

[0095] The prediction module 3 is configured to call the trained multi-task learning network model during the encoding process of the standard encoder, input the 64×64 CU into the trained multi-task learning network model, obtain a first prediction result, and perform CU division according to the first prediction result; in response to determining that the first prediction result is a division, divide the 64×64 CU into 4 32×32 CUs, input the 32×32 CU into the trained multi-task learning network model, obtain a second prediction result, and perform CU division according to the second prediction result.

[0096] Referring now to FIG6 , a schematic diagram illustrating the structure of a computer device 600 suitable for implementing an electronic device (e.g., the server or terminal device shown in FIG1 ) according to an embodiment of the present application is shown. The electronic device shown in FIG6 is merely an example and should not limit the functionality or scope of use of the embodiments of the present application.

[0097] As shown in FIG6 , a computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 603 or a program loaded from a storage unit 609 into a random access memory (RAM) 604. Various programs and data required for the operation of the device 600 are also stored in the RAM 604. The CPU 601, GPU 602, ROM 603, and RAM 604 are connected to each other via a bus 605. An input / output (I / O) interface 606 is also connected to the bus 605.

[0098] The following components are connected to the I / O interface 606: an input section 607 including a keyboard, a mouse, and the like; an output section 608 including a display such as a liquid crystal display (LCD), a speaker, and the like; a storage section 609 including a hard disk and the like; and a communication section 610 including a network interface card such as a LAN card or a modem. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 may also be connected to the I / O interface 606 as needed. A removable medium 612, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 611 as needed, so that a computer program read therefrom can be installed into the storage section 609 as needed.

[0099] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion 610, and / or installed from a removable medium 612. When the computer program is executed by the central processing unit (CPU) 601 and the graphics processing unit (GPU) 602, the above-mentioned functions defined in the method of the present application are performed.

[0100] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable medium, or any combination thereof. Computer-readable media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, apparatuses, or components, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or component. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution apparatus, device, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0101] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based device that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] The modules involved in the embodiments described in this application may be implemented in software or hardware, and may also be set in a processor.

[0104] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device: obtains the screen content video, encodes the screen content video using a standard encoder, and directly divides the 128×128 coding tree unit CTU into 64×64 coding units CU; constructs a multi-task learning network model and trains it to obtain a trained multi-task learning network model. The multi-task learning network model includes a backbone network, a first sub-network, and a second sub-network. The first sub-network and the second sub-network are respectively connected to the backbone network. The backbone network is used to extract CU features, input the CU features into the first sub-network, predict the CU partition type and its corresponding prediction probability, and input the CU features into the second sub-network. The coding mode and its corresponding prediction probability are predicted, and the CU partition type is used as the prediction result, or the prediction result is comprehensively determined according to the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU; in the encoding process of the standard encoder, the trained multi-task learning network model is called, and the 64×64 CU is input into the trained multi-task learning network model to obtain a first prediction result, and CU partitioning is performed according to the first prediction result; in response to determining that the first prediction result is a partition, the 64×64 CU is divided into 4 32×32 CUs, and the 32×32 CU is input into the trained multi-task learning network model to obtain a second prediction result, and CU partitioning is performed according to the second prediction result.

[0105] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application. Industrial Applicability

[0106] The present invention provides a method and device for fast CU segmentation within an H.266 VVC screen content frame based on multi-task learning. By acquiring a screen content video, a 128×128 CTU is directly divided into 64×64 CUs. A multi-task learning network model is constructed, which includes a backbone network, a first sub-network, and a second sub-network. The backbone network is used to extract CU features. The CU features are input into the first and second sub-networks to obtain the CU segmentation type and coding mode. The prediction result can be comprehensively determined by combining the coding mode and its prediction probability with the segmentation type of adjacent CUs. The 64×64 CU is input into the trained multi-task learning network model to obtain a first prediction result. If the first prediction result is a segmentation, it is further divided into four 32×32 CUs, and the four 32×32 CUs are input into the trained multi-task learning network model to obtain a second prediction result. This solves the problem of high complexity in H.266 VVC screen content intra-frame coding and has good industrial applicability.

Claims

1. A method for fast intra-frame CU segmentation of H.266 VVC screen content based on multi-task learning, characterized in that: The following steps are involved: Acquire a screen content video, encode the screen content video using a standard encoder, and directly divide a coding tree unit CTU of a size of 128×128 into a coding unit CU of a size of 64×64; Constructing a multi-task learning network model and training it to obtain a trained multi-task learning network model, wherein the multi-task learning network model includes a backbone network, a first sub-network and a second sub-network, wherein the first sub-network and the second sub-network are respectively connected to the backbone network, wherein the backbone network is used to extract CU features, input the CU features into the first sub-network, predict the CU partition type and its corresponding prediction probability, input the CU features into the second sub-network, predict the coding mode and its corresponding prediction probability, use the CU partition type as a prediction result, or comprehensively determine the prediction result according to the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU; In the encoding process of the standard encoder, the trained multi-task learning network model is called, the CU of the size of 64×64 is input into the trained multi-task learning network model, a first prediction result is obtained, and CU division is performed according to the first prediction result; in response to determining that the first prediction result is division, the CU of the size of 64×64 is divided into four CUs of the size of 32×32, the CU of the size of 32×32 is input into the trained multi-task learning network model, a second prediction result is obtained, and CU division is performed according to the second prediction result; The performing CU division according to the first prediction result specifically includes: In response to determining that the CU partition type of the first prediction result is not partitioned, terminating the rate-distortion optimization search process; In response to determining that the CU partition type of the first prediction result is partition, the CU of size 64×64 is divided into four CUs of size 32×32; The performing CU division according to the second prediction result specifically includes: In response to determining that the CU partition type of the second prediction result is not partitioned, terminating the rate-distortion optimization search process; In response to determining that the CU partition type of the second prediction result is quadtree partition, four CUs of size 16×16 are obtained; In response to determining that the CU partition type of the second prediction result is horizontal binary tree partition, two CUs of size 16×32 are obtained; In response to determining that the CU partition type of the second prediction result is vertical binary tree partition, two CUs of size 32×16 are obtained; In response to determining that the CU partition type of the second prediction result is horizontal ternary tree partition, two 8×32-sized CUs are obtained. CU and 1 CU of size 16×32; In response to determining that the CU partition type of the second prediction result is vertical ternary tree partition, two CUs of 32×8 size and one CU of 32×16 size are obtained.

2. The method for fast intra-frame CU segmentation of H.266 VVC screen content based on multi-task learning according to claim 1, characterized in that: The backbone network includes a first convolutional layer, a second convolutional layer, a first pooling layer, a third convolutional layer, a fourth convolutional layer, and a second pooling layer connected in sequence. The convolution kernel size of the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer is 3×3, the step size is 1, the padding is 1, and the number of channels is 64, 64, 128, and 128, respectively.

3. The method for fast intra-frame CU segmentation of H.266 VVC screen content based on multi-task learning according to claim 1, characterized in that: The first subnetwork includes a fifth convolutional layer, a sixth convolutional layer and three first fully connected layers connected in sequence, the convolution kernel size of the fifth convolutional layer and the sixth convolutional layer are both 1×1, the step size is 1, the padding is 1, the number of channels is 256 and 256 respectively, the number of neurons in the three first fully connected layers is 16384, 512, 2 or 6 respectively, and the dropout ratio is 0.

3.

4. The method for fast intra-frame CU segmentation of H.266 VVC screen content based on multi-task learning according to claim 1, characterized in that: The second sub-network includes a seventh convolutional layer, an eighth convolutional layer and three second fully connected layers connected in sequence, the convolution kernel size of the seventh convolutional layer and the eighth convolutional layer are both 1×1, the step size is 1, the padding is 1, the number of channels is 256 and 256 respectively, the number of neurons in the three second fully connected layers are 16384, 512, and 4 respectively, and the dropout ratio is 0.

25.

5. The method for fast intra-frame CU segmentation of H.266 VVC screen content based on multi-task learning according to claim 1, characterized in that: The taking the CU partition type as a prediction result or comprehensively determining the prediction result according to the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU specifically includes: In response to determining that there is no contradiction between the CU partition type and the encoding mode, taking the CU partition type as a prediction result; In response to determining that there is a contradiction between the CU partition type and the coding mode, a joint judgment is performed according to the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU to determine the prediction result, specifically including: In response to determining that the CU partition type is not partitioned and the coding mode is a non-allocation mode, a judgment is made according to the prediction probability corresponding to the coding mode to determine whether the prediction probability of the coding mode is greater than a threshold and greater than the prediction probability corresponding to the CU partition type, and the left and upper CUs of this CU are both partitioned, if so, the CU partition type with the largest prediction probability is selected as the prediction result, otherwise the CU partition type in the prediction result is not partitioned; In response to determining that the CU partition type is partition, the encoding mode is other than the non-allocation mode, Determine whether the prediction probability corresponding to the CU partition type is greater than a threshold and greater than the prediction probability corresponding to the encoding mode. If so, the CU partition type in the prediction result is partition; otherwise, the CU partition type in the prediction result is partition.

6. The method for fast intra-frame CU segmentation of H.266 VVC screen content based on multi-task learning according to claim 1, characterized in that: The loss function used in the training process of the multi-task learning network model is shown in the following formula: Among them, a represents the weight of the main task CU division, b represents the weight of the auxiliary task coding mode, and w1 represents the CU division type q CU The proportion of CU partition type q CU The corresponding values ​​are labels of CUs of different sizes 0, 1 or 0, 1, 2, 3, 4, 5, p CU Indicates CU partition type q CU The corresponding prediction probability; w2 represents the coding mode q M The proportion of coding mode q M The corresponding value is the coding mode label of CU 0, 1, 2, 3, p M Indicates the encoding mode q M The corresponding prediction probability, N represents the number of batches of training samples.

7. A device for fast CU segmentation within a H.266 VVC screen content frame based on multi-task learning, applying the method described in any one of claims 1 to 6, characterized in that: include: The encoding module is configured to obtain a screen content video, encode the screen content video using a standard encoder, and directly divide a coding tree unit CTU of a size of 128×128 into a coding unit CU of a size of 64×64; A model construction module is configured to construct and train a multi-task learning network model to obtain a trained multi-task learning network model, wherein the multi-task learning network model includes a backbone network, a first sub-network, and a second sub-network, wherein the first sub-network and the second sub-network are respectively connected to the backbone network, wherein the backbone network is used to extract CU features, input the CU features into the first sub-network, predict the CU partition type and its corresponding prediction probability, input the CU features into the second sub-network, predict the coding mode and its corresponding prediction probability, use the CU partition type as a prediction result, or comprehensively determine the prediction result based on the CU partition type and its corresponding prediction probability, the coding mode and its corresponding prediction probability, and the partition type of the adjacent CU; The prediction module is configured to call the trained multi-task learning network model during the encoding process of the standard encoder, input the 64×64 CU into the trained multi-task learning network model, obtain a first prediction result, and perform CU division according to the first prediction result; in response to determining that the first prediction result is a division, divide the 64×64 CU into 4 32×32 CUs, input the 32×32 CU into the trained multi-task learning network model, obtain a second prediction result, and perform CU division according to the second prediction result.

8. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images