Video encoding method and apparatus

Abstract

The application discloses a video encoding method, system, device and equipment. The method represents the partition structure of a coding tree unit (CTU) by a hierarchical CU partition map (HCPM), represents the encoding mode type of each CU in the CTU by a hierarchical CU mode map (HCMM), applies multi-task learning to simultaneously accelerate the encoding processes of multiple modules of a video encoder, including the CU depth decision process and the CU mode decision process, and uses the HCPM and the HCMM as learning targets of a multi-task learning model. Based on the prediction results of the CU depth and the encoding mode of the model, the partition mode and the encoding mode of the CTU are determined. In this way, one depth model is used to predict multiple targets, the unnecessary CU partition process and the CU intra-frame mode decision process can be quickly skipped, and therefore, the encoding complexity under a high encoding quality can be effectively reduced.

Description

[0001] This application claims priority to the Chinese Patent Application No. 202210926773.7, filed on August 3, 2022, and entitled "Video encoding method and apparatus", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of video processing, in particular to a video encoding method, device and system, a video encoding prediction model construction method and device, and an electronic device. BACKGROUND

[0003] The video compression rate and encoding complexity that a video encoder can provide are important indicators affecting the competitiveness of video products (such as live broadcast, on-demand, video call, etc.).

[0004] A typical video encoding method adopts the following process: a CU partition structure of a whole CTU is expressed in the form of hierarchical coding unit (CU) partition of a picture (HCPM); a prediction HCPM is learned through an early-terminated hierarchical CNN (ETH-CNN); a time domain correlation of CU partition is learned through an early-terminated long short-term memory LSTM network (ETH-LSTM); and the CU partition is predicted by combining the ETH-LSTM and ETH-CNN networks, so as to improve the video compression rate and reduce the encoding complexity in the inter-frame mode.

[0005] However, in the process of implementing the present application, the inventors found that the above-mentioned scheme at least has the following problems: due to the existence of unnecessary CU partition process and CU intra-frame mode decision process, the encoding complexity is high. SUMMARY

[0006] The present application provides a video encoding method to solve the problem of high encoding complexity in the prior art. The present application further provides a video encoding device and system, a video encoding prediction model construction method and device, and an electronic device.

[0007] The present application provides a video encoding method, comprising:

[0008] obtaining a coding tree unit of a video frame;

[0009] obtaining a prediction residual image and a prediction reconstructed image of the coding tree unit;

[0010] The prediction model through multi-task learning obtains a hierarchical coding unit partition map and a hierarchical coding unit mode map of the coding tree unit according to the predicted residual image and the predicted reconstruction image, the hierarchical coding unit partition map comprising a coding unit partition probability, and the hierarchical coding unit mode map comprising a coding unit mode probability.

[0011] According to the partition probability and the coding mode probability, a partition mode and a coding mode of the coding tree unit are determined.

[0012] Optionally, the prediction model through multi-task learning obtains the hierarchical coding unit partition map and the hierarchical coding unit mode map of the coding tree unit according to the predicted residual image and the predicted reconstruction image, comprising:

[0013] The prediction model obtains shared features according to the predicted residual image and the predicted reconstruction image.

[0014] The hierarchical coding unit partition map is obtained according to the shared features.

[0015] The hierarchical coding unit mode map is obtained according to the shared features.

[0016] Optionally, the prediction model obtains shared features according to the predicted residual image and the predicted reconstruction image, comprising:

[0017] The prediction model obtains multiple levels of shared features according to the predicted residual image and the predicted reconstruction image.

[0018] The hierarchical coding unit partition map is obtained according to the shared features, comprising:

[0019] The hierarchical coding unit partition map is obtained according to multiple levels of shared features.

[0020] The hierarchical coding unit mode map is obtained according to the shared features, comprising:

[0021] The hierarchical coding unit mode map is obtained according to multiple levels of shared features.

[0022] Optionally, the prediction model obtains the hierarchical coding unit partition map and the hierarchical coding unit mode map of the coding tree unit according to the predicted residual image and the predicted reconstruction image, comprising:

[0023] According to the predicted residual image and the predicted reconstruction image, normalized data of the predicted residual image and the predicted reconstruction image is obtained.

[0024] The hierarchical coding unit partition map and the hierarchical coding unit mode map are obtained according to the normalized data.

[0025] Optionally, the obtaining the prediction residual image and the prediction reconstructed image of the coding tree unit comprises:

[0026] taking a reconstructed image generated by inter-coding the coding tree unit as the prediction reconstructed image;

[0027] determining the prediction residual image according to the prediction reconstructed image.

[0028] Optionally, the determining the partition mode and the coding mode of the coding tree unit according to the partition probability and the coding mode probability comprises:

[0029] determining whether to skip the coding process of the coding unit at the current depth and / or the partition mode coding process of the coding unit according to the partition probability;

[0030] determining whether to skip the intra-coding process or the inter-coding process of the coding unit according to the coding mode probability.

[0031] Optionally, the method further comprises:

[0032] determining the first partition probability threshold, the second partition probability threshold and / or the coding mode probability threshold corresponding to the coding unit according to the depth of the coding unit and the coding quantization parameter.

[0033] The application also provides a video coding prediction model processing method, comprising:

[0034] obtaining a training data set; the training data comprises a corresponding relationship between residual images and reconstructed images and annotation data of a hierarchical coding unit partition graph and annotation data of a hierarchical coding unit mode graph;

[0035] constructing a multi-task learning prediction model;

[0036] training model parameters of the prediction model according to the training data set.

[0037] The application also provides a video coding device, comprising:

[0038] a coding tree unit obtaining unit configured to obtain a coding tree unit of a video frame;

[0039] a first coding unit configured to obtain a prediction residual image and a prediction reconstructed image of the coding tree unit;

[0040] a prediction unit configured to obtain, by a prediction model trained by multi-task learning, a hierarchical coding unit partition map and a hierarchical coding unit mode map of the coding tree unit according to the prediction residual image and the prediction reconstructed image, the hierarchical coding unit partition map comprising a partition probability of a coding unit, and the hierarchical coding unit mode map comprising a coding mode probability of the coding unit;

[0041] a second coding unit configured to determine a partition mode and a coding mode of the coding tree unit according to the partition probability and the coding mode probability.

[0042] The application also provides an electronic device comprising:

[0043] a processor and a memory;

[0044] the memory is configured to store a program for implementing the video coding method, and the device is powered on and runs the program of the method by the processor.

[0045] The application also provides a video processing system comprising:

[0046] the video coding device and a video decoding device.

[0047] The application also provides a video live streaming method comprising:

[0048] obtaining a coding tree unit of a video frame included in a live streaming video;

[0049] obtaining a prediction residual image and a prediction reconstructed image of the coding tree unit;

[0050] obtaining, by a prediction model trained by multi-task learning, a hierarchical coding unit partition map and a hierarchical coding unit mode map of the coding tree unit according to the prediction residual image and the prediction reconstructed image, the hierarchical coding unit partition map comprising a partition probability of a coding unit, and the hierarchical coding unit mode map comprising a coding mode probability of the coding unit;

[0051] determining a partition mode and a coding mode of the coding tree unit according to the partition probability and the coding mode probability, and generating coding data according to the partition mode and the coding mode of the coding tree unit.

[0052] The application also provides a video on demand video coding method comprising:

[0053] obtaining a coding tree unit of a video frame included in a video on demand video;

[0054] obtaining a prediction residual image and a prediction reconstructed image of the coding tree unit;

[0055] The prediction model trained by multi-task learning obtains a hierarchical coding unit partition map and a hierarchical coding unit mode map of the coding tree unit according to the prediction residual image and the prediction reconstruction image, the hierarchical coding unit partition map includes a partition probability of a coding unit, and the hierarchical coding unit mode map includes a coding mode probability of the coding unit.

[0056] The partition mode and the coding mode of the coding tree unit are determined according to the partition probability and the coding mode probability, and coding data is generated according to the partition mode and the coding mode of the coding tree unit.

[0057] The application further provides a video coding method for a video call, comprising:

[0058] A coding tree unit of a video frame included in the video call is obtained.

[0059] A prediction residual image and a prediction reconstruction image of the coding tree unit are obtained.

[0060] The prediction model trained by multi-task learning obtains a hierarchical coding unit partition map and a hierarchical coding unit mode map of the coding tree unit according to the prediction residual image and the prediction reconstruction image, the hierarchical coding unit partition map includes a partition probability of a coding unit, and the hierarchical coding unit mode map includes a coding mode probability of the coding unit.

[0061] The partition mode and the coding mode of the coding tree unit are determined according to the partition probability and the coding mode probability, and coding data is generated according to the partition mode and the coding mode of the coding tree unit.

[0062] The application further provides a video coding method for a video conference, comprising:

[0063] A coding tree unit of a video frame included in the video conference is obtained.

[0064] A prediction residual image and a prediction reconstruction image of the coding tree unit are obtained.

[0065] The prediction model trained by multi-task learning obtains a hierarchical coding unit partition map and a hierarchical coding unit mode map of the coding tree unit according to the prediction residual image and the prediction reconstruction image, the hierarchical coding unit partition map includes a partition probability of a coding unit, and the hierarchical coding unit mode map includes a coding mode probability of the coding unit.

[0066] The partition mode and the coding mode of the coding tree unit are determined according to the partition probability and the coding mode probability, and coding data is generated according to the partition mode and the coding mode of the coding tree unit.

[0067] The application further provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions are executed on a computer, the computer is caused to perform the various methods described above.

[0068] The application further provides a computer program product comprising instructions which, when executed on a computer, cause the computer to perform the various methods described above.

[0069] Compared with the prior art, the application has the following advantages:

[0070] The video encoding method provided by the embodiments of the application represents the partition structure of a coding tree unit (CTU) by a hierarchical CU partition map (HCPM) and represents the encoding mode type of each CU in the CTU by a hierarchical CU mode map (HCMM), and uses multi-task learning to simultaneously accelerate the encoding processes of multiple modules of a video encoder, including the CU depth decision process and the CU mode decision process. The multi-task learning model takes the HCPM and the HCMM as learning targets, and determines the partition mode and the encoding mode of the coding tree unit based on the prediction results of the CU depth and the encoding mode based on the model. In this way, one depth model is used to predict multiple targets, and the unnecessary CU partition process and the CU intra mode decision process can be quickly skipped, so that the encoding complexity under a high encoding quality can be effectively reduced. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 A flowchart of an embodiment of the video encoding method provided by the application is shown in the figure;

[0072] Figure 2 An encoding flowchart of an embodiment of the video encoding method provided by the application is shown in the figure;

[0073] Figure 3 An HCPM structure example of an embodiment of the video encoding method provided by the application is shown in the figure;

[0074] Figure 4 An HCMM structure example of an embodiment of the video encoding method provided by the application is shown in the figure;

[0075] Figure 5 An MTL-CNN model structure of an embodiment of the video encoding method provided by the application is shown in the figure;

[0076] Figure 6 A CommonNet_X network structure of an embodiment of the video encoding method provided by the application is shown in the figure;

[0077] Figure 7 A Task0_Net network structure of an embodiment of the video encoding method provided by the application is shown in the figure;

[0078] Figure 8 The Task1_Net network structure diagram of an embodiment of the video encoding method provided in the application is shown in the following figure:

[0079] Figure 9 The training sample generation example diagram of an embodiment of the video encoding method provided in the application is shown in the following figure:

[0080] Figure 10 The specific flowchart of an embodiment of the video encoding method provided in the application is shown in the following figure. DETAILED DESCRIPTION

[0081] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the application. However, the application can be practiced in a large number of other ways than those described herein, and one skilled in the art can make similar substitutions without departing from the scope of the application, and therefore the application is not limited to the specific implementation disclosed below.

[0082] In the present application, a video encoding method and device, and an electronic device are provided. In the following embodiments, various schemes are described one by one in detail.

[0083] Video encoding refers to converting a file in an original video format into another video format file through compression technology. Video encoding includes intra-frame encoding and inter-frame encoding. Objects in an image often have spatial correlation, and objects in adjacent images often have temporal correlation. Intra-frame encoding is compression using spatial redundancy of an image, and inter-frame encoding is compression using correlation between frames of video information. The compression rate of inter-frame encoding is usually higher than that of intra-frame encoding.

[0084] Video encoding divides the image of a video frame into coding tree units (CTUs), which are basic encoding units and can be recursively divided into coding units (CUs). As shown in the following figure, each coding tree unit can be recursively divided using a quadtree structure. Figure 2

[0085] In the existing encoding framework, the encoder recursively attempts encoding of different CU depths to find a CU size suitable for the video content where the current CTU is located. For example, when the CTU size is 64, the encoder attempts encoding of 5 different depths (64x64 to 4x4 CUs) from Depth 0 to Depth 4. Except for Depth 4, which does not attempt a quadtree partitioning mode, CUs of all depths are encoded in inter-frame mode, intra-frame mode, and quadtree partitioning mode, and the rate-distortion cost (RD0Cost) of different modes is calculated. Finally, the optimal encoding mode and CU depth are selected according to the rate-distortion cost.

[0086] ​The inventors of the present application find that the existing coding mode has redundant CU depth decision coding process, and the coding complexity is high. For example, in the area of image flatness and basically no texture features, the optimized coding block generally belongs to large CU, but in order to find this CU size, the encoder makes many coding attempts of small CUs, and by solving the rate distortion cost (RDO Cost) of different CU sizes and comparing, the optimized CU depth is obtained. This process is called "CU depth search process".

[0087] The inventors of the present application analyze the complexity and optimization space of the existing coding process through experiments. In order to understand the proportion of inter-frame coding in the total coding complexity and the proportion of inter-frame coding complexity in different partition depths, the inventors first divide the coding module into "inter-frame module" (inter module), "B-frame intra-frame module" (intra module (B frame)), "I-frame intra-frame module" (intra module (I frame)) and "other module" (other), and calculate the proportion of each module in the total coding time. Then, according to the two categories of inter-frame and intra-frame, the time proportion of the two coding modes is calculated at different depths. Among them, the inter-frame module refers to the coding unit coded by inter-frame coding mode, the B-frame intra-frame module refers to the coding unit coded by B-frame intra-frame coding mode, the I-frame intra-frame module refers to the coding unit coded by I-frame intra-frame coding mode, and the coding unit coded by other coding mode is the other module.

[0088] The optimization space of CU depth search process refers to the wasted calculation resources and time cost in order to find a more suitable optimization depth, which can actually be avoided by other algorithms. In order to understand the optimization space in the CU partition process, the inventors divide the CU into "malignant non-split" CU (Malignant_nonSplit” CU) and "malignant split" CU (Malignant_split” CU). Among them, "malignant non-split" CU represents that the current CU is the optimal CU depth or CU size, but the encoder makes quadtree partition on the current CU, and also makes intra / inter-frame mode coding on the sub-CU obtained by partitioning; while "malignant split" CU represents that the current CU is not the optimal CU size, but the encoder makes intra / inter-frame mode coding on the current CU in order to find the optimal CU depth. In the experiment process, the inventors calculate the proportion of the two types of CUs in different depth CUs in the actual coding process, and use their proportion to measure the optimization space in the CU depth search process. The higher the proportion of the two types of CUs, the larger the optimization space of the current depth CU.

[0089] To find a suitable encoding mode for the current CU, the encoder will try all encoding modes in order, encode for each mode and calculate the RD Cost, and finally select the encoding mode with the smallest RD Cost as the optimal encoding mode of the current CU.

[0090] To understand the optimization space of the CU mode selection process, the inventors simplify the encoding modes into inter-frame encoding and intra-frame encoding. And divide the CU into "Malignant_intra" CU and "Malignant_inter" CU. Among them, "Malignant_intra" CU refers to the CU whose optimal encoding mode is intra-frame encoding mode, but has tried inter-frame encoding mode; while "Malignant_inter" CU refers to the CU whose optimal encoding mode is inter-frame encoding mode, but has tried intra-frame encoding mode. The higher the proportion of these two types of CUs, the greater the optimization space of the CU mode selection process, that is, the proportion of these two types of CUs can represent the optimization space of the mode selection process.

[0091] Experimental data shows that in the existing encoding method, the decision of inter-frame mode occupies as high as 51.54% of the encoding time, and the intra-frame mode decision of I frame and B frame each occupies less than 10% of the time. The inter-frame mode decision and the intra-frame mode decision of B frame belong to the main part of the inter-frame CU depth decision process, therefore, the inter-frame CU depth decision process almost occupies more than 60% of the time complexity of the existing encoding process. In terms of complexity distribution, from the average results, except for the CU with depth 4 (Depth 4) which only accounts for 1.99% of the complexity, the encoding complexity of other depths is basically similar (13% ~ 15%), indicating that most CUs will not choose Depth 4, and the encoding acceleration algorithm will be difficult to play an effect on the CU with depth 4. In addition, it is found that on each CU depth, compared with the intra-frame mode, the inter-frame mode occupies most of the complexity of mode selection.

[0092] The experimental data also show that the optimization space under small QP (quantization parameter) is larger than that under QP, and small block CUs still have a large optimization space. From the average results, a large number of CUs exist in the case of waste of computing resources, and the proportion of “Malignant_nonSplit” CUs and “Malignant_split” CUs is 100%, 88.05%, 85.18% and 82.63%, 78.99%, 61.60% at depths D0-D2. The smaller the QP value, the smaller the quantization step, and the higher the quantization accuracy. In the case of the same image quality, the data generated is larger. From the different QPs, smaller QPs are more difficult to accurately predict the depth of the CU because the image quality coding is more accurate and more strict, so from the results, the optimization space under small QP is larger than that under QP. From different CU depths, larger CUs have larger optimization spaces than small CUs. When predicting the depth of small CUs, more reference information is available than large CUs, so the existing coding acceleration algorithm has used this information to make a part of the depth prediction algorithm for small CUs. However, from the statistical results, small CUs still have a large optimization space.

[0093] The experimental data also show that the intraframe interframe mode search process still has optimization space. The optimization space of mode selection at different depths (Depth 0-Depth 4) is different, but overall, the optimization space of “Malignant_intra” CUs is higher than that of “Malignant_inter” CUs, indicating that some algorithms in the existing encoder are used to skip the intraframe mode search process of some CUs. Except for Depth3, the proportion of “Malignant_intra” CUs at other depths is 100%, indicating that the existing fast skip CU interframe mode search algorithm of VVenc is only effective at Depth 3. Therefore, the intraframe interframe mode search process of the existing coding framework still has optimization space.

[0094] First embodiment

[0095] Please refer to Figure 1 , which is a flowchart of an embodiment of the video coding method of the present application. In this embodiment, the method can include the following steps:

[0096] Step S101: Obtain the coding tree unit of the video frame.

[0097] Video encoding is based on blocks, a frame of video is divided into different blocks, and then each block is encoded separately. In this embodiment, a frame of image is divided into multiple coding tree units (CTU), for example, for a high-definition video, the frame resolution is 1920x1080, if it is divided into 64x64 size CTU, there are 30 CTU in each row, and the whole frame image can be divided into 30x17 CTU.

[0098] The coding tree unit is a basic encoding unit, which can be divided into coding units (CU) downward, and each tree coding block can be recursively divided by using the structure of quadtree nested multi-type tree.

[0099] Step S103: Obtain the prediction residual image and the prediction reconstructed image of the coding tree unit.

[0100] In the model inference stage, the current video frame has not yet started encoding, so the real prediction residual (Resi) and the real reconstructed image (Rec) of the CTU cannot be obtained. The method provided in the embodiment of the present application first performs a simple and rough encoding on each CTU in the current video frame through step S103 before starting to encode the current video frame through step S105, which is called pre-encoding, to obtain the prediction residual image and the prediction reconstructed image generated by pre-encoding.

[0101] In one example, the video encoder has strict requirements on time performance, in order to minimize the time overhead generated by pre-encoding, step S103 can be implemented in the following way: the reconstructed image generated by inter-frame encoding of the coding tree unit is used as the prediction reconstructed image, and the prediction residual image is determined according to the prediction residual image. By using this processing method, only the CU with depth 0 (the largest CU) is encoded in inter mode (Inter_ME), and the time-consuming CU depth search and mode search process in actual encoding is omitted. Experiments show that the pre-encoding method only increases about 3% of the time, and the residual image generated by pre-encoding is used as the input data, which also shows good results on the model trained next.

[0102] In one example, the method provided by the embodiments of the present application is used in a video coding system adopting a versatile video coding standard (VVC, such as H.266). Compared with traditional coding standards, VVC emphasizes more on the versatility of the coding standard, but the coding complexity is greatly increased, so the method provided by the embodiments of the present application can be used to reduce the coding complexity. In this case, a video coding mode with lower complexity can be used to obtain the prediction residual image and the prediction reconstructed image of the coding tree unit. For example, a pre-coding is performed by using a coder of a high efficiency video coding standard (H.265 / HEVC), an advanced video coding standard (H.264 / AVC), or a lower standard.

[0103] Step S105: obtaining, by the prediction model trained by multi-task learning, the hierarchical coding unit partition map and the hierarchical coding unit mode map of the coding tree unit according to the prediction residual image and the prediction reconstructed image.

[0104] In the embodiments, the CU partition structure of the entire coding tree unit (CTU) is represented in the form of a hierarchical CU partition map (HCPM), and the coding mode type of each CU in the CTU is represented in the form of a hierarchical CU mode map (HCMM). The hierarchical coding unit partition map includes the partition probability of each coding unit, and the hierarchical coding unit mode map includes the coding mode probability of each coding unit, such as the inter-frame coding probability or the intra-frame coding probability. By using the prediction model trained by multi-task learning, the prediction residual image and the prediction reconstructed image generated by the pre-coding process are used as inputs to predict two tasks: 1) whether the current CU needs to be partitioned into small block coding; and 2) whether the optimal coding mode of the current CU is an inter-frame coding mode, so that the decision mode of the CU partition and the selection mode of the CU coding mode can be optimized, thereby reducing the coding complexity in the inter-frame mode.

[0105] In this step, the prediction residual image and the prediction reconstructed image generated by the pre-coding are used to approximate the real prediction residual and reconstructed image. The prediction residual generated by the pre-coding and the reconstructed image after the pre-coding are used as the input data of the prediction model (also referred to as the multi-task learning model), and the hierarchical coding unit partition map and the hierarchical coding unit mode map of the coding tree unit are output by the prediction model. The hierarchical coding unit partition map output by the prediction model includes the partition probability (probability_split, the predicted value of the HCPM) of each depth CU, and the hierarchical coding unit mode map output by the prediction model includes the probability (probability_inter, the predicted value of the HCMM) of selecting the inter-frame mode as the optimal mode of each CU, which is used as the coding mode probability of the coding unit output by the prediction model. inter_mode ​

[0106] The hierarchical CU partitioning graph (HCPM) represents the partitioning structure of the CTU. This structure is used as the output structure of the prediction model. The purpose is to enable the model to obtain the partitioning structure of the entire CTU by predicting the corresponding HCPM, without having to call the model separately for each CU, thus effectively reducing time overhead.

[0107] HCPM structure as Figure 3 As shown in the figure, (a) represents the true label of HCPM, and (b) represents the predicted HCPM value of the model. The HCPM structure uses y1(U) to represent whether the first-level (depth 0) CU is partitioned, y1(U) = 1 represents partitioning, and y1(U) = 0 represents no partitioning; y2(U i i = {1, 2, 3, 4} represents whether the four CUs of the second level are divided. Similarly, y3(U i,j i,j={1,2,3,4} represents whether the 16 CUs in the third level are divided. The HCPM values ​​predicted by the model represent the probabilities that each layer's CU needs to be partitioned and encoded. Furthermore, in the HCPM structure, if the parent CU of the previous layer is not partitioned, the partition labels of the four child CUs in the next layer will be invalid. As shown in Figure (a) of the HCPM annotation data (Ground-truth of HCPM), if y2(U2) = 0 in the second level (representing that the CU is not partitioned), then the four child CUs corresponding to that CU will be invalid in the third level. The loss of these invalid CUs will not be considered during the model training loss calculation stage.

[0108] To meet the needs of multi-task models, the method provided in this application proposes a Hierarchical CU Mode Map (HCMM) based on the HCPM structure. For example... Figure 4 As shown, HCMM adopts a structure similar to HCPM. Compared to HCPM, HCMM can also make decisions for CUs at level 4 (depth 3), and M1(U) and M2(U) in HCMM are also possible. i )i={1,2,3,4}、M3(U i,j )i,j={1,2,3,4}、M4(U i,j,ki, j, k = {1, 2, 3, 4} represent the optimization modes of each depth CU, and the value of 1 represents the inter mode, and the value of 0 represents the intra mode. Since the CU division structure and the mode decision have an inseparable relationship, that is, if the current CU is not divided, there is no smaller block CU at the corresponding position, and there is no corresponding intra / inter mode decision process. Therefore, the invalid CU in the HCPM structure will also be invalid in the HCMM, and the invalid CU will finally not be counted in the loss calculation. The prediction model (such as the MTL-CNN model) proposed in the embodiment of the application aims to predict the HCPM and the HCMM, and similar to the predicted HCPM, the predicted HCMM can include, may respectively represent the probability of each layer CU selecting the inter coding as the optimization mode.

[0109] The prediction model can be a deep model, and the network structure thereof can be a multi-task learning convolutional neural network (MTL-CNN) or a multi-task learning recurrent neural network.

[0110] In one example, step S105 can include the following steps: obtaining shared features according to the prediction residual image and the prediction reconstruction image; obtaining the hierarchical coding unit division map according to the shared features; and obtaining the hierarchical coding unit mode map according to the shared features.

[0111] In a specific implementation, the prediction model can include a shared feature extraction unit, a first prediction unit, and a second prediction unit. Step S105 can be implemented in the following manner: the shared feature extraction unit is used to obtain shared features according to the prediction residual image and the prediction reconstruction image; the first prediction unit is used to obtain the hierarchical coding unit division map according to the shared features; and the second prediction unit is used to obtain the hierarchical coding unit mode map according to the shared features. The shared features refer to features that are used to obtain the hierarchical coding unit division map and the hierarchical coding unit mode map, that is, the hierarchical coding unit division map can be obtained according to the shared features, and the hierarchical coding unit mode map can also be obtained according to the shared features.

[0112] In this embodiment, the prediction model is a model of a multi-task learning convolutional neural network (MTL-CNN). The backbone of the MTL-CNN model can adopt a shared-bottom multi-task (Shared-Bottom multi-task) DNN structure commonly used in multi-task learning. This network structure uses a shared layer to learn the same features in the shallow layer for different tasks after the input layer, and then each task uses a separate network to learn the respective deep layer features on the basis of the same features learned in the bottom, and outputs the respective target task results.

[0113] Figure 5 The basic structure of the MTL-CNN model is shown, including three layers: an input layer, a shared layer, and a multi-task layer. In the input layer, the Resi CTU and the Rec CTU are residual images and reconstructed images obtained by pre-encoding the CTU, and the size of both can be 64x64 and the number of channels can be 1.

[0114] In one example, step S105 can further include the following steps: obtaining normalized data of the prediction residual image and the prediction reconstructed image according to the prediction residual image and the prediction reconstructed image. Since the data distribution of the prediction residual image and the prediction reconstructed image is quite different, in order to accelerate the convergence speed of the model and improve the performance of the model on multiple tasks, the input layer can use a standardization method to standardize the data of the Resi CTU (residual image) and the Rec CTU (reconstructed image) so that they conform to a standard normal distribution.

[0115] In specific implementation, the prediction model can include a data standardization unit configured to obtain normalized data of the prediction residual image and the prediction reconstructed image according to the prediction residual image and the prediction reconstructed image.

[0116] In specific implementation, the standardization method can be a z-score method, and the z-score method is used to standardize the data of the residual image and the reconstructed image, for example, the following formula is used: In this formula, x * represents the standardized data, x represents the data before standardization, μ represents the mean of multiple data samples, and σ represents the standard deviation of all data. The data samples, also known as training samples, include the prediction residual image and the prediction reconstructed image, and the mean of multiple data samples includes the mean of multiple prediction residual images and the mean of multiple prediction reconstructed images, or the mean of multiple prediction residual images and multiple prediction reconstructed images.

[0117] The shared layer is responsible for extracting common or shared features from the shallow layers of multi-task data. In one example, obtaining shared features based on the predicted residual image and the predicted reconstructed image includes: obtaining shared features at multiple levels based on the predicted residual image and the predicted reconstructed image, with different levels corresponding to different coding unit partitioning depths; obtaining the hierarchical coding unit partitioning map based on the shared features includes: obtaining the hierarchical coding unit partitioning map based on the shared features at multiple levels; obtaining the hierarchical coding unit pattern map based on the shared features includes: obtaining the hierarchical coding unit pattern map based on the shared features at multiple levels.

[0118] In this embodiment, the shared feature extraction unit includes: multiple levels of shared feature extraction units, with different levels corresponding to different coding unit partitioning depths; the first prediction unit obtains the hierarchical coding unit partitioning map based on the shared features of multiple levels; and the second prediction unit obtains the hierarchical coding unit pattern map based on the shared features of multiple levels.

[0119] In one example, the hierarchical coding unit partitioning map is obtained based on the shared features corresponding to the first, second, and third levels, respectively; and the hierarchical coding unit pattern map is obtained based on the shared features corresponding to the first, second, third, and fourth levels, respectively.

[0120] In specific implementation, the first prediction unit can obtain the hierarchical coding unit partitioning map based on the shared features corresponding to the first, second, and third levels respectively; and the second prediction unit can obtain the hierarchical coding unit pattern map based on the shared features corresponding to the first, second, third, and fourth levels respectively.

[0121] like Figure 5 As shown, the normalized residual image and reconstructed image output from the input layer are processed by four parallel common networks (CommonNet_X, X = 0, 1, 2, 3) to extract shared features at different levels, which can correspond to the three levels of HCPM and the four levels of HCMM, respectively.

[0122] In one example, obtaining shared features at different levels based on the predicted residual image and the predicted reconstructed image may include the following steps: obtaining overall features of the predicted residual image based on the predicted residual image and different coding quantization parameters; obtaining overall features of the predicted reconstructed image based on the predicted reconstructed image and different coding quantization parameters; obtaining a first overall feature based on the overall features of the predicted residual image and the overall features of the predicted reconstructed image; obtaining a second overall feature based on the first overall feature; and obtaining the shared features based on the second overall feature.

[0123] In the implementation, the shared feature extraction unit of different levels includes a first pooling layer, a second pooling layer, a feature fusion layer, a first shared feature extraction layer, and an output layer. The first pooling layer is configured to obtain overall features of the predicted residual image according to the predicted residual image. The second pooling layer is configured to obtain overall features of the predicted reconstructed image according to the predicted reconstructed image. The feature fusion layer is configured to obtain first overall features according to the overall features of the predicted residual image and the overall features of the predicted reconstructed image. The first shared feature extraction layer is configured to obtain second overall features according to the first overall features. The output layer is configured to obtain the shared features according to the second overall features.

[0124] Figure 6 The CommonNet_X network structure is shown. The residual image and the reconstructed image are respectively subjected to average pooling (AvgPool) to extract overall features of data, and then are spliced together. Finally, a convolutional neural network (CNN) with a kernel size of 3*3 is used to extract shallow shared features of the residual image and the reconstructed image. The shared features are transmitted to the next layer through a normalization layer (Batch Normalization, BN) and an activation function (RELU).

[0125] In an example, the pooling layers of different levels correspond to different pooling layer parameters. The pooling layer parameters can be kernel sizes, and can also be steps. In order to extract features of different granularities or levels, different pooling layer parameters are used for the average pooling layers of different CommonNet_X networks. Table 1 below shows the specific parameter setting in this embodiment.

[0126] Table 1: Average pooling layer parameter setting of CommonNet_X network

[0127]

[0128] In this embodiment, the multi-task layer includes two networks Task0_Net and Task1_Net of different tasks, respectively, for further learning different features between the respective tasks from the shared features, and respectively predicting the HCPM and the HCMM.

[0129] Figure 7 and 8The network structures of Task0_Net and Task1_Net are shown. In this embodiment, the basic structures of the two networks are the same, both of which take the shared features of the shared layer as the output, and extract deeper features through a convolutional neural network layer (CNN). The CNN can include three non-overlapping CNNs (the step length of the 4x4 CNN is 4x4, and the step length of the 2x2 CNN is 2x2). Then, the deep features are flattened into one-dimensional vectors, and the one-dimensional vectors of each branch (including global and local features) are spliced to obtain a vector Vector, so that the subsequent fully connected layer (FC Layer) can select appropriate features from the global and local features. The fully connected layer includes two hidden layers (f x-1 ,f x-2 ) and an output layer, which aims to integrate the features vector extracted by the task layer (CNN) through the two hidden layers, and finally predict HCPM or HCMM through the output layer.

[0130] In this embodiment, the difference between the network structures of Task0_Net and Task1_Net is that Task1_Net includes four branches, while Task0_Net only has three branches, because the output of Task0_Net network is to predict HCPM, and the HCPM structure has only three levels (as shown in the figure), each branch is responsible for predicting a level, and the prediction results of 1, 2x2 and 4x4 output by different branches correspond to the Figure 4 They respectively represent the probability of each layer CU needing to do the partition mode. And HCMM contains four levels, and the prediction results of 1, 2x2, 4x4 and 8x8 output by the four branches of the corresponding Task1_Net network correspond to the They respectively represent the probability of each layer CU selecting inter-frame coding as the optimization mode.

[0131] ​​​To train the multi-task prediction model, 111 YUV test sequences are used as the database for establishing the data set, including test sequences of different resolutions, namely: SIF (352x240), CIF (352x288), NTSC (720x486), 4CIF (704x576), 240p (416x240), 480p (832x480), 720p (1280x720), 1080p (1920x1080), WQXGA (2560x1600), 4K (4096x2160). The 111 sequences are divided into three non-overlapping databases, which are used as the training set database, the validation set database and the test set database, respectively. The training set database contains 83 sequences, which are used to construct the training set. The validation set database contains 10 sequences, which are used to construct the validation set. The test set contains 18 standard test sequences provided by JCT-VC, which are used to construct the test set.

[0132] In one example, the training data set of the prediction model is obtained in the following manner: a plurality of coding tree units and a plurality of coding quantization parameters are obtained; and a residual image and a reconstructed image of the plurality of coding tree units are obtained according to the plurality of coding quantization parameters.

[0133] For example, four coding quantization parameters QP{22, 27, 32, 37} are used to encode all the sequences in the database under the random access (RA) fast gear of the encoder (using random_access.cfg configuration). After encoding, the prediction residual (Resi), the reconstructed image after encoding (Rec) and the binary label of the CU corresponding to the CTU of different sequences in the database are generated (corresponding to the HCPM structure) and (corresponding to the HCMM structure), d represents the depth of the current CU, and i represents the CU label of the depth d in the CTU. Wherein is used to identify the d-depth i-th CU whether division is needed, and the label 1 represents that division into small blocks is needed, otherwise, the label 0 represents that division into small blocks is not needed. is used to identify the category of the optimal coding mode, and the label 1 represents that the optimal coding mode is the inter mode, and the label 0 represents that the optimal mode is the intra mode.

[0134] The training data of the multi-task prediction model includes the correspondence between the residual image and the reconstructed image and the annotation data of the hierarchical coding unit division graph and the annotation data of the hierarchical coding unit mode graph.

[0135] Figure 9A generation example of a training sample is shown, the encoder encodes the prediction residual Resi of the CTU , the coded reconstructed image Rec, the label data of HCPM {ls0, ls1, ls2} in the Figure 9 , and the label data of HCMM {lm0, lm1, lm2, lm3} in the Figure 9 , i.e. Resi, Rec, the label data of 21 CU divisions + the label data of 85 CU coding modes as a training sample. After encoding the video sequence database, a total of millions of training samples, millions of verification samples and millions of test samples are obtained.

[0136] It should be noted that different ways can be used to obtain the input data (residual image and reconstructed image) of the model in the model training stage and the model inference stage, or the same way can be used to obtain the input data of the model. For example, in the model training stage, the CUs of multiple depths can be encoded to obtain the real residual image and the real reconstructed image of the coding tree unit; while in the model training stage, only the CUs of depth 0 (the largest CU) are encoded in Inter_ME mode to obtain the residual image and the reconstructed image of the coding tree unit with slightly lower precision, which can save the time-consuming CU depth search and mode search process in actual encoding to meet the higher demand of the encoder on time performance. In actual application, it can be determined according to the demand of the encoder on time performance to determine which way to obtain the input data of the model.

[0137] Step S107: determining the division mode and the coding mode of the coding tree unit according to the division probability and the coding mode probability.

[0138] This step determines the division mode of the coding tree unit according to the division probability, and determines the coding mode of the coding tree unit according to the coding mode probability. The division mode includes the way in which the coding tree unit is divided into coding units, and the coding mode includes the coding mode of each coding unit.

[0139] In this embodiment, step S107 can include the following steps:

[0140] Step S1071: determining whether to skip the encoding process of the coding unit at the current depth and / or the coding process of the division mode of the coding unit according to the division probability.

[0141] According to the division probability, it can be determined whether to skip the encoding process of the coding unit at the current depth, or whether to skip the coding process of the division mode of the coding unit, or whether to skip the encoding process of the coding unit at the current depth and the coding process of the division mode of the coding unit.

[0142] In one example, step S1071 can be implemented as follows: Based on the partitioning probability and a first partitioning probability range (e.g., 70% to 96%), determine whether to skip the encoding process of the coding unit at the current depth; based on the partitioning probability and a second partitioning probability range (e.g., 5% to 30%), determine whether to skip the coding unit partitioning pattern encoding process. This approach allows for the determination, based on a pre-set first partitioning probability range, whether to skip the encoding process of the CU at the current depth (i.e., only partitioning the CU); and based on a pre-set second partitioning probability range, whether to skip the coding unit partitioning pattern encoding process of the CU at the current depth (i.e., not performing deeper sub-block encoding on the current CU).

[0143] In another example, step S1071 can be implemented as follows: determine whether to skip the encoding process of the coding unit at the current depth based on the partition probability and the first partition probability threshold; determine whether to skip the coding unit partitioning mode encoding process based on the partition probability and the second partition probability threshold.

[0144] The method provided in this application can determine whether to skip the encoding process of the CU at the current depth (i.e., only perform partitioning pattern encoding on the CU) based on the probability of partitioning at each depth predicted by the multi-task learning model (probability_split, the predicted value of HCPM) and a pre-set first partitioning probability threshold (TH_UP). Furthermore, it can also determine whether to skip the CU partitioning pattern encoding process (i.e., not perform deeper sub-block encoding on the current CU) based on probability_split and a second partitioning probability threshold (TH_DOWN).

[0145] In this embodiment, the partition probability corresponding to the current CU can be read from the HCPM based on the depth and location information of the current CU in the CTU. split ).like Figure 10 As shown, in this embodiment, if the partitioning probability is greater than the first partitioning probability threshold (TH_UP), it is determined that the encoding process of the CU at the current depth is skipped, and only partitioning mode encoding is performed on the CU. This is called the "early skip CU encoding strategy". If the partitioning probability is less than or equal to the first partitioning probability threshold, it is determined that the encoding process of the CU at the current depth cannot be skipped, and the encoding process of the CU at the current depth needs to be performed, and inter-frame encoding is performed on the CU. In specific implementation, if part of the current CU exceeds the image boundary, it is determined that the encoding process of the CU at the current depth is skipped.

[0146] like Figure 10As shown, in the embodiment, if the division probability is less than or equal to a second division probability threshold (TH_DOWN), it is determined to skip the CU division mode encoding process, i.e., no deeper sub-block encoding is performed on the current CU, which is referred to as an "early skip CU division strategy". If the division probability is greater than the second division probability threshold, it is determined that the CU division mode encoding process cannot be skipped, and deeper sub-block encoding, such as quad-tree encoding, needs to be performed on the current CU.

[0147] The method provided by the embodiment of the present application can determine whether to skip the encoding process of the CU at the current depth according to the CU division probability, and skip the encoding unit division mode encoding process, so that unnecessary CU encoding processes and unnecessary encoding unit division mode encoding processes can be quickly skipped, thereby reducing the encoding complexity.

[0148] Step S1073: determining whether to skip the intra-frame encoding process or the inter-frame encoding process of the encoding unit according to the encoding mode probability.

[0149] In one example, step S1073 can be implemented in the following manner: determining whether to skip the intra-frame encoding process or the inter-frame encoding process of the encoding unit according to the encoding mode probability and an encoding mode probability range. In this way, it can be determined whether to skip the intra-frame encoding process or the inter-frame encoding process of the CU according to the pre-set encoding mode probability range (e.g., 56-78%).

[0150] In another example, the encoding mode probability includes an inter-frame mode probability, and step S1073 can include the following step: determining whether to skip the intra-frame encoding process of the CU according to the inter-frame mode probability and an inter-frame mode probability threshold.

[0151] In the embodiment, the probability (probability inter_mode ) that the current CU selects the inter-frame mode as the optimal mode can be read from the HCMM according to the depth and position information of the current CU in the CTU. Figure 10 As shown, in the embodiment, if the inter-frame mode probability is greater than an inter-frame mode probability threshold, it is determined to skip the intra-frame encoding process of the CU, and only inter-frame encoding is performed on the CU, which is referred to as an "early skip CU intra-frame encoding strategy". If the division probability is less than or equal to the inter-frame mode probability threshold, it is determined that the intra-frame encoding process of the CU cannot be skipped, and the intra-frame encoding needs to be performed on the CU. In a specific implementation, if the position of the current CU partially exceeds the image boundary, it is determined to skip the encoding process of the CU at the current depth.

[0152] The method provided in this application determines whether to skip the CU intra-frame coding process based on the inter-frame mode probability and the inter-frame mode probability threshold. This allows for the determination of whether to skip the CU intra-frame coding process based on the CU inter-frame mode probability and the inter-frame mode probability threshold. Therefore, unnecessary CU intra-frame coding processes can be skipped quickly, thereby reducing coding complexity.

[0153] In specific implementation, the coding mode probability may also include: intra-frame mode probability; step S1073 may include the following step: determining whether to skip the CU inter-frame coding process based on the intra-frame mode probability and the intra-frame mode probability threshold. This processing method allows for the determination of whether to skip the CU inter-frame coding process based on the CU intra-frame mode probability and the intra-frame mode probability threshold; therefore, unnecessary CU inter-frame coding processes can be quickly skipped, thereby reducing coding complexity.

[0154] like Figure 10 As shown, in one example, when determining the partitioning mode and encoding mode of the coding tree unit, step S107 may include the following steps:

[0155] Step S1101: Obtain the partition probability of the coding unit and the inter-frame mode probability based on the hierarchical coding unit partitioning diagram and the hierarchical coding unit pattern diagram.

[0156] Step S1102: Based on the partitioning probability and the first partitioning probability threshold, determine whether to skip the encoding process of the CU at the current depth.

[0157] Step S1103: Based on the partitioning probability and the second partitioning probability threshold, determine whether to skip the CU partitioning mode encoding process.

[0158] Step S1104: Based on the inter-frame mode probability and coding mode probability threshold, determine whether to skip the CU intra-frame coding process.

[0159] The method provided in this application embodiment can quickly skip unnecessary CU encoding processes at the current depth, CU partitioning mode encoding processes, and CU intra-frame encoding processes through steps S1101 to S1104, thereby reducing encoding complexity.

[0160] In specific implementation, step S107 may further include the following step: determining the coding mode based on coding quality assessment parameters. The coding quality assessment parameters include, but are not limited to, rate-distortion loss (RD COST).

[0161] After the current coding unit is processed, it is determined whether there is another coding unit at the same depth. If there is another coding unit at the same depth, the next coding unit at the same depth is obtained, and the next coding unit at the same depth is processed through the steps S1101 to S1104 until all coding units in the hierarchical coding unit division graph are processed.

[0162] In one example, the method can further include the step of determining the first division probability threshold, the second division probability threshold and / or the coding mode probability threshold corresponding to the coding unit according to the depth of the coding unit and the coding quantization parameter.

[0163] The setting of the first division probability threshold and the second division probability threshold has a strong correlation with the coding quantization parameter (QP) and the depth of the CU. In order to set more reasonable double thresholds for different coding QPs and different CU depths, the inventors have conducted test experiments. For example, four different QPs {22, 27, 32, 37} are used, different values are set for the first division probability threshold and the second division probability threshold, a test sequence is encoded under the random access RA (random_access.cfg) fast gear, and the precision and recall of the above-mentioned "early skip CU coding strategy" and "early skip CU division strategy" under different QPs, different CU depths and different thresholds are counted. These two indicators reflect the accuracy of the algorithm decision and the effectiveness of reducing the coding complexity. The specific experimental configuration is shown in Table 2.

[0164]

[0165] The change of the first division probability threshold (TH_UP) will affect the precision and recall of the "early skip CU encoding strategy", which are calculated by formula (2). In the decision of skipping the encoding of the current CU, A in formula 2 represents that the current CU needs to be divided into small blocks for encoding by the traditional search method of the encoder, B represents that the current CU meets the first division probability threshold condition (i.e. the division probability is greater than the first division probability threshold), and the skip CU encoding strategy will be executed, Prob(A|B) and Prob(B|A) represent conditional probabilities, N(*) represents the total number of CUs corresponding to the event, the greater the recall, the greater the proportion of events that execute the skip CU encoding, and the more the encoder complexity is reduced, and the greater the precision, the higher the correctness of the algorithm strategy of executing the skip CU encoding, and the less the compression performance of the encoder is reduced. It is verified by experiments that, in order to guarantee the encoding compression performance and achieve better reduction of encoding complexity, the first division probability threshold TH_UP is set in the above manner, which can guarantee that the precision is higher than 90% and the recall is 30% to 70% under each QP and each depth in video encoding.

[0166] The change of the second division probability threshold (TH_DOWN) will affect the precision and recall of the "early skip CU division strategy", which are calculated by formula (2). In the decision of skipping the division of the current CU, A in formula 2 represents that the current CU does not need to be divided into small blocks for encoding by the traditional search method of the encoder, B represents that the current CU meets the second division probability threshold condition (i.e. the division probability is less than or equal to the first division probability threshold), and the skip CU division encoding will be executed, Prob(A|B) and Prob(B|A) represent conditional probabilities, N(*) represents the total number of CUs corresponding to the event, the greater the recall, the greater the proportion of events that execute the skip CU division, and the more the encoder complexity is reduced, and the greater the precision, the higher the correctness of the algorithm strategy of executing the skip CU division, and the less the compression performance of the encoder is reduced. According to experimental verification, in order to guarantee the encoding compression performance and achieve better reduction of encoding complexity, the second division probability threshold is set in the above manner, which can guarantee that the precision is higher than 90% and the recall is up to 77% under each QP and each depth in video encoding.

[0167] In order to select the optimal coding mode probability threshold (TH_SKIP_INTRA), the precision and recall of the "skip intra prediction mode strategy" at different values of the coding mode probability threshold (TH_SKIP_INTRA) can be tested at 4 different QPs {22, 27, 32, 37}, respectively, wherein the precision and recall are calculated by formula (2). In the test, event A represents that the optimal mode of the current CU is an inter mode through the traditional search method of the encoder, event B represents that the current CU meets the threshold condition of skipping intra coding and performs the skip intra mode coding, Prob(A|B) and Prob(B|A) represent conditional probabilities, and N(*) represents the total number of CUs corresponding to the event. The greater the recall is, the greater the proportion of CUs that perform the "skip intra prediction mode strategy" is, and the more the encoder complexity is reduced. The greater the precision is, the higher the correctness of the "skip intra prediction mode strategy" is, and the less the compression performance of the encoder is reduced. According to experimental verification, in order to guarantee the compression performance while achieving the best effect of reducing the coding complexity, the coding mode probability threshold TH_SKIP_INTRA is encoded in the above manner, which can guarantee that the precision is higher than 97% and the recall is up to 86% at each QP and each depth.

[0168] From the above embodiments, it can be seen that the video encoding method provided by the embodiments of the present application represents the division structure of the coding tree unit (CTU) through the hierarchical CU division map (HCPM) and represents the coding mode type of each CU in the CTU through the hierarchical CU mode map (HCMM), uses multi-task learning to simultaneously accelerate the encoding processes of multiple modules of the video encoder, including the CU depth decision process and the CU mode decision process, uses the HCPM and the HCMM as learning targets for a multi-task learning model, and determines the division mode and the coding mode of the coding tree unit based on the prediction results of the CU depth and the coding mode by the model. Using this processing manner, one depth model is used to predict multiple targets, which can quickly skip unnecessary CU division processes and CU intra mode decision processes; therefore, the encoding complexity at a high encoding quality can be effectively reduced.

[0169] Second embodiment

[0170] In the above embodiments, a video encoding method is provided, and a video encoding device corresponding thereto is also provided by the present application. The device corresponds to the embodiments of the above method. The same parts of the present embodiment as the first embodiment are not described again, and please refer to the corresponding parts in the first embodiment.

[0171] The video coding device provided in the application comprises: a coding tree unit acquisition unit, a first coding unit, a prediction unit, and a second coding unit.

[0172] The coding tree unit acquisition unit is configured to acquire a coding tree unit of a video frame; the first coding unit is configured to acquire a prediction residual image and a prediction reconstruction image of the coding tree unit; the prediction unit is configured to acquire, by using a multi-task learning prediction model, a hierarchical coding unit partitioning map and a hierarchical coding unit mode map of the coding tree unit according to the prediction residual image and the prediction reconstruction image, the hierarchical coding unit partitioning map comprising a partitioning probability of a coding unit, and the hierarchical coding unit mode map comprising a coding mode probability of the coding unit; and the second coding unit is configured to determine a partitioning mode and a coding mode of the coding tree unit according to the partitioning probability and the coding mode probability.

[0173] In one example, the prediction unit comprises: a shared feature extraction unit configured to acquire shared features by using the prediction model according to the prediction residual image and the prediction reconstruction image; a first prediction unit configured to acquire the hierarchical coding unit partitioning map according to the shared features; and a second prediction unit configured to acquire the hierarchical coding unit mode map according to the shared features.

[0174] In one example, the shared feature extraction unit is specifically configured to acquire shared features of multiple levels by using the prediction model according to the prediction residual image and the prediction reconstruction image; the first prediction unit is specifically configured to acquire the hierarchical coding unit partitioning map according to the shared features of the multiple levels; and the second prediction unit is specifically configured to acquire the hierarchical coding unit mode map according to the shared features of the multiple levels.

[0175] In one example, the first prediction unit is specifically configured to acquire the hierarchical coding unit partitioning map according to shared features corresponding to a first level, a second level, and a third level respectively; and the second prediction unit is specifically configured to acquire the hierarchical coding unit mode map according to shared features corresponding to the first level, the second level, the third level, and a fourth level respectively.

[0176] In one example, the shared feature extraction unit comprises: a first overall feature extraction unit, a second overall feature extraction unit, a third overall feature extraction unit, a fourth overall feature extraction unit, and a shared feature acquisition unit.

[0177] The first overall feature extraction unit is configured to obtain an overall feature of the prediction residual image according to the prediction residual image and different coding quantization parameters; the second overall feature extraction unit is configured to obtain an overall feature of the prediction reconstruction image according to the prediction reconstruction image and different coding quantization parameters; the third overall feature extraction unit is configured to obtain a first overall feature according to the overall feature of the prediction residual image and the overall feature of the prediction reconstruction image; and the fourth overall feature extraction unit is configured to obtain a second overall feature according to the first overall feature; and the shared feature acquisition unit is configured to obtain the shared feature according to the second overall feature.

[0178] In one example, the prediction unit is specifically configured to obtain normalized data of the prediction residual image and the prediction reconstruction image according to the prediction residual image and the prediction reconstruction image; and obtain the hierarchical coding unit division map and the hierarchical coding unit mode map according to the normalized data.

[0179] In one example, the first coding unit is specifically configured to take a reconstruction image generated by inter-frame coding of the coding tree unit as the prediction reconstruction image; and determine the prediction residual image according to the prediction reconstruction image.

[0180] In one example, the second coding unit is specifically configured to determine whether to skip the coding process of the coding unit at the current depth and / or the coding process of the coding unit division mode according to the division probability; and determine whether to skip the intra-frame coding process or the inter-frame coding process of the coding unit according to the coding mode probability.

[0181] In one example, the apparatus further comprises: determining the first division probability threshold, the second division probability threshold and / or the coding mode probability threshold corresponding to the coding unit according to the depth of the coding unit and the coding quantization parameter.

[0182] Third embodiment

[0183] In the above embodiments, a video coding method is provided, and a video coding prediction model processing method is also provided by the present application. The method corresponds to the embodiments of the above method. The same parts of the present embodiment as the first embodiment are not described again, please refer to the corresponding parts in the first embodiment.

[0184] The video coding prediction model processing method provided by the present application comprises:

[0185] Step 1: Obtain a training data set.

[0186] The training data comprises a corresponding relationship between the residual image and the reconstruction image and the annotation data of the hierarchical coding unit division map and the annotation data of the hierarchical coding unit mode map.

[0187] The method for preparing the training data set is described in detail in Embodiment One, please refer to the relevant part in Embodiment One, which will not be repeated here.

[0188] Step 2: Construct a prediction model for multi-task learning.

[0189] The input data, output data, basic network structure and optimized network structure of the prediction model are described in detail in Embodiment One, please refer to the relevant part in Embodiment One, which will not be repeated here.

[0190] Step 3: According to the training data set, training the model parameters of the prediction model.

[0191] This step is to learn the model parameters of the prediction model from the training data set by machine learning, and store the model parameters. In the prediction stage, the relevant prediction is carried out according to the model parameters.

[0192] As can be seen from the above embodiment, the video coding prediction model processing method provided by the embodiment of the application constructs a prediction model for multi-task learning, so that the model is used to obtain a hierarchical coding unit division graph and a hierarchical coding unit mode graph of a coding tree unit according to a prediction residual image and a prediction reconstruction image of the coding tree unit, the hierarchical coding unit division graph includes a division probability of a coding unit, and the hierarchical coding unit mode graph includes a coding mode probability of the coding unit; and the division mode and the coding mode of the coding tree unit are determined according to the division probability and the coding mode probability. Using this processing method, a deep model is used to predict multiple targets, which can quickly skip unnecessary CU division process and CU intra mode decision process; therefore, the coding complexity under high coding quality can be effectively reduced.

[0193] Fourth embodiment

[0194] In the above embodiment, a video coding prediction model processing method is provided, and a video coding prediction model processing device is also provided. The device corresponds to the above-mentioned method. The embodiment of the present application has the same content as the first embodiment, and the same part will not be repeated. Please refer to the corresponding part in Embodiment One.

[0195] The video coding prediction model processing device provided by the application comprises a training data acquisition unit, a model construction unit and a model training unit.

[0196] The training data acquisition unit is configured to acquire a training data set; the training data comprises a corresponding relationship between residual images and annotation data of a hierarchical coding unit division graph and annotation data of a hierarchical coding unit mode graph; the model construction unit is configured to construct a prediction model for multi-task learning; and the model training unit is configured to train model parameters of the prediction model according to the training data set.

[0197] The fifth embodiment

[0198] The present application also provides an electronic device. Since the device embodiment is basically similar to the method embodiment, it is described more simply, and the relevant parts refer to the part of the method embodiment. The device embodiment described below is only illustrative.

[0199] The electronic device of the present embodiment comprises a processor and a memory; the memory is configured to store a program for implementing the video encoding method or the video encoding prediction model processing method; the device is powered on and runs the program of the method through the processor.

[0200] The sixth embodiment

[0201] In the above-mentioned embodiments, a video encoding method is provided, and the present application also provides a video encoding system corresponding thereto. The system corresponds to the embodiments of the above-mentioned method. The same parts of the present embodiment as the first embodiment are not described again, and please refer to the corresponding part in the first embodiment.

[0202] The video encoding system provided by the present application comprises any of the above-mentioned video encoding devices and a video decoding device. The video decoding device belongs to the existing technical field, and a relatively mature video decoding technology can be used, which is not described again here. The video encoding system can be used in live streaming scenarios, on-demand scenarios, video conference scenarios, etc.

[0203] The seventh embodiment

[0204] In the above-mentioned embodiments, a video encoding method is provided, and the present application also provides a video live streaming method for a live streaming scenario. The method corresponds to the embodiments of the above-mentioned method. The same parts of the present embodiment as the first embodiment are not described again, and please refer to the corresponding part in the first embodiment.

[0205] The video live streaming method provided by the present application comprises the following steps:

[0206] Step 201: acquiring a coding tree unit of a video frame included in a live video.

[0207] The live video also comprises a plurality of video frames, and the processing method of the video frames is as described in the first embodiment, which is not described again here.

[0208] Step 203: obtaining a prediction residual image and a prediction reconstructed image of the coding tree unit.

[0209] Step 205: obtaining, by the prediction model of multi-task learning, a hierarchical coding unit partition map and a hierarchical coding unit mode map of the coding tree unit according to the prediction residual image and the prediction reconstructed image, the hierarchical coding unit partition map comprising a partition probability of a coding unit, and the hierarchical coding unit mode map comprising a coding mode probability of the coding unit.

[0210] Step 207: determining a partition mode and a coding mode of the coding tree unit according to the partition probability and the coding mode probability, so as to generate coding data according to the partition mode and the coding mode of the coding tree unit.

[0211] By adopting the processing manner, the unnecessary CU partition process and CU intra mode decision process can be quickly skipped, and thus the encoding complexity under a high encoding quality can be effectively reduced, and the live video fluency under a high picture quality can be improved.

[0212] Eighth embodiment

[0213] In the above embodiments, a video encoding method is provided, and the present application also provides a video encoding method for video on demand. The method corresponds to the above-mentioned embodiments. The same parts of the present embodiment as those of the first embodiment will not be described herein again, and please refer to the corresponding parts in the first embodiment.

[0214] The video encoding method for video on demand provided by the present application comprises the following steps:

[0215] Step 301: obtaining a coding tree unit of a video frame included in the video on demand.

[0216] The video on demand also comprises a plurality of video frames, and the processing manner of the video frames is as described in the first embodiment, which will not be described herein again.

[0217] Step 303: obtaining a prediction residual image and a prediction reconstructed image of the coding tree unit.

[0218] Step 305: obtaining, by the prediction model of multi-task learning, a hierarchical coding unit partition map and a hierarchical coding unit mode map of the coding tree unit according to the prediction residual image and the prediction reconstructed image, the hierarchical coding unit partition map comprising a partition probability of a coding unit, and the hierarchical coding unit mode map comprising a coding mode probability of the coding unit.

[0219] Step 307: determining the partition mode and the coding mode of the coding tree unit according to the partition probability and the coding mode probability, to generate coding data according to the partition mode and the coding mode of the coding tree unit.

[0220] With this processing manner, unnecessary CU partition process and CU intra mode decision process can be quickly skipped; therefore, the encoding complexity under high encoding quality can be effectively reduced, so as to improve the on-demand video fluency under high picture quality.

[0221] Ninth embodiment

[0222] In the above embodiments, a video encoding method is provided, and the present application also provides a video encoding method for video call. The method corresponds to the above-mentioned embodiments. The same parts of the present embodiment as the first embodiment are not described again, please refer to the corresponding parts in the first embodiment.

[0223] The video encoding method for video call provided by the present application comprises the following steps:

[0224] Step 401: acquiring a coding tree unit of a video frame included in a video call.

[0225] The video call also includes a plurality of video frames, and the processing manner of the video frames is as described in the first embodiment, which is not described again here.

[0226] Step 403: acquiring a prediction residual image and a prediction reconstruction image of the coding tree unit.

[0227] Step 405: acquiring a hierarchical coding unit partition map and a hierarchical coding unit mode map of the coding tree unit according to the prediction residual image and the prediction reconstruction image through a multi-task learning prediction model, the hierarchical coding unit partition map comprising a partition probability of a coding unit, and the hierarchical coding unit mode map comprising a coding mode probability of the coding unit.

[0228] Step 407: determining the partition mode and the coding mode of the coding tree unit according to the partition probability and the coding mode probability, to generate coding data according to the partition mode and the coding mode of the coding tree unit.

[0229] With this processing manner, unnecessary CU partition process and CU intra mode decision process can be quickly skipped; therefore, the encoding complexity under high encoding quality can be effectively reduced, so as to improve the on-demand video fluency under high picture quality.

[0230] Tenth embodiment

[0231] In the above embodiment, a video coding method is provided, and the present application also provides a video coding method for video conference. The method corresponds to the above embodiment. The same parts of the present embodiment as the first embodiment are not described again, please refer to the corresponding parts in the first embodiment.

[0232] The present application provides a video coding method for video conference, comprising the following steps:

[0233] Step 501: Obtain the coding tree unit of the video frame included in the video conference.

[0234] The call video also includes a plurality of video frames, and the processing method of the video frames is as described in the first embodiment, which is not described again here.

[0235] Step 503: Obtain the prediction residual image and the prediction reconstruction image of the coding tree unit.

[0236] Step 505: Obtain the hierarchical coding unit division graph and the hierarchical coding unit mode graph of the coding tree unit according to the prediction residual image and the prediction reconstruction image through the multi-task learning prediction model, the hierarchical coding unit division graph includes the division probability of the coding unit, and the hierarchical coding unit mode graph includes the coding mode probability of the coding unit.

[0237] Step 507: Determine the division mode and the coding mode of the coding tree unit according to the division probability and the coding mode probability, so as to generate coding data according to the division mode and the coding mode of the coding tree unit.

[0238] With this processing method, unnecessary CU division process and CU intra mode decision process can be quickly skipped; therefore, the encoding complexity under high encoding quality can be effectively reduced, so as to improve the fluency of the video call picture under high picture quality.

[0239] Although the present application is disclosed with the preferred embodiments, it is not intended to limit the present application, any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application, therefore, the protection scope of the present application should be subject to the scope defined by the claims of the present application.

[0240] In a typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memories.

[0241] The memory can include non-persistent memory in computer readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer readable media.

[0242] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media, such as modulated data signals and carriers.

[0243] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as a method, system or computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A method of video coding, the method comprising: The method comprises the following steps: obtaining a coding tree unit of a video frame; obtaining a prediction residual image and a prediction reconstruction image of the coding tree unit; obtaining shared features of multiple levels by a prediction model of multi-task learning according to the prediction residual image and the prediction reconstruction image; obtaining a hierarchical coding unit division map of the coding tree unit according to the shared features corresponding to the first level, the second level and the third level respectively; and obtaining a hierarchical coding unit mode map of the coding tree unit according to the shared features corresponding to the first level, the second level, the third level and the fourth level respectively, wherein the hierarchical coding unit division map comprises a division probability of a coding unit, and the hierarchical coding unit mode map comprises a coding mode probability of the coding unit; determining a division mode and a coding mode of the coding tree unit according to the division probability and the coding mode probability.

2. The method of claim 1, wherein, The method of obtaining the prediction residual image and the prediction reconstruction image of the coding tree unit comprises the following steps: taking a reconstruction image generated by inter-frame coding on the coding tree unit as the prediction reconstruction image; determining the prediction residual image according to the prediction reconstruction image.

3. The method of claim 1, wherein, The method of determining the division mode and the coding mode of the coding tree unit according to the division probability and the coding mode probability comprises the following steps: determining whether to skip the coding process of the coding unit at the current depth and / or the coding process of the coding unit division mode according to the division probability; determining whether to skip the intra-frame coding process or the inter-frame coding process of the coding unit according to the coding mode probability.

4. A method for video coding prediction model processing, the method comprising: The method comprises the following steps: obtaining a training data set; the training data comprises a corresponding relationship between residual images and reconstruction images and labeled data of a hierarchical coding unit division map and labeled data of a hierarchical coding unit mode map; constructing a prediction model of multi-task learning, wherein the prediction model is used to obtain shared features of multiple levels according to residual images and reconstruction images; obtain a hierarchical coding unit division map of a coding tree unit according to the shared features corresponding to the first level, the second level and the third level respectively; and obtain a hierarchical coding unit mode map of the coding tree unit according to the shared features corresponding to the first level, the second level, the third level and the fourth level respectively, wherein the hierarchical coding unit division map comprises a division probability of a coding unit, and the hierarchical coding unit mode map comprises a coding mode probability of the coding unit; training model parameters of the prediction model according to the training data set.

5. A video encoding apparatus, comprising: The method comprises the following steps: a coding tree unit obtaining unit is configured to obtain a coding tree unit of a video frame; a first coding unit is configured to obtain a prediction residual image and a prediction reconstruction image of the coding tree unit; a prediction unit is configured to obtain shared features of multiple levels by a prediction model of multi-task learning according to the prediction residual image and the prediction reconstruction image; obtain a hierarchical coding unit division map of a coding tree unit according to the shared features corresponding to the first level, the second level and the third level respectively; and obtain a hierarchical coding unit mode map of the coding tree unit according to the shared features corresponding to the first level, the second level, the third level and the fourth level respectively, wherein the hierarchical coding unit division map comprises a division probability of a coding unit, and the hierarchical coding unit mode map comprises a coding mode probability of the coding unit; The second encoding unit is configured to determine a partition mode and an encoding mode of the coding tree unit according to the partition probability and the encoding mode probability.

6. An electronic device, comprising: The method comprises: a processor and a memory; the memory is configured to store a program for implementing the method according to any one of claims 1-4, and the device is powered on and runs the program of the method by the processor.

7. A video processing system characterized by The method comprises: The video encoding apparatus according to claim 5; and a video decoding apparatus.

8. A method for video coding for video live streaming, characterized in that, The method comprises: obtaining a coding tree unit of a video frame included in a live video; obtaining a prediction residual image and a prediction reconstruction image of the coding tree unit; obtaining shared features of multiple levels by a multi-task learning prediction model according to the prediction residual image and the prediction reconstruction image; obtaining a hierarchical coding unit partition map of the coding tree unit according to the shared features corresponding to the first level, the second level and the third level respectively; and obtaining a hierarchical coding unit mode map of the coding tree unit according to the shared features corresponding to the first level, the second level, the third level and the fourth level respectively, wherein the hierarchical coding unit partition map comprises a partition probability of a coding unit, and the hierarchical coding unit mode map comprises an encoding mode probability of the coding unit. determining a partition mode and an encoding mode of the coding tree unit according to the partition probability and the encoding mode probability, and generating encoding data according to the partition mode and the encoding mode of the coding tree unit.

9. A method for video coding for video on demand, characterized by, The method comprises: obtaining a coding tree unit of a video frame included in a live video; obtaining a prediction residual image and a prediction reconstruction image of the coding tree unit; obtaining shared features of multiple levels by a multi-task learning prediction model according to the prediction residual image and the prediction reconstruction image; obtaining a hierarchical coding unit partition map of the coding tree unit according to the shared features corresponding to the first level, the second level and the third level respectively; and obtaining a hierarchical coding unit mode map of the coding tree unit according to the shared features corresponding to the first level, the second level, the third level and the fourth level respectively, wherein the hierarchical coding unit partition map comprises a partition probability of a coding unit, and the hierarchical coding unit mode map comprises an encoding mode probability of the coding unit. determining a partition mode and an encoding mode of the coding tree unit according to the partition probability and the encoding mode probability, and generating encoding data according to the partition mode and the encoding mode of the coding tree unit.

10. A method of video coding for video call, characterized by, The method comprises: obtaining a coding tree unit of a video frame included in a live video; obtaining a prediction residual image and a prediction reconstruction image of the coding tree unit; obtaining shared features of multiple levels by a multi-task learning prediction model according to the prediction residual image and the prediction reconstruction image; obtaining a hierarchical coding unit partition map of the coding tree unit according to the shared features corresponding to the first level, the second level and the third level respectively; and obtaining a hierarchical coding unit mode map of the coding tree unit according to the shared features corresponding to the first level, the second level, the third level and the fourth level respectively, wherein the hierarchical coding unit partition map comprises a partition probability of a coding unit, and the hierarchical coding unit mode map comprises an encoding mode probability of the coding unit. According to the division probability and the coding mode probability, a division mode and a coding mode of the coding tree unit are determined to generate coding data according to the division mode and the coding mode of the coding tree unit.

11. A method for video coding for video conferencing, characterized in that, Comprise: Obtain the coding tree unit of the video frame included in the video conference; Obtain the prediction residual image and the prediction reconstruction image of the coding tree unit; Through the prediction model of multi-task learning, obtain the shared features of multiple levels according to the prediction residual image and the prediction reconstruction image; obtain the hierarchical coding unit division graph of the coding tree unit according to the shared features corresponding to the first level, the second level and the third level respectively; obtain the hierarchical coding unit mode graph of the coding tree unit according to the shared features corresponding to the first level, the second level, the third level and the fourth level respectively, the hierarchical coding unit division graph comprises the division probability of the coding unit, and the hierarchical coding unit mode graph comprises the coding mode probability of the coding unit; According to the division probability and the coding mode probability, a division mode and a coding mode of the coding tree unit are determined to generate coding data according to the division mode and the coding mode of the coding tree unit.

Images