Video encoding method and apparatus, video decoding method and apparatus, and electronic device

Abstract

The present application discloses a video encoding method and apparatus, a video decoding method and apparatus, and an electronic device, belonging to the technical field of video compression. The method includes: dividing a CU in a video frame in at least one division direction to obtain sub-block information of the CU in each division direction; predicting the CU in a network model ISP mode according to the reference information, sub-block information of the CU, and a pre-established network prediction model, and predicting the CU in a conventional ISP mode; determining the syntax element information of the CU according to the prediction errors of the network model ISP mode and the conventional ISP mode corresponding to each division direction; and sending the syntax element information of the CU, where the syntax element information includes division information and ISP mode indication information. In this way, as long as the prediction error of the network model ISP mode of the CU in the video frame is less than the prediction error of the conventional ISP mode, it is expected to improve the overall compression ratio of the video frame and save the bit rate.

Description

Background Art

[0002] An important objective of video encoding and decoding technology is to compress video data into a form using a lower bit rate while avoiding or minimizing video quality loss. In related technologies, in order to improve the encoding performance, an IntraSub-Partition (ISP) mode has been proposed to encode Coding Units (CUs) in a video frame. However, the encoding performance of this solution still needs to be improved. Summary of the Invention

[0003] Embodiments of the present application provide a video encoding method and apparatus, a video decoding method and apparatus, and an electronic device to improve the encoding performance of video frames.

[0004] In a first aspect, embodiments of the present application provide a video encoding method, including:

[0005] Partitioning a Coding Unit (CU) in a video frame in at least one partitioning direction to obtain sub-block information of the CU in each partitioning direction;

[0006] Predicting the CU using a model IntraSub-Partition (ISP) mode according to the reference information of the CU, the sub-block information, and a pre-established network prediction model, where the reference information at least includes reference pixels adjacent to the CU in the reconstructed pixel region of the video frame;

[0007] Predicting the CU using a conventional ISP mode;

[0008] Determining syntax element information of the CU according to the prediction errors of the network model ISP mode corresponding to each partitioning direction and the prediction error of the conventional ISP mode, where the syntax element information includes partitioning information and ISP mode indication information, and the ISP mode indication information is used to indicate whether to use the network model ISP mode for decoding;

[0009] Sending the syntax element information of the CU.

[0010] In some embodiments, predicting the sub-block using a network model ISP mode according to the encoding end reference information of the CU, the sub-block information, and a pre-established network prediction model includes:

[0011] Determining a target network prediction model according to the sub-block information and the correspondence between the sub-block information in the partitioning direction and the network prediction model;

[0012] Predicting the CU using a network model ISP mode according to the target network prediction model and the encoding end reference information of the CU.

[0013] In some embodiments, the target network prediction model includes N sub-network prediction units, where N is the number of sub-blocks of the CU in the partitioning direction. Predicting the CU using the network model ISP mode according to the target network prediction model and the coding-end reference information of the CU includes:

[0014] Input the coding-end reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction to obtain the model prediction result of the i-th sub-block. The target sub-block is selected from the sub-blocks that have been predicted in the CU in the partitioning direction. i is an integer greater than 1. The model prediction result of the first sub-block is obtained by inputting the coding-end reference information into the first sub-network prediction unit for prediction;

[0015] When i is less than N, update i to i + 1, and execute the step of inputting the coding-end reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction until i is equal to N.

[0016] In some embodiments, the target sub-block is all the sub-blocks that have been predicted, or the target sub-block is the sub-block that has been predicted and is the closest to the i-th sub-block.

[0017] In some embodiments, the coding-end reference information further includes quantization parameter information, and

[0018] The syntax element information further includes the quantization parameter information.

[0019] In some embodiments, the input of each sub-network prediction unit further includes the conventional prediction result of the corresponding sub-block, where the conventional prediction result refers to the prediction result obtained by using the conventional ISP mode; and

[0020] The syntax element information further includes the result inference information of the conventional prediction results of each sub-block.

[0021] In some embodiments, the reference pixels include M pixel reference rows and M pixel reference columns, where M is an integer greater than zero.

[0022] In some embodiments, the target network prediction model is trained according to the following steps:

[0023] Divide the obtained CU sample into N sub-block samples in the partitioning direction;

[0024] Taking the reference information of the CU sample as the input and the pixels of the i-th sub-block sample as the output, training the i-th sub-network unit in the preset network model, after determining that the prediction loss value of the i-th sub-network unit is lower than the first set value, updating i to i + 1, and performing the step of training the i-th sub-network unit in the preset network model with the reference information of the CU sample as the input and the pixels of the i-th sub-block sample as the output, until it is determined that the prediction loss value of the N-th sub-network unit is lower than the first set value, taking the current preset model as the target network prediction model; or

[0025] Taking the reference information of the CU sample as the input and the pixels of N sub-block samples as the output, training N sub-network units in the preset network model synchronously, and when it is determined that the total prediction loss value of the N sub-network units is lower than the second set value, taking the current preset model as the target network prediction model.

[0026] In a second aspect, an embodiment of the present application provides a video decoding method, including:

[0027] Receiving syntax element information of a coding unit CU in a video frame, where the syntax element information includes partitioning information and an intra-sub-block ISP mode indication information, and the ISP mode indication information is used to indicate whether to perform decoding using the network model ISP mode;

[0028] Determining sub-block information of the CU in a partitioning direction according to the size of the CU and the partitioning information;

[0029] In response to the indication of using the network model ISP mode for decoding by the ISP mode indication information, predicting the CU using the network model ISP mode according to the reference information of the CU, the sub-block information, and a pre-established network prediction model, where the reference information at least includes reference pixels adjacent to the CU in the reconstructed pixel region of the video frame;

[0030] Reconstructing the CU according to the prediction result.

[0031] In some embodiments, predicting the CU using the network model ISP mode according to the decoding-end reference information of the CU, the sub-block information, and a pre-established network prediction model includes:

[0032] Determining a target network prediction model according to the sub-block information and the established correspondence between the sub-block information in the partitioning direction and the network prediction model;

[0033] Predicting the CU using the network model ISP mode according to the target network prediction model and the decoding-end reference information of the CU.

[0034] In some embodiments, the target network prediction model includes N sub-network prediction units, where N is the number of the sub-blocks. Predicting the CU by using the network model ISP mode according to the target network prediction model and the decoding end reference information of the CU includes:

[0035] Inputting the decoding end reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction to obtain the model prediction result of the i-th sub-block. The target sub-block is selected from the sub-blocks that have been predicted in the CU in the partitioning direction. i is an integer greater than 1. The model prediction result of the first sub-block is obtained by inputting the decoding end reference information into the first sub-network prediction unit for prediction;

[0036] When i is less than N, update i to i + 1, and execute the step of inputting the decoding end reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction until i is equal to N.

[0037] In some embodiments, the target sub-block is all the sub-blocks that have been predicted, or the target sub-block is a sub-block that has been predicted and is the closest to the i-th sub-block.

[0038] In some embodiments, the syntax element information further includes quantization parameter information, and

[0039] the decoding end reference information further includes the quantization parameter information.

[0040] In some embodiments, the syntax element information further includes result inference information of the conventional prediction results of each sub-block. The conventional prediction result refers to the prediction result obtained by performing conventional prediction in the ISP mode, and further includes:

[0041] Determine the conventional prediction results of each sub-block according to the result inference information;

[0042] Use the conventional prediction result of each sub-block as an input to the corresponding sub-network prediction unit.

[0043] In some embodiments, the reference pixels include M pixel reference rows and M pixel reference columns, where M is an integer greater than zero.

[0044] In a third aspect, an embodiment of the present application provides a video encoding apparatus, including:

[0045] A partitioning module, configured to partition an encoding unit CU in a video frame in at least one partitioning direction to obtain sub-block information of the CU in each partitioning direction;

[0046] The first prediction module is configured to perform prediction on the CU according to the reference information of the CU, the sub-block information, and a pre-established network prediction model, and adopt the in-model-frame sub-block ISP mode to predict the CU, where the reference information at least includes the reference pixels adjacent to the CU in the reconstructed pixel region of the video frame;

[0047] The second prediction module is configured to perform prediction on the CU by using the conventional ISP mode;

[0048] The determination module is configured to determine the syntax element information of the CU according to the prediction errors of the network model ISP modes corresponding to each partitioning direction and the prediction error of the conventional ISP mode, where the syntax element information includes partitioning information and ISP mode indication information, and the ISP mode indication information is used to indicate whether to perform decoding by using the network model ISP mode;

[0049] The sending module is configured to send the syntax element information of the CU.

[0050] In some embodiments, the first prediction module is specifically configured to:

[0051] Determine a target network prediction model according to the sub-block information and the correspondence between the sub-block information in the partitioning direction and the network prediction model established;

[0052] Perform prediction on the CU by using the network model ISP mode according to the target network prediction model and the reference information at the encoding end of the CU.

[0053] In some embodiments, the target network prediction model includes N sub-network prediction units, where N is the number of sub-blocks of the CU in the partitioning direction, and the first prediction module is specifically configured to:

[0054] Input the reference information at the encoding end and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction to obtain the model prediction result of the i-th sub-block, where the target sub-block is selected from the sub-blocks that have been predicted in the partitioning direction of the CU, i is an integer greater than 1, and the model prediction result of the first sub-block is obtained by inputting the reference information at the encoding end into the first sub-network prediction unit for prediction;

[0055] When i is less than N, update i to i + 1, and execute the step of inputting the reference information at the encoding end and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction until i is equal to N.

[0056] In some embodiments, the target sub-block is all the sub-blocks that have been predicted, or the target sub-block is the sub-block that has been predicted and is the closest to the i-th sub-block.

[0057] In some embodiments, the encoding end reference information further includes quantization parameter information, and

[0058] the syntax element information further includes the quantization parameter information.

[0059] In some embodiments, the input of each sub-network prediction unit further includes the conventional prediction result of the corresponding sub-block, where the conventional prediction result refers to the prediction result obtained by using the conventional ISP mode for prediction; and

[0060] the syntax element information further includes the result inference information of the conventional prediction results of each sub-block.

[0061] In some embodiments, the reference pixels include M pixel reference rows and M pixel reference columns, where M is an integer greater than zero.

[0062] Fourthly, an embodiment of the present application provides a video decoding device, including:

[0063] a receiving module, configured to receive the syntax element information of a coding unit CU in a video frame, where the syntax element information includes partitioning information and an intra-sub-block ISP mode indication information, and the ISP mode indication information is used to indicate whether to use the network model ISP mode for decoding;

[0064] a determining module, configured to determine the sub-block information of the CU in a partitioning direction according to the size of the CU and the partitioning information;

[0065] a prediction module, configured to, in response to the use of the network model ISP mode for decoding indicated by the ISP mode indication information, perform prediction on the CU by using the network model ISP mode according to the reference information of the CU, the sub-block information, and a pre-established network prediction model, where the reference information at least includes the reference pixels adjacent to the CU in the reconstructed pixel region of the video frame;

[0066] a reconstruction module, configured to reconstruct the CU according to the prediction result.

[0067] In some embodiments, the prediction module is specifically configured to:

[0068] determine a target network prediction model according to the sub-block information and the established correspondence between the sub-block information in the partitioning direction and the network prediction model;

[0069] perform prediction on the CU by using the network model ISP mode according to the target network prediction model and the decoding end reference information of the CU.

[0070] In some embodiments, the target network prediction model includes N sub-network prediction units, where N is the number of the sub-blocks, and the prediction module is specifically configured to:

[0071] Input the decoded end reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction to obtain the model prediction result of the i-th sub-block. The target sub-block is selected from the sub-blocks that have been predicted in the CU in the partitioning direction. i is an integer greater than 1. The model prediction result of the first sub-block is obtained by inputting the decoded end reference information into the first sub-network prediction unit for prediction;

[0072] When i is less than N, update i to i + 1 and execute the step of inputting the decoded end reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction until i is equal to N.

[0073] In some embodiments, the target sub-block is all the sub-blocks that have been predicted, or the target sub-block is the sub-block that has been predicted and is the closest to the i-th sub-block.

[0074] In some embodiments, the syntax element information further includes quantization parameter information, and

[0075] the decoded end reference information further includes the quantization parameter information.

[0076] In some embodiments, the syntax element information further includes result inference information of the conventional prediction results of each sub-block. The conventional prediction result refers to the prediction result obtained by performing prediction in the conventional ISP mode, and further includes:

[0077] Determine the conventional prediction results of each sub-block according to the result inference information;

[0078] Use the conventional prediction result of each sub-block as an input to the corresponding sub-network prediction unit.

[0079] In some embodiments, the reference pixels include M pixel reference rows and M pixel reference columns, where M is an integer greater than zero.

[0080] In a fifth aspect, an embodiment of the present application provides an electronic device, including: at least one processor, and a memory communicatively connected to the at least one processor, wherein:

[0081] The memory stores instructions executable by the at least one processor. When the instructions are executed by the at least one processor, the at least one processor is enabled to execute the above video encoding method or video decoding method.

[0082] In a sixth aspect, an embodiment of the present application provides a storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the above video encoding method or video decoding method.

[0083] In the embodiments of the present application, the CUs in a video frame are partitioned in at least one partitioning direction to obtain the sub-block information of the CUs in each partitioning direction. According to the reference information of the CUs, the sub-block information, and a pre-established network prediction model, the CUs are predicted using the network model ISP mode and the CUs are predicted using the conventional ISP mode. Then, according to the prediction errors of the network model ISP mode corresponding to each partitioning direction and the prediction errors of the conventional ISP mode, the syntax element information of the CUs is determined. The syntax element information includes partitioning information and ISP mode indication information. The ISP mode indication information is used to indicate whether to use the network model ISP mode for decoding. Then, the syntax element information of the CUs is sent. In this way, a brand-new network model ISP mode is proposed, and it can compete with the conventional ISP mode for intra prediction as a sub-mode of the ISP mode. As long as the prediction error of the network model ISP mode of the CUs in the video frame is less than the prediction error of the conventional ISP mode, it is expected to improve the overall compression ratio of the video frame and save the bit rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0085] Figure 1 It is a schematic diagram of an intra prediction mode provided by an embodiment of the present application;

[0086] Figure 2 It is a schematic diagram of the relative positions of a CU and reference pixels provided by an embodiment of the present application;

[0087] Figure 3 It is a schematic diagram of the partitioning of a CU provided by an embodiment of the present application;

[0088] Figure 4 It is another schematic diagram of the partitioning of a CU provided by an embodiment of the present application;

[0089] Figure 5 It is a schematic diagram of the network structure of a network prediction model provided by an embodiment of the present application;

[0090] Figure 6 It is another schematic diagram of the network structure of a network prediction model provided by an embodiment of the present application;

[0091] Figure 7 It is another schematic diagram of the network structure of a network prediction model provided by an embodiment of the present application;

[0092] Figure 8 It is another schematic diagram of the network structure of a network prediction model provided by an embodiment of the present application;

[0093] Figure 9 Flow chart of a video encoding method provided by an embodiment of the present application;

[0094] Figure 10 Schematic diagram of the correspondence between sub-block information in different partitioning directions and a network prediction model provided by an embodiment of the present application;

[0095] Figure 11 Flow chart of a video decoding method provided by an embodiment of the present application;

[0096] Figure 12 Schematic diagram of the structure of a video encoding device provided by an embodiment of the present application;

[0097] Figure 13 Schematic diagram of the structure of a video decoding device provided by an embodiment of the present application;

[0098] Figure 14 Schematic diagram of the hardware structure of an electronic device for implementing a video encoding method or a video decoding method provided by an embodiment of the present application. Detailed implementation manners

[0099] In order to improve the encoding performance of video frames, an embodiment of the present application provides a video encoding method and device, a video decoding method and device, and an electronic device.

[0100] The following describes the preferred embodiments of the present application with reference to the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. And without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0101] For the convenience of understanding the present application, in the technical terms involved in the present application:

[0102] The conventional ISP mode refers to predicting each sub-block obtained by dividing the CU using the traditional ISP mode. Its prediction process is generally as follows: The CU is divided into multiple sub-blocks in a dividing direction, and each sub-block is predicted in turn. During one prediction process, the prediction modes of all sub-blocks are the same. Among them, when predicting the first sub-block, the reconstructed pixels around the CU can be obtained as reference pixels, and the current prediction mode is used for prediction to obtain the predicted value of the first sub-block. When predicting subsequent sub-blocks (i.e., non-first sub-blocks), a part of the reconstructed pixels of the previous sub-block and a part of the reconstructed pixels around the CU can be obtained as reference pixels, and the current prediction mode is used for prediction to obtain the predicted value of the current sub-block. After predicting all sub-blocks using the current prediction mode, the prediction cost value of the CU is calculated as the cost value of the current prediction mode in the corresponding dividing direction. Then, the cost values of multiple prediction modes in each dividing direction are compared, and the prediction mode with the smallest cost value is selected as the final sub-block prediction mode of the CU, and the dividing direction with the smallest cost value is selected as the final dividing direction of the CU.

[0103] The network model ISP mode refers to predicting each sub-block obtained by dividing the CU using a network prediction model.

[0104] The video encoding and video decoding solutions provided in the embodiments of the present application belong to a part of intra-frame prediction. Therefore, intra-frame prediction is introduced first.

[0105] Video is formed by continuously playing many static images, and each static image can be regarded as a video frame. Since the pixel values of adjacent pixel points in a video frame are usually relatively close and the color generally does not change suddenly, spatial correlation can be used to compress the video frame, and this compression technology is intra-frame prediction.

[0106] Intra-frame prediction is to predict the pixel value of the current pixel point based on the pixel values of the pixel points around the current pixel point (i.e., reference pixels) within a video frame. Currently, the intra-frame prediction modes include Direct Current (DC) mode, Planar mode, and multiple ordinary angle modes, and the numbers of the ordinary angle modes range from 2 to 66.

[0107] In addition to these modes, for the case where the reference pixels adjacent to the long side of a non-square block have stronger correlation than the reference pixels adjacent to the short side, some wide-angle modes are added, which can make the reference pixels be selected as the reference pixels adjacent to the long side as much as possible. If N is 66, all intra-frame prediction modes including wide-angle modes Figure 1 are shown in the figure. Among them, 2 to 66 are ordinary angle modes, and angle modes -13 to 1 and 67 to 81 are wide-angle modes, which represent different prediction directions respectively. Modes 18 and 50 correspond to the horizontal direction and the vertical direction respectively.

[0108] Figure 2 This is a schematic diagram of the relative positions of a CU and reference pixels provided by an embodiment of the present application. There are n reference rows above the CU: TR1, TR2, TR3...TRn, and n reference columns on the left: LR1, LR2, LR3...LRn. The reference pixels of the CU are obtained from these reference rows and reference columns.

[0109] The general process of intra-frame prediction is as follows: Find the corresponding reference pixels on each reference row or reference column in the direction indicated by the prediction mode. If the reference pixels on all the found reference rows or reference columns are integer pixels, select the integer pixel value on the reference line with the smallest cost value as the predicted value of the current pixel; if there is a fractional pixel among the reference pixels on a certain reference row or reference column, first perform interpolation through the integer pixels on both sides (or above and below) of this fractional pixel, then compare the cost values based on the interpolated values, and select the reference pixel value with the smallest cost value as the predicted value of the current pixel.

[0110] In addition, there is a special intra-frame prediction mode: the ISP mode. First, the CU is divided, and then, predictions are made for each sub-block obtained by the division respectively.

[0111] Next, the division of the CU will be introduced first.

[0112] 1. Division conditions of the CU.

[0113] Generally, only CUs larger than 4*4 are divided. And, 4*8 and 8*4 CUs are divided into 2 sub-blocks, and CUs of other sizes are divided into 4 sub-blocks.

[0114] 2. Division methods of the CU.

[0115] Generally, the division directions include the vertical direction and the horizontal direction. Each CU can only choose to be divided in one division direction at a time, and cannot be divided in both the vertical direction and the horizontal direction at the same time. And, the sizes of the sub-blocks obtained by dividing the CU are the same, and each sub-block contains at least 16 pixel points, that is, the width * height of each sub-block >= 16.

[0116] Figure 3 This is a schematic diagram of the division of a CU provided by an embodiment of the present application. Among them, the width W * height H of the CU is 4*8 or 8*4. The CU can be divided into 2 sub-blocks in the vertical direction or can be divided into 2 sub-blocks in the horizontal direction. Figure 4 This is another schematic diagram of the division of a CU provided by an embodiment of the present application. Among them, the CU is of a size other than 4*8 or 8*4. The CU can be divided into 4 sub-blocks in the vertical direction or can be divided into 4 sub-blocks in the horizontal direction.

[0117] Next, the prediction process of the ISP mode will be introduced.

[0118] The process is generally as follows: The CU is divided into multiple sub-blocks in a partitioning direction, and each sub-block is predicted in turn. The prediction mode of all sub-blocks is the same during one prediction process. Among them, when predicting the first sub-block, the reconstructed pixels around the CU can be obtained as reference pixels, and the current prediction mode is used for prediction to obtain the predicted value of the first sub-block. When predicting a non-first sub-block, a part of the reconstructed pixels of the previous sub-block and a part of the reconstructed pixels around the CU can be obtained as reference pixels, and the current prediction mode is used for prediction to obtain the predicted value of the current sub-block. After predicting all sub-blocks using the current prediction mode, the prediction cost value of the CU is calculated as the cost value of the current prediction mode in the corresponding partitioning direction. Then, the cost values of multiple prediction modes in each partitioning direction are compared, and the prediction mode with the smallest cost value is selected as the final sub-block prediction mode of the CU, and the partitioning direction with the smallest cost value is selected as the final partitioning direction of the CU.

[0119] However, the inventors found that when predicting sub-blocks in the ISP mode, only the first reference row on the upper side and the first reference column on the left side of the CU are considered, and other reference rows and reference columns are not effectively utilized, which is not conducive to improving the coding efficiency.

[0120] In order to improve the coding efficiency, the embodiments of the present application provide a brand-new network model ISP mode, which is used as a sub-mode of the IPS mode and competes with the conventional ISP mode.

[0121] First, the network prediction model used in the network model ISP mode will be introduced below.

[0122] 1. Input and output of the network prediction model.

[0123] Select n reference rows from the upper side of the CU from near to far, and select n reference columns from the left side of the CU from near to far. These n reference rows can be adjacent to each other or not, and these n reference columns can be adjacent to each other or not. It is only necessary that the row numbers of these n reference rows and the column numbers of these n reference columns are the same. These n reference rows and these n reference columns can form an L-shaped reference pixel region. The reference region is processed into several sub-regions with the same size, and the pixel data of each sub-region are spliced as an initial input of the network prediction model. In addition, side information such as quantization parameter information (QPmap) can also be used as an initial input of the network prediction model to better guide the network prediction model to make predictions.

[0124] When the network prediction model includes multiple sub-network prediction units (each sub-network prediction unit is used to predict a sub-block), in addition to the above initial input, the input of each sub-network prediction unit may further include the conventional prediction result corresponding to the sub-block (i.e., the prediction result obtained by using the conventional ISP mode for prediction), and the model prediction result of the sub-block that has been predicted in the corresponding division direction of the CU (i.e., the prediction result obtained by the sub-network prediction unit).

[0125] The output of the network prediction model is the pixel values of each sub-block in the CU. After all sub-blocks are predicted, the pixel values of each sub-block can be reconstructed into a CU.

[0126] 2. Network structure of the network prediction model.

[0127] In specific implementation, the network prediction model may include multiple sub-network prediction units, and the sub-network prediction units correspond to the sub-blocks in the CU one by one. Based on the output of the previous sub-network prediction units connected to each sub-network prediction unit, two network prediction models with the following structures can be constructed:

[0128] The first type: Each sub-network prediction unit except the first sub-network prediction unit refers to the prediction result of the previous sub-network prediction unit (i.e., each sub-block except the first sub-block refers to the prediction result of the previous sub-block for prediction).

[0129] Taking n = 4 as an example, Figure 5 is a schematic diagram of the network structure of a network prediction model provided by an embodiment of the present application. Among them, the prediction network includes N sub-network prediction units, and the Nth sub-network prediction unit is used to predict the Nth sub-block of the CU. Each sub-network prediction unit except the first sub-network prediction unit refers to the prediction result of the previous sub-network prediction unit for prediction.

[0130] The second type: Each sub-network prediction unit except the first sub-network prediction unit refers to the prediction results of all the previous sub-network prediction units (i.e., each sub-block except the first sub-block refers to the prediction results of all the previous sub-blocks for prediction).

[0131] Taking n = 4 as an example, Figure 6 is a schematic diagram of the network structure of another network prediction model provided by an embodiment of the present application. Among them, the prediction network includes N sub-network prediction units, and the Nth sub-network prediction unit is used to predict the Nth sub-block of the CU. Each sub-network prediction unit except the first sub-network prediction unit refers to the prediction results of all the previous sub-network prediction units for prediction.

[0132] Since the pixel values of adjacent pixels in a video frame are relatively close, regardless of which of the above structures is used, the current sub-block can be predicted by means of the prediction results of adjacent sub-blocks of the current sub-block, thereby improving the prediction accuracy of the current sub-block.

[0133] In specific implementation, each sub-network prediction unit may be composed of a convolutional structure and / or a fully connected structure. Moreover, for smaller-sized sub-blocks, there are more fully connected layers and fewer or no convolutional layers in the sub-network prediction unit; for larger-sized sub-blocks, there are more convolutional layers and fewer or no fully connected layers in the sub-network prediction unit. Additionally, the positions of the convolutional layer and the fully connected layer in the sub-network prediction unit can be freely arranged.

[0134] Assume that the size of the CU is 8*4 and it is horizontally divided into two 8*2 sub-blocks. The network structure of the network prediction model selects the first type, and 4 reference rows and 4 reference columns are selected for the reference pixel region.

[0135] In some embodiments, each sub-network prediction unit in the network prediction model uses a fully connected structure, and the network structure of the network prediction model can be as Figure 7 shown. Among them, the reference pixel region is divided into 7 small regions of 4*4, and these small regions are merged into 7 channels as an initial input of the network prediction model. The Flatten layer is used to convert the input data into a one-dimensional vector. The first fully connected layer FC in the sub-network prediction unit 1 is used to extract features from the output of Flatten. The activation layer Relu is used to perform non-linear processing on the output of Flatten. The second FC is used to extract features from the output of Relu. After that, the output result of the second FC is reshaped to obtain the prediction result of sub-block 1. C represents concat, which is used to fuse the prediction result of the sub-network prediction unit 1 and the input of the network prediction model and use it as the input of the sub-network prediction unit 2. The first FC in the sub-network prediction unit 2 is used to extract features from the input data. Relu is used to perform non-linear processing on the output of the first FC. The second FC is used to extract features from the output of Relu. Then, the output result of the second FC is reshaped to obtain the prediction result of sub-block 2. Finally, based on the prediction results of sub-block 1 and sub-block 2, reconstruction is performed to obtain the prediction result of the CU.

[0136] In some embodiments, each sub-network prediction unit in the network prediction model uses a convolutional structure, and the network structure of the network prediction model can be as Figure 8As shown in the figure. Among them, the reference pixel region is divided into 7 small regions of 4*4, and these small regions are merged into 7 channels as an initial input of the network prediction model. The first convolutional layer Conv in the sub-network prediction unit 1 is used to extract features from the input data, the activation layer Relu is used to perform non-linear processing on the output of the first Conv, and the second Conv is used to extract features from the output of Relu. After that, the output result of the second Conv is reshaped to obtain the prediction result of sub-block 1. C represents concat, which is used to fuse the prediction result of the sub-network prediction unit 1 and the input of the network prediction model and then use it as the input of the sub-network prediction unit 2. The first Conv in the sub-network prediction unit 2 is used to extract features from the input data, Relu is used to perform non-linear processing on the output of the first Conv, and the second Conv is used to extract features from the output of Relu. After that, the output result of the second Conv is reshaped to obtain the prediction result of sub-block 2. Finally, based on the prediction results of sub-block 1 and sub-block 2, reconstruction is performed to obtain the prediction result of the CU.

[0137] 3. Training of the network prediction model.

[0138] In the training stage, each sub-block has its own label and corresponding loss function, and the label is the original pixel value of the sub-block.

[0139] The training strategies include the following two:

[0140] a. First, train the sub-network prediction unit corresponding to the first sub-block. When the loss function of the sub-network prediction unit corresponding to the first sub-block reaches the convergence state (such as the value of the loss function is less than the first set value), then train the sub-network prediction unit corresponding to the second sub-block, and repeat this process until the sub-network prediction units corresponding to all sub-blocks reach the convergence state, and then the network prediction model can be obtained.

[0141] b. Train the sub-network prediction units corresponding to all sub-blocks simultaneously. When the sum of the loss functions corresponding to the sub-network prediction units of all sub-blocks reaches the convergence state (such as the sum of the values of the loss functions is less than the second set value) to obtain the network prediction model, end the training to obtain the network prediction model.

[0142] Next, the operations at the encoding end are introduced.

[0143] In order to notify the decoding end whether to use the network model ISP mode or the conventional ISP mode for decoding, the encoding end can transmit a frame-level or block-level flag to the decoding end. For example, when flag = 1, it indicates that the network model ISP mode is used, and when flag = 0, it indicates that the conventional ISP mode is used.

[0144] When the ISP mode is enabled at the encoding end, the sub-blocks obtained by dividing the CU are predicted using the network model ISP mode and the conventional ISP mode respectively. Then, the mode with the smaller cost is selected as the best ISP mode by comparing the rate-distortion costs. Additionally, when predicting using the network model ISP mode, if the network prediction model needs to use the conventional prediction result of the sub-block, the sub-block must be predicted using the conventional ISP mode first.

[0145] For example, set a frame-level syntax flag enable_cnn_ISP. When enable_cnn_ISP is 1, it indicates that the network model ISP mode can be enabled for the current frame. And assume that the network prediction model does not need the conventional prediction result. Then, when the ISP mode is enabled at the encoding end, the sub-blocks obtained by dividing the CU can be predicted using the network model ISP mode and the conventional ISP mode respectively. Assume that the rate-distortion cost of the network model ISP mode is RDcost1 and the rate-distortion cost of the conventional ISP mode is RDcost2, and RDcost1 < RDcost2, then the network model ISP mode is determined as the best ISP mode. To transmit the best ISP mode to the decoding end, a syntax apply_cnn_ISP_mode can be set. Setting apply_cnn_ISP_mode to 1 means using the network model ISP mode.

[0146] Next, the operations at the decoding end are introduced.

[0147] The decoding end can receive the ISP enable flag and the ISP mode indication information sent by the encoding end. Among them, the ISP enable flag is used to indicate the enabling of the ISP mode, and the ISP mode indication information is used to indicate whether to enable the network model ISP mode. If the ISP enable flag indicates the enabling of the ISP mode and the ISP mode indication information indicates the enabling of the network model ISP mode, then the network model ISP mode is used for prediction at the decoding end.

[0148] Continuing with the above example, that is, when enable_cnn_ISP is 1 and apply_cnn_ISP_mode is 1, the decoding end can use the network model ISP mode to predict the current CU.

[0149] The following uses specific embodiments to illustrate the video encoding method proposed in this application. Figure 9 It is a flowchart of a video encoding method provided by an embodiment of this application, including the following steps.

[0150] In step S901, the CU in the video frame is divided in at least one division direction to obtain the sub-block information of the CU in each division direction.

[0151] Among them, the sub-block information of the CU in each partitioning direction includes information such as sub-block size and the number of sub-blocks.

[0152] In specific implementation, some CUs with certain sizes can only be partitioned in the horizontal direction or the vertical direction, and some CUs with certain sizes can be partitioned in both the horizontal direction and the vertical direction. Even if a CU can be partitioned in two partitioning directions, once a partitioning direction is selected, the CU can only be partitioned in this direction, and cannot be partitioned in both directions simultaneously during one partitioning process. For example, a CU can be partitioned into 4 sub-blocks in the horizontal direction, or can be partitioned into 4 sub-blocks in the vertical direction, but cannot be partitioned once in the horizontal direction and once in the vertical direction to obtain 4 sub-blocks.

[0153] In step S902, according to the reference information of the CU, the sub-block information in each partitioning direction, and the pre-established network prediction model, the network model ISP mode is used to predict the CU.

[0154] Among them, the reference information at least includes the reference pixels adjacent to the CU in the reconstructed pixel region of the video frame. Moreover, the reference pixels can include M pixel reference rows and M pixel reference columns, where M is an integer greater than zero. That is to say, the network model ISP mode can use more than one reference row and reference column for prediction. In this way, a larger reference pixel region can be used for prediction, which is beneficial to improving the prediction accuracy.

[0155] Considering that the sizes of the sub-blocks obtained by partitioning a CU in the horizontal direction and the vertical direction are usually different, if the same network prediction model is used for the sub-blocks obtained by partitioning in the horizontal direction and the vertical direction, the prediction effect may not be good. Moreover, the number of sub-blocks obtained by partitioning CUs with different sizes may also be different, and it is also difficult for the same network prediction model to predict CUs containing different numbers of sub-blocks.

[0156] Therefore, multiple network prediction models can be pre-trained based on the partitioning direction and sub-block information. Figure 10 This is a schematic diagram of the correspondence relationship between the sub-block information in different partitioning directions and the network prediction model provided by the embodiment of the present application. Among them, different partitioning directions correspond to different network prediction models, different numbers of sub-blocks obtained in the same partitioning direction correspond to different network prediction models, and for the same number of sub-blocks obtained in the same partitioning direction, different network prediction models can be used for sub-blocks with different sizes. In this way, it is beneficial to improve the prediction accuracy of the network prediction model and improve the coding efficiency.

[0157] It should be noted that Figure 10This is only an example and does not limit the correspondence between the sub-block information and the network prediction model in different partitioning directions in the embodiments of the present application.

[0158] In specific implementation, when predicting the CU using the network model ISP mode according to the reference information of the CU, the sub-block information in each partitioning direction, and the pre-established network prediction model, the target network prediction model can be determined first according to the correspondence between the sub-block information in this partitioning direction and the network prediction model. Then, the CU is predicted using the network model ISP mode according to the target network prediction model and the reference information of the CU.

[0159] See Figure 10 , assuming that the CU is partitioned into 4 sub-blocks in the horizontal direction, and the size of the sub-blocks is 4×8, then the network prediction model h5 can be determined as the target network prediction model. Then, the 4 sub-blocks of 4×8 are predicted using the target network prediction model h5. Assuming that the CU is partitioned into 2 sub-blocks in the vertical direction, and the size of the sub-blocks is 8×8, then the network prediction model v3 can be determined as the target network prediction model. Then, the 2 sub-blocks of 8×8 are predicted using the target network prediction model v3.

[0160] In specific implementation, the sub-network prediction units in the target network prediction model and the sub-blocks in a certain partitioning direction are in one-to-one correspondence. Generally, the reference information of the CU is input into the first sub-network prediction unit for prediction, and the model prediction result of the first sub-block can be obtained. Starting from the i-th sub-network prediction unit (i>1), the reference information and the model prediction result of the target sub-block can be input into the i-th sub-network prediction unit for prediction to obtain the model prediction result of the i-th sub-block, where the target sub-block is selected from the sub-blocks that have been predicted in this partitioning direction of the CU. If i is less than the number of sub-blocks N, then i can be updated to i + 1, and the step of inputting the reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction to obtain the model prediction result of the i-th sub-block is looped until i is equal to N, and the prediction ends.

[0161] In some embodiments, the target sub-blocks are all the sub-blocks that have been predicted. That is, starting from the second sub-block, each sub-block is predicted with reference to the prediction results of all the previously predicted sub-blocks in front of it. In this way, the spatial information of all the previous sub-blocks can be used to predict the current sub-block, which is beneficial to improving the prediction accuracy of the current sub-block.

[0162] In some embodiments, the target sub-block is the sub-block that has completed prediction and is the closest to the i-th sub-block. That is, starting from the second sub-block, each sub-block only refers to the closest sub-block that has completed prediction in front of it for prediction. In this way, the spatial information of the sub-block most similar to the current sub-block can be used to predict the current sub-block, which is beneficial to improving both the prediction accuracy and the prediction speed of the current sub-block.

[0163] The training process of the target network prediction model will be introduced below.

[0164] In some embodiments, the obtained CU sample can be divided into N sub-block samples in the corresponding partitioning direction. Then, using the reference information of the CU sample as the input and the pixels of the i-th sub-block sample as the output, the i-th sub-network unit in the preset network model is trained. After determining that the prediction loss value of the i-th sub-network unit is lower than the first set value, i is updated to i + 1, and the step of training the i-th sub-network unit in the preset network model with the reference information of the CU sample as the input and the pixels of the i-th sub-block sample as the output is executed. This is done until it is determined that the prediction loss value of the N-th sub-network unit is lower than the first set value, and the current preset model is used as the target network prediction model.

[0165] In some embodiments, the obtained CU sample can be divided into N sub-block samples in the corresponding partitioning direction. Then, using the reference information of the CU sample as the input and the pixels of the N sub-block samples as the output, the N sub-network units in the preset network model are trained synchronously. When it is determined that the total prediction loss value of the N sub-network units is lower than the second set value, the current preset model is used as the target network prediction model.

[0166] In step S903, the CU is predicted using the conventional ISP mode.

[0167] In step S904, according to the prediction errors of the network model ISP mode corresponding to each partitioning direction and the prediction error of the conventional ISP mode, the syntactic element information of the CU is determined. The syntactic element information includes partitioning information and ISP mode indication information.

[0168] Specifically, during implementation, the minimum prediction error can be selected from the prediction errors of the network model ISP mode corresponding to each partitioning direction and the prediction error of the conventional ISP mode. Based on the minimum prediction error, the syntactic element information of the CU is determined. The syntactic element information includes partitioning information and ISP mode indication information. The ISP mode indication information is used to indicate whether to use the network model ISP mode for decoding.

[0169] Among them, if the CU is only allowed to be partitioned in one partitioning direction, the partitioning information may only include the ISP enabling flag, and the ISP enabling flag is used to indicate the enabling of the ISP mode (i.e., partitioning the CU); if the CU is allowed to be partitioned in two partitioning directions, the partitioning information may include the ISP enabling flag and the partitioning direction corresponding to the minimum prediction error (the target partitioning direction). In short, based on the size of the CU and the partitioning information, the decoding end can know the sub-block information of the CU in the target partitioning direction.

[0170] In step S905, send the syntax element information of the CU.

[0171] In some embodiments, the reference information may further include quantization parameter information. Correspondingly, the syntax element information may also include quantization parameter information. In this way, the encoding end can know the quality of the reference pixels, which is beneficial to guiding prediction and improving the encoding efficiency.

[0172] In some embodiments, the input of each sub-network prediction unit further includes the conventional prediction result of the corresponding sub-block. The conventional prediction result refers to the prediction result obtained by using the conventional ISP mode. Correspondingly, the syntax element information may also include the result inference information of the conventional prediction results of each sub-block. Among them, the result inference information is used to infer the conventional prediction result.

[0173] Since the data volume of the result inference information is smaller than that of the conventional prediction result, sending the result inference information of the conventional prediction result is beneficial to reducing the data volume to be encoded, thereby improving the encoding speed.

[0174] The video decoding method proposed in this application will be described below with specific embodiments. Figure 11 It is a flowchart of a video decoding method provided by an embodiment of this application, including the following steps.

[0175] In step S1101, receive the syntax element information of the CU in the video frame. The syntax element information includes partitioning information and ISP mode indication information.

[0176] Among them, the ISP mode indication information is used to indicate whether to use the network model ISP mode for decoding.

[0177] In step S1102, determine the sub-block information of the CU in one partitioning direction according to the size of the CU and the partitioning information.

[0178] In some embodiments, the partitioning information only contains the ISP enabling flag, and the ISP enabling flag is used to indicate the enabling of the ISP mode.

[0179] Based on the ISP enabling flag, the decoding end can know that the ISP mode is enabled for the CU. If the size of the CU only allows division in one division direction, the decoding end can determine the sub-block information of the CU in this division direction.

[0180] In some embodiments, the division information includes an ISP enabling flag and a target division direction.

[0181] Based on the ISP enabling flag, the decoding end can know that the ISP mode is enabled for the CU. If the size of the CU allows division in two division directions, one division direction can be excluded based on the target division direction, so that the decoding end can determine the sub-block information of the CU in the target division direction.

[0182] In step S1103, in response to decoding using the network model ISP mode indicated by the ISP mode indication information, the CU is predicted using the network model ISP mode according to the reference information, sub-block information of the CU, and the pre-established network prediction model.

[0183] Among them, the reference information at least includes the reference pixels adjacent to the CU in the reconstructed pixel region of the video frame. And the reference pixels can include M pixel reference rows and M pixel reference columns, where M is an integer greater than zero. That is to say, the network model ISP mode can use more than one reference row and reference column for prediction. In this way, a larger reference pixel region can be used for prediction, which is beneficial to improving the prediction accuracy.

[0184] In specific implementation, the target network prediction model can be determined according to the sub-block information and the corresponding relationship between the sub-block information in the corresponding division direction and the network prediction model. Then, according to the target network prediction model and the reference information of the CU, the CU is predicted using the network model ISP mode.

[0185] Generally, the sub-network prediction units in the target network prediction model are in one-to-one correspondence with the sub-blocks in a certain division direction. The reference information of the CU is input into the first sub-network prediction unit for prediction, and the model prediction result of the first sub-block can be obtained. Starting from the i-th sub-network prediction unit (i>1), the reference information and the model prediction result of the target sub-block can be input into the i-th sub-network prediction unit for prediction, and the model prediction result of the i-th sub-block can be obtained, where the target sub-block is selected from the sub-blocks that have been predicted in the CU in this division direction. Then, if i is less than the number of sub-blocks N, i is updated to i + 1, and the step of inputting the reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction to obtain the model prediction result of the i-th sub-block is looped until i is equal to N, and the prediction ends.

[0186] In some embodiments, the target sub-blocks are all the sub-blocks for which prediction has been completed. That is, starting from the second sub-block, each sub-block is predicted by referring to the prediction results of all the previously completed and predicted sub-blocks in front of it. In this way, the spatial information of all the previous sub-blocks can be used to predict the current sub-block, which is beneficial to improving the prediction accuracy of the current sub-block.

[0187] In some embodiments, the target sub-block is the one that has been completed in prediction and is the closest to the i-th sub-block. That is, starting from the second sub-block, each sub-block is predicted by referring only to the one previously completed and predicted sub-block that is closest to itself. In this way, the spatial information of the sub-block that is most similar to itself can be used to predict the current sub-block, which is beneficial to both improving the prediction accuracy of the current sub-block and the prediction speed of the current sub-block.

[0188] In addition, the training process of the target network prediction model is the same as that of the encoding end and will not be elaborated here.

[0189] In step S1104, according to the prediction result, the CU is reconstructed.

[0190] Specifically in implementation, by reconstructing the prediction results of each sub-block, the CU can be obtained.

[0191] In some embodiments, the syntax element information further includes quantization parameter information. Correspondingly, the reference information may also include quantization parameter information. In this way, the decoding end can know the quality of the decoded pixels, which is beneficial to guiding prediction and improving the decoding efficiency.

[0192] In some embodiments, the syntax element information further includes result inference information of the conventional prediction results of each sub-block. The conventional prediction result refers to the prediction result obtained by performing prediction in the conventional ISP mode. Then the encoding end can also determine the conventional prediction results of each sub-block according to the result inference information, and use the conventional prediction result of each sub-block as an input to the corresponding sub-network prediction unit. In this way, it is beneficial to improving the prediction effect of each sub-block.

[0193] Based on the same technical concept, the embodiments of the present application further provide a video encoding device. The principle of the video encoding device to solve problems is similar to the above video encoding method. Therefore, the implementation of video encoding can refer to the implementation of the video encoding method, and the repeated parts will not be elaborated. Figure 12 FIG. is a schematic structural diagram of a video encoding device provided by an embodiment of the present application, including a partitioning module 1201, a first prediction module 1202, a second prediction module 1203, a determination module 1204, and a sending module 1205.

[0194] The partitioning module 1201 is configured to partition the coding unit CU in the video frame in at least one partitioning direction to obtain sub-block information of the CU in each partitioning direction;

[0195] The first prediction module 1202 is configured to perform prediction on the CU by using the model intra-block ISP mode according to the reference information of the CU, the sub-block information, and a pre-established network prediction model, where the reference information at least includes reference pixels adjacent to the CU in the reconstructed pixel region of the video frame;

[0196] The second prediction module 1203 is configured to perform prediction on the CU by using the conventional ISP mode;

[0197] The determination module 1204 is configured to determine the syntax element information of the CU according to the prediction errors of the network model ISP modes corresponding to each partitioning direction and the prediction error of the conventional ISP mode, where the syntax element information includes partitioning information and ISP mode indication information, and the ISP mode indication information is used to indicate whether to perform decoding by using the network model ISP mode;

[0198] The sending module 1205 is configured to send the syntax element information of the CU.

[0199] In some embodiments, the first prediction module 1202 is specifically configured to:

[0200] Determine a target network prediction model according to the sub-block information and the established correspondence between the sub-block information in the partitioning direction and the network prediction model;

[0201] Perform prediction on the CU by using the network model ISP mode according to the target network prediction model and the reference information of the CU.

[0202] In some embodiments, the target network prediction model includes N sub-network prediction units, where N is the number of sub-blocks of the CU in the partitioning direction, and the first prediction module 1202 is specifically configured to:

[0203] Input the reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction to obtain the model prediction result of the i-th sub-block, where the target sub-block is selected from the sub-blocks that have completed prediction of the CU in the partitioning direction, i is an integer greater than 1, and the model prediction result of the first sub-block is obtained by inputting the reference information into the first sub-network prediction unit for prediction;

[0204] When i is less than N, update i to i + 1, and perform the step of inputting the reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction until i is equal to N.

[0205] In some embodiments, the target sub-block is all the sub-blocks that have completed prediction, or the target sub-block is a sub-block that has completed prediction and is the closest to the i-th sub-block.

[0206] In some embodiments, the reference information further includes quantization parameter information, and

[0207] the syntax element information further includes the quantization parameter information.

[0208] In some embodiments, the input of each sub-network prediction unit further includes the conventional prediction result of the corresponding sub-block, where the conventional prediction result refers to the prediction result obtained by using the conventional ISP mode; and

[0209] the syntax element information further includes the result inference information of the conventional prediction results of each sub-block.

[0210] In some embodiments, the reference pixels include M pixel reference rows and M pixel reference columns, where M is an integer greater than zero.

[0211] In some embodiments, the target network prediction model is trained according to the following steps:

[0212] Divide the obtained CU sample into N sub-block samples in the division direction;

[0213] Using the reference information of the CU sample as the input and the pixels of the i-th sub-block sample as the output, train the i-th sub-network unit in the preset network model. After determining that the prediction loss value of the i-th sub-network unit is lower than the first set value, update i to i + 1, and execute the step of using the reference information of the CU sample as the input and the pixels of the i-th sub-block sample as the output to train the i-th sub-network unit in the preset network model until it is determined that the prediction loss value of the N-th sub-network unit is lower than the first set value, then use the current preset model as the target network prediction model; or

[0214] Using the reference information of the CU sample as the input and the pixels of the N sub-block samples as the output, train the N sub-network units in the preset network model synchronously. When it is determined that the total prediction loss value of the N sub-network units is lower than the second set value, use the current preset model as the target network prediction model.

[0215] Based on the same technical concept, an embodiment of the present application further provides a video decoding device. The principle of the video decoding device to solve the problem is similar to that of the above video decoding method. Therefore, for the implementation of video decoding, reference can be made to the implementation of the video decoding method, and the repeated parts will not be described again. Figure 13 The structural schematic diagram of a video decoding device provided by an embodiment of the present application includes a receiving module 1301, a determining module 1302, a predicting module 1303, and a reconstructing module 1304.

[0216] A receiving module 1301, configured to receive syntax element information of a coding unit (CU) in a video frame, where the syntax element information includes partitioning information and an intra sub-block (ISP) mode indication information, and the ISP mode indication information is used to indicate whether to perform decoding using a network model ISP mode;

[0217] A determining module 1302, configured to determine sub-block information of the CU in a partitioning direction according to the size of the CU and the partitioning information;

[0218] A predicting module 1303, configured to, in response to the indication of performing decoding using a network model ISP mode by the ISP mode indication information, perform prediction on the CU using a network model ISP mode according to the reference information of the CU, the sub-block information, and a pre-established network prediction model, where the reference information at least includes reference pixels adjacent to the CU in a reconstructed pixel region of the video frame;

[0219] A reconstructing module 1304, configured to reconstruct the CU according to a prediction result.

[0220] In some embodiments, the predicting module 1303 is specifically configured to:

[0221] Determine a target network prediction model according to the sub-block information and a correspondence relationship between the sub-block information in the partitioning direction and the network prediction model established;

[0222] Perform prediction on the CU using a network model ISP mode according to the target network prediction model and the reference information of the CU.

[0223] In some embodiments, the target network prediction model includes N sub-network prediction units, N being the number of sub-blocks, and the predicting module 1303 is specifically configured to:

[0224] Input the reference information and a model prediction result of a target sub-block into an i-th sub-network prediction unit for prediction to obtain a model prediction result of an i-th sub-block, where the target sub-block is selected from sub-blocks that have completed prediction of the CU in the partitioning direction, i is an integer greater than 1, and a model prediction result of a first sub-block is obtained by inputting the reference information into a first sub-network prediction unit for prediction;

[0225] When i is less than N, update i to i + 1, and perform the step of inputting the reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction until i is equal to N.

[0226] In some embodiments, the target sub-block is all sub-blocks that have completed prediction, or the target sub-block is a sub-block that has completed prediction and is the closest to the i-th sub-block.

[0227] In some embodiments, the syntax element information further includes quantization parameter information, and

[0228] the decoding end reference information further includes the quantization parameter information.

[0229] In some embodiments, the syntax element information further includes result inference information of the conventional prediction results of each sub-block, where the conventional prediction results refer to the prediction results obtained by performing prediction in the conventional ISP mode, and further includes:

[0230] Determine the conventional prediction results of each sub-block according to the result inference information;

[0231] Use the conventional prediction result of each sub-block as an input to the corresponding sub-network prediction unit.

[0232] In some embodiments, the reference pixels include M pixel reference rows and M pixel reference columns, where M is an integer greater than zero.

[0233] The division of modules in the embodiments of the present application is illustrative, merely a logical function division. In actual implementation, there may be other division methods. In addition, each functional module in the embodiments of the present application can be integrated in one processor, or can exist separately physically, or two or more modules can be integrated in one module. The coupling between each module can be realized through some interfaces, and these interfaces are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and can be located in one place, or distributed to different positions of the same or different devices. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0234] After introducing the contrast learning method and device of the exemplary embodiments of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0235] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method, or a program product. Therefore, various aspects of the present application can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "system" here.

[0236] In some possible embodiments, the electronic device according to the present application may at least include at least one processor and at least one memory. Among them, the memory stores program code, and when the program code is executed by the processor, the processor executes the methods according to various exemplary embodiments of the present application described above in this specification.

[0237] Reference is made below to Figure 14 describe the electronic device 140 implemented according to this embodiment of the present application. Figure 14 The illustrated electronic device 140 is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0238] As Figure 14 shown, the electronic device 140 is presented in the form of a general-purpose electronic device. The components of the electronic device 140 may include, but are not limited to: the above-mentioned at least one processor 141, the above-mentioned at least one memory 142, and a bus 143 connecting different system components (including the memory 142 and the processor 141).

[0239] The bus 143 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a processor, or a local bus using any of the multiple bus structures.

[0240] The memory 142 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1421 and / or a cache memory 1422, and may further include a read-only memory (ROM) 1423.

[0241] The memory 142 may further include a program / utility 1425 having a set (at least one) of program modules 1424. Such program modules 1424 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. The implementation of a network environment may be included in each or some combination of these examples.

[0242] The electronic device 140 may also communicate with one or more external devices 144 (such as a keyboard, a pointing device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 140, and / or communicate with any device (such as a router, a modem, etc.) that enables the electronic device 140 to communicate with one or more other electronic devices. Such communication may be carried out through the input / output (I / O) interface 145. Moreover, the electronic device 140 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 146. As shown in the figure, the network adapter 146 communicates with other modules for the electronic device 140 through the bus 143. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in combination with the electronic device 140, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0243] In an exemplary embodiment, there is also provided a computer-readable storage medium including instructions, such as the memory 142 including instructions, and the above instructions can be executed by the processor 141 to complete the above contrast learning method. Optionally, the storage medium may be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0244] In an exemplary embodiment, there is also provided a computer program product. When the computer program product is executed by an electronic device, the electronic device can implement any exemplary method provided in this application.

[0245] Moreover, the computer program product may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0246] In an embodiment of the present application, a program product for a video encoding method or a video decoding method may be implemented using a CD-ROM and include program code, and may be run on a computing device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0247] A readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0248] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0249] The program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network such as a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0250] It should be noted that although several units or subunits of the apparatus are mentioned in the foregoing detailed description, such a division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more of the above-described units may be embodied in one unit. Conversely, the features and functions of one unit described above may be further divided and embodied by a plurality of units.

[0251] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0252] Those skilled in the art should understand that the embodiments of the present application may be provided as a method, a system, or a computer program product. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) that contain computer-usable program code.

[0253] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one or more flows or blocks Figure 1 one or more blocks.

[0254] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in Figure 1 one or more flows or blocks Figure 1 one or more blocks.

[0255] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are performed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more flows or blocks Figure 1 one or more blocks.

[0256] While the preferred embodiments of the present application have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present application.

[0257] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A video encoding method, characterized in that, Including: Dividing a coding unit (CU) in a video frame in at least one division direction to obtain sub-block information of the CU in each division direction; Predicting the CU by using an in-frame sub-block ISP mode of a network model according to the reference information of the CU, the sub-block information, and a pre-established network prediction model, where the reference information at least includes reference pixels adjacent to the CU in a reconstructed pixel region of the video frame, and the ISP mode of the network model means predicting each sub-block obtained by dividing the CU by using the network prediction model; Predicting the CU by using a conventional ISP mode, where the conventional ISP mode means predicting each sub-block obtained by dividing the CU by using a traditional ISP mode; Determining syntax element information of the CU according to prediction errors of the ISP mode of the network model and prediction errors of the conventional ISP mode corresponding to each division direction, where the syntax element information includes division information and ISP mode indication information, and the ISP mode indication information is used to indicate whether to perform decoding by using the ISP mode of the network model; Sending the syntax element information of the CU.

2. The method according to claim 1, characterized in that, Predicting the sub-block by using the ISP mode of the network model according to the reference information of the CU, the sub-block information, and a pre-established network prediction model, including: Determining a target network prediction model according to the sub-block information and a correspondence relationship between the sub-block information in the division direction and the network prediction model established; Predicting the CU by using the ISP mode of the network model according to the target network prediction model and the reference information of the CU.

3. The method according to claim 2, wherein The target network prediction model includes N sub-network prediction units, where N is the number of sub-blocks of the CU in the division direction. Predicting the CU by using the ISP mode of the network model according to the target network prediction model and the reference information of the CU includes: Inputting the reference information and a model prediction result of a target sub-block into the i-th sub-network prediction unit for prediction to obtain a model prediction result of the i-th sub-block, where the target sub-block is selected from sub-blocks that have been predicted in the division direction of the CU, i is an integer greater than 1, and a model prediction result of the first sub-block is obtained by inputting the reference information into the first sub-network prediction unit for prediction; When i is less than N, updating i to i + 1 and executing the step of inputting the reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction until i is equal to N.

4. The method according to claim 3, wherein The target sub-block is all sub-blocks that have been predicted, or the target sub-block is a sub-block that has been predicted and is the closest to the i-th sub-block.

5. The method according to claim 3, wherein The reference information further includes quantization parameter information, and The syntax element information further includes the quantization parameter information.

6. The method according to any one of claims 3-5, characterized in that, An input of each sub-network prediction unit further includes a conventional prediction result of a corresponding sub-block, where the conventional prediction result is a prediction result obtained by using the conventional ISP mode; and The syntax element information further includes result inference information of the conventional prediction results of each sub-block.

7. The method according to claim 1, wherein The reference pixels include M pixel reference rows and M pixel reference columns, where M is an integer greater than zero.

8. The method according to claim 3, wherein Train the target network prediction model according to the following steps: Divide the obtained CU sample into N sub-block samples in the division direction; Use the reference information of the CU sample as the input and the pixels of the i-th sub-block sample as the output to train the i-th sub-network unit in the preset network model. After determining that the prediction loss value of the i-th sub-network unit is lower than the first set value, update i to i + 1, and execute the step of using the reference information of the CU sample as the input and the pixels of the i-th sub-block sample as the output to train the i-th sub-network unit in the preset network model until it is determined that the prediction loss value of the N-th sub-network unit is lower than the first set value, and then use the current preset model as the target network prediction model; or Use the reference information of the CU sample as the input and the pixels of the N sub-block samples as the output to train the N sub-network units in the preset network model synchronously. After determining that the total prediction loss value of the N sub-network units is lower than the second set value, use the current preset model as the target network prediction model.

9. A video decoding method, characterized in that, It includes: Receive the syntax element information of the coding unit CU in the video frame. The syntax element information includes division information and the in-frame sub-block ISP mode indication information. The ISP mode indication information is used to indicate whether to use the network model ISP mode for decoding. The network model ISP mode means using the network prediction model to predict each sub-block obtained by dividing the CU; Determine the sub-block information of the CU in a division direction according to the size of the CU and the division information; In response to the indication of using the network model ISP mode for decoding by the ISP mode indication information, use the network model ISP mode to predict the CU according to the reference information of the CU, the sub-block information, and the pre-established network prediction model. The reference information at least includes the reference pixels adjacent to the CU in the reconstructed pixel region of the video frame; Reconstruct the CU according to the prediction result.

10. The method according to claim 9, characterized in that Using the network model ISP mode to predict the CU according to the reference information of the CU, the sub-block information, and the pre-established network prediction model includes: Determine the target network prediction model according to the sub-block information and the established correspondence between the sub-block information in the division direction and the network prediction model; Predict the CU using the network model ISP mode according to the target network prediction model and the reference information of the CU.

11. The method according to claim 10, characterized in that The target network prediction model includes N sub-network prediction units, where N is the number of sub-blocks. Using the network model ISP mode to predict the CU according to the target network prediction model and the reference information of the CU includes: Input the reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction to obtain the model prediction result of the i-th sub-block. The target sub-block is selected from the sub-blocks that have been predicted in the CU in the partitioning direction. i is an integer greater than 1. The model prediction result of the first sub-block is obtained by inputting the reference information into the first sub-network prediction unit for prediction; When i is less than N, update i to i + 1, and execute the step of inputting the reference information and the model prediction result of the target sub-block into the i-th sub-network prediction unit for prediction until i is equal to N.

12. The method according to claim 11, wherein The target sub-block is all the sub-blocks that have been predicted, or the target sub-block is the one sub-block that has been predicted and is the closest to the i-th sub-block.

13. The method according to claim 11, wherein The syntactic element information further includes quantization parameter information, and The reference information further includes the quantization parameter information.

14. The method according to any one of claims 11-13, characterized in that The syntactic element information further includes result inference information of the conventional prediction results of each sub-block. The conventional prediction result refers to the prediction result obtained by performing prediction in the conventional ISP mode, and further includes: Determine the conventional prediction results of each sub-block according to the result inference information; Use the conventional prediction result of each sub-block as an input to the corresponding sub-network prediction unit.

15. The method according to claim 9, wherein The reference pixels include M pixel reference rows and M pixel reference columns, where M is an integer greater than zero.

16. A video encoding device, characterized in that, Comprising: A partitioning module for partitioning the coding unit CU in the video frame in at least one partitioning direction to obtain the sub-block information of the CU in each partitioning direction; A first prediction module for predicting the CU using the model intra-sub-block ISP mode according to the reference information of the CU, the sub-block information, and a pre-established network prediction model. The reference information at least includes the reference pixels adjacent to the CU in the reconstructed pixel region of the video frame. The network model ISP mode refers to using the network prediction model to predict each sub-block obtained by partitioning the CU; A second prediction module for predicting the CU using the conventional ISP mode. The conventional ISP mode refers to using the traditional ISP mode to predict each sub-block obtained by partitioning the CU; A determination module for determining the syntactic element information of the CU according to the prediction error of the network model ISP mode and the prediction error of the conventional ISP mode corresponding to each partitioning direction. The syntactic element information includes partitioning information and ISP mode indication information. The ISP mode indication information is used to indicate whether to perform decoding using the network model ISP mode; A sending module for sending the syntactic element information of the CU.

17. A video decoding device, characterized in that, Comprising: A receiving module for receiving the syntactic element information of the coding unit CU in the video frame. The syntactic element information includes partitioning information and intra-sub-block ISP mode indication information. The ISP mode indication information is used to indicate whether to perform decoding using the network model ISP mode. The network model ISP mode refers to using the network prediction model to predict each sub-block obtained by partitioning the CU; A determination module, configured to determine sub-block information of the CU in a partitioning direction according to the size of the CU and the partitioning information; A prediction module, configured to, in response to decoding being performed in the network model ISP mode indicated by the ISP mode indication information, perform prediction on the CU in the network model ISP mode according to the reference information of the CU, the sub-block information, and a pre-established network prediction model, where the reference information at least includes reference pixels adjacent to the CU in the reconstructed pixel region of the video frame; A reconstruction module, configured to reconstruct the CU according to the prediction result.

18. An electronic device, characterized in that, Comprising: At least one processor, and a memory communicatively connected to the at least one processor, where: The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method according to any one of claims 1-8 or 9-15.

19. A storage medium, characterized in that, When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1-8 or 9-15.

Images