Video encoding method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a video coding method, device, electronic equipment and storage medium, and belongs to the field of video coding. The method comprises: for any picture group in a video sequence, determining a first type picture and a second type picture in the picture group based on the reference times of pictures; for the first type picture, dividing the CTU of the first type picture based on the division depth value corresponding to the first type picture to obtain a plurality of CUs, and performing video coding in units of the CUs; the division depth value is used to indicate the maximum division depth of the CU division of the CTU; for the second type picture, dividing the CTU of the second type picture based on the division depth value corresponding to the second type picture to obtain a plurality of CUs, and performing video coding in units of the CUs. The scheme can adaptively select the division depth for different pictures, and then the selected division depth can be used for CU division of the pictures. The CU division mode can improve the coding performance and speed.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of video coding, and in particular, to a video coding method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Video coding refers to converting a video in an original format into another format by using an encoder through compression technology. Storing and transmitting a video in a compressed form not only saves storage space, but also reduces bandwidth costs.

[0003] Among them, Versatile Video Coding (VVC) as a new generation of video coding standard is an important technical innovation in the field of video coding. In VVC, an image is first divided into multiple coding tree units (CTUs), and then each CTU is further recursively divided into multiple coding units (CUs). Illustratively, recursive division refers to first dividing a CTU into multiple CUs using a multi-type tree, and then dividing these CUs at least once using a multi-type tree.

[0004] In VVC, dividing CUs is an important process. For example, accurate CU division not only brings greater coding performance gain for compression tasks, but also improves coding speed. In other words, how to divide CUs has a crucial impact on coding performance and coding speed, so there is an urgent need for a new video coding method to achieve effective division of CUs. SUMMARY

[0005] The present disclosure provides a video coding method, device, electronic device, and storage medium. The technical solutions of the present disclosure are as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a video coding method is provided, comprising:

[0007] For any image group in a video sequence to be coded, based on the number of times an image is referenced, first and second types of images are determined in the image group;

[0008] For the first type of image, based on a division depth value corresponding to the first type of image, a CTU of the first type of image is divided to obtain multiple CUs; video coding is performed in units of CUs of the first type of image; wherein the division depth value is used to indicate the maximum division depth of CU division of the CTU;

[0009] For the second type of image, based on the partition depth value corresponding to the second type of image, the CTU of the second type of image is partitioned to obtain a plurality of CUs; and the CUs of the second type of image are taken as units for video encoding.

[0010] In some embodiments, the determining the first type of image and the second type of image in the group of images based on the reference times of the images comprises:

[0011] determining, as the first type of image, the I frames and the P frames that are reference frames in the group of images, and determining, as the second type of image, the B frames in the group of images; or

[0012] determining, as the first type of image, the I frames, the P frames and the B4 frames that are reference frames in the group of images, and determining, as the second type of image, the remaining B frames in the group of images.

[0013] In some embodiments, the first type of image corresponds to a partition depth value greater than a partition depth value corresponding to the second type of image.

[0014] In some embodiments, the method further comprises:

[0015] setting the partition depth value in the sequence parameter set (SPS) of each image in the group of images, and setting the partition depth value in the PPS of the first type of image.

[0016] In some embodiments, the setting the partition depth value in the PPS of the first type of image comprises:

[0017] setting the partition depth value in the picture header included in the PPS of the first type of image.

[0018] In some embodiments, the partitioning the CTU of the first type of image based on the partition depth value corresponding to the first type of image comprises:

[0019] partitioning the first type of image into a plurality of CTUs using a multi-type tree;

[0020] For any one of the plurality of CTUs, based on the partition depth value included in the PPS of the first type of image, the CTU is partitioned using a multi-type tree to obtain at least one CU.

[0021] In some embodiments, the partitioning the CTU of the second type of image based on the partition depth value corresponding to the second type of image comprises:

[0022] partitioning the second type of image into a plurality of CTUs using a multi-type tree;

[0023] For any one of the CTUs, the CTU is partitioned by a multi-type tree to obtain at least one CU based on a partition depth value included in the SPS of the second type image.

[0024] According to a second aspect of the embodiments of the present disclosure, a video encoding apparatus is provided, comprising:

[0025] A determining module is configured to determine, for any one of image groups in a video sequence to be encoded, first type images and second type images in the image group based on the number of times of being referenced by images.

[0026] An encoding module is configured to, for the first type images, partition CTUs of the first type images based on partition depth values corresponding to the first type images to obtain a plurality of CUs, and perform video encoding in units of CUs of the first type images; wherein the partition depth value is used to indicate a maximum partition depth of CU partitioning of the CTU.

[0027] The encoding module is further configured to, for the second type images, partition CTUs of the second type images based on partition depth values corresponding to the second type images to obtain a plurality of CUs, and perform video encoding in units of CUs of the second type images.

[0028] In some embodiments, the determining module is configured to:

[0029] determine I frames and P frames in the image group as reference frames as the first type images, and determine B frames in the image group as the second type images; or,

[0030] determine I frames, P frames and B4 frames in the image group as reference frames as the first type images, and determine remaining B frames in the image group as the second type images.

[0031] The partition depth value corresponding to the first type images is greater than the partition depth value corresponding to the second type images.

[0032] In some embodiments, the apparatus further comprises a setting module; the setting module is configured to:

[0033] set the partition depth value in sequence parameter sets (SPS) of each image in the image group, and set the partition depth value in a PPS of the first type images.

[0034] In some embodiments, the setting module is configured to:

[0035] set the partition depth value in a picture header included in the PPS of the first type images.

[0036] In some embodiments, the encoding module is configured to:

[0037] divide the first type image into a plurality of CTUs using the multi-type tree;

[0038] For any one of the plurality of CTUs, the CTU is divided using the multi-type tree based on a division depth value included in the PPS of the first type image, to obtain at least one CU.

[0039] In some embodiments, the encoding module is further configured to:

[0040] divide the second type image into a plurality of CTUs using the multi-type tree;

[0041] For any one of the plurality of CTUs, the CTU is divided using the multi-type tree based on a division depth value included in the SPS of the second type image, to obtain at least one CU.

[0042] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, which comprises:

[0043] one or more processors;

[0044] a memory for storing program code executable by the processor;

[0045] The processor is configured to execute the program code to implement the video encoding method according to the first aspect or any possible implementation manner of the first aspect.

[0046] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which, when program code in the computer readable storage medium is executed by a processor of an electronic device, enables the electronic device to perform the video encoding method according to the first aspect or any possible implementation manner of the first aspect.

[0047] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, which comprises a computer program, and the computer program, when executed by a processor, implements the video encoding method according to the first aspect or any possible implementation manner of the first aspect.

[0048] The embodiments of the present disclosure propose a new video encoding scheme, for each group of pictures in a video sequence to be encoded, the scheme adaptively selects a division depth for different pictures in the group of pictures based on the number of times the pictures are referenced; in other words, the scheme can automatically select the optimal division depth for different pictures in the same group of pictures.

[0049] In detail, for the first type of images in the image group, the scheme divides the CTUs of the first type of images into a plurality of CUs based on the partition depth value corresponding to the first type of images, and then encodes the video in units of the CUs; for the second type of images in the image group, the scheme divides the CTUs of the second type of images into a plurality of CUs based on the partition depth value corresponding to the second type of images, and then encodes the video in units of the CUs.

[0050] Since the scheme can perform CU partitioning on each image according to the optimal partition depth, video encoding based on this CU partitioning manner can significantly improve the encoding performance and encoding speed.

[0051] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure without imposing on the disclosure any undue limitations.

[0053] Figure 1 is a schematic diagram of a multi-type tree partitioning according to an exemplary embodiment.

[0054] Figure 2 is a schematic diagram of another multi-type tree partitioning according to an exemplary embodiment.

[0055] Figure 3 is a schematic diagram of another multi-type tree partitioning according to an exemplary embodiment.

[0056] Figure 4 is a schematic diagram of another multi-type tree partitioning according to an exemplary embodiment.

[0057] Figure 5 is a schematic diagram of an implementation environment of a video encoding method according to an exemplary embodiment.

[0058] Figure 6 is a flowchart of a video encoding method according to an exemplary embodiment.

[0059] Figure 7 is a schematic diagram of an inter-image reference relationship according to an exemplary embodiment.

[0060] Figure 8 is a flowchart of another video encoding method according to an exemplary embodiment.

[0061] Figure 9is a block diagram of a video encoding apparatus according to an exemplary embodiment.

[0062] Figure 10 is a block diagram of a terminal 1000 according to an exemplary embodiment. DETAILED DESCRIPTION

[0063] In order for the ordinary person skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings.

[0064] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0065] The information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals involved in the present disclosure are authorized by the user or authorized by all parties, and the collection, use, and processing of related data need to comply with relevant laws, regulations, and standards in relevant countries and regions.

[0066] Some abbreviations or key terms related to the embodiments of the present disclosure will be introduced first.

[0067] CTU: referred to as a tree coding unit or coding tree unit. In VVC, an image is divided into multiple CTUs (such as 4 CTUs obtained by quadtree division), and each CTU is further divided into multiple CUs.

[0068] For example, each CTU is recursively divided into multiple CUs of different sizes using a multi-type tree (MTT). In other words, the CTU is treated as the root node of a quadtree for CU division, and each leaf node of the quadtree can be further divided into CUs by MTT. That is, the CTU is divided by the multi-type tree, and the leaf nodes after the division can be further divided by the multi-type tree.

[0069] CU: referred to as a coding unit, wherein the CU can be square or rectangular, and a CTU can contain only one CU (i.e., no division) or be divided into multiple CUs.

[0070] Multi-type tree: including binary tree, ternary tree and quad tree; wherein, the quad tree division refers to dividing a square into 4 same small squares, and the specific division manner is as shown in Figure 1 Fig. 2. The division manners of binary tree division and ternary tree division are as shown in Figure 2 and Figure 3 . Among them, the binary tree division includes vertical binary tree division (left diagram in Figure 2 ) and horizontal binary tree division (right diagram in Figure 2 ), and the ternary tree division includes vertical ternary tree division (left diagram in Figure 3 ) and horizontal ternary tree division (right diagram in Figure 3 ).

[0071] Exemplarily, Figure 4 a CTU is divided into multiple CUs by a multi-type tree. As shown in Figure 4 , the CTU is first divided once by a quad tree to obtain 4 CUs; then, the 4 CUs are further divided by a multi-type tree, and the subsequent obtained CUs will be further divided.

[0072] Figure 5 is a schematic diagram of an implementation environment of a video encoding method according to an example embodiment.

[0073] Referring to Figure 5 , the implementation environment includes a video encoding device 501 and a video decoding device 502.

[0074] Among them, the video encoding device 501 is also referred to as an encoding device, which is used to execute the video encoding method provided by the embodiments of the present disclosure. The video decoding device 502 is also referred to as a decoding device.

[0075] In the embodiments of the present disclosure, the video encoding device 501 is used to perform video encoding, and the obtained video code stream after encoding is transmitted to the video decoding device 502 through a network. And the video decoding device 502 is used to decode the received video code stream to obtain an image which can be displayed on a display screen. Exemplarily, the above-mentioned image is a screen content image, which is not limited by the present disclosure.

[0076] In one possible implementation, the types of video encoding device 501 and video decoding device 502 include, but are not limited to, mobile terminals and fixed terminals. Mobile terminals include, but are not limited to, smartphones, tablets, laptops, e-readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), etc.; while fixed terminals include, but are not limited to, desktop computers.

[0077] Based on the aforementioned implementation environment, this disclosure proposes a novel video coding scheme. This scheme, based on the number of times an image is referenced, can adaptively select the partitioning depth for different images. Consequently, the video coding device can effectively partition the image into control units (CUs) according to the selected partitioning depth. This CU partitioning method can significantly improve coding performance and speed. The following embodiments will provide a detailed description of this video coding scheme.

[0078] Figure 6 This is a flowchart illustrating a video encoding method according to an exemplary embodiment, such as... Figure 6 As shown, this video encoding method is applied to electronic devices, such as... Figure 5 The video encoding device shown. The video encoding method includes the following steps:

[0079] In 601, for any group of images in the video sequence to be encoded, a first class of images and a second class of images are determined in the group of images based on the number of times the images are referenced.

[0080] The video sequence consists of several temporally consecutive images. When compressing it, the video sequence is first divided into several small groups of pictures (GOPs). In addition, a GOP is a group of consecutive images, usually consisting of one I-frame and multiple B / P frames.

[0081] Figure 7 The reference relationships between the images are shown. For example... Figure 7 As mentioned above, I-frames and P-frames can be used as reference frames. In detail, I-frames serve as keyframes and do not need to reference other frames; P-frames are inter-coded frames that use previous I-frames or P-frames for predictive coding; while B-frames are inter-coded frames that require bidirectional predictive coding using previous and / or subsequent I-frames or P-frames.

[0082] In the embodiments of the present disclosure, the first type of image refers to an image that is referenced most frequently in the image group, and the second type of image refers to a remaining image in the image group other than the first type of image.

[0083] In 602, for the first type of image, based on the partition depth value corresponding to the first type of image, the CTU of the first type of image is partitioned to obtain a plurality of CUs; and the CUs of the first type of image are taken as a unit for video encoding; wherein the partition depth value is used to indicate the maximum partition depth for CU partitioning of the CTU.

[0084] In the embodiments of the present disclosure, the partition depth value refers to the parameter value of the MaxMttDepth parameter. Wherein the parameter value of the MaxMttDepth parameter is used to indicate the maximum depth of MTT partitioning starting from a leaf node of a quadtree; in other words, the parameter value represents the maximum multi-type tree partitioning allowed depth starting from a leaf node of a quadtree.

[0085] In 603, for the second type of image, based on the partition depth value corresponding to the second type of image, the CTU of the second type of image is partitioned to obtain a plurality of CUs; and the CUs of the second type of image are taken as a unit for video encoding.

[0086] The embodiments of the present disclosure propose a new video encoding scheme. For each image group in a video sequence to be encoded, the scheme adaptively selects a partition depth for different images in the image group based on the number of times the images are referenced; in other words, the scheme can automatically select the optimal partition depth for different images in the same image group. Because the scheme can perform CU partitioning on each image according to the optimal partition depth, video encoding based on this CU partitioning method can significantly improve encoding performance and encoding speed.

[0087] In some embodiments, the number of times the images are referenced includes:

[0088] determining I frames and P frames in the image group as reference frames as the first type of image, and determining B frames in the image group as the second type of image; or,

[0089] determining I frames, P frames and B4 frames in the image group as reference frames as the first type of image, and determining the remaining B frames in the image group as the second type of image;

[0090] Wherein the partition depth value corresponding to the first type of image is greater than the partition depth value corresponding to the second type of image.

[0091] Generally, the quality of the image with more reference times is more important. Therefore, the embodiment of the present disclosure distinguishes the importance of the images in the image group based on the reference times of the images, and sets a larger division depth value for the more important images and a smaller division depth value for the remaining images. This CU division method can retain more image information of important images, and has better coding quality.

[0092] In some embodiments, the method further comprises:

[0093] setting the division depth value in the sequence parameter set (SPS) of each image in the image group, and setting the division depth value in the PPS of the first type of image.

[0094] In some embodiments, the setting of the division depth value in the PPS of the first type of image comprises:

[0095] setting the division depth value in the picture header included in the PPS of the first type of image.

[0096] The embodiment of the present disclosure proposes to set the MaxMttDepth parameter in the PH of the PPS, and set the priority of the MaxMttDepth parameter value included in the PH to be higher than the MaxMttDepth parameter value included in the SPS, that is, the MaxMttDepth parameter value included in the PH will cover the corresponding parameter value included in the SPS, which provides further guarantee for improving the coding performance and coding speed. In other words, the embodiment of the present disclosure achieves the purpose of adaptively selecting the division depth for different images by setting the MaxMttDepth parameter in the PH.

[0097] In some embodiments, the division of the CTU of the first type of image based on the division depth value corresponding to the first type of image comprises:

[0098] dividing the first type of image into a plurality of CTUs using a multi-type tree;

[0099] For any one of the plurality of CTUs, based on the division depth value included in the PPS of the first type of image, the CTU is divided using a multi-type tree to obtain at least one CU.

[0100] For the image with more reference times, the CU division based on the adaptively determined optimal division depth can improve the coding quality and coding speed.

[0101] In some embodiments, the division of the CTU of the second type of image based on the division depth value corresponding to the second type of image comprises:

[0102] dividing the second type of image into a plurality of CTUs using a multi-type tree;

[0103] For any one of the plurality of CTUs, based on a partition depth value included in the SPS of the second type of image, the CTU is partitioned by a multi-type tree to obtain at least one CU.

[0104] For the remaining images other than the images with a large number of references, CU partitioning is performed based on the adaptively determined optimal partition depth, which can improve the coding quality and coding speed.

[0105] All the optional technical solutions described above can be combined in any manner to form optional embodiments of the disclosure, which will not be described here.

[0106] The above Figure 6 The above Figure 8 is a flowchart of another video encoding method according to an exemplary embodiment, as Figure 8 shown, the video encoding method is applied to an electronic device, such as Figure 5 the video encoding device shown. The video encoding method includes the following steps:

[0107] In 801, for any one of the image groups in the video sequence to be encoded, based on the number of references of the images, the video encoding device determines the first type of image and the second type of image in the image group.

[0108] Generally, each GOP includes multiple images, including I frames, P frames and B frames. In inter prediction, images can be encoded by referring to other images, wherein I frames are independently encoded as key frames, P frames can be forwardly referenced, and B frames can be forwardly or backwardly referenced. Generally, the quality of images with a large number of references is more important. Therefore, for each GOP in the video sequence to be encoded, the I frames and P frames included in the GOP are regarded as images with a large number of references, and the B frames included in the GOP are regarded as remaining images.

[0109] Exemplarily, the images with a large number of references and the remaining images can be distinguished by the reference relationship between images as Figure 7 shown. In Figure 7 , the arrow direction is from the referenced image to the image to be encoded by referring to other images, such as B4 frame which is referenced by B2 frame, B3 frame and B5 frame. In addition, the numerical sequence of B1 frame to B7 frame is used to indicate the playing order.

[0110] In the embodiments of the present disclosure, the first type of image is the image with a large number of reference times, and the second type of image is the remaining image. In other embodiments, for each GOP in the video sequence to be encoded, the present disclosure also supports taking the I frame, P frame and B4 frame included in the GOP as the image with a large number of reference times, and taking the remaining B frames in the GOP as the remaining image, and the present disclosure does not make any limitation thereto. The B4 frame refers to the fourth played B frame in the GOP. For example, the remaining B frames are B1 frame, B2 frame, B3 frame, B5 frame, B6 frame and B7 frame. Figure 7

[0111] In other words, based on the reference times of the images, the first type of image and the second type of image can be determined in the GOP by the following manner: for each GOP, the I frame and P frame in the GOP as reference frames are determined as the first type of image, and the B frames in the GOP are determined as the second type of image; or, the I frame, P frame and B4 frame in the GOP as reference frames are determined as the first type of image, and the remaining B frames in the GOP are determined as the second type of image.

[0112] In 802, the video encoding device sets the partition depth value in the SPS of each image in the GOP; and sets the partition depth value in the PPS of the first type of image.

[0113] In VVC, the parameter set is an independent data unit, which contains the common information of different levels of coding units of the video, including Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH) and Adaptation Parameter Set (APS).

[0114] It should be noted that the PPS includes the PH and the APS; wherein the PH is used to carry the frequently changed coding parameter information. In addition, the SPS contains a plurality of parameters related to the CU partition, and the MaxMttDepth parameter is included in these parameters, which is used to limit the multi-type tree partition of the CU. The parameter represents the maximum multi-type tree partition allowed depth from a leaf node of a quadtree.

[0115] ​In the embodiments of the present disclosure, in order to achieve better coding performance and coding speed, a larger MaxMttDepth parameter value is used for the image with more reference times, for example, the parameter value is 3; a smaller MaxMttDepth parameter value is used for the remaining image, for example, the parameter value is 2. In other words, the partition depth value corresponding to the first type of image is greater than the partition depth value corresponding to the second type of image. In addition, the larger the MaxMttDepth parameter value used by the image, the better the coding quality is generally.

[0116] Exemplarily, in the embodiments of the present disclosure, one image corresponds to one PPS parameter set. In order to adaptively select different MaxMttDepth parameter values for different images in the video sequence to be coded, the embodiments of the present disclosure select to set the MaxMttDepth parameter value in the PH of the PPS.

[0117] In some embodiments, during the coding of an image, when the MaxMttDepth parameter value needs to be used, it can be first checked whether the MaxMttDepth parameter value is included in the PH of the PPS corresponding to the image; if the MaxMttDepth parameter value is included in the PH of the PPS of the image, the parameter value is used to guide the CU partition; if the MaxMttDepth parameter value is not included in the PH of the PPS corresponding to the image, the MaxMttDepth parameter value in the SPS is used to guide the CU partition. In addition, when the MaxMttDepth parameter value is included in both the PH and the SPS, the priority of the parameter value corresponding to the PH is higher than the priority of the parameter value corresponding to the SPS.

[0118] Exemplarily, the MaxMttDepth parameter value set in the SPS is 2, and the MaxMttDepth parameter value set in the PH is 3. Among them, the MaxMttDepth parameter value included in the SPS is used for all images in each group of images, such as I frames, P frames and B frames; and for the remaining images, the MaxMttDepth parameter value is not set in the PH included in the corresponding PPS. For example, for B frames, the MaxMttDepth parameter value is not set in the corresponding PH.

[0119] In addition, since the I frames and the P frames are referred to more frequently, the image information of the I frames and the P frames is relatively more important, and therefore for the image with more reference times, the embodiments of the present disclosure use a larger MaxMttDepth parameter value, so that the I frames and the P frames retain more image information, thereby ensuring the coding quality.

[0120] In 803, for the first type of image, the video coding device partitions the CTU of the first type of image based on the partition depth value corresponding to the first type of image, to obtain a plurality of CUs; and video coding is performed in units of the CUs of the first type of image.

[0121] In some embodiments, the partitioning of the CTU of the first type image based on the partition depth value corresponding to the first type image includes but is not limited to the following manners:

[0122] partitioning the first type image into a plurality of CTUs using a multi-type tree; for example, partitioning a CTU into 4 CUs using a quad-tree; then, for any one of the plurality of CTUs obtained by the partitioning, partitioning the CTU using a multi-type tree based on the partition depth value included in the PPS of the first type image to obtain at least one CU. Wherein, the partitioning of the CTU using a multi-type tree can be partitioning the CTU using at least one of a binary tree, a ternary tree or a quad-tree, and the present disclosure does not limit this.

[0123] In 804, for the second type image, the video coding device partitions the CTU of the second type image based on the partition depth value corresponding to the second type image to obtain a plurality of CUs; and performs video coding in units of the CUs of the second type image.

[0124] In some embodiments, the partitioning of the CTU of the second type image based on the partition depth value corresponding to the second type image includes but is not limited to the following manners: partitioning the second type image into a plurality of CTUs using a multi-type tree; for any one of the plurality of CTUs, partitioning the CTU using a multi-type tree based on the partition depth value included in the SPS of the second type image to obtain at least one CU.

[0125] In the embodiments of the present disclosure, when encoding the images in the group of pictures, for the images with more reference times, the MaxMttDepth parameter value included in the corresponding PH is used for CU partitioning; and for the remaining images, the MaxMttDepth parameter value included in the corresponding SPS is used for CU partitioning.

[0126] The process of adaptively selecting the partition depth value for different images is illustrated below.

[0127] 1. When the MaxMttDepth parameter value needs to be used for encoding the I frames and P frames in the group of pictures, the MaxMttDepth parameter value included in the corresponding PH is preferentially checked, at this time, the MaxMttDepth parameter value in the corresponding PH of the I frames and P frames exists, if the value is 3, the MaxMttDepth parameter value with the value of 3 is used for CU partitioning.

[0128] 2、In the case that MaxMttDepth parameter value is needed to be used in the encoding of B frame image, since MaxMttDepth parameter value is not included in the corresponding PH of B frame, but MaxMttDepth parameter value is included in the corresponding SPS, if the value of MaxMttDepth parameter value is 2, MaxMttDepth parameter value with the value of 2 is used for CU division.

[0129] 3、Encoding is performed according to MaxMttDepth parameter value with the value of 3 corresponding to I frame and P frame, and MaxMttDepth parameter value with the value of 2 corresponding to B frame.

[0130] The embodiment of the present disclosure proposes a new video encoding scheme, for each picture group in a video sequence to be encoded, the scheme adaptively selects the division depth for different pictures in the picture group based on the reference times of the pictures; in other words, the scheme can automatically select the optimal division depth for different pictures in the same picture group. In detail, for the first type of pictures in the picture group, the scheme divides the CTU of the first type of pictures to obtain a plurality of CUs based on the division depth value corresponding to the first type of pictures, and then encodes the video in units of the CUs; for the second type of pictures in the picture group, the scheme divides the CTU of the second type of pictures to obtain a plurality of CUs based on the division depth value corresponding to the second type of pictures, and then encodes the video in units of the CUs. Since the scheme can perform CU division on each picture according to the optimal division depth, video encoding based on this CU division manner can significantly improve the encoding performance and encoding speed.

[0131] For example, the pictures with more reference times use larger division depth values, and the remaining pictures use smaller division depth values, and the larger the MaxMttDepth parameter value used by the pictures, the better the encoding quality is in general, and the more the corresponding encoding time is. The embodiment of the present disclosure uses different MaxMttDepth parameter values for different pictures, which guarantees the encoding quality (the pictures with more reference times use larger division depth values) and also takes into account the encoding speed (the remaining pictures use smaller division depth values), reduces the encoding complexity, and saves the calculation overhead.

[0132] Figure 9 is a block diagram of a video encoding apparatus according to an exemplary embodiment. Referring to Figure 9 , the apparatus comprises:

[0133] The determining module 901 is configured to, for any picture group in a video sequence to be encoded, determine the first type of pictures and the second type of pictures in the picture group based on the reference times of the pictures.

[0134] The encoding module 902 is configured to, for the first type of image, perform partitioning on a CTU of the first type of image based on a partitioning depth value corresponding to the first type of image, to obtain a plurality of CUs; and perform video encoding in units of the CUs of the first type of image; wherein the partitioning depth value is used to indicate a maximum partitioning depth for partitioning the CTU into CUs.

[0135] The encoding module 902 is further configured to, for the second type of image, perform partitioning on a CTU of the second type of image based on a partitioning depth value corresponding to the second type of image, to obtain a plurality of CUs; and perform video encoding in units of the CUs of the second type of image.

[0136] The embodiments of the present disclosure propose a new video encoding scheme. For each image group in a video sequence to be encoded, the scheme adaptively selects a partitioning depth for different images in the image group based on the number of times the images are referenced. In other words, the scheme can automatically select an optimal partitioning depth for different images in the same image group. In detail, for a first type of image in the image group, the scheme performs partitioning on a CTU of the first type of image based on a partitioning depth value corresponding to the first type of image, to obtain a plurality of CUs, and then performs video encoding in units of the CUs. For a second type of image in the image group, the scheme performs partitioning on a CTU of the second type of image based on a partitioning depth value corresponding to the second type of image, to obtain a plurality of CUs, and then performs video encoding in units of the CUs. Since the scheme can perform CU partitioning on each image according to an optimal partitioning depth, video encoding based on this CU partitioning manner can significantly improve encoding performance and encoding speed.

[0137] In some embodiments, the determining module is configured to:

[0138] determine, as the first type of image, I frames and P frames in the image group that are reference frames, and determine, as the second type of image, B frames in the image group; or

[0139] determine, as the first type of image, I frames, P frames and B4 frames in the image group that are reference frames, and determine, as the second type of image, remaining B frames in the image group;

[0140] The partitioning depth value corresponding to the first type of image is greater than the partitioning depth value corresponding to the second type of image.

[0141] In some embodiments, the apparatus further includes a setting module.

[0142] The setting module is configured to perform either of the following: setting the partitioning depth value in a sequence parameter set (SPS) of each image in the image group; and setting the partitioning depth value in a PPS of the first type of image.

[0143] The division depth value is set in a sequence parameter set (SPS) of each image in the image group; and the division depth value is set in a PPS of the second type image.

[0144] In some embodiments, the setting module is configured to:

[0145] The division depth value is set in a picture header included in the PPS of the first type image.

[0146] In some embodiments, the encoding module is configured to:

[0147] The first type image is divided into a plurality of CTUs by using a multi-type tree;

[0148] For any one of the plurality of CTUs, the CTU is divided by using a multi-type tree based on the division depth value included in the PPS of the first type image, to obtain at least one CU.

[0149] In some embodiments, the encoding module is further configured to:

[0150] The second type image is divided into a plurality of CTUs by using a multi-type tree;

[0151] For any one of the plurality of CTUs, the CTU is divided by using a multi-type tree based on the division depth value included in the SPS of the second type image, to obtain at least one CU.

[0152] All the optional technical solutions described above can be combined to form optional embodiments of the disclosure, which will not be described here.

[0153] It should be noted that the video encoding apparatus provided in the above embodiments is only used as an example to illustrate the division of the above functional units, and in actual applications, the above functions can be completed by different functional units according to needs, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the video encoding apparatus and the video encoding method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be described here.

[0154] As to the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described here in detail.

[0155] In an exemplary embodiment, an electronic device is also provided, which includes:

[0156] One or more processors;

[0157] a memory for storing the processor-executable program code;

[0158] The processor is configured to execute the program code to implement the video encoding method.

[0159] Exemplarily, in response to the electronic device being provided as a terminal, Figure 10 is a block diagram of a terminal 1000 according to an exemplary embodiment.

[0160] Generally, the terminal 1000 includes a processor 1001 and a memory 1002.

[0161] The processor 1001 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1001 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by a display screen. In some embodiments, the processor 1001 can further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

[0162] The memory 1002 can include one or more computer-readable storage media, which can be non-transitory. The memory 1002 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one program code for being executed by the processor 1001 to implement the video encoding method provided by the method embodiments in the present disclosure.

[0163] In some embodiments, terminal 1000 can also optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1003 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1008.

[0164] The peripheral device interface 1003 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002 and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002 and the peripheral device interface 1003 can be implemented on a separate chip or circuit board, and the present embodiment is not limited in this regard.

[0165] The radio frequency circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1004 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G and 5G), wireless local area networks and / or WiFi (Wireless Fidelity) networks. In some embodiments, the radio frequency circuit 1004 can also include NFC (Near Field Communication) related circuitry, and the present disclosure is not limited in this regard.

[0166] The display screen 1005 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 is further configured to capture touch signals on or above the surface of the display screen 1005. The touch signals can be input to the processor 1001 as control signals for processing. In this case, the display screen 1005 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 1005 can be one, arranged on the front panel of the terminal 1000; in other embodiments, the display screen 1005 can be at least two, arranged on different surfaces of the terminal 1000 or in a folding design; in other embodiments, the display screen 1005 can be a flexible display screen, arranged on a curved surface or a folding surface of the terminal 1000. Even, the display screen 1005 can also be arranged in an irregular shape other than a rectangle, i.e., a special-shaped screen. The display screen 1005 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0167] The camera assembly 1006 is configured to capture images or videos. Optionally, the camera assembly 1006 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is arranged on the front panel of the terminal, and the rear-facing camera is arranged on the back of the terminal. In some embodiments, the rear-facing camera is at least two, which is any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1006 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0168] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1001 for processing, or input to the radio frequency circuit 1004 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1000. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0169] The power supply 1008 is used to power the various components in the terminal 1000. The power supply 1008 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 1008 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0170] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on terminal 1000 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0171] In one exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1002 including instructions, which can be executed by a processor 1001 of a terminal 1000 to complete the video encoding method described above. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0172] In one exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the video encoding method described above.

[0173] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0174] It should be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of video coding, the method comprising: The method includes: For any group of images in the video sequence to be encoded, a first class of images and a second class of images are determined in the group of images based on the number of times the images are referenced; A segmentation depth value is set in the sequence parameter set (SPS) of each image in the image group; and a segmentation depth value is set in the image header included in the image parameter set (PPS) of the first type of images; wherein, for any image in the image group, when both the SPS and the image header of the image include a segmentation depth value, the parameter priority corresponding to the image header is higher than the parameter priority corresponding to the SPS. For the first type of image, the first type of image is divided into multiple coding tree units (CTUs); for any one of the multiple CTUs, the CTU is divided based on the partitioning depth value included in the image header of the PPS of the first type of image to obtain multiple coding units (CUs); video encoding is performed in units of CUs of the first type of image; wherein, the partitioning depth value is used to indicate the maximum partitioning depth for partitioning the CTU into CUs. For the second type of image, the second type of image is divided into multiple CTUs; for any one of the multiple CTUs, the CTU is divided based on the division depth value included in the SPS of the second type of image to obtain multiple CUs; ​​video encoding is performed on the CUs of the second type of image as units.

2. The video coding method of claim 1, wherein, The determination of the first and second class images in the image group based on the number of times the image is referenced includes: The I-frames and P-frames in the image group, which serve as reference frames, are identified as images of the first type, and the B-frames in the image group are identified as images of the second type; or, The I-frame, P-frame, and B4-frame in the image group, which serve as reference frames, are identified as the first type of image, and the remaining B-frames in the image group are identified as the second type of image. The segmentation depth value corresponding to the first type of image is greater than the segmentation depth value corresponding to the second type of image.

3. A video encoding apparatus, comprising: The device includes: The determination module is configured to, for any group of images in the video sequence to be encoded, determine a first class of images and a second class of images in the group of images based on the number of times the images are referenced; The setting module is configured to set a segmentation depth value in the sequence parameter set (SPS) of each image in the image group; and to set a segmentation depth value in the image header included in the PPS of the first type of image; wherein, for any image in the image group, when both the SPS and the PPS of the image include a segmentation depth value in their image headers, the parameter priority corresponding to the image header is higher than the parameter priority corresponding to the SPS. The encoding module is configured to, for the first type of image, divide the first type of image into multiple coding tree units (CTUs); for any one of the multiple CTUs, divide the CTU based on the partitioning depth value included in the image header of the PPS of the first type of image to obtain multiple coding units (CUs); and perform video encoding in units of CUs of the first type of image; wherein the partitioning depth value is used to indicate the maximum partitioning depth for partitioning the CTU into CUs. The encoding module is further configured to, for the second type of image, divide the second type of image into multiple CTUs; for any one of the multiple CTUs, divide the CTU based on the division depth value included in the SPS of the second type of image to obtain multiple CUs; ​​and perform video encoding in units of CUs of the second type of image.

4. The video encoding apparatus according to claim 3, characterized in that, The determining module is configured as follows: The I-frames and P-frames in the image group, which serve as reference frames, are identified as images of the first type, and the B-frames in the image group are identified as images of the second type; or, The I-frame, P-frame, and B4-frame in the image group, which serve as reference frames, are identified as the first type of image, and the remaining B-frames in the image group are identified as the second type of image. The segmentation depth value corresponding to the first type of image is greater than the segmentation depth value corresponding to the second type of image.

5. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the video encoding method as described in claim 1 or 2.

6. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the video encoding method as described in claim 1 or 2.

Images