Regarding partitioning prediction

Abstract

A method for segmentation prediction implemented by an encoding / decoding device. The method includes obtaining a probability of the segmentation type to be implemented when segmenting an image; and skipping one or more segmentation types based on the obtained probabilities when segmenting encoding / decoding blocks of the image during a conversion between a video media file and a bitstream. Corresponding apparatus and a non-transitory computer-readable medium are also provided.

Description

[0001] Cross-reference to related applications

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 106,603, filed October 28, 2020, by Lemon Corporation, entitled “Regarding Partition Prediction,” which is incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to video encoding and decoding, and more particularly to encoder optimization and block partitioning prediction in image / video encoding and decoding. Background Technology

[0004] Digital video accounts for the largest share of bandwidth usage on the Internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention

[0005] The disclosed aspects / embodiments provide techniques for segmenting codec blocks using information received from a convolutional neural network (CNN). These techniques allow codec devices (e.g., encoders) to segment codec blocks more efficiently than known techniques. For example, the technique may include: ignoring partition types determined to be below a predetermined threshold when segmenting codec blocks; deriving the probability of the partition type to be implemented after identifying all possible segmentation structures of the codec block and considering a predetermined number of partition types and a predetermined depth; and / or examining the segmentation result of each sub-block in the codec block using partition types above the predetermined threshold. Therefore, encoding / decoding time is improved compared to existing encoding / decoding solutions.

[0006] The first aspect relates to a method implemented by an encoding / decoding device. The method includes obtaining a probability of a partition type implemented when segmenting an image, and skipping one or more partition types based on the obtained probabilities when segmenting encoding / decoding blocks during conversion between a video media file and a bitstream.

[0007] Optionally, in any of the foregoing aspects, another implementation of the aspect provides a probability of obtaining the partition type to be implemented, and further includes deriving the probability of the partition type to be implemented based on the probability that the partition type boundary coincides with the sub-block boundary of the codec block.

[0008] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides a method for determining which partition types are below a predetermined threshold based on probability, and skipping one or more partition types determined to be below the predetermined threshold when segmenting codec blocks.

[0009] Optionally, in any of the foregoing aspects, another implementation of that aspect provides partitioning types including non-partitioning type, quadtree partitioning type, binary tree partitioning type, and ternary tree partitioning type.

[0010] Optionally, in any of the foregoing aspects, another implementation of that aspect provides that the probability of the j-th sub-block boundary from the sub-block boundary is represented as... Where p represents the probability of the partition type, Non represents the non-partition partition type, QT represents the quadtree partition type, BT represents the binary tree partition type, TT represents the ternary tree partition type, j is an element of {1, 2, 3, ..., K}, and K represents the total number of sub-block boundaries in the encoding / decoding block.

[0011] Optionally, in any of the foregoing aspects, another implementation of the aspect provides that the probability of a partition type boundary coinciding with a sub-block boundary is based on an a×b probability vector, where a represents the total number of partition types of the sub-blocks of the codec block, and where b represents the total number of sub-block boundaries in the codec block.

[0012] Optionally, in any of the foregoing aspects, another implementation of the aspect provides that the probability of the implemented partitioning type is based on one of the following: all sub-block boundaries in the code-decode block, a subset of sub-block boundaries in the code-decode block, the average of the probabilities, or a weighted average of the probabilities.

[0013] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides a basis... Derive one of the probabilities from the partition type, where p represents the probability of the partition type, i represents the sub-block boundary from the sub-block boundary, ST represents one of the quadtree partition type, horizontal binary tree partition type, vertical binary tree partition type, horizontal ternary tree partition type and vertical ternary tree partition type, and x represents the number of sub-block boundaries in the codec block.

[0014] Optionally, in any of the foregoing aspects, another implementation of the aspect provides that obtaining the probability of the implemented partition type includes receiving input from a convolutional neural network that identifies all possible partition structures of the codec block and takes into account a predetermined number of partition types and a predetermined depth.

[0015] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides a probability distribution for deriving all possible segmentation structures based on the input, wherein the probability distribution includes the probability of each implemented segmentation type.

[0016] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides an input that identifies all possible segmentation structures of the codec block based on the sum of possible segmentations of non-partition type, quadtree partition type, horizontal binary tree partition type, vertical binary tree partition type, horizontal ternary tree partition type, and vertical ternary tree partition type.

[0017] Optionally, in any of the foregoing aspects, another implementation of the aspect provides that obtaining the probability of the implemented partition type includes receiving input from a convolutional neural network that identifies the partition type of the codec block.

[0018] Optionally, in any of the foregoing aspects, another implementation of the aspect provides determining which partition types are higher than a predetermined threshold, and examining the partitioning results of each sub-block in the codec block using partition types determined to be higher than the predetermined threshold.

[0019] The second aspect relates to an apparatus for encoding and decoding video data, including a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to: obtain a probability of a partition type to be implemented; and when partitioning a codec block, skip one or more partition types based on the obtained probability.

[0020] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides that one or more processors are further configured to derive the probability of the partition type being implemented based on the probability that the partition type boundary coincides with the sub-block boundary of the codec block.

[0021] Alternatively, in any of the foregoing aspects, another embodiment of the aspect provides that one or more processors are further configured to receive input from a convolutional neural network, the input identifying all possible segmentation structures of the codec block and taking into account a predetermined number of segmentation types and a predetermined depth.

[0022] Optionally, in any of the foregoing aspects, another embodiment of the aspect provides that one or more processors are further configured to determine which partition types are higher than a predetermined threshold, and to examine the partitioning results of each sub-block in the codec block using the partition types determined to be higher than the predetermined threshold.

[0023] The third aspect relates to a non-transitory computer-readable medium including a computer program product for use by an encoding / decoding apparatus, the computer program product including computer-executable instructions stored on the non-transitory computer-readable medium, which, when executed by one or more processors, cause the encoding / decoding apparatus to: obtain a probability of a partition type to be implemented; and skip one or more partition types based on the obtained probability when partitioning an encoding / decoding block.

[0024] Alternatively, in any of the foregoing aspects, another embodiment of the aspect provides instructions that also cause the codec apparatus to derive the probability of the partition type being implemented based on the probability that the partition type boundary coincides with the sub-block boundary of the codec block.

[0025] Optionally, in any of the foregoing aspects, another embodiment of the aspect provides instructions that the codec apparatus derives a probability distribution of all possible segmentation structures based on input from a convolutional neural network, the input identifying all possible segmentation structures of the codec block and taking into account a predetermined number of segmentation types and a predetermined depth, wherein the probability distribution includes the probability of each implemented segmentation type.

[0026] For the sake of brevity, any of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create new embodiments within the scope of this disclosure.

[0027] These and other features will become clearer from the following detailed description, taken in conjunction with the accompanying drawings and claims. Attached Figure Description

[0028] To gain a more complete understanding of this disclosure, reference is now made to the following brief description in conjunction with the accompanying drawings and detailed description, wherein the same reference numerals denote the same parts.

[0029] Figure 1 This is a schematic diagram of an encoder.

[0030] Figure 2 This is a schematic diagram of the type division in a multi-tree structure.

[0031] Figure 3 This is a diagram illustrating the directional patterns used in video encoding and decoding.

[0032] Figures 4A-4B It is a mapping between intra-prediction directions and intra-prediction modes with additional directional modes.

[0033] Figure 5 This is a diagram illustrating wide-angle prediction.

[0034] Figures 6A-6D The mapping of reference samples for the location-related intra-prediction combination (PDPC) applied under various prediction modes is shown.

[0035] Figures 7A-7B Examples of two segmentation probabilities are shown.

[0036] Figure 8 An example of the low-frequency non-separable quadratic transform (LFNST) is shown.

[0037] Figure 9A The forward and inverse reduction transformations (RT) are shown.

[0038] Figure 9B The horizontal and vertical transformations at each subblock transform (SBT) location are shown.

[0039] Figure 10 An example of a 64×64 block and its internal basic boundaries (aligned with the boundaries of a 4×4 block) is shown.

[0040] Figures 11A-11E Examples of five possible partitions of a 16×16 codec block are shown.

[0041] Figure 12 This is an example illustration of the integrated partitioning type of a codec block with a depth of 2.

[0042] Figure 13 An example of mapping in implicit partitioning prediction is shown.

[0043] Figure 14 An example of mapping in implicit partitioning prediction is shown.

[0044] Figure 15 Examples of CNN architectures for explicit and implicit split prediction are shown.

[0045] Figure 16 A table is shown illustrating the performance of the proposed implicit and explicit segmentation predictions at specific thresholds.

[0046] Figure 17 A table showing the proposed scheme performs well compared to another method is presented.

[0047] Figure 18 This is a method for encoding and decoding video data according to embodiments of the present disclosure.

[0048] Figure 19 This is a method for encoding and decoding video data according to embodiments of the present disclosure.

[0049] Figure 20 This is a method for encoding and decoding video data according to embodiments of the present disclosure.

[0050] Figure 21 This is a block diagram illustrating an example video codec system that can utilize the techniques disclosed herein.

[0051] Figure 22 This is a block diagram illustrating an example of a video encoder, which can be... Figure 21 The video encoder in the system shown.

[0052] Figure 23 This is a block diagram illustrating an example of a video decoder, which can be... Figure 21The video decoder in the system shown.

[0053] Figure 24 This is a block diagram illustrating an example video processing system in which various techniques disclosed herein can be implemented.

[0054] Figure 25 This is a block diagram of a video processing device.

[0055] Figure 26 This is a method for encoding and decoding video data according to embodiments of the present disclosure. Detailed Implementation

[0056] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the illustrative embodiments, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but modifications can be made within the full scope of the appended claims and their equivalents.

[0057] Universal Video Coding (VVC), also known as H.266, is a video compression standard finalized on July 6, 2020, by the Joint Video Experts Team (JVET). JVET is a joint video expert team comprised of the Video Coding Experts Group (VCEG) working group of Study Group 16 of the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the Moving Picture Experts Group (MPEG) working group of the Joint Technical Committee (JTC) 1 of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). VVC is the successor to High-Efficiency Video Coding (HEVC), also known as ITU-T H.265. The use of the term VVC in some descriptions is for ease of understanding only and not to limit the scope of the disclosed technology. Thus, the technology described herein is also applicable to other video codec protocols and designs.

[0058] This disclosure relates to image / video codec techniques. Specifically, this disclosure relates to encoder optimization and block partitioning prediction in image / video codecs. The techniques in this disclosure can be applied to existing video codec standards, such as, for example, HEVC, VVC, or the third-generation audio-video codec standard (AVS3). These techniques can also be applied to future video codec standards or video encoders.

[0059] Video codec standards have primarily evolved through the development of well-known ITU-T and ISO / IEC standards. ITU-T developed H.261 and H.263, while ISO / IEC developed the MPEG-1 and MPEG-4 vision standards. The two organizations jointly developed the H.262 / MPEG-2 video, H.264 / MPEG-4 Advanced Video Coding (AVC), and H.265 / HEVC standards. Since H.262, video codec standards have been based on a hybrid video codec architecture, employing temporal prediction plus transform coding. To explore future video codec technologies beyond HEVC, VCEG and MPEG jointly established JVET in 2015. Since then, JVET has adopted many new methods and applied them to a reference software called the Joint Exploration Model (JEM). In April 2018, JVET was created between VCEG (Q6 / 16) and ISO / IEC JTC1 SC29 / WG11 (MPEG) to research VVC standards. The goal of the VVC standard is to achieve a 50% reduction in bit rate compared to HEVC. VVC version 1 was finalized in July 2020.

[0060] The latest publicly available draft of VVC, Draft Universal Video Codec 10, can be found at the following URL: http: / / phenix.it-sudparis.eu / jvet / doc_end_user / current_document.php?id=10399. The latest publicly available reference software for VVC, called the VVC Test Model (VTM), can be found at the following URL: https: / / vcgit.hhi.fraunhofer.de / jvet / VVCSoftware_VTM / tags / VTM-10.0.

[0061] Figure 1This is a diagram of encoder 100, suitable for implementing VVC technology. Encoder 100 includes three loop filters: a deblocking filter (DF) 102, a sample adaptive offset (SAO) 104, and an adaptive loop filter (ALF) 106. Unlike DF 102, which uses predefined filters, SAO 104 and ALF 106 reduce the mean square error between original and reconstructed samples by adding offsets and applying finite impulse response (FIR) filters, respectively, using the original samples of the current image, by signaling the offset and the encoding / decoding side information of the filter coefficients. ALF 106 is located at the final processing stage of each image and can be considered as a tool to attempt to capture and repair artifacts caused by previous stages.

[0062] The encoder 100 also includes an intra-frame prediction component 108 and a motion estimation / compensation component 110 configured to receive input video. The intra-frame prediction component 108 is configured to perform intra-frame prediction, while the ME / MC component 110 is configured to perform inter-frame prediction using a reference image obtained from the reference image buffer 112. Residual blocks from the inter-frame or intra-frame prediction are fed into a transform component 114 and a quantization component 116 to generate quantized residual transform coefficients, which are then fed into an entropy codec component 118. The entropy codec component 118 entropy codes and decodes the prediction results and the quantized transform coefficients and sends them to a video decoder (not shown). Quantized components output from the quantization component 116 can be fed into an inverse quantization component 120, an inverse transform component 122, and a reconstruction (REC) component 124. The REC component 124 is capable of outputting images to a DF 102, a SAO 104, and an ALF 106 for filtering before these images are stored in the reference image buffer 112.

[0063] Images / strips / pieces are partitioned into a series of codec tree units (CTUs). The CTU concept discussed in this paper is the same as that in HEVC. For an image with a three-sample array (e.g., in a non-monochrome case), a CTU consists of an N×N block of luma samples and two corresponding chroma sample blocks. The maximum allowable size of the luma block in a CTU is specified as 128×128 (although the maximum size of the luma transform block is 64×64).

[0064] In HEVC, the CTU is divided into coding units (CUs) using a quadtree structure called the codec tree to accommodate various local characteristics. The decision of whether to use inter-image (temporal) prediction or intra-image (spatial) prediction to encode and decode image regions is made at the leaf CU level. Depending on the PU partitioning type, each leaf CU can be further divided into one, two, or four prediction units (PUs). Within a PU, the same prediction process is applied, and relevant information is transmitted to the decoder based on the PU. After obtaining residual blocks by applying the prediction process based on the PU partitioning type, the leaf CUs can be partitioned into transform units (TUs) according to another quadtree structure similar to the codec tree of the CUs. A key feature of the HEVC structure is its multiple partitioning concept, including CUs, PUs, and TUs.

[0065] In VVC, the concept of multiple segmentation unit types is replaced by a quadtree using a nested multi-type tree (MTT) with binary and ternary partitioning structures. That is, the MTT with binary and ternary partitioning structures removes the separation between the concepts of CU, PU, ​​and TU, except in a few cases where the CU might be larger than the PU, such as when the size of the CU exceeds the maximum transform length. The MTT with binary and ternary partitioning structures supports greater flexibility in the shape of the CU segmentation. In the codec tree structure, the CU can be square or rectangular. The CTU is first segmented by a quadtree (also known as a quadtree or quad tree) structure. Then, the leaf nodes of the quadtree can be further segmented using a multi-type tree structure.

[0066] Figure 2 This is a schematic diagram of partitioning type 200 in a multi-tree structure. For example... Figure 2 As shown, the multi-type tree structure has four partitioning types: vertical binary partition (SPLIT_BT_VER) 202, horizontal binary partition (SPLIT_BT_HOR) 204, vertical ternary partition (SPLIT_TT_VER) 206, and horizontal ternary partition (SPLIT_TT_HOR) 208. The leaf nodes of the multi-type tree are called codec units (CUs), and unless the CU is too large for the maximum transform length, this partitioning is used for prediction and transform processing without any further partitioning. This means that in most cases, in a quadtree codec block structure using nested multi-type trees, the CU, PU, ​​and TU have the same block size. An anomaly occurs when the maximum supported transform length is less than the width or height of the CU color component.

[0067] Figure 3This is an illustration of directional mode 300 used in video encoding and decoding. To capture arbitrary edge directions presented in natural video, the number of intra-frame directional modes has been expanded from the 33 used by HEVC to 65. Additional directional modes are... Figure 3 The image is depicted with unnumbered arrows, and the planar and DC modes remain unchanged. These denser directional intra-prediction modes are applicable to all block sizes as well as luma and chroma intra-prediction.

[0068] Traditional intra-frame prediction direction is defined as clockwise from 45 degrees to -135 degrees, such as... Figure 3 As shown. In the second version of the VTM test model (VTM2), for non-square blocks, several traditional angular intra-prediction modes are adaptively replaced with wide-angle intra-prediction modes. The replaced modes are signaled using the original method and remapped to the wide-angle mode index after parsing. The total number of intra-prediction modes remains unchanged at 67, and the intra-mode encoding and decoding remain unchanged.

[0069] In HEVC, each intra-frame codec block has a square shape, and the length of each side is a power of 2. Therefore, no division is needed when generating intra-frame predictions using Directional Codec (DC) mode. In VVV, blocks can have rectangular shapes, which generally requires division for each block. To avoid division in DC prediction, only the longer sides are used to calculate the average of non-square blocks.

[0070] Traditional angular intra-prediction directions are defined as clockwise from 45 degrees to -135 degrees. In VTM2, for non-square blocks, several traditional angular intra-prediction modes are adaptively replaced with wide-angle intra-prediction modes. The replaced modes are signaled using the original method and remapped to the wide-angle mode index after parsing. The total number of intra-prediction modes for a given block remains unchanged at 67, and the intra-mode encoding / decoding remains unchanged.

[0071] Figure 4A and Figure 4B These are mappings 400 and 450 between the intra-prediction direction and the intra-prediction mode with additional directional modes. To support all prediction directions, a top reference of length 2W+1 and a left reference of length 2H+1 are defined, where W represents the width and H represents the height.

[0072] In the wide-angle directional mode, the mode number of the replaced mode depends on the aspect ratio of the block. Table 1 shows the replaced intra-prediction modes.

[0073] Table 1: Intra-prediction modes replaced by wide-angle mode

[0074] condition Replaced intra-prediction mode W / H==2 Patterns 2, 3, 4, 5, 6, 7 W / H > 2 Patterns 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 W / H==1 none H / W = 1 / 2 Patterns 61, 62, 63, 64, 65, 66 H / W < 1 / 2 Patterns 57, 58, 59, 60, 61, 62, 63, 64, 65, 66

[0075] Figure 5 This is a diagram illustrating a 500° wide-angle prediction. (Example) Figure 5 As shown, in the case of using wide-angle intra-frame prediction, two vertically adjacent prediction samples can use two non-adjacent reference samples. Therefore, a low-pass reference sample filter and edge smoothing are applied to wide-angle prediction to reduce the increased spacing Δp. α The negative impact, where Δp represents the distance between two reference points (e.g., reference point 504) at angle α.

[0076] In VTM2, the intra-prediction results for planar modes are further modified using the Position-Dependent Intra-Prediction Combination (PDPC) method. PDPC is an intra-prediction method that combines unfiltered boundary reference samples with HEVC-type intra-prediction using filtered boundary reference samples. PDPC is applicable to the following intra-modal modes without signaling notification: planar, DC, horizontal, vertical, lower-left angle modes and their eight adjacent angle modes, and upper-right angle mode and its eight adjacent angle modes.

[0077] The prediction sample pred(x, y) is predicted using a linear combination of the intra-frame prediction mode (DC, plane, angle) and the reference sample according to the following equation:

[0078] pred(x, y) = (wL × R) -1，y +wT×R x，-1 -wTL×R -1，-1 +(64-wL-wT+wTL)×pred(x,y)+32)>>6

[0079] Where w represents the weighting factor, L represents the left side, R represents the right side, T represents the top, and R x，-1 and R -1，y Let R represent the reference points located at the top and left of the current sample point (x, y), respectively, and R... -1，-1 This represents the reference sample point located at the top left corner of the current block.

[0080] When PDPC is applied to DC, planar, horizontal, and vertical intra-frame modes, no additional boundary filter is required, unlike the case of HEVC DC mode boundary filter or horizontal / vertical mode edge filter.

[0081] Figures 6A-6D Reference samples (R) of PDPC applied to various prediction modes are shown. x，-1 R -1，y and R -1，-1 The mappings 600 and 610 are defined. The predicted sample point pred(x', y') is located at (x', y') within the prediction block 650. The reference sample point R x，-1The coordinates x are given by x = x' + y' + 1, and the reference sample point R is used. -1，y The coordinates y are similarly given by y = x' + y' + 1.

[0082] The PDPC weights depend on the prediction pattern, as shown in Table 2.

[0083] Table 2: Examples of PDPC weights based on prediction patterns

[0084]

[0085] In JVET-M0102, Intra Subblock Partitioning (ISP) was proposed, which divides the lumen intra-prediction block vertically or horizontally into 2 or 4 sub-partitions based on the block size dimension, as shown in Table 3.

[0086] Table 3: Total number of sub-partitions depending on block size.

[0087] Block size Number of sub-partitions 4×4 No division 4×8 and 8×4 2 All other cases 4

[0088] Figures 7A-7B Examples of two segmentation possibilities 700 and 710 are shown. All sub-partitions 730 of the codec block 740 satisfy the condition of having at least 16 samples. Figures 7A-7B In the diagram, W represents the width of the codec block 740, and H represents the height of the codec block 740.

[0089] For each of these sub-partitions 730, a residual signal is generated by entropy decoding of the coefficients sent by the encoder, followed by inverse quantization and inverse transform. The sub-partition 730 is then intra-predicted, and the corresponding reconstructed samples are finally obtained by adding the residual signal to the prediction signal. Therefore, the reconstructed values ​​of each sub-partition can be used to generate the prediction for the next sub-partition, and this process is repeated. All sub-partitions 730 share the same intra-frame mode.

[0090] Based on the intra-frame pattern and partitioning used, two different processing orders are employed, referred to as the normal order and the reverse order. In the normal order, the first sub-partition 730 to be processed is the sub-partition containing the top-left sample of the CU, and then it continues downwards (horizontal partitioning) or to the right (vertical partitioning). As a result, the reference samples used to generate the sub-partition prediction signal are located only to the left and top of the row. On the other hand, the reverse processing order either starts from the sub-partition containing the bottom-left sample of the CU and continues upwards, or starts from the sub-partition containing the top-right sample of the CU and continues to the left.

[0091] In addition to the Discrete Cosine Transform II (DCT-II or DCT-2) already used in HEVC, the Multiple Transform Selection (MTS) scheme is used for residual encoding and decoding of inter-frame and intra-frame codec blocks. MTS uses several selected transforms from Discrete Cosine Transform Version 8 (DCT8) / Discrete Cosine Transform Version 7 (DCT7). The newly introduced transform matrices are DST-VII and DCT-VIII. Table 4 shows the basis functions of the selected DST / DCT.

[0092] Table 4: Transformation Types and Basis Functions

[0093]

[0094] There are two ways to enable MTS: one is explicit MTS, and the other is implicit MTS.

[0095] Implicit MTS is a recent tool in VVC. The derivation of the variable implicitMtsEnabled is as follows:

[0096] Whether implicit MTS is enabled depends on the value of the variable implicitMtsEnabled. The derivation of the variable implicitMtsEnabled is as follows:

[0097] - If sps_mts_enabled_flag equals 1, and one or more of the following conditions are true, then implicitMtsEnabled is set to 1:

[0098] -IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT (i.e., ISP is enabled).

[0099] -cu_sbt_flag equals 1 (i.e., ISP is enabled), and Max(nTbW, nTbH) is less than or equal to 32.

[0100] -sps_explicit_mts_intra_enabled_flag equals 0 (i.e., explicit MTS is disabled), CuPredMode[0][xTbY][yTbY] equals MODE_INTRA, lfnst_idx[x0][y0] equals 0, and intra_mip_flag[x0][y0] equals 0.

[0101] Otherwise, implicitMtsEnabled is set to 0.

[0102] Based on Tables 39 and 40 in VVC, the derivation of the variable trTypeHor for the horizontal transform kernel and the variable trTypeVer for the vertical transform kernel is as follows:

[0103] - If one or more of the following conditions are true, then trTypeHor and trTypeVer are set to 0 (i.e., DCT2).

[0104] -cIdx is greater than 0 (i.e., for the chromaticity component).

[0105] –IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT, and lfnst_idx is not equal to 0

[0106] Otherwise, if implicitMtsEnabled equals 1, the following applies:

[0107] – If cu_sbt_flag equals 1, then trTypeHor and trTypeVer are specified in Table 40 according to cu_sbt_horizontal_flag and cu_sbt_pos_flag.

[0108] Otherwise (cu_sbt_flag equals 0), the derivation of trTypeHor and trTypeVer is as follows:

[0109] trTypeHor=(nTbW>=4&&nTbW<=16)? 1:0 (1188)

[0110] trTypeVer=(nTbH>=4&&nTbH<=16)? 1:0 (1189)

[0111] – Otherwise, according to mts_idx, trTypeHor and trTypeVer are specified in Table 39.

[0112] The derivation of variables nonZeroW and nonZeroH is as follows:

[0113] – If ApplyLfnstFlag equals 1, nTbW is greater than or equal to 4, and nTbH is greater than or equal to 4, then the following applies:

[0114] nonZeroW=(nTbW==4||nTbH==4)? 4:8 (1190)

[0115] nonZeroH=(nTbW==4||nTbH==4)? 4:8 (1191)

[0116] Otherwise, the following applies:

[0117] nonZeroW=Min(nTbW,(trTypeHor>0)?16:32) (1192)

[0118] nonZeroH=Min(nTbH,(trTypeVer>0)?16:32) (1193)

[0119] To control the MTS scheme, a flag is used to specify whether explicit intra / inter-frame MTS exists in the bitstream. Furthermore, two separate enable flags are specified at the Sequence Parameter Set (SPS) level for intra-frame and inter-frame use to indicate whether explicit MTS is enabled. When MTS is enabled at the SPS level, the CU-level transform index can be signaled to indicate whether MTS should be applied. Here, MTS applies only to luma. The CU-level MTS CU-level index (represented by mts_idx) is signaled when the following conditions are met.

[0120] – Both width and height are less than or equal to 32

[0121] –CBF brightness rating equals 1

[0122] -Non-TS

[0123] -Non-ISP

[0124] –Non-SBT

[0125] –Disable LFNST

[0126] - It has a non-zero coefficient and is not in the DC position (top left corner of the block).

[0127] - There are no non-zero coefficients outside the 16×16 region in the upper left corner.

[0128] If the first binary bit (bin) of mts_idx is zero, DCT2 is applied in both directions. However, if the first binary bit of mts_idx is 1, additional signaling is applied to the other two binary bits to indicate the transform type in the horizontal and vertical directions, respectively. The transform and signaling mapping table is shown in Table 5. An 8-bit master transform core is used when considering transform matrix precision. Therefore, all transform cores used in HEVC remain the same, including 4-point DCT-2 and DST-7, as well as 8-point, 16-point, and 32-point DCT-2. Furthermore, other transform cores include 64-point DCT-2, 4-point DCT-8, and 8-point, 16-point, and 32-point DST-7 and DCT-8, all using an 8-bit master transform core.

[0129] Table 5: Signaling Notifications of MTS

[0130]

[0131] To reduce the complexity of large-size DST-7 and DCT-8, for DST-7 blocks and DCT-8 blocks with a size equal to 32 (width or height, or both width and height), the high-frequency transform coefficients are set to zero. Only the coefficients in the 16×16 lower-frequency region are retained.

[0132] Similar to HEVC, the residual of a block can be encoded and decoded using the transform skip mode. To avoid redundancy in syntax encoding and decoding, when the CU-level MTS_CU_flag is not equal to 0, the transform skip flag is not signaled. The block size limit for transform skip is the same as that of MTS in JEM4, which means that transform skip is applicable to a CU when both the block width and height are equal to or less than 32.

[0133] Figure 8 An example of the low-frequency non-separable quadratic transform (LFNST) 800 is shown. The LFNST 800 was introduced in JVET-K0099, and a mapping of 4 transform sets (instead of 35 transform sets) was introduced in JVET-L0133. In this JVET-N0193, a 16×64 (further reduced to 16×48) matrix and a 16×16 matrix are used for 8×8 blocks and 4×4 blocks respectively. For the sake of convenient marking, the 16×64 (which can be further reduced to 16×48) transform is denoted as LFNST 8×8, and the 16×16 transform is denoted as LFNST 4×4.

[0134] Figure 9A An example of the forward and inverse reduced transform (RT) 900 is shown. The main idea of the RT is to map an N-dimensional vector to an R-dimensional vector in a different space, where R / N (R < N) is the reduction factor. The RT matrix is an R×N matrix as follows.

[0135]

[0136] Where the R rows of the transform are the R bases of the N-dimensional space. The inverse transform matrix of the RT is the transpose of its forward transform.

[0137] In this disclosure, an LFNST 8×8 matrix with a reduction factor of 4 (1 / 4 size) is applied. Therefore, a 16×64 direct matrix is ​​used instead of the 64×64 matrix, which is the size of a regular 8×8 inseparable transform matrix. In other words, a 64×16 inverse LFNST matrix is ​​used on the decoder side to generate the core (primary) transform coefficients within the top-left region of the 8×8 matrix. The forward LFNST 8×8 uses a 16×64 (or 8×64 for an 8×8 block) matrix such that it produces non-zero coefficients only within the top-left 4×4 region of a given 8×8 region. In other words, when LFNST is applied, this 8×8 region will have only zero coefficients outside the top-left 4×4 region. For the LFNST 4×4, a 16×16 (or 8×16 for a 4×4 block) direct matrix multiplication is applied.

[0138] The reverse LFNST is conditionally applied when the following two conditions are met:

[0139] a. The block size is greater than or equal to a given threshold (W>=4 && H>=4);

[0140] b. The change skip mode flag is set to 0.

[0141] If both the width (W) and height (H) of the transform coefficient block are greater than 4, then apply LFNST 8×8 to the top-left 8×8 region of the transform coefficient block. Otherwise, apply LFNST 4×4 to the top-left min(8,W)×min(8,H) region of the transform coefficient block.

[0142] If the LFNST index is equal to 0, then LFNST is not applied. Otherwise, LFNST is applied, and its kernel is selected using the LFNST index. The LFNST selection method and the encoding / decoding of the LFNST index will be explained later.

[0143] In addition, LFNST is applied to intra-CUs within intra-frame and inter-frame stripes, and for both luma and chroma. When dual-tree is enabled, separate signaling is used to notify the LFNST indexes for luma and chroma. For inter-frame stripes (dual-tree disabled), a single signaling is used to notify a single LFNST index, and that index is used for both luma and chroma.

[0144] At the 13th JVET conference, Intra-Frame Sub-Partitioning (ISP) was adopted as a new intra-frame prediction mode. When selecting the ISP mode, LFNST is disabled, and no signaling notification is given for the LFNST index, because the performance improvement gained even if LFNST is applied to every feasible partition block is negligible. Furthermore, disabling LFNST on the ISP prediction residuals may reduce encoding / decoding complexity.

[0145] The LFNST matrix is ​​selected from four transform sets, each consisting of two transforms. The choice of which transform set to apply is determined from the intra-prediction mode, as described below.

[0146] 1) If one of the three cross-component linear model (CCLM) modes is indicated, then transform set 0 is selected.

[0147] 2) Otherwise, perform the transformation set selection according to Table 6.

[0148] Table 6: Transform Set Selection Table

[0149]

[0150] The index for accessing Table 6 (denoted as IntraPredMode) has a range of [-14, 83], which is the transform mode index for wide-angle intra-frame prediction.

[0151] In JVET-N0217, Affine Ray Weighted Intra Prediction (ALWIP, also known as Matrix-Based Intra Prediction (MIP)) was proposed.

[0152] Two tests were conducted in JVET-N0217. In Test 1, ALWIP was limited to 8KB of memory, with a maximum of 4 multiplications per sample. Test 2 was similar to Test 1, but with a further simplified design in terms of memory requirements and model architecture.

[0153] • A single set of matrices and offset vectors for all block shapes.

[0154] • The number of patterns for all block shapes has been reduced to 19.

[0155] • Reduce memory requirements to 5760 10-bit values, or 7.20 kilobytes.

[0156] • Linear interpolation of the predicted samples is performed in a single step in each direction, instead of iterative interpolation in the first test.

[0157] For inter-frame prediction CUs where cu_cbf equals 1, signaling can be used to notify cu_sbt_flag to indicate whether decoding is performed on the entire residual block or on a sub-part of the residual block. In the former case, the inter-frame MTS information is further parsed to determine the transform type of the CU. In the latter case, the inferred adaptive transform is used to partially encode and decode the residual block, and the remaining parts of the residual block are zeroed out. SBT is not applied to the combined inter-frame-intra-frame mode and triangular prediction mode.

[0158] Figure 9BThe horizontal and vertical transformations 950° for each SBT location are shown, where w represents the width and h represents the height. In the sub-block transformations, position-dependent transformations are applied to the luminance transform blocks in SBT-V and SBT-H (the chroma transform block (TB) always uses DCT-2). The two locations in SBT-H and SBT-V are associated with different core transformations. More specifically, in Figure 9B The code specifies the horizontal and vertical transformations for each SBT position. For example, the horizontal and vertical transformations 950 for SBT-V position 0 are DCT-8 and DST-7, respectively. When one side of the residual TU is greater than 32, the corresponding transformation is set to DCT-2. Therefore, the sub-block transformation 950, together with the residual block TU tiling, the codec block flag (cbf), and the horizontal and vertical transformations 950, can be considered a syntax shortcut for the case where the main residual of the block is on one side of the block.

[0159] In VTM, the encoder uses a greedy search to find a suitable segmentation structure for each codec region. Specifically, the encoder calculates the rate-distortion (RD) cost of the codec region for each allowed segmentation pattern, as described above, and selects the segmentation pattern with the lowest cost. The number of allowed segmentation patterns for each codec region can be very large and independent of the region content, resulting in a significant burden on the encoder.

[0160] Current encoder algorithms used for searching for optimal partitions suffer from the following problems. First, current block partitioning / segmentation schemes in VTM are based on greedy search. Greedy search requires trying a large number of partitioning possibilities, making it very time-consuming. Second, there is no fast algorithm suitable for fast encoder partitioning decisions for video content.

[0161] This paper discloses techniques for segmenting codec blocks using information received from convolutional neural networks (CNNs). These techniques allow codec devices (e.g., encoders) to segment codec blocks more efficiently than known techniques. For example, the technique may include: ignoring partition types determined to be below a predetermined threshold when segmenting codec blocks; deriving the probability of the partition type being implemented after identifying all possible segmentation structures of the codec block and considering a predetermined number of partition types and a predetermined depth; and / or examining the segmentation results of each sub-block in the codec block using partition types above the predetermined threshold. Therefore, encoding / decoding time is improved compared to existing encoding / decoding solutions.

[0162] The items listed below should be considered as examples to illustrate general concepts. These examples should not be interpreted narrowly. Furthermore, these embodiments can be combined in any way.

[0163] In one example, the avg function avg(x1,x2,…,x) nReturns x1, x2, ..., x n The average value. In this disclosure, certain partition types (segmentation methods) are skipped based on the probability of certain partition types derived explicitly or implicitly.

[0164] In the implicit approach, the embodiment can first predict the probability of having different edge types at sub-block boundaries, and then derive the probability of a partition based on the probability of the sub-block boundaries. When the probability of a partition is below a predetermined threshold, the encoder will skip checking that partition. In the explicit approach, the embodiment can directly predict the probability distribution of all possible partition structures in the integrated partition space. Then, the embodiment can sort all partition structures according to the probability of all partition structures. To achieve speedup, the encoder only checks the top K partitions that fall within those partition structures and skips the rest, where K is an integer, variable, or function.

[0165] The partition types considered include at least one of non-partition (NS), quadtree (QT) partition, binary tree horizontal (BTH) partition, binary tree vertical (BTV) partition, ternary tree horizontal (TTH) partition, and ternary tree vertical (TTV) partition. These partitions can be applied to one or more color components, such as applying only to the luma component of a two-tree partition and the current encoding / decoding color component is luma; and applying to the two chroma components of a two-tree partition and the current encoding / decoding color component is chroma; or applying to the three color components in the odd case.

[0166] In the implicit method, to determine whether a block partition should be skipped, the process is as follows: the input is the block to be tested and the partition type, and then the probability of that partition is calculated. The calculation process will be further explained below. Then, if the derived probability is greater than a predetermined threshold, the partition is checked. Otherwise, the partition is skipped. In the implicit method, the process is as follows: the input is the block to be tested and the partition type, and then the encoder determines whether the partition of the block is among the top K partitions in the sorted partition structure. When the answer is yes, the encoder checks the partition. Otherwise, the partition is skipped.

[0167] For implicit-based partitioning type prediction, the following steps are implemented.

[0168] 1. A method is proposed to derive the probability of partitioning types based on the probability of sub-block boundaries.

[0169] a. In one example, a sub-block boundary is defined as the boundary of a sub-region within a video processing unit (e.g., a CTU / block to be encoded or decoded).

[0170] i. In one example, a subregion is defined as M×N, where M and / or N are predefined, signaled, or inferred integers.

[0171] 1) In one example, M = N = 4.

[0172] 2) In one example, M and / or N are set to the width and / or height of the minimum CU, respectively, where the minimum CU is signaled in the bitstream.

[0173] 3) In one example, M and / or N are set to the width and / or height of the minimum CU, respectively, where the minimum CU is partitioned according to one or more partition types (e.g., QT / BT / TT).

[0174] 4) In one example, the settings for M and / or N can depend on the color components.

[0175] b. Figure 10 An example of a 64×64 box 1000 and its internal basic boundaries 1010 (aligned with the boundaries of a 4×4 block) is shown. The basic boundaries are aligned with N×N (e.g., 4×4) blocks within an M×M block. Therefore, the number of basic boundaries is (M / N)×(M / N)×2-(M / N)×2 (e.g., Figure 10 (480 boundary in the middle).

[0176] 2. Defines the probability of the j-th sub-block boundary within a set of sub-block boundaries for a given partition type from a set of allowed partition types.

[0177] a. In one example, a set of allowed partition types is determined based on the block dimension / location / allowed partition type notified by signaling in the bitstream.

[0178] b. In one example, for a basic boundary represented by index j, j∈{1,2,3,...,K}, assume its type can be one of four cases: non-partitioning (also known as no partitioning), QT partitioning, BT partitioning, and TT partitioning. And the probability distribution of the sub-block boundary j can be represented as...

[0179] c. In one example, the probability distribution of sub-block boundaries is derived from the convolutional neural network through the training process.

[0180] d. In one example, the probability distribution of the sub-block boundaries is derived from other learning-based methods, such as support vector machines, linear regression, etc.

[0181] e. In one example, the probability distribution of the sub-block boundaries is derived using a manual method.

[0182] 3. Calculate the probability of the partition to be tested based on the sub-block boundaries associated with the partition to be tested.

[0183] a. In one example, the probability of the partition to be tested is defined as a function of the probability of being located at the boundaries of all sub-blocks in the partition.

[0184] i. In one example, the function is defined as the average of the probabilities.

[0185] 1) Alternatively, the function can be defined as the average of probabilities.

[0186] 2) Alternatively, the function can be defined as a weighted average of probabilities.

[0187] 3) Alternatively, the function can be defined as the maximum probability function.

[0188] ii. Alternatively, utilize only the selective sub-block boundaries located within the partitioned sub-blocks.

[0189] b. Figures 11A-11E Examples of five possible partitions 1100 of a 16×16 codec frame 1102 are shown. In such... Figure 11A In one example shown, the partition to be tested is a QT partition, and the block size is 16×16. Let the relevant sub-block boundaries of the QT partition be i1, i2, ..., i8. Then, the probability of the QT partition can be calculated as follows:

[0190] c. In such Figure 11B In one example shown, the partition to be tested is a BTH partition with a block size of 16×16. The basic boundaries of the sub-blocks in the BTH partition are denoted as i1, i2, ..., i4. Then, the probability of the BTH partition can be calculated as follows:

[0191] d. In such Figure 11C In one example shown, the partition to be tested is a BTV partition with a block size of 16×16. Let the relevant sub-block boundaries of the BTV partition be denoted as i1, i2, ..., i4. Then, the probability of the BTV partition can be calculated as follows:

[0192] e. In such Figure 11D In one example shown, the partition to be tested is a TTH partition with a block size of 16×16. Let the relevant sub-block boundaries of the TTH partition be i1, i2, ..., i8. Then, the probability of the TTH partition can be calculated as follows:

[0193] f. In such Figure 11E In one example shown, the partition to be tested is a TTV partition with a block size of 16×16. Let the relevant sub-block boundaries of the TTV partition be i1, i2, ..., i8. Then, the probability of the TTV partition can be calculated as follows:

[0194] 4. The encoder may also choose to examine the top N partitions (e.g., the N partition types with the highest probabilities) after sorting the probabilities of all possible partitions of the current block.

[0195] a. in Figures 11A-11E In one example shown, the encoder first derives the probabilities of five possible partitions: p QT p BTH p BTV p TTH and p TTV The encoder then sorts the five values ​​and selects the two partitions with the highest probabilities to examine.

[0196] b. In one example, non-partitions can always be checked, except for the first N partition types.

[0197] 5. The prediction process for different color component classifications can be independent.

[0198] a. In one example, the luminance partitioning type is predicted based on the luminance sub-block boundary probability, while the partitioning type of the two chrominance components is derived based on the chrominance sub-block probability.

[0199] b. In one example, the partition type of each component is predicted based on the sub-block boundary probability of the corresponding component.

[0200] c. Alternatively, the above methods can be applied to dual-tree cases and / or local dual-tree cases and / or split-plane encoding and decoding.

[0201] 6. The prediction process for different color component classification types can be the same.

[0202] a. In one example, the partition type is predicted based on the sub-block boundary probability of only one color component (e.g., luminance).

[0203] b. In one example, the partition type is predicted based on the sub-block boundary probabilities of all three color components.

[0204] i. In one example, the probability of a partition type can be based on a weighted average of all components.

[0205] 1) In one example, all weights can be equal.

[0206] 2) In one example, the weights may not be equal.

[0207] 3) In one example, the weights can depend on the color format.

[0208] c. Alternatively, the above method can be applied to the case of a single tree or internal stripes.

[0209] 7. Whether and / or how the proposed method is applied may depend on the location of the block (e.g., codec tree block (CTB) / CU) and / or the decoding information associated with the block (e.g., depth, block width / height, prediction mode, etc.).

[0210] a. In one example, the proposed method can be applied only to blocks that are not located at the boundaries of video processing units (e.g., pictures / strips / pieces / subpictures).

[0211] b. In one example, when a block is located at the boundary of a video processing unit (e.g., a picture / strip / piece / subpicture), the probability is derived based on a different rule than those described above.

[0212] For explicit partitioning type prediction, the following is implemented.

[0213] 8. Propose a method based on the output of CNN or any other method to derive the probability distribution of all possible segmentation structures in the integrated segmentation space of the video processing unit.

[0214] a. In one example, consider all possible segmentation structures of a video processing unit within a specific depth.

[0215] i. Figure 12 This is an example illustration of an integrated partition type 1200 for a codec block with a depth of 2. The current block selects QT, BTH, and TTH partitions in the top, middle, and bottom rows, respectively. The possible partitions of the first sub-block derived from the QT, BTH, and TTH partitions are shown. In one example, such as... Figure 12 As shown, all possible segmentation structures for a video processing unit within a depth of 2 are considered. The possible segmentation types for the current block can be NS, QT, BTH, BTV, TTH, and TTV. In the case of the current block being a QT segmentation type, each derived sub-block can be selected from NS, QT, BTH, BTV, TTH, and TTV, as shown below. Figure 12 As shown in the first line of the diagram. Therefore, the number of possible partitions originating from this situation is 6 × 6 × 6 × 6 = 1296. Similarly, the number of possible partitions originating from BTH / BTV is 5 × 5 = 25, and the number of possible partitions originating from TTH / TTV is 5 × 5 × 5 = 125. Therefore, the total number of partitions within the current block's depth of 2 is 1296 + 25 + 25 + 125 + 125 + 1 = 1597.

[0216] 9. Any suitable method can be used to derive the probability of the ensemble partition type, as described in the last bullet point.

[0217] a. In one example, a CNN is used to derive probabilities.

[0218] i. Alternatively, the CNN output can indicate the probability.

[0219] b. In one example, other learning-based methods, such as linear regression, support vector machines, or decision trees, can be used to derive probabilities.

[0220] 10. The encoder may choose to examine the partitions included in the top K integrated partitions for each sub-block within the video processing unit after sorting the probabilities of the integrated partition types of the video processing unit.

[0221] a. In one example, the encoder may choose to sort the K partitions based on the numbers output by the CNN.

[0222] b. In one example, K can be a variable.

[0223] c. In one example, K can be derived from block statistics.

[0224] i. In one example, K can be larger for blocks with larger gradients or variances.

[0225] The discussion focuses on CNN-based partitioning prediction for intra-frame acceleration of VVC.

[0226] An example method for implicit partitioning prediction is as follows.

[0227] In implicit partitioning prediction, a CNN is trained to predict the probability of having different edge types at sub-block boundaries. In this disclosure, a sub-block is defined as a 4×4 block within a 64×64 codec unit. Therefore, the CNN only needs to run once over the entire 64×64 codec unit and can then make decisions for any block contained within the 64×64 codec unit because their partition boundaries are always aligned with some 4×4 boundary.

[0228] Figure 13 An example of mapping 1300 in implicit partitioning prediction is shown. The total number of 4×4 boundaries within a 64×64 codec unit can be calculated to be 480. For each sub-block boundary, its type can be one of {SPLIT_QT, SPLIT_BT, SPLIT_TT, NON_SPLIT}. Therefore, the ground truth probability to be predicted is a 480×4 vector 1310, as shown... Figure 13 As shown. Using Figure 13 The CNN shown is used to predict a 480×4 probability vector 1310.

[0229] Considering the significant role of reference samples in the top row and left column during intra-frame prediction, which greatly influence the prediction of the current block's partition, the CNN input is a 65×65 block, including the current 64×64 encoder-decoder block and its reference samples. First, the input is fed into convolutional layers, and the feature map is scaled down using max pooling with a stride of 4. The purpose of max pooling is to aggregate the main features and increase the effective receptive field. Furthermore, it reduces computational complexity because subsequent convolutions can operate at much lower precision. Then, the feature map after max pooling is further passed through four residual blocks. Note that, for the same purpose mentioned earlier, a max pooling layer with a stride of 2 is inserted after every two residual blocks. Finally, a fully connected layer is used, followed by a shaping operation that maps the features extracted from the last pooling layer into a 4×480 probability vector, where 4 represents the number of partition types the sub-block boundary can have, and 480 is the number of 4×4 boundaries within the 64×64 encoder-decoder unit. Since this is a multi-class classification problem, the commonly used cross-entropy loss is chosen to drive the training process. Specifically, the cross-entropy loss of each sub-block boundary is calculated and then averaged as the final loss.

[0230] Figure 14 An example of mapping 1400 in implicit partition prediction is shown. After training is complete, the probability distribution of each sub-block boundary can be obtained. Note that the softmax operation is used to convert the CNN output into effective probabilities. Using these probabilities, it becomes simple to compute the probability (or score) of each partition during RD checking. For example, the probabilities of the sub-block boundaries contained in a partition can be averaged and treated as the probability of the partition. Then the encoder can skip checking partitions whose probabilities are below a predetermined threshold.

[0231] To perform explicit partition prediction, the integration of the partition space must first be determined. Without loss of generality, the size of the codec unit is restricted to 32×32, and all possible partition structures within a depth ≤ 2 are discussed (assuming the depth of 32×32 is 0).

[0232] Considering the possible partitions of a 32×32 block at depth = 1, there are 5 candidates: {SPLIT_QT, SPLIT_BT_HOR, SPLIT_BT_VER, SPLIT_TT_HOR, SPLIT_TT_VER}. To calculate the possible partitions at depth = 2, let's take a sub-block derived from SPLIT_QT as an example. Specifically, SPLIT_QT will generate four square sub-blocks, and each sub-block can be further selected from {NON_SPLIT, SPLIT_QT, SPLIT_BT_HOR, SPLIT_BT_VER, SPLIT_TT_HOR, SPLIT_TT_VER}. In other words, each sub-block can have 6 types of partitions, resulting in 6×6×6×6 = 1296 possible partition structures. Similarly, the total number of segment structures originating from SPLIT_BT_HOR (or SPLIT_BT_VER) will be 5 × 5 = 25, and the total number of segment structures originating from SPLIT_TT_HOR (or SPLIT_TT_VER) will be 5 × 5 × 5 = 125. Adding these numbers together, we get the number of segment structures for a 32 × 32 codec unit with a depth ≤ 2: 1296 + 25 × 2 + 125 × 2 + 1 = 1597.

[0233] Figure 15 An example of a CNN architecture 1500 for explicit and implicit partitioning prediction is shown. Explicit and implicit predictions share the same module located before the vertical dashed line and have different designs after the vertical dashed line. N+1 represents the input block size. For explicit partitioning prediction, N+1 equals 33, and Me (representing the total number of partitioning structures considered in this disclosure) equals 1597. For implicit partitioning prediction, N+1 equals 65, and Mi, representing the number of internal sub-block boundaries considered in this disclosure, equals 480.

[0234] Explicit split prediction is essentially a multi-class classification problem. Therefore, a CNN is trained to solve this problem, such as... Figure 15 As shown. The main structure of the CNN is the same as that of the CNN used for implicit partitioning prediction, and has been introduced in previous sections. The last layer, the fully connected layer, maps the features extracted from the third pooling layer to a 1597 probability vector. Then, the cross-entropy loss is calculated and used to drive the training process.

[0235] After training is complete and the probability distribution of segmentation structures is obtained, these segmentation structures are sorted according to their probabilities. The encoder can then decide to examine the top K segmentation structures falling within these structures and skip the rest to achieve encoder speedup. In practice, it is beneficial to divide the 32×32 blocks into several groups based on their activity (characterized by sample gradients or sample variance) and set different k-thresholds for different groups. Specifically, since it is more difficult to predict segmentation structures in blocks with high activity, the k-threshold for these blocks should be set higher, and vice versa.

[0236] The following discusses some exemplary setups. PyTorch is used as the training platform, an open-source machine learning architecture that accelerates the path from research prototypes to production deployment. For training and validation data, the DIV2K dataset, containing 800 training images and 100 validation images, was used. A separate model was trained for each test quadratic programming (QP). The trained CNN model was then plugged into VVC's reference software, VTM-7.0, for performance evaluation. In VTM, the CNN model's inference is based on LibTorch, the C++ interface provided by PyTorch.

[0237] We used the intra-frame (AI) configuration and the JVET-recommended general test conditions of QP 22, 27, 32, and 37. We adopted the conditions defined in (1) respectively. Incremental bit rate (BD-BR) and ΔET are used to evaluate the encoding / decoding efficiency and complexity reduction. In (1), EncTanchor and EncTproposed represent the encoding time of the VTM-7.0 anchor and the VTM-7.0 plus the proposed method, respectively. Note that the CNN runtime is already included in EncTproposed. For the test sequences, we consider commonly used test sequences, referred to as classes A1, A2, B, C, D, and E.

[0238]

[0239] Figure 16 Table 1600 shows the performance of the proposed implicit and explicit segmentation predictions at specific thresholds. Implicit segmentation prediction reduces encoding time by an average of 52.3% and BD-BR loss by 0.46%, while explicit segmentation prediction reduces encoding time by an average of 39.5% and BD-BR loss by 0.28%.

[0240] The proposed method was compared with several recently published works, and the results were presented. Figure 14The results show that, although the compared works are based on different software versions (e.g., VTM-4.0, VTM-6.1, or VTM-7.0), the differences between them in terms of partitioning and runtime are small. For example, according to Frank Bossen, Xiang Li, and Karsten Suehring, “Ahg report: Test Model Software Development (ahg3),” JVET-Q0003, January 2020, the runtime difference between VTM-6.2 and VTM-7.0 is less than 5%.

[0241] Figure 17 Table 1700 shows the superior performance of the proposed scheme compared to other methods. These other methods are from Sang-Hyo Park and Je-Won Kang, “Context-Based Ternary Tree Decision-Making Approach for General Video Coding in Fast Intra-Frame Encoding and Decoding,” IEEE Access vol.7, pp.172597-172605, 2019, and also from A Tissier, W Hamidouche, J Vanney, F Galpinz, and D Menard, “CNN-Driven Complexity Reduction for VVC Intra-Frame Encoders,” IEEE International Conference on Image Processing (ICIP) 2020. IEEE, 2020, pp.3139-3143. Specifically, the method disclosed in this disclosure achieves greater coding complexity reduction and produces less BD-BR loss.

[0242] Figure 18 This is a method 1800 for encoding and decoding video data according to embodiments of the present disclosure. Method 1800 can be performed by an encoding / decoding device (e.g., an encoder) having a processor and memory. Method 1800 can be implemented when determining how to segment or divide encoding / decoding blocks.

[0243] In block 1802, the encoding / decoding device derives the probability of the partition type being implemented based on the probability that the partition type boundary coincides with the sub-block boundary of the encoding / decoding block. In one embodiment, the partition type includes a non-partition type, a quadtree partition type, a binary tree partition type, and a ternary tree partition type.

[0244] In one embodiment, the probability of the j-th sub-block boundary originating from a sub-block boundary is expressed as: Where p represents the probability of the partition type, Non represents the non-partition partition type, QT represents the quadtree partition type, BT represents the binary tree partition type, TT represents the ternary tree partition type, j is an element of {1, 2, 3, ..., K}, and K represents the total number of sub-block boundaries in the encoding / decoding block.

[0245] In one embodiment, the probability that the partition type boundary coincides with the sub-block boundary is based on an a×b probability vector, where a represents the total number of partition types of the sub-blocks in the codec block, and b represents the total number of sub-block boundaries in the codec block.

[0246] In one embodiment, the probability of the partition type being implemented is based on all sub-block boundaries in the codec block. In one embodiment, the probability of the partition type being implemented is based on a subset of sub-block boundaries in the codec block. In one embodiment, the probability of the partition type being implemented is based on the average probability. In one embodiment, the probability of the partition type being implemented is based on a weighted average probability. In one embodiment, the probability of the partition type being implemented is based on a function of the maximum probability.

[0247] In one embodiment, based on Derive one of the probabilities from the partition type, where p represents the probability of the partition type, i represents the sub-block boundary from the sub-block boundary, ST represents one of the quadtree partition type, horizontal binary tree partition type, vertical binary tree partition type, horizontal ternary tree partition type and vertical ternary tree partition type, and x represents the number of sub-block boundaries in the codec block.

[0248] In box 1804, the codec determines which partition types are below a predetermined threshold. For example, the codec determines that a certain number of partition types are below the predetermined threshold.

[0249] In block 1806, when a codec block is segmented, the codec means ignores segmentation types determined to be below a predetermined threshold. In one embodiment, the codec means is configured to normalize a probability vector corresponding to the probability that a segmentation type boundary coincides with a sub-block boundary of the codec block. In one embodiment, the codec means is configured to perform a rate-distortion (RD) check to segment the codec block using segmentation types determined to be equal to or higher than the predetermined threshold.

[0250] Figure 19 This is a method 1900 for encoding and decoding video data according to embodiments of the present disclosure. Method 1900 can be performed by an encoding / decoding device (e.g., an encoder) having a processor and memory. Method 1900 can be implemented when determining how to divide or segment encoding / decoding blocks.

[0251] In block 1902, the encoding / decoding device receives input that identifies all possible segmentation structures of the encoding / decoding block and takes into account a predetermined number of segmentation types and a predetermined depth. In one embodiment, input is received from a convolutional neural network. In one embodiment, the predetermined number of segmentation types is five, wherein the segmentation types include a non-segmentation type, a quadtree segmentation type, a binary tree segmentation type, and a ternary tree segmentation type, and wherein the predetermined depth is 2.

[0252] In one embodiment, the input identifying all possible segmentation structures of the codec block is based on the sum of possible segmentations of no segmentation type, quadtree segmentation type, horizontal binary tree segmentation type, vertical binary tree segmentation type, horizontal ternary tree segmentation type, and vertical ternary tree segmentation type.

[0253] In block 1904, the codec device derives a probability distribution of all possible segmentation structures based on the input, wherein the probability distribution includes the probability of each implemented segmentation type. In one embodiment, the codec device is configured to segment codec blocks based on the probability distribution.

[0254] Figure 20 This is a method 2000 for encoding and decoding video data according to embodiments of the present disclosure. Method 2000 can be performed by an encoding / decoding device (e.g., an encoder) having a processor and memory. Method 2000 can be implemented when determining how to divide or segment coded blocks.

[0255] In block 2002, the codec device receives input identifying the partition type of the codec block. In one embodiment, the input is received from a convolutional neural network. In one embodiment, the partition type above a predetermined threshold is determined based on K, where K represents a variable or block statistic. In another embodiment, the partition type above a predetermined threshold is determined based on K, where the value of K depends on the gradient or variance of the codec block.

[0256] In block 2004, the codec device determines which partition types exceed a predetermined threshold. In block 2006, the codec device examines the partitioning results of each sub-block within the codec block using partition types determined to be above the predetermined threshold. Thereafter, the codec device is configured to partition or divide the codec block.

[0257] Figure 21 This is a block diagram illustrating an example video codec system 2100 that can utilize the techniques disclosed herein.

[0258] like Figure 21 As shown, the video encoding / decoding system 2100 may include a source device 2110 and a destination device 2120. The source device 2110 generates encoded / decoded video data, and this source device 2110 may be referred to as a video encoding / decoding device. The destination device 2120 can decode the encoded / decoded video data generated by the source device 2110, and the destination device 2120 may be referred to as a video decoding device.

[0259] The source device 2110 may include a video source 2112, a video encoder 2114, and an input / output (I / O) interface 2116.

[0260] Video source 2112 may include sources such as video capture devices, interfaces for receiving video data from video content providers, and / or computer graphics systems for generating video data, or combinations of these sources. Video data may include one or more pictures. Video encoder 2114 encodes and decodes the video data from video source 2112 to generate a bitstream. The bitstream may include a sequence of bits forming an encoded representation of the video data. The bitstream may include encoded pictures and related data. The encoded picture is an encoded representation of a picture. Related data may include sequence parameter sets, picture parameter sets, and other syntax structures. I / O interface 2116 may include a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be transmitted directly to destination device 2120 via network 2130 through I / O interface 2116. Encoded video data may also be stored on storage medium / server 2140 for access by destination device 2120.

[0261] Destination device 2120 may include I / O interface 2126, video decoder 2124 and display device 2122.

[0262] I / O interface 2126 may include a receiver and / or a modem. I / O interface 2126 may acquire encoded / decoded video data from source device 2110 or storage medium / server 2140. Video decoder 2124 may decode the encoded / decoded video data. Display device 2122 may display the decoded video data to a user. Display device 2122 may be integrated with destination device 2120 or may be external to destination device 2120 that is configured to interface with an external display device.

[0263] The video encoder 2114 and the video decoder 2124 can operate according to video compression standards, such as the High Efficiency Video Codec (HEVC) standard, the Universal Video Codec (VVM) standard, and other current and / or additional standards.

[0264] Figure 22 This is a block diagram illustrating an example of a video encoder 2200, which can be... Figure 21 The video encoder 2114 in the system 2100 shown.

[0265] The video encoder 2200 can be configured to perform any or all of the technologies disclosed herein. Figure 22 In the example, the video encoder 2200 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video encoder 2200. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0266] The functional components of the video encoder 2200 may include a segmentation unit 2201, a prediction unit 2202 (which may include a mode selection unit 2203, a motion estimation unit 2204, a motion compensation unit 2205, and an intra-frame prediction unit 2206), a residual generation unit 2207, a transform unit 2208, a quantization unit 2209, an inverse quantization unit 2210, an inverse transform unit 2211, a reconstruction unit 2212, a buffer 2213, and an entropy encoding / decoding unit 2214.

[0267] In other examples, the video encoder 2200 may include more, fewer, or different functional components. In one example, the prediction unit 2202 may include an intra-block copy (IBC) unit. The IBC unit can perform prediction in IBC mode, where at least one reference picture is the picture containing the current video block.

[0268] Furthermore, some components, such as the motion estimation unit 2204 and the motion compensation unit 2205, can be highly integrated, but for illustrative purposes, in Figure 22 The examples are shown separately.

[0269] The segmentation unit 2201 can segment an image into one or more video blocks. Figure 21 The video encoder 2114 and video decoder 2124 can support various video block sizes.

[0270] The mode selection unit 2203 can select one of the encoding / decoding modes (e.g., intra-frame or inter-frame) based on the error result, and provide the resulting intra-frame or inter-frame codec block to the residual generation unit 2207 to generate residual block data, and to the reconstruction unit 2212 to reconstruct the codec block for use as a reference image. In some examples, the mode selection unit 2203 can select a combination of intra-frame and inter-frame prediction modes (CIIP), where the prediction is based on the inter-frame prediction signal and the intra-frame prediction signal. In the case of inter-frame prediction, the mode selection unit 2203 can also select the resolution of the block's motion vector (e.g., sub-pixel or integer pixel precision).

[0271] To perform inter-frame prediction on the current video block, motion estimation unit 2204 can generate motion information for the current video block by comparing one or more reference frames from buffer 2213 with the current video block. Motion compensation unit 2205 can determine the predicted video block for the current video block based on the motion information and decoded samples of images from buffer 2213 other than the image associated with the current video block.

[0272] The motion estimation unit 2204 and the motion compensation unit 2205 can perform different operations on the current video block, depending on whether the current video block is in an I-band, P-band, or B-band. I-bands (or I-frames) have the lowest compressibility but do not require decoding of other video frames. S-bands (or P-frames) can be decompressed using data from the previous frame and are more compressible than I-frames. B-bands (or B-frames) can use the previous and next frames as data references to achieve the highest data compression.

[0273] In some examples, motion estimation unit 2204 can perform unidirectional prediction on the current video block, and can search for reference images in list 0 or list 1 for reference video blocks of the current video block. Motion estimation unit 2204 can then generate a reference index indicating the reference image in list 0 or list 1, which contains the reference video block and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 2204 can output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 2205 can generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0274] In other examples, motion estimation unit 2204 can perform bidirectional prediction on the current video block. Motion estimation unit 2204 can search for a reference video block for the current video block in the reference images in list 0, and can also search for another reference video block for the current video block in list 1. Motion estimation unit 2204 can then generate a reference index indicating the reference images in lists 0 and 1 containing the reference video blocks, and a motion vector indicating the spatial displacement between the reference video blocks and the current video block. Motion estimation unit 2204 can output the reference index and motion vector of the current video block as motion information for the current video block. Motion compensation unit 2205 can generate a predicted video block for the current video block based on the reference video blocks indicated by the motion information of the current video block.

[0275] In some examples, the motion estimation unit 2204 can output a complete set of motion information for use in the decoder's decoding process.

[0276] In some examples, the motion estimation unit 2204 may not output the complete set of motion information for the current video. Instead, the motion estimation unit 2204 may refer to motion information signaling from another video block to inform the motion information of the current video block. For example, the motion estimation unit 2204 may determine that the motion information of the current video block is sufficiently similar to the motion information of neighboring video blocks.

[0277] In one example, the motion estimation unit 2204 may indicate a value in the syntax structure associated with the current video block that indicates to the video decoder 2300 that the current video block has the same motion information as another video block.

[0278] In another example, motion estimation unit 2204 may identify another video block and motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. Video decoder 2124 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0279] As discussed above, the video encoder 2124 can predictively signal motion vectors. Two examples of predictive signaling notification techniques that can be implemented by the video encoder 2114 include Advanced Motion Vector Prediction (AMVP) and Merge Pattern Signaling Notification.

[0280] Intra-prediction unit 2206 can perform intra-prediction on the current video block. When intra-prediction unit 2206 performs intra-prediction on the current video block, it can generate prediction data for the current video block based on decoded samples from other video blocks in the same frame. The prediction data for the current video block may include the predicted video block and various syntax elements.

[0281] The residual generation unit 2207 can generate residual data for the current video block by subtracting (e.g., indicated by a minus sign) multiple predicted video blocks from the current video block. The residual data for the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0282] In other examples, such as in skip mode, there may be no residual data for the current video block, and the residual generation unit 2207 may not perform the subtraction operation.

[0283] The transform processing unit 2208 can generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to the residual video blocks associated with the current video block.

[0284] After the transform processing unit 2208 generates a transform coefficient video block associated with the current video block, the quantization unit 2209 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0285] The inverse quantization unit 2210 and the inverse transform unit 2211 can apply inverse quantization and inverse transform to the transform coefficient video block, respectively, to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 2212 can add the reconstructed residual video block to the corresponding samples from one or more predicted video blocks generated by the prediction unit 2202 to produce a reconstructed video block associated with the current block, which is stored in the buffer 2213.

[0286] After the video block is reconstructed by the reconstruction unit 2212, a loop filtering operation can be performed to reduce the video block effect in the video block.

[0287] The entropy encoding / decoding unit 2214 can receive data from other functional components of the video encoder 2200. When the entropy encoding / decoding unit 2214 receives data, it can perform one or more entropy encoding / decoding operations to generate entropy encoded / decoded data and output a bitstream including the entropy encoded / decoded data.

[0288] Figure 23 This is a block diagram illustrating an example of a video decoder 2300, which can be... Figure 21 The video decoder 2124 in the video encoding and decoding system 2100 shown.

[0289] The video decoder 2300 can be configured to perform any or all of the technologies disclosed herein. Figure 23 In the example, the video decoder 2300 includes multiple functional components. The techniques described in this disclosure can be shared among the various components of the video decoder 2300. In some examples, the processor can be configured to perform any or all of the techniques described in this disclosure.

[0290] exist Figure 23 In the example, the video decoder 2300 includes an entropy decoding unit 2301, a motion compensation unit 2302, an intra-frame prediction unit 2303, an inverse quantization unit 2304, an inverse transform unit 2305, a reconstruction unit 2306, and a buffer 2307. In some examples, the video decoder 2300 can perform functions typically associated with the video encoder 2114. Figure 21 The encoding and decoding process described is the opposite of the decoding process.

[0291] Entropy decoding unit 2301 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded blocks of video data). Entropy decoding unit 2301 can decode the entropy-encoded video data, and based on the entropy-decoded video data, motion compensation unit 2302 can determine motion information including motion vectors, motion vector precision, reference image list index, and other motion information. Motion compensation unit 2302 can determine such information, for example, by executing AMVP and Merge mode signaling notifications.

[0292] The motion compensation unit 2302 can generate motion compensation blocks and can perform interpolation based on an interpolation filter. The identifier of the interpolation filter to be used at sub-pixel precision can be included in the syntax element.

[0293] The motion compensation unit 2302 can use an interpolation filter, such as that used by the video encoder 2114 during the encoding and decoding of a video block, to calculate the interpolation of sub-integer pixels of the reference block. The motion compensation unit 2302 can determine the interpolation filter used by the video encoder 2114 based on the received syntax information, and use the interpolation filter to generate the prediction block.

[0294] The motion compensation unit 2302 may use some syntax information to determine the size of the blocks used to encode and decode the (multiple) frames and / or (multiple) stripes of the encoded and decoded video sequence, segmentation information describing how each macroblock of the image of the encoded and decoded video sequence is segmented, a mode indicating how each segment is encoded and decoded, one or more reference frames (and a list of reference frames) for each inter-frame encoded and decoded block, and other information used to decode the encoded and decoded video sequence.

[0295] Intra-prediction unit 2303 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Inverse quantization unit 2303 performs inverse quantization, i.e., dequantization, on the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 2301. Inverse transform unit 2303 applies an inverse transform.

[0296] The reconstruction unit 2306 can add the residual block to the corresponding prediction block generated by the motion compensation unit 2202 or the intra-frame prediction unit 2303 to form a decoded block. If necessary, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in the buffer 2307 to provide a reference block for subsequent motion compensation / intra-frame prediction, and also generates the decoded video for presentation on the display device.

[0297] Figure 24This is a block diagram illustrating an example video processing system 2400 in which various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 2400. System 2400 may include an input 2402 for receiving video content. The video content may be received in a raw or uncompressed format, such as 8 or 10-bit multi-component pixel values, or it may be in a compressed or encoded format. Input 2402 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, Passive Optical Network (PON), etc., and wireless interfaces such as Wi-Fi or cellular interfaces.

[0298] Video processing system 2400 may include codec component 2404 which can implement the various codecs or codec methods described in this document. Codec component 2404 can reduce the average bit rate of the video from input 2402 to the output of codec component 2404 to produce a codec representation of the video. Codec techniques are therefore sometimes referred to as video compression or video transcoding techniques. The output of codec component 2404 may be stored or transmitted via a communication connection such as represented by component 2406. The bitstream (or codec) representation of the video received at input 2402, whether stored or communicated, can be used by component 2408 to generate pixel values ​​or transmit as displayable video to display interface 2410. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “codec” operations or tools, it will be understood that codec tools or operations are used at the encoder, and the corresponding decoding tools or operations that inversely represent the codec results will be performed by the decoder.

[0299] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Attachment), PCI, IDE, etc. The technologies described in this document can be found in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of performing digital data processing and / or video display.

[0300] Figure 25This is a block diagram of a video processing apparatus 2500. Apparatus 2500 can be used to implement one or more methods described herein. Apparatus 2500 can be embodied in a smartphone, tablet, computer, Internet of Things (IoT) receiver, etc. Apparatus 2500 may include one or more processors 2502, one or more memories 2504, and video processing hardware 2506. The processors(multiple) 2502 may be configured to implement one or more methods described herein. The memories(multiple) 2504 may be used to store data and code for implementing the methods and techniques described herein. The video processing hardware 2506 may be used to implement some of the techniques described herein in a hardware circuit system. In some embodiments, the video processing hardware 2506 may be at least partially included in the processor 502 (e.g., a graphics coprocessor).

[0301] Figure 26 This is a method 2600 for encoding and decoding video data according to an embodiment of the present disclosure. Method 2600 can be performed by an encoding / decoding device (e.g., an encoder) having a processor and memory. Method 2600 can be implemented when determining how to partition or segment codec blocks. In block 2602, the encoding / decoding device derives the probability of the partitioning type implemented when segmenting the image. In block 2604, the encoding / decoding device skips one or more partitioning types based on the obtained probabilities when segmenting codec blocks during the conversion between video media files and bitstreams. In one embodiment, method 2600 may utilize or combine one or more features or processes of other methods disclosed herein.

[0302] In this document, the terms "digital media (e.g., images or videos)" or "video processing" can refer to media encoding / decoding, media decoding, media compression, or media decompression. The term "media" can refer to video, audio, or images. For example, a video compression algorithm can be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, and vice versa. The bitstream representation of the current video block can, for example, correspond to bits juxtaposed or scattered in different places within the bitstream, as defined by the syntax. For example, a macroblock can be encoded / decoded based on the error residual values ​​after transformation and encoding / decoding, and also using bits from the header and other fields in the bitstream. Furthermore, during the conversion, the decoder can, based on this determination, parse the bitstream knowing that some fields may or may not be present, as described in the solutions above. Similarly, the encoder can determine whether to include or exclude certain syntax fields and generate the encoding / decoding representation accordingly by including or excluding syntax fields from the encoding / decoding representation.

[0303] The disclosed and other solutions, examples, embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this document and their structural equivalents), or in a combination of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded and decoded on a computer-readable medium for use by a data processing apparatus to operate or control the operation of the data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances affecting machine-readable propagation signals, or a combination of one or more of them. The term "data processing apparatus" includes all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an operating environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. Propagation signals are artificially generated signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, generated to encode and decode information for transmission to a suitable receiver device.

[0304] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language (including compiled or interpreted languages) and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., a file storing one or more modules, subroutines, or code sections). Computer programs can be deployed to run on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected through a communications network.

[0305] The processes and logic described in this document can be executed by one or more programmable processors running one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic can also be executed by dedicated logic circuits, and the devices can be implemented as dedicated logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits).

[0306] Processors suitable for running computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from, transfer data to, or receive data from and transfer data to such mass storage devices. However, a computer does not require such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically EPROM (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and optical disc read-only memory (CD ROM) and digital versatile optical disc read-only memory (DVD-ROM) disks. The processor and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0307] While this patent document contains numerous details, these details should not be construed as limiting any subject matter or potentially claimed scope, but rather as descriptions of features specific to particular embodiments of a particular art. Certain features described in this patent document within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be excluded from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0308] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring the operations to be performed in the specific order shown or in a sequential manner, or as performing all shown operations to achieve the desired result. Furthermore, the separation of various system components in the embodiments described in this patent document should not be construed as requiring such separation in all embodiments.

[0309] Only some implementation methods and examples are described, and other implementation methods, enhancements and variations can be made based on the content described and shown in this patent document.

Claims

1. A method implemented by a video encoding / decoding device, comprising: Obtain the probability that a segmentation type is implemented when segmenting an image; Based on the probability, determine which of the partitioning types are below a predetermined threshold; as well as During the conversion between video media files and bitstreams, one or more partition types among the partition types are skipped based on the obtained probability when segmenting the codec blocks of the image, wherein one or more partition types among the partition types determined to be below the predetermined threshold are skipped when segmenting the codec blocks. The probability of a partition type being implemented is obtained by deriving the probability of a partition type being implemented based on the probability that the partition type boundary coincides with the sub-block boundary of the codec block.

2. The method according to claim 1, wherein, The partitioning types include non-partitioning type, quadtree partitioning type, binary tree partitioning type, and ternary tree partitioning type.

3. The method according to claim 1, wherein, The probability of originating from the j-th sub-block boundary of the sub-block boundary is expressed as: , where p represents the probability of the partitioning type, Non represents the non-partitioning partitioning type, QT represents the quadtree partitioning type, BT represents the binary tree partitioning type, TT represents the ternary tree partitioning type, j is an element of {1, 2, 3, ... K}, and K represents the total number of sub-block boundaries in the encoding / decoding block.

4. The method according to claim 1 or 3, wherein, The probability that the partition type boundary coincides with the sub-block boundary is based on an a×b probability vector, where a represents the total number of partition types of the sub-blocks in the codec block, and b represents the total number of sub-block boundaries in the codec block.

5. The method according to any one of claims 1 and 3-4, wherein, The probability of a partition type being implemented is based on one of the following: all sub-block boundaries in the codec block, a subset of sub-block boundaries in the codec block, the average probability, or a weighted average probability.

6. The method according to any one of claims 1-5, wherein, based on Derive one of the probabilities from the partition type, where p represents the probability of the partition type, i represents the sub-block boundary from the sub-block boundary, ST represents one of the quadtree partition type, horizontal binary tree partition type, vertical binary tree partition type, horizontal ternary tree partition type and vertical ternary tree partition type, and x represents the number of sub-block boundaries in the codec block.

7. The method according to claim 1, wherein, Obtaining the probability that a partition type is implemented involves receiving input from a convolutional neural network, which identifies all possible partition structures of the codec block and takes into account a predetermined number of partition types and a predetermined depth.

8. The method of claim 7, further comprising deriving a probability distribution of all possible segmentation structures based on the input, wherein the probability distribution includes the probability of each of the segmentation types being implemented.

9. The method according to claim 8, wherein, The input representing all possible segmentation structures of the codec block is the sum of possible segmentations based on non-segmentation type, quadtree segmentation type, horizontal binary tree segmentation type, vertical binary tree segmentation type, horizontal ternary tree segmentation type, and vertical ternary tree segmentation type.

10. The method according to claim 1, wherein, Obtaining the probability that a partition type is implemented includes receiving input from a convolutional neural network, the input identifying the partition type of the codec block.

11. The method of claim 10, further comprising determining which of the partition types are above a predetermined threshold, and examining the segmentation result of each sub-block in the codec block using partition types determined to be above the predetermined threshold.

12. An apparatus for encoding and decoding video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, When the instruction is executed by the processor, the processor: Obtain the probability that a segmentation type is implemented when segmenting an image; Based on the probability, determine which of the partitioning types are below a predetermined threshold; and During the conversion between video media files and bitstreams, one or more partition types among the partition types are skipped based on the obtained probability when segmenting the codec blocks of the image, wherein one or more partition types among the partition types determined to be below the predetermined threshold are skipped when segmenting the codec blocks. The probability of a partition type being implemented is obtained by deriving the probability of a partition type being implemented based on the probability that the partition type boundary coincides with the sub-block boundary of the codec block.

13. The apparatus according to claim 12, wherein, The one or more processors are further configured to receive input from a convolutional neural network, the input identifying all possible segmentation structures of the codec block and taking into account a predetermined number of segmentation types and a predetermined depth.

14. The apparatus according to claim 12, wherein, The one or more processors are further configured to determine which of the partition types are higher than a predetermined threshold, and to examine the segmentation results of each sub-block in the codec block using partition types that are determined to be higher than the predetermined threshold.

15. A non-transitory computer-readable medium comprising computer-executable instructions used by an encoding / decoding apparatus, which, when executed by one or more processors, cause the encoding / decoding apparatus to: Obtain the probability that a segmentation type is implemented when segmenting an image; Based on the probability, determine which of the partitioning types are below a predetermined threshold; as well as During the conversion between video media files and bitstreams, one or more partition types among the partition types are skipped based on the obtained probability when segmenting the codec blocks of the image, wherein one or more partition types among the partition types determined to be below the predetermined threshold are skipped when segmenting the codec blocks. The probability of a partition type being implemented is obtained by deriving the probability of a partition type being implemented based on the probability that the partition type boundary coincides with the sub-block boundary of the codec block.

16. The non-transitory computer-readable medium according to claim 15, wherein, The instructions also cause the codec to derive a probability distribution of all possible segmentation structures based on input from a convolutional neural network, the input identifying all possible segmentation structures of the codec block and taking into account a predetermined number of segmentation types and a predetermined depth, wherein the probability distribution includes the probability of each of the segmentation types being implemented.

17. A non-transitory computer-readable storage medium for storing a bitstream of visual media files, said bitstream being generated by a method performed by means of encoding / decoding video data, wherein, The non-transitory computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to: Determine the probability that a segmentation type will be implemented when segmenting an image; Based on the probability, determine which of the partitioning types are below a predetermined threshold; When segmenting the codec blocks of the image, one or more segmentation types among the segmentation types are skipped based on the obtained probability, wherein when segmenting the codec blocks, one or more segmentation types among the segmentation types determined to be below the predetermined threshold are skipped; as well as The bit stream is generated based on the determination. The probability of a partition type being implemented is obtained by deriving the probability of a partition type being implemented based on the probability that the partition type boundary coincides with the sub-block boundary of the codec block.

18. A method for storing a video bitstream, comprising: Determine the probability that a segmentation type will be implemented when segmenting an image; Based on the probability, determine which of the partitioning types are below a predetermined threshold; When segmenting the codec blocks of the image, one or more segmentation types among the segmentation types are skipped based on the obtained probability, wherein when segmenting the codec blocks, one or more segmentation types among the segmentation types determined to be below the predetermined threshold are skipped; The bit stream is generated based on the determination; as well as The bitstream is stored in a non-transitory computer-readable recording medium. The probability of a partition type being implemented is obtained by deriving the probability of a partition type being implemented based on the probability that the partition type boundary coincides with the sub-block boundary of the codec block.

Images