Method, apparatus and medium for encoding block of image data

Abstract

The invention relates to a method and apparatus for encoding a block of image data, and a medium. A method includes determining a set of partitioning probabilities associated with a block of image data based at least on a neural network; partitioning the block of image data into two or more smaller prediction units based at least on the determined set of partitioning probabilities; and encoding the block of image data at least based on the partition.

Description

[0001] This application is a divisional application of the invention patent application with application number 201880080802.4, application date December 12, 2018, and invention name “Texture-based partitioning decision for video compression”. Technical Field

[0002] The present principles relate to the field of video compression. Background Art

[0003] In the HEVC video compression standard (International Telecommunication Union, ITU-T H.265 High Efficiency Video Coding), pictures are divided into so-called Coding Tree Units (CTUs), which typically have a size of 64x64, 128x128 or 256x256 pixels.

[0004] Each CTU is represented by a coding tree in the compressed domain. This is a quadtree partition of the CTU, where each leaf is called a coding unit (CU), such as Figure 1 shown.

[0005] Each CU is then provided with some intra or inter prediction parameters (prediction information). To do this, it is spatially partitioned into one or more prediction units (PUs), each of which is assigned some prediction information. Intra or inter coding modes are assigned at the CU level, see Figure 2 .

[0006] The coding unit is partitioned into prediction units according to the partition type signaled in the bitstream. For intra coding units, only Figure 3 The partition types 2Nx2N and NxN are shown in . This means that only square prediction units are used in intra coding units.

[0007] Instead, inter-frame coding units can use Figure 3 All partition types shown.

[0008] According to the HEVC standard, following the "transform tree", coding units are also divided into so-called transform units in a recursive manner. The transform tree is thus a quadtree partitioning of coding units and transform units are the leaves of the transform tree. A transform unit encapsulates a square transform block for each picture component corresponding to the square spatial region considered. A transform block is a square block of samples in a single component, where the same transform is applied.

[0009] Emerging video compression tools include coding tree unit representation in the compressed domain, which is proposed to represent picture data in a more flexible way in the compressed domain. The advantage of this flexible representation of the coding tree is that it provides higher compression efficiency compared to the CU / PU / TU arrangement of the HEVC standard.

[0010] The Quadtree plus Binary Tree (QTBT) coding tool provides this increased flexibility. It contains a coding tree where coding units can be partitioned in a quadtree and binary tree manner. Figure 4 This coding tree representation of the coding tree unit is illustrated in FIG. The QTBT representation on the block is Figure 5 Shown in.

[0011] The partitioning of coding units is decided at the encoder side through a rate-distortion optimization process, which involves determining the QTBT representation of the CTU with the minimum rate-distortion cost.

[0012] In QTBT technology, the CU has a square or rectangular shape. The size of the coding unit is always a power of 2, usually from 4 to 256.

[0013] In addition to the various rectangular shapes used for coding units, this new CTU representation has the following different features compared to HEVC.

[0014] First, the QTBT decomposition of a CTU is performed in two stages: first, the CTU is split in a quadtree manner, and then the leaves of each quadtree can be further divided in a binary manner. Figure 4 The right side of is illustrated, where the solid line represents the quadtree decomposition stage and the dashed line represents the binary decomposition of the spatial decomposition embedding into the quadtree leaves.

[0015] Second, within an intra slice, the luma and chroma block partition structures are separated and determined independently.

[0016] Next, CU partitioning into prediction units or transform units is no longer used. In other words, each coding unit is systematically composed of a single prediction unit (previously 2Nx2N prediction unit partition type) and a single transform unit (not divided into transform trees).

[0017] However, QTBT technology needs to further improve compression efficiency. Summary of the invention

[0018] These and other drawbacks and deficiencies of the prior art are addressed by at least one of the described embodiments, which are directed to methods and apparatus for encoding or decoding a block of video data.In at least one embodiment, a texture-based technique is proposed to generate a set of partition probabilities for a segmentation.

[0019] According to at least one general embodiment described herein, a method is provided, comprising: determining a set of partition probabilities associated with an image data block based at least on a neural network; partitioning the image data block into two or more smaller prediction units based at least on the determined set of partition probabilities; and encoding the image data block based at least on the partitioning.

[0020] According to another general embodiment described herein, there is provided an apparatus for encoding an image data block, comprising: a memory, and a processor, configured to: determine a set of partition probabilities associated with the image data block based at least on a neural network; partition the image data block into two or more smaller prediction units based at least on the determined set of partition probabilities; and encode the image data block based at least on the partitioning.

[0021] According to another aspect described herein, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the above method embodiment.

[0022] These and other aspects, features and advantages of the present principles will become apparent from a reading of the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 An example of the concept of a coding tree unit and a coding tree representing a compressed picture is illustrated.

[0024] Figure 2 An example of dividing a coding tree unit into coding units, prediction units, and transform units is illustrated.

[0025] Figure 3 An example of partitioning of a coding unit is illustrated.

[0026] Figure 4 An example quadtree plus binary tree coding tree unit representation is illustrated.

[0027] Figure 5 An embodiment of a quadtree plus binary tree representation on a coding block is illustrated.

[0028] Figure 6 A triple mode partitioning of a block is illustrated.

[0029] Figure 7 Different partitioning of the block for handling block non-uniformity is illustrated.

[0030] Figure 8 The diagram illustrates the set of all coding unit partitioning modes supported in the video coding scheme described herein.

[0031] Fig. 9 An example embodiment of the entire encoding of a coding tree unit (CTU) with deep learning based partitioning decisions is illustrated.

[0032] Fig.10 One embodiment of a standard, general-purpose video compression scheme is illustrated.

[0033] Fig.11 An example of pixels input to a convolutional neural network (CNN) is shown.

[0034] Fig.12 The figure shows the boundary information as an auxiliary input channel of CNN.

[0035] Fig.13 An example of the resulting vector of the boundary is illustrated.

[0036] Fig.14 An example of the basic principle of partitioning is illustrated (left) and the corresponding score represented by each pixel (right).

[0037] Fig.15 Exemplary segmentation positions L with corresponding boundary probabilities i, j, k, . . . are illustrated.

[0038] Fig.16 Illustrated is a graph of exemplary probabilities of quadtree partitions at different levels in the tree, averaged over a large test set of 19336 65x65 patches at QP=32 for JEM (quadtree / bintree).

[0039] Fig.17 Illustrated are exemplary probabilities of not splitting at different levels in the tree, averaged over a large test set of 19336 65x65 patches at QP=32 for JEM (quadtree / bintree).

[0040] Fig.18 An exemplary embodiment of a flow chart for compressing a CTU including a CNN-based decision is illustrated.

[0041] Fig.19 One embodiment of a method using the described aspects is illustrated.

[0042] Fig. 20 One embodiment of an apparatus utilizing the described aspects is illustrated.

[0043] Fig.21 Plotted are vectors of probabilities, averaged separately on each side of the horizontal boundary.

[0044] Fig. 22 A decision diagram for the no-split decision is illustrated.

[0045] Fig.23 Illustrated is testing quadtree partitioning when both vertical and horizontal boundaries are absent.

[0046] Fig.24 An example of testing a binary horizontal split is shown.

[0047] Fig.25An example of an asymmetric top split test is shown. DETAILED DESCRIPTION

[0048] An approach to improved entropy coding efficiency of transform coefficients in a coding unit is described to provide good compression with minimal increase in encoder design complexity.

[0049] For clarity, in this specification, "division", "segmentation" and "segmentation" all mean the same, which is the action of linearly dividing a block of pixels. Similarly, "segmentation" and "partitioning" mean the same, the grouping of pixels as a result of the division, segmentation or segmentation of a block or sub-block.

[0050] In the HEVC video compression standard, a picture is divided into so-called Coding Tree Units (CTUs), which typically have a size of 64x64, 128x128 or 256x256 pixels.

[0051] Each CTU is represented by a coding tree in the compressed domain. This is a quadtree partition of the CTU, where each leaf is called a coding unit (CU), see Figure 1 .

[0052] Each CU is then given some intra or inter prediction parameters (prediction information). For this purpose, it is spatially partitioned into one or more prediction units (PUs), each of which is assigned some prediction information. Intra or inter coding modes are assigned at CU level, see Figure 2 .

[0053] The partitioning of coding units into prediction units is done according to the partitioning type signaled in the bitstream. For intra coding units, only Figure 3 The illustrated partition types are 2Nx2N and NxN. This means that only square prediction units are used in intra coding units.

[0054] Instead, inter-frame coding units can use Figure 3 All partition types shown.

[0055] According to the HEVC standard, following the "transform tree", coding units are also divided in a recursive manner into so-called transform units. The transform tree is thus a quadtree partitioning of coding units and transform units are the leaves of the transform tree. A transform unit encapsulates a square transform block for each picture component corresponding to a square spatial area under consideration. A transform block is a square block of samples in a single component, where the same transform is applied.

[0056] Emerging video compression tools include coding tree unit representation in the compressed domain to represent picture data in a more flexible way in the compressed domain. The advantage of this flexible representation of the coding tree is that it provides increased compression efficiency compared to the CU / PU / TU arrangement of the HEVC standard.

[0057] The Quadtree plus Binary Tree (QTBT) coding tool provides this increased flexibility. It consists in a coding tree in which a coding unit can be partitioned in both a quadtree (4 sub-units) and a binary tree (2 sub-units) manner. Figure 4 This coding tree representation of a coding tree unit is illustrated.

[0058] The partitioning of the coding units is decided at the encoder side by a rate-distortion optimization process which aims at determining the QTBT representation of the CTU with minimum rate-distortion cost.

[0059] In QTBT technology, a CU has a square or rectangular shape. The size of a coding unit is usually a power of 2, typically from 4 to 256.

[0060] In addition to the various rectangular shapes used for coding units, this new CTU representation has the following different characteristics compared to HEVC.

[0061] First, the QTBT decomposition of a CTU consists of two stages: first, the CTU is split in a quadtree manner, and then each quadtree leaf can be further divided in a binary manner. Figure 4 The right side of is illustrated, where the solid line represents the quadtree decomposition stage and the dashed line represents the binary decomposition spatially embedded in the quadtree leaves.

[0062] Next, within an intra slice, the luma and chroma block partition structures are separated and determined independently.

[0063] Furthermore, CU partitioning into prediction units or transform units is no longer adopted. In other words, each coding unit systematically consists of a single prediction unit (previously 2Nx2N prediction unit partition type) and a single transform unit (not partitioned into transform trees).

[0064] The described embodiments focus on the field of picture coding unit representation in compression and aim at further improved compression efficiency.

[0065] A ternary tree or ternary partitioning is a partitioning of a coding unit or subunit into three parts. One advantage is that all subblocks can be powers of 2. The main disadvantage is that even only two are necessary, for example when the boundaries of objects pass close to each other (see Figure 7 , trifurcated partitioning and asymmetric partitioning), does not provide much flexibility in optimally choosing the partitioning of a block because it forces three sub-blocks.

[0066] The coding unit partitioning mode, called horizontal or vertical ternary tree partitioning mode, consists in dividing the coding unit (CU) into three sub-coding units (sub-CUs), whose respective sizes are equal to 1 / 4, 1 / 2 and 1 / 4 of the parent CU size in the considered spatial partitioning direction. Figure 6 Example in. Figure 8 shows the set of all coding unit partitioning modes supported in the video coding scheme described herein.

[0067] The problem solved by at least one of the described embodiments is how to improve and / or speed up the rate-distortion optimization (RDO) performed at the encoder, which corresponds to the decision made by the encoder to select the best tool for the criteria to compress a given block, macroblock or coding tree unit (CTU). Instead of checking the bitrate / distortion at each block level to find the best partitioning, a CNN (Convolutional Neural Network) network, for example, provides a probability score for each possible type of partitioning into smaller blocks. The calculation of this score is very fast because it can be highly parallelized and has lower complexity than the classic RDO operation. This is an exemplary solution, but the described embodiments can receive any partitioning score as input.

[0068] These embodiments are incorporated into the framework of video compression, particularly in the encoding stage, where a decision is made to divide the images of a given input video sequence into blocks. Since H.264 / MPEG4-AVC, hybrid codecs have divided images into blocks of different sizes to adapt their compression tools to textures and objects. In H.265 / HEVC, for example, images are first divided into 64x64 non-overlapping coding tree units (CTUs), which are then potentially divided into multiple smaller coding units (CUs) up to 8x8 pixels. Although the standard only specifies the syntax for signaling the size of these CUs, the encoder still requires an efficient way to select and optimize the size on the image, depending on its ability to compress different areas. This process is included in the so-called rate-distortion optimization (RDO), which aims to find the best compromise between the target quality of the reconstructed area and the bits required for transmission. Basically, if the CTU contains a well-predicted and reconstructed signal, there is no need to divide it into small blocks that will result in more syntax. However, when highly detailed textures are difficult to encode and result in residuals with high energy, it makes sense to split them into small blocks, as the cost of the syntax is worth it compared to the cost reduction in the residuals. Full RDO then consists in evaluating every possible block partition of the CTU, which is quite complex. The reference software encoder uses a subset of full RDO, considering only fixed causal parts in a CTU and adding some heuristics, but the combinatoriality is still high.

[0069] The classic encoder performs a depth-first tree search recursive process to determine the coding tree associated with the CTU. At each depth level, the coding unit partitioning mode (no partitioning, quadtree block partitioning, binary block partitioning or other partitioning) that gives the lowest rate-distortion cost is selected.

[0070] In the described embodiment, a set or table of probabilities of segmentation computed from texture analysis of pixels in and surrounding a block of image data is used to predict when to segment a given block into smaller blocks.

[0071] To perform RDO with acceptable complexity, several approaches have been implemented in existing codecs.

[0072] First, the search is split into two passes: the first pass exercises almost all prediction mode and partitioning options, but uses a fast method to evaluate the energy of the potential residuals. Then, the partitions are fixed, and the second pass accurately evaluates the cost of a subset of the selected best modes, transformations, or other tools.

[0073] Furthermore, heuristics may be used to avoid a completely exhaustive search over all possible combinations of CU partitions, coding modes, prediction modes, and transform indices. As an example, in the case where evaluation of the parent CU and / or the first spatial partitioning mode results in a quantization residual equal to zero (e.g., skip mode or CBF flag equal to 0), the spatial partitioning of a given CU into several sub-CUs will not be evaluated.

[0074] Heuristics can be used to avoid unlikely partitions, for example, by considering partitions of previously encoded neighboring regions.

[0075] The method proposed in this paper enables faster partition selection while keeping the coding efficiency as high as possible.

[0076] Existing fast encoding methods involving CNN-based methods for RDO acceleration have several limitations. In this method, within the framework of HEVC, a convolutional network is used to decide whether to split a given block.

[0077] However, the output of the CNN is a tuple of values ​​that gives information about the probability of whether to split the current block into 4 smaller quadtree blocks. The embodiments described herein enable multiple types of splits to be considered: for example, quadtree, binary tree, as detailed below.

[0078] And, all blocks from 8x8 to 32x32 are first sampled into 8x8 matrices by averaging to provide 8x8 matrices as input to the CNN. Although this clearly shows the advantage in terms of implementation, i.e., unique CNN for all block sizes, it reduces the performance in terms of efficiency in distinguishing the boundaries of each level of the partition tree.

[0079] One idea of ​​the proposed embodiment is to use texture-based segmentation prediction to improve rate-distortion optimization on the encoder side of existing and future codecs. Specifically, this technique allows the system to compute better decisions on partitioning an image in terms of accuracy and speed.

[0080] The described embodiment is Fig. 9 A portion of the frame is shown. Fig. 9 The overall processing on the encoder side is presented. Additional inventions deal with the CNN (820) structure itself and the interaction with the encoder and heuristics (870).

[0081] The described embodiments focus on a way to use the output vector of a CNN to make decisions about which segmentation type to consider for each block by the RDO, without having to consider heuristics and speedups that already exist in the encoder.

[0082] It takes as input a vector of boundary probabilities (830) and outputs a decision (true; false) (860) for each segmentation type considered for rate-distortion optimization (RDO).

[0083] Also note that for the described embodiments, the output of an example CNN is given, but any boundary prediction (texture / object analysis) method can be used to perform segmentation prediction and decision making.

[0084] The described embodiments are intended to guide encoder decisions and can therefore be applied to encoders and improve encoding decisions. Fig.10 They enable partitioning of the coding tree units while avoiding exhaustively computing the rate and distortion of all candidate block partitions, which means applying all steps for each candidate block partition.

[0085] The description of the present solution is based on an exemplary version of reference software designed by the Joint Video Exploration Team (JVET) to design the next MPEG / ITU standard for video coding, currently known as H.266.

[0086] In this framework, like in HEVC, the image is divided into non-overlapping coding tree units (CTUs). The CTU then corresponds to the largest block that can be considered for coding a given area. In the current version, the CTU is of size 256X256 pixels.

[0087] Then, when the texture is difficult to predict and encode on the surface of the entire CTU, smaller blocks can be considered. This allows different prediction / transform / coding tools to be chosen per block that are suitable for the texture. This comes at the cost of transmitting more syntax elements to describe those choices and block shapes. Therefore, rate-distortion optimization is performed to select the best mode and the best partitioning for a given quality and / or bitrate target. A balance needs to be found between the cost of the residual after prediction and transformation and the cost of the syntax elements describing the coding tools. The larger the block on the detail area, the larger the energy of the residual. On the other hand, the smaller the block, the higher the syntax element cost.

[0088] In HEVC, the structure of block partitions follows a quadtree. Therefore, for each tree depth, the encoder can decide whether to encode the current block directly or divide it into four sub-blocks. This is typically done by evaluating the cost of encoding the block at its current size and comparing it to the sum of the 4 costs of the sub-blocks. If, for a given quality, it costs fewer bits to transmit the syntax of a smaller block, the encoder will continue to evaluate the next depth level in the coding tree. This requires trying different prediction modes and transforms of the standard and other tools for all block sizes until the encoder reaches a block size that minimizes the cost, so the complexity of the encoder is high compared to the decoder.

[0089] The coding of the considered CTU as a 256x256 block is compared with the coding of a CTU spatially divided into two or more smaller blocks. For this purpose, a depth-first recursive rate-distortion optimization process is performed. It leads to a representation of the CTU in the form of a so-called coding tree. A typical coding tree obtained with the reference model is Figure 4 Shown in.

[0090] The leaves of the coding tree correspond to CUs (coding units, also referred to as blocks in this document) that are not divided into smaller CUs. They are encoded by prediction, residual transform, quantization and entropy coding. The overall rate-distortion optimization includes the RD optimal selection of coding parameters associated with non-partitioned CUs.

[0091] Moreover, each node of the coding tree represents a potential CU that is spatially divided into smaller sub-CUs according to a given partitioning mode. The partitioning mode that results in the minimum rate-distortion cost is selected among all the partitioning mode candidates. Some syntax elements indicate the partitioning mode that results in each corresponding CU. During the overall rate-distortion minimization process, the bits used to signal the partitioning mode of each CU are taken into account when calculating the rate cost.

[0092] For HEVC, a given CU may have two partitioning modes: no partitioning or quadtree partitioning.

[0093] In the case of the H.266 encoder, the complexity is even higher for a number of reasons:

[0094] -Block size range up to 256x256

[0095] - Consider different block shapes: square and rectangular blocks

[0096] -More forecasting tools

[0097] - More transformations

[0098] The following paragraphs describe exemplary CNN-based structures to illustrate the types of inputs used by the described embodiments.

[0099] In contrast to existing solutions, the input to the CNN is a block or patch of pixels that corresponds to the entire CTU plus the causal L-shaped pixels adjacent to the CTU, such as Fig.11 As shown. For example, with a CTU of size 64x64, the input patch is of size (64+N)x(64+N), where N is the size of the causal boundary. In practice, we choose to take N=1. The reason for adding the boundary is that the encoder itself uses this causal boundary to perform prediction of the block.

[0100] In the first embodiment, the causal boundary comes from the original frame. In this case, one advantage is that the CNN processing of each CTU can be done in parallel because it only uses the original frame.

[0101] In a second embodiment, the causal boundary comes from the decoded neighborhood. In this case, an advantage is that the CNN uses the same input as the encoder for prediction and is therefore more accurate. However, in this embodiment, the processing of each CTU cannot be done in parallel because it uses decoded frames.

[0102] In HEVC or H266, the partitioning of the top and left neighbors has some impact on the coding cost of the partitioning of the current CTU. For example, in HEVC, the partition flag at each level of the tree is encoded based on context entropy that depends on the partition depth of the neighborhood.

[0103] To take this information into account in CNN, auxiliary channels can be added (the main channels of the input are texture and causal boundaries). Fig.12 An example of using boundary information as an auxiliary input channel is shown.

[0104] The output of the CNN corresponds to a vector where each value represents a possible boundary in a given CTU. In H.266, the minimum block size is 4x4, and only pixel boundaries on a 4x4 grid can be considered for block segmentation. Therefore, for each potential boundary, the CNN outputs a probability for it to correspond to an actual boundary between two blocks. For example, for the case where the CTU size is 64x64, a vector of size 480x1 is considered. Fig.13Depicting an exemplary mapping between potential 4 pixel long boundaries (310) in a CTU and their corresponding values ​​in an output vector (320). Each boundary, such as boundary #0 (330), is assigned an index that may be in a predetermined order. In the example shown, the CTU is of size 64x64, which results in a vector of 480 values. Evaluating different boundary probabilities (i.e., the size and shape of the block) at each tree depth will help decide whether to split the block, and in the event of a split, which split should be performed.

[0105] The described embodiment takes as input a vector of boundary probabilities. In this example, the vector is calculated using a CNN structure. However, the following embodiments can use any information that gives the probability of segmentation at each possible segmentation position.

[0106] At each tree depth, the encoder needs to decide whether to split, which type of split, or not to split.

[0107] Just a reminder, Fig. 9 This part of the framework described in Fig. 9 It takes the boundary probability vector 830 as input and outputs a decision (true; false) for each segmentation type to be considered by the rate-distortion optimization (RDO) 860.

[0108] Fig.14 An exemplary real partition is shown on the left, which is the result of encoding using the H.266 candidate encoder, and the corresponding probabilities depicted in grayscale. The whiter the edge, the higher the probability of partitioning at each location. Note that this corresponds to the representation of mapping the probabilities from the vector to the corresponding boundary locations in the CTU.

[0109] In order to obtain the probability of boundaries, it is proposed to consider all boundary locations involved in each segmentation. Fig.15 An exemplary CTU is depicted where we consider a partition along L, and the corresponding index in the vector that will be considered for computing the probability of the partition.

[0110] Then, the probability corresponds to

[0111]

[0112] Where L represents the total boundary of a given segmentation and V represents the vector of probabilities. Note that L can contain a "cross".

[0113] The described embodiment proposes to distinguish between three main options:

[0114] - is boundary: the probability is high enough to predict the boundary

[0115] -Not a boundary: The probability that this boundary does not exist is high enough that it cannot be considered.

[0116] -Uncertainty: Probability does not allow conclusions to be drawn.

[0117] In another embodiment, the probability of segmentation is calculated more conservatively. In the above formula, the probability of segmentation is the average probability given the boundary. However, to be conservative, we can redefine the probability as the minimum of the averages on each side of the boundary. For example, in Fig.21 From the vector of probabilities, the averages are taken on each side of the horizontal boundary (0 and 1 in the figure): L0 = (a0 + b0 + c0) / 3 and L1 = (a1 + b1 + c1) / 3.

[0118] Then the probability of having a horizontal boundary is

[0119] p(L)=min(p(L0),p(L1))(Equation p2)

[0120] With this approach, it is ensured that a binary split is indeed required (and not just a half split from another vertical then horizontal split combination). In particular, for the quadtree, the probability of each boundary (horizontal and vertical) is redefined in this way.

[0121] For a given segmentation, we consider p(L) and generate segmentation predictions for each segmentation type, which in the exemplary case of JEM can be expressed as:

[0122] splitPred[s]∈{false,undefined,true},s∈{NO_SPLIT,QT_SPLT,HOR,VER}

[0123] Therefore, each segmentation type requires two thresholds p(L): a decision threshold Th for the segmentation S , another decision for "no split" Th NS If any of these conditions are not met, then bounds prediction remains undefined and classic RDO is executed.

[0124] To better understand the relationship between probability and threshold, Fig.16 On the y-axis is shown the proportion of correct boundaries, depending on the "boundary present" threshold Th set for 3 different levels in the partition tree S (x-axis) position. Fig.17 Shows about Th NS The corresponding "no border / no split" statistics.

[0125] Since the boundaries considered are the horizontal and vertical middle boundaries in JEM (quadtree / binarytree), these graphs show the average ratio of these two boundaries.

[0126] For example, for a segmentation threshold of 0.8 with probability p at depth 1, about 90% of the samples are above it and are then well predicted. For a no segmentation threshold of 0.3 at depth 1, about 80% of the samples are well predicted.

[0127] Boundary determination step ( Fig. 9 850 in takes as input the boundary predictions detailed in the previous section. It outputs a segmentation decision for each segmentation type to be performed by RDO: yes or no, depending on the encoder constraints.

[0128] In the case of HEVC, a quadtree drives the partitioning, and the unique probability of splitting at each level corresponds to a division into 4 square blocks of equal area.

[0129] To evaluate the probability of a partition, the proposed method then considers all indices of the vectors that form a "cross" that divides the CU into four.

[0130] Fig.18 The entire process of compressing a CTU at the encoder side in the exemplary case of HEVC is depicted. In this simple example, only one partitioning type is considered.

[0131] It takes a CTU (410) as input and outputs compressed data (460). First, the function 415 that returns the next CU is called, which returns the parameters of the current CU to be processed. In the first call, if the entire CTU can be encoded as a CU, then it will return a CU with the scale of the CTU. In the exemplary case of HEVC, in an intra-coded picture, the largest CU is 32×32, then the first CU to be processed is the upper left 32×32 block in the 64×64 CTU. This function should also take into account the situation at the picture boundary.

[0132] The first step (420) includes evaluating whether the CU can be split. For example, a CU of the smallest size cannot be further split.

[0133] The description of the boundary prediction above ( Fig. 9 The segmentation probability calculation (425) described in step 840 in FIG. 1 takes the current CU as input and outputs a score p representing the probability of segmentation. s In the exemplary case of HEVC, the only possible type of partitioning corresponds to quadtree partitioning.

[0134] If p s Better than for segmentation determination Th S If the predetermined threshold is greater than , then module 465 automatically splits the CU without checking its cost (430). If not, then its cost is checked by module 445.

[0135] In this case, p s With another threshold Th NS is compared to decide whether to encode it directly without checking the probability of segmentation. s <Th NS , then the CU is not directly partitioned and encoded by module 450.

[0136] If Th NS <p s <Th S , then the costs of the sub-CUs are collected to be compared with the cost of the current CU, as is classically done in classical encoders. A decision is made (450), and the CU is either encoded 455 or split 465, and the next CU is processed.

[0137] It can be noted that when p s >Th S When p , there is no need to perform RD cost checks at the current level, which is a time gain at the encoder. s >Th NS The same is true when the sub-CU check can be omitted, so this method is faster compared to classic RDO.

[0138] When all CUs of the current CTU are encoded, compressed data is output (470).

[0139] The following paragraphs describe some practical options that depend on the codec framework.

[0140] In future video coding, several types of partitioning will appear. In the case of the reference software JEM designed by JVET, a hybrid structure combining quadtree and binary tree has been adopted.

[0141] For each CU, three types of partitioning can occur:

[0142] -Quarter: Four square blocks

[0143] - Binary Horizontal: 2 equal surface blocks with horizontal borders. Depending on the shape of the parents, they can be square or rectangular.

[0144] - Binary Perpendicular: 2 equal surface blocks with perpendicular borders. Depending on the shape of the parents, they can be square or rectangular.

[0145] Table 1 details the proposed strategies for segmentation probability in JEM, depending on the boundary prediction p s (or p(L)), which is related to the segmentation threshold Th S and the non-segmentation threshold Th NSFor comparison, in JEM, quadtree splitting as well as vertical and horizontal binary splitting are considered. At the top of the tree, a quadtree splitting can be selected. Once a binary splitting is selected, quadtree splitting cannot be performed at the next level. This table takes this constraint of the codec into account in the third column. Whether quadtree splitting is available or not, for the same p s , the possible choices of segmentation may be different. The last column provides the output of the system described in this patent, i.e., for each segmentation type {NS (no segmentation), V (vertical), H (horizontal), Q (quadtree)}, a Boolean value {true, false} checking the rate-distortion optimization.

[0146] At the end of the processing described in these embodiments, the RDO obtains as input a set or table of Boolean values ​​that informs whether each segmentation type is included in the rate-distortion check.

[0147] Table 1: Segmentation strategies in JEM, depending on boundary prediction, threshold, and type of available segmentation

[0148]

[0149]

[0150] exist Figures 22 to 25 In , we give several examples of decision graphs for split decisions, using the previously defined probabilities and two thresholds: t m is the low threshold, determining whether there is no boundary, t M is a high threshold, which determines whether there is a boundary. x represents the probability of having an edge at x (x is v (vertical), h (horizontal), t (top), b (bottom), l (left), r (right)), calculated using equation p2. The value S x represents the probability calculated using equation p1.

[0151] exist Fig. 22 In , the decision graph for the no-split decision is shown. No-split is tested when there are no vertical and horizontal boundaries. In addition, if asymmetric splits are available, the absence of 4 splits (horizontal top, horizontal bottom, vertical left, and vertical right) is also tested. Fig.23 In , the quadtree split is tested in the absence of vertical and horizontal boundaries. Note that the presence / absence of a boundary is calculated using equation p2, which is a conservative probability for the boundary, using the minimum probability on each side of the boundary.

[0152] exist Fig.24In Figure 1, an example of testing a binary horizontal split (the same logic can be applied to a binary vertical by swapping h and v). If a quadtree is possible at this topological level, the split is tested only if a quadtree split is not possible, otherwise the split is tested if no horizontal boundary exists.

[0153] exist Fig.25 In , an example of testing an asymmetric TOP split (the same logic applies to horizontal bottom, vertical left, or vertical right). If there is no top border, then the split is tested.

[0154] Other codec candidates for H.266 suggest adding other types of partitioning, namely, asymmetric binary trees and ternary trees. Asymmetric binary trees provide the possibility of partitioning a square or rectangular block into 2 sub-blocks with different surfaces. The partitioning can be vertical or horizontal. The sub-block size should conform to the transform size, typically 1 / 4 and 3 / 4 of the parent block size. Ternary trees partition blocks into 3 sub-blocks. The partitioning can be vertical or horizontal. The sub-block size should also conform to the transform size, typically 1 / 4, 1 / 2 and 1 / 4 of the parent block size. As with JEM, the proposed embodiment can be adapted to take into account the partitioning type and the output decision for each partitioning type. Figure 8 The types of segmentation considered by the described embodiments are shown.

[0155] So far, the segmentation prediction and decision at each level of the tree has been described in detail. This section gives how to manage the processing of the entire tree. If we take the example of JEM within a frame, then consider 5 levels.

[0156] Then you can set the threshold depending on:

[0157] -Levels in the tree

[0158] - Type of partitioning (quadtree, binary, asymmetric, etc.)

[0159] - Block size / shape

[0160] -Quantization Parameter (QP)

[0161] - Color Components

[0162] - Strip type

[0163] - Temporal levels in random access and low latency modes.

[0164] An example of setting thresholds by tree level is given by:

[0165] t m [n]=max(0,α(n+1) β )

[0166] tM [n]=min(1,max(t m [n]+∈,α′(n+1) β ′))

[0167] Here, β, α, β, and ε are parameters that set the trade-off between speedup and performance. Exemplary values ​​are α=0.1, β=0.5, α'=0.9, β'=0, and ε=0.1. In the equations, n represents the segmentation level. For example, the first level of the segmentation tree is 0. After each segmentation, the tree level increases by 1. Alternatively, for quadtree levels, the level increases by 2, and for binary segmentation, the level increases by 1. In the first equation, the low threshold (decision boundary does not exist) follows an increasing function of the tree level, which means that the absence of the boundary increases with the depth of the tree. In the second equation, we note that the high threshold (decision boundary existence) is always higher than the low threshold. In this case, the minimum uncertainty is controlled by ε.

[0168] As for the thresholds it depends on the size / shape of the blocks. Therefore they can be stored in a 2D table or as a set:

[0169] 4 8 16 32 64 4 <![CDATA[T 4x4 ]]> … … … <![CDATA[T 4x64 ]]> 8 … … … … … 16 … … … … … 32 … … … … … 64 <![CDATA[T 64x4 ]]> … … … <![CDATA[T 64x64 ]]>

[0170] To simplify the scheme and to be consistent with experiments, the thresholds can be symmetric for horizontal / vertical blocks, i.e., T ixj =T jxi , thus reducing the number of thresholds to be optimized.

[0171] One advantage of the proposed methods is that they can speed up the decision time of the encoder to determine the partitioning of a block.

[0172] The foregoing embodiments have been described with respect to an encoder or encoding operation.Decoder The segmentation generated at the encoder will be simply explained using the described embodiments in the same way as segmentation generated using RDO processing or any other type of segmentation embodiments.

[0173] Fig.19 One embodiment of a method 1900 for encoding a block of video data is shown. The method begins at start block 1901 and proceeds to block 1910 for generating a set of partition probabilities using a texture-based analysis of the block of image data based on partition boundary probabilities. Control then proceeds from block 1910 to block 1920 to partition the block of image data into two or more smaller blocks using the set of partition probabilities. Control then proceeds from block 1920 to block 1930 to encode at least one smaller block.

[0174] Fig. 20An embodiment of an apparatus 2000 for encoding or decoding a block of video data is shown. The apparatus comprises a processor 2010 having input and output ports and being in signal communication with a memory 2020 also having input and output ports. The apparatus may perform the aforementioned method embodiments or any variation thereof.

[0175] The functions of the various elements shown in the drawings may be provided using dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of separate processors (some of which may be shared). In addition, explicit use of the term "processor" or "controller" should not be interpreted as referring exclusively to hardware capable of executing software, but may implicitly include, but not be limited to, digital signal processor ("DSP") hardware, read-only memory ("ROM") for storing software, random access memory ("RAM"), and non-volatile memory.

[0176] Other conventional and / or custom hardware may also be included. Similarly, any switches shown in the figures are conceptual only. Their functions may be implemented through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the specific technique being selectable by the implementer as more specifically understood from the context.

[0177] This description exemplifies the present concept. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the present concept and are included within its scope.

[0178] All examples and conditional language recited herein are intended for teaching purposes to aid the reader in understanding the present principles and concepts contributed by the inventor(s) to further the art, and are to be construed as not being limited to such specifically recited examples and conditions.

[0179] In addition, all statements describing the principles, aspects and embodiments of the present principles and specific examples thereof herein are intended to cover their structural and functional equivalents. In addition, it is intended that such equivalents include both currently known equivalents and equivalents developed in the future, that is, any elements developed that perform the same function, regardless of structure.

[0180] Thus, for example, those skilled in the art will recognize that the block diagrams presented herein represent conceptual diagrams of illustrative circuitry embodying the present principles. Similarly, it will be recognized that any flow charts, flow diagrams, state transition diagrams, pseudo-code and the like represent various processes that may be substantially represented in a computer-readable medium and thus executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0181] In the claims herein, any means expressed as a means for performing a particular function is intended to encompass any manner of performing that function, including, for example, a) a combination of circuit elements that perform that function or b) any form of software, thus including firmware, microcode, etc., combined with appropriate circuitry for executing that software to perform the function. The present principles defined by such claims reside in the fact that the functions provided by the various recited means are combined together in the manner required by the claims. Any means that can provide those functions are therefore considered equivalent to those shown herein.

[0182] References in the specification to "one embodiment" or "an embodiment" of the present principles and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the present principles. Therefore, the appearance of the phrase "in one embodiment" or "in an embodiment" and any other variations appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

Claims

1. A method comprising: determining a set of partition probabilities associated with the image data block based at least on the neural network; partitioning the image data block into two or more smaller prediction units based at least on the determined set of partition probabilities; as well as The block of image data is encoded based at least on the partition.

2. The method according to claim 1, wherein: The neural network is configured to determine the set of partition probabilities further based on the image data block and one or more causal pixels adjacent to the image data block.

3. The method according to claim 1, wherein: The neural network is configured to determine the set of partition probabilities further based on information indicating how neighboring blocks of the image data block are partitioned.

4. The method according to claim 1, wherein: Determining a set of partition probabilities associated with the block of image data includes determining respective probability scores associated with the set of partition probabilities based on the neural network.

5. The method according to claim 4, wherein: The set of partition probabilities is determined by comparing the respective probability scores associated with the set of partition probabilities to a predetermined threshold.

6. The method according to claim 5, wherein: The predetermined threshold depends on the quantization parameter.

7. The method according to claim 5, wherein: The predetermined threshold depends on the tree depth.

8. An apparatus for encoding an image data block, comprising: Memory, and The processor is configured as: determining a set of partition probabilities associated with the image data block based at least on a neural network; partitioning the image data block into two or more smaller prediction units based at least on the determined set of partition probabilities; as well as The block of image data is encoded based at least on the partition.

9. The device according to claim 8, wherein: The neural network is configured to determine the set of partition probabilities further based on the image data block and one or more causal pixels adjacent to the image data block.

10. The device according to claim 8, wherein: The neural network is configured to determine the set of partition probabilities further based on information indicating how neighboring blocks of the image data block are partitioned.

11. The device according to claim 8, wherein: The processor being configured to determine a set of partition probabilities associated with the block of image data includes the processor being configured to determine respective probability scores associated with the set of partition probabilities based on the neural network.

12. The device according to claim 11, wherein The set of partition probabilities is determined by comparing the respective probability scores associated with the set of partition probabilities to a predetermined threshold.

13. The device according to claim 12, wherein: The predetermined threshold depends on the quantization parameter.

14. The device according to claim 12, wherein: The predetermined threshold depends on a tree depth, a block size, a block shape or a slice type associated with the image data block.

15. A non-transitory computer readable medium having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 7.

Images