Block partitioning acceleration based on reinforcement learning

Abstract

Methods and apparatus are provided for block segmentation acceleration based on reinforcement learning by a neural network. In one embodiment, the neural network receives input including at least one of neighbor features, parent features, coding unit information, and spatial features to determine block segmentation. In another embodiment, spatial features used by the neural network include a concatenation of two direction gradient histograms are used to determine block segmentation. In another embodiment, information is signaled from an encoder to a decoder for a neural network determination of block segmentation.

Description

[0001] Cross-reference to related applications This application claims the benefit of European serial number 23306885.7, filed on 31 October 2023, which is incorporated herein by reference in its entirety. Technical Field

[0002] At least one of these embodiments generally relates to a method or apparatus for video encoding or decoding, compression or decompression. Background Technology

[0003] To achieve high compression efficiency, image and video coding schemes typically employ prediction (including motion vector prediction) and transform to utilize spatial and temporal redundancy in the video content. Generally, intra-frame or inter-frame prediction is used to leverage intra-frame or inter-frame correlations, and then the differences between the original and predicted images (often denoted as prediction error or prediction residual) are transformed, quantized, and entropy-coded. To reconstruct the video, the compressed data is decoded through the inverse process corresponding to entropy coding, quantization, transform, and prediction. Summary of the Invention

[0004] At least one of these embodiments generally relates to a method or apparatus for video encoding or decoding, and more particularly, to a method or apparatus for using reinforcement learning (RL) to predict segmentation patterns in a hybrid block-based video codec.

[0005] According to a first aspect, a method is provided. The method includes the steps of: performing segmentation prediction of video blocks using a neural network; segmenting the video blocks based on the segmentation prediction; and encoding the segmented video blocks.

[0006] According to the second aspect, another method is provided. This method includes the steps of: performing segmentation prediction of video blocks using a neural network; segmenting the video blocks based on the segmentation prediction; and decoding the segmented video blocks.

[0007] According to another aspect, an apparatus is provided. The apparatus includes a processor and a memory. The processor can be configured to operate on digital video data according to the aforementioned method.

[0008] According to another aspect, an apparatus is provided. The apparatus includes a processor and a memory. The processor can be configured to encode video blocks or decode video data by performing any of the methods described above.

[0009] According to another general aspect of at least one embodiment, an apparatus is provided, including means according to any of the decoding embodiments; and at least one of the following: (i) an antenna configured to receive a signal including a video block, (ii) a band limiter configured to limit the received signal to a band including the video block, or (iii) a display configured to display an output representing the video block.

[0010] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that contains data content generated according to any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, a signal comprising video data generated according to any of the described encoding embodiments or variations is provided.

[0012] According to another general aspect of at least one embodiment, video data or bitstreams are formatted to include data content generated according to any of the described encoding embodiments or variations.

[0013] According to another general aspect of at least one embodiment, a computer program product including instructions is provided that, when the program is executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0014] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments to be read in conjunction with the accompanying drawings. Attached Figure Description

[0015] Figure 1 The illustration shows an example of the segmentation of the image's luminance channel and two chrominance channels.

[0016] Figure 2 The illustration shows an example of a split tree starting from a 64×64 root block in HEVC.

[0017] Figure 3 The illustration shows the example rate distortion optimization process in a loop to find the splitting or prediction pattern of the coding unit.

[0018] Figure 4 The illustration shows an example of the top / left adjacent block of the coding unit.

[0019] Figure 5 The illustration shows the adjacent blocks of the current coding unit in the VVC of the fourth sub-implementation according to the first variant embodiment.

[0020] Figure 6The illustration shows the adjacent blocks of the current coding unit in the VVC of an alternative fourth sub-implementation according to the first variant embodiment.

[0021] Figure 7 The illustration shows the adjacent blocks of the current coding unit in the VVC of the fifth sub-implementation according to the first variant embodiment.

[0022] Figure 8 The illustration shows the variant neighborhood block considerations: (a) considering only the middle top / left adjacent blocks, and (b) considering all top and left adjacent blocks as neighborhood features.

[0023] Figure 9 The diagram shows the adjacent blocks of the current coding unit.

[0024] Figure 10 The illustration shows texture information being scaled down / up to various sizes according to one embodiment.

[0025] Figure 11 The illustration shows texture information being scaled down / up to various sizes according to another embodiment.

[0026] Figure 12 The diagram illustrates the directional gradient histogram for each subdivision of the current coding unit and its causal boundary.

[0027] Figure 13 The illustration shows an example of training database collection during encoding.

[0028] Figure 14 The illustration shows an example training process in which the agent interacts with the environment and loops through each round.

[0029] Figure 15 The illustration shows an example of a flag that controls model prediction within the encoder.

[0030] Figure 16 An example of the prediction process is illustrated.

[0031] Figure 17 An embodiment of the first method under the described aspects is illustrated.

[0032] Figure 18 An embodiment of the second method under the described aspects is illustrated.

[0033] Figure 19 An embodiment of the apparatus under the described aspects is illustrated.

[0034] Figure 20 The diagram illustrates a standard, universal video compression scheme.

[0035] Figure 21 The diagram illustrates a standard, universal video decompression scheme.

[0036] Figure 22 The illustration depicts a processor-based system for encoding / decoding under various aspects of the general description. Detailed Implementation

[0037] The embodiments described herein are in the field of video compression and generally relate to video compression as well as video encoding and decoding, and more specifically, to a method or apparatus for using reinforcement learning (RL) to predict segmentation patterns in a hybrid block-based video codec.

[0038] To achieve high compression efficiency, image and video coding schemes typically employ block-based prediction, including motion vector prediction, and perform transforms to utilize spatial and temporal redundancy in the video content. Intra-frame or inter-frame prediction is usually used to leverage intra-frame or inter-frame correlations, and then the differences between the original and predicted blocks (often denoted as prediction error or prediction residuals) are transformed, quantized, and entropy-coded. To reconstruct the video, the compressed data is decoded through the inverse process corresponding to entropy coding, quantization, transform, and prediction.

[0039] In the HEVC (High Efficiency Video Coding) video compression standard, motion-compensated time prediction is used to take advantage of the redundancy between successive images in a video.

[0040] Therefore, motion vectors are associated with each prediction unit (PU). Each CTU (coding tree unit) is represented by a coding tree in the compressed domain. This is a quadtree partition of the CTU, where each leaf is called a coding unit (CU).

[0041] Then, each CU is given some intra-frame or inter-frame prediction parameters (prediction information). For this purpose, it is spatially divided into one or more prediction units (PUs), and each PU is assigned some prediction information. Intra-frame or inter-frame coding modes are assigned at the CU level.

[0042] The described embodiments are situated within the context of video compression. Specifically, the embodiments focus on utilizing reinforcement learning (RL) to predict segmentation patterns in hybrid block-based video codecs such as H.265 / HEVC, H.266 / VVC, Enhanced Compression Model (ECM), or Basic Video Coding (EVC). In this description, the term "segmentation pattern" refers to different ways in which the current block can be divided into sub-blocks. For example, in H.266, there are six segmentation patterns: Quadtree (QT) splitting divides the current block into four equal-sized non-overlapping sub-blocks; Binary Tree splitting divides the current block horizontally (BTH) or vertically (BTV) into two equal-sized non-overlapping sub-blocks; Tritree splitting divides the current block into three non-overlapping sub-blocks, one of which is twice the size of the other two in the horizontal (TTH) or vertical (TTV) direction; and No Split (NS) does not split the current block. In this description, for a given frame, the term "channel splitting" refers to the eventual decomposition of the channel into blocks, which are obtained by hierarchically dividing the channel into sub-blocks via a splitting mode selected during the encoding process. As illustrated, Figure 1 An example is depicted where the luminance channel and two chrominance channels of a given frame are split via H.266 version VTM-10.0.

[0043] The encoding process aims to find the optimal split pattern for each coded block (CB) through rate-distortion optimization (RDO). By trying all possibilities, the encoder selects the split that results in the minimum cost. For a given original CB and a given split pattern, the cost is the weighted sum of the distortion between the original CB and the distortion of the reconstructed original CB caused by encoding via the split pattern, plus the bit rate used to encode the original CB via the split pattern. Therefore, this process significantly increases the overall encoding time.

[0044] However, the segmentation process can be accelerated by integrating neural networks (NNs) into the block-based video codec to predict the appropriate splitting pattern for each block. For example, NN-based splitting prediction can work as follows: During the encoding step, for each luma / chroma block, instead of having the encoder find the appropriate splitting pattern, the trained NN directly infers the appropriate splitting pattern to be applied to that block from information about neighboring blocks, the spatial representation of the block, and other relevant features.

[0045] The goal is to improve compression efficiency, that is, to reduce the bit rate while maintaining quality, or equivalently, to improve quality while maintaining the bit rate.

[0046] In the described embodiments, a block segmentation process was investigated, wherein the segmentation pattern can be predicted via a reinforcement learning agent model independent of the coded block size, prediction pattern (such as intra / inter frame), and channel components.

[0047] In video codecs such as HEVC, VVC, and others, several techniques have been proposed to enhance the block segmentation process, employing deep learning and even deep reinforcement learning. These methods aim to address the challenge of accelerating the segmentation task while minimizing encoder complexity. State-of-the-art methods have emerged to address this problem efficiently and effectively, focusing on strategies that accelerate the segmentation process while striving to maintain high quality. These advances in deep learning and reinforcement learning have paved the way for significant improvements in the performance and computational efficiency of video codecs.

[0048] Reinforcement learning-based depth-per-segmentation pattern prediction model In HEVC, the first approach uses deep reinforcement learning to address the challenge of code block splitting. This approach involves training three deep Q-learning network (DQN) models, one for each depth (depths 0, 1, and 2 only), as follows: Figure 2 As shown, the compression benefits associated with splitting the coded blocks are anticipated. The encoder then uses the model's output to guide the decision-making process.

[0049] The training of the three Q-networks is performed sequentially, starting with a 16×16 block of the Q-network at a depth of 2 and progressing to a 64×64 encoding unit (CU) of the Q-network at a depth of 0. Each model takes the current CU block and its corresponding quantization parameter (QP) as input and uses the Q-network at the next depth (not considering depth 2) that has already been trained to compute the cumulative reward for that action.

[0050] Furthermore, the reward assignment for non-split actions always results in a zero reward, leading to a terminating state in subsequent states. In HEVC, quad-split is the only supported mode, and as a result, each element in the Q-network only requires a single output to provide an estimate of the expected rate-distortion cost reduction for the split action. If the estimate is positive, the split action is taken; otherwise, the current CU remains unchanged.

[0051] To reduce the coding complexity in the VVC standard, the second approach introduces a fast method for coding unit splitting based on deep reinforcement learning. This method processes only 32×32 CUs, where the CU splitting scenario at a specific node represents the state, the selection of the splitting pattern represents the action, the change in rate-distortion cost acts as an immediate reward or penalty, and the encoder acts as an agent making sequential coding decisions. The model receives a 32×32 CU size and its corresponding QP as input, and outputs a Q-value for each of six possible splits (NS, QT, BTH, BTV, TTH, TTV) (or the accumulated reward if the action is selected), then selects the split with the largest Q-value for the current input.

[0052] The methods mentioned do demonstrate their ability to accelerate encoding and enhance the segmentation process. However, a noteworthy constraint is that a new model is required to handle each block size. In other words, these methods are heavily dependent on the specific block size being considered.

[0053] Hierarchical Deep Learning Methods for Segmentation Pattern Prediction The problem of accelerating segmentation is also addressed in VP9, ​​where the third method proposes a hierarchical fully convolutional network (H-FCN) and uses four matrices (M0, M1, M2, M3) corresponding to the four depths of the VP9 segmentation tree to represent the segmentation tree. Each matrix element represents the type of merging of groups of four blocks at their corresponding position and level within a 64×64 block. "Completely merging four blocks" means there is no splitting, no merging corresponds to a quad split, and horizontal merging and vertical merging correspond to the horizontal and vertical splitting of the four blocks of interest, respectively.

[0054] For example, M0 represents the merging of four 4×4 blocks, M1 represents the merging of four non-overlapping groups of 8×8 blocks, M2 represents the merging of four non-overlapping groups of 16×16 blocks, and M3 represents the merging of four non-overlapping groups of 32×32 blocks.

[0055] H-FCN has a main branch and four output branches derived from the main branch. The model takes a 64×64 block as input and each branch outputs a merged selection of each of the four depths.

[0056] The fourth approach proposes a multi-stage exit CNN model with an early exit mechanism to accelerate the coding process in intra-mode VVC configurations. In this approach, the process of splitting a 128×128 coding tree unit (CTU) into a 64×64 coding unit (CU) can be designated as Stage 1. Similarly, the subsequent subdivision of these 64×64 CUs into 32×32 CUs can be considered Stage 2, and so on. In the default settings of intra-mode Universal Video Coding (VVC), all 128×128 CTUs are forced to undergo splitting into 64×64 CUs, resulting in exclusive support for quadtree mode in Stage 1. For Stage 2, both non-split and quadtree modes are applicable. Subsequent stages offer up to six mode possibilities, including non-split, quadtree, horizontal binary, vertical binary, horizontal ternary, and vertical ternary. For all these modes, it is ensured that the minimum width or height of the CU remains at 4.

[0057] In addition to computational complexity, these methods suffer from consistency issues, and predictions may be contradictory due to limitations in capturing global or long-range contextual information, which may affect the understanding of relationships between sub-blocks within the input CU. Furthermore, they are size-independent methods.

[0058] The encoding process is as follows: Figure 3 The combination problem is shown in the figure. In VVC, determining the optimal CU partition involves an exhaustive rate-distortion optimization (RDO) search, in which the RD cost of all potential CUs is evaluated and the combination with the lowest RD cost is selected. For each tested mode, the cost J is calculated by adding the distortion D between the raw and decoded pixels to the necessary bit rate R and balancing these two terms by λ, as shown in equation (1).

[0059] The splitting process is the head of the combinatorial tree, where it recursively loops on the CU to determine the optimal split, which is time-consuming. This process can be accelerated by inserting a NN.

[0060] As discussed in previous chapters, this problem exists and has been addressed by researchers using various approaches. However, all methods in the literature are size-dependent, and each of the proposed models takes inputs of a different size. In this case, CUs with similar textures but different sizes cannot be handled by a uniform robust neural network. Alternatively, in these embodiments, the solution can cover all CU sizes by representing all CUs by considering the features that the encoder takes into account when encoding a given CU.

[0061] Subsequently, a general, size-independent reinforcement learning method was proposed, in which all CUs are processed by the proposed framework. The key to this method is creating representations for all CUs.

[0062] Representation of CU based on feature set Because neural networks take fixed-size inputs, and these embodiments are designed to use a single neural network for CUs of any size, the representation of the CUs fed into the neural network must be independent of the CU size in these embodiments. Therefore, the representation of the CU can be projected into the dimension of a vector V, where the representation of all CUs can be based on several feature sets. The case where the vector V consists of four feature sets {“neighbor features”, “parent features”, “CU information”, and “spatial features”} is described in detail in subsequent sections. V = {Neighbor features, Parent features, CU information, Spatial features}.

[0063] In another variant embodiment, vector V may consist of only a subset of the four feature sets {"neighbor features", "parent features", "CU information", and "spatial features"}. For example, vector V may consist exclusively of "neighbor features" and "parent features". As another example, vector V may consist exclusively of "parent features", "CU information", and "spatial features".

[0064] In another variant embodiment, vector V may include four feature sets {"neighbor features", "parent features", "CU information", and "spatial features"} in addition to other feature sets. For example, let us consider, in a hybrid block-based video codec of interest, "no split" belongs to the set of possible splits for a given CU. Let us also consider that the hybrid block-based video codec includes a neural network (NN) that takes vector V of a given CU to infer the most likely split for that CU, excluding "no split". For a given CU, the NN operates after testing "no split" and before testing any other split. In this case, vector V may include {"neighbor features", "parent features", "CU information", and "spatial features"}, as well as features representing the intra / inter-frame prediction mode selected to predict the current CU when testing "no split". Another example can be derived from this case. Vector V may include {"neighbor features", "parent features", "CU information", and "spatial features"}, features representing the intra / inter-frame prediction mode selected to predict the current CU when testing "no split", and features representing the number of bits required to write the quantization transform coefficients of the current CU into the bitstream when testing "no split".

[0065] Neighborhood features (NF) for segmentation structure understanding During the encoding of a particular coding unit (CU), the encoder can utilize adjacent blocks. These adjacent blocks can correspond to previously processed CUs.

[0066] Features of the top and left adjacent blocks In the first variant embodiment, the top and left adjacent blocks can be considered. For example, Figure 4 Examples of the top and left neighboring blocks of a current W×H CU are depicted. These neighboring blocks may have undergone their own individual encoding processes. Therefore, for each of the top and left neighboring blocks, valuable data such as rate-distortion cost or quadtree depth (the number of quadtree splits required to move from the root block (CTU) to that block in the split tree) can be used. For example, the neighboring features included in the vector V could be the rate-distortion cost of the top neighboring block. Rate distortion cost of the left adjacent block The quadtree depth of the top adjacent block The quadtree depth of the block to the left In this case, NF = .

[0067] Normalization of rate-distortion cost per pixel and through "no splitting" cost In a first sub-implementation of the first variant embodiment, the rate-distortion cost of each adjacent block can be normalized pixel-wise and by the "unsplit" cost of that adjacent block. For example, following Figure 4 The rate-distortion cost of the top adjacent block can be as follows. in W is the rate-distortion cost of the top adjacent block of the current CU, and W / 2 and W / 2 are the width and height of the top adjacent block, respectively. It is the "no-split" cost of the top adjacent block of the current CU.

[0068] Normalization of distortion cost per pixel In a second sub-implementation of the first variant, the rate-distortion cost of each adjacent block can be exclusively normalized per pixel. For example, following Figure 4 The rate-distortion cost of the top adjacent block can be as follows.

[0069] Normalization of rate-distortion costs through "no-split" costs In a third sub-implementation of the first variant, the rate-distortion cost of each adjacent block can be exclusively normalized by the "unsplit" cost of that adjacent block. For example, following Figure 4 The rate-distortion cost of the top adjacent block can be as follows.

[0070] Rate-distortion cost of adjacent blocks as leaves of the current segmentation tree In the fourth sub-implementation of the first variant embodiment, when RDO is run on the current CU, given the current state of the segmentation of the current frame, the top neighbor block can correspond to a "leaf" CU containing the pixel above the pixel at the top left of the current CU. The rate-distortion cost of the top neighbor block can then correspond to its "no-splitting" cost. The left neighbor block can correspond to a "leaf" CU containing the pixel to the left of the pixel at the top left of the current CU. The rate-distortion cost of the left neighbor block can then correspond to its "no-splitting" cost. Note that in the current segmentation state of the current frame, since the top / left neighbor blocks correspond to the leaves of the segmentation tree, their "no-splitting" cost is always the minimum cost in the test segmentation. An example of the fourth sub-implementation of the first variant embodiment is in... Figure 5 Described in the text.

[0071] Note that in this fourth sub-implementation of the first variant embodiment, any alternative criterion can be used to locate the top / left adjacent block of the current CU. For example, the top adjacent block can correspond to a "leaf" CU containing the pixel above the pixel at the center of the first row of the current CU. The left adjacent block can correspond to a "leaf" CU containing the pixel to the left of the pixel at the center of the first column of the current CU. An example of this alternative fourth sub-implementation is shown in... Figure 6 Presented in the middle.

[0072] In the case where the current CU and its neighboring block are generated by the same split, the rate-distortion cost of the neighboring block. In the fifth sub-implementation of the first variant embodiment, such as Figure 7 As shown in (a), if the current CU and the region above it are generated by the same split, then the top adjacent block can correspond to a CU that satisfies conditions (i) and (ii). (i) It is a descendant of index j generated by that split, where The current CU is a descendant of index i generated by this split. (ii) It contains the pixels above the pixel at the top left of the current CU. Figure 7 As illustrated in (b), if the current CU and its left-hand region are generated by the same split, then the left-hand adjacent block can correspond to a CU that meets conditions (i) and (iii). (iii) It contains the pixel to the left of the pixel at the top left of the current CU.

[0073] Note that in the fifth sub-implementation of the first variant embodiment, (ii) and (iii) can be adapted in any way. For example, (ii) can become "it contains the pixel above the pixel at the center of the first row of the current CU." And (iii) can become "it contains the pixel to the left of the pixel at the center of the first column of the current CU."

[0074] In this fifth sub-implementation of the first variant, the rate-distortion cost of the top adjacent block can correspond to the cost of its selected split. The rate-distortion cost of the left adjacent block can correspond to the cost of its selected split.

[0075] Add the MTT depth of each adjacent block to the adjacent features. In the sixth sub-implementation of the first variant, the MTT depth of the top adjacent block and the MTT depth of the left adjacent block can be parts of the adjacent features of the current CU. For example, the adjacent features included in the vector V can be , , , MTT depth of the top adjacent block MTT depth of the block to the left .Then, NF = .

[0076] Features of multiple top adjacent blocks and multiple left adjacent blocks In a second variant embodiment, multiple top adjacent blocks and multiple left adjacent blocks can be considered. For example, Figure 8 Examples are presented involving multiple top and left neighboring blocks of the current W×H CU. These neighboring blocks may have already undergone their own individual encoding procedures. Therefore, for each of these neighboring blocks, valuable data such as rate-distortion cost or quadtree depth can be used.

[0077] exist Figure 8 In (a), the adjacent features included in vector V can be the rate-distortion cost of the middle and top adjacent blocks. Rate distortion cost of adjacent blocks on the left side of the middle The quadtree depth of adjacent blocks at the top and middle The quadtree depth of the middle left adjacent block In this case, NF = .

[0078] exist Figure 8 In (b), the top adjacent block of index 1 can correspond to a "leaf" CU containing the pixel above the pixel at the upper left of the current CU. Then, the rate-distortion cost of the top adjacent block of index 1 can correspond to its "no-split" cost. The top adjacent block of index 4 can correspond to a "leaf" CU containing the pixel above the pixel at the upper right of the current CU. Then, the rate-distortion cost of the top adjacent block of index 4 can correspond to its "no-split" cost. The left adjacent block of index 1 can correspond to a "leaf" CU containing the pixel to the left of the pixel at the upper left of the current CU. Then, the rate-distortion cost of the left adjacent block of index 1 can correspond to its "no-split" cost. The left adjacent block of index 4 can correspond to a "leaf" CU containing the pixel to the left of the pixel at the lower left of the current CU. Then, the rate-distortion cost of the left adjacent block of index 4 can correspond to its "no-split" cost. In this case, NF = .

[0079] Consider the characteristics of the top / left adjacent blocks of the entire row and column of the current CU. In the first sub-implementation of the second variant, the adjacent features included in vector V can be all the top and left blocks, such as... Figure 8 As shown in (b), the rate-distortion cost includes the top adjacent block of index 1. Rate distortion cost of the top adjacent block of index 2 Rate distortion cost of the top adjacent block of index 3 Rate distortion cost of the top adjacent block of index 4 Rate distortion cost of the left adjacent block of index 1 Rate distortion cost of the left adjacent block of index 2 Rate distortion cost of the left adjacent block of index 3 Rate distortion cost of the left adjacent block of index 4 The quadtree depth of the top adjacent block of index 1 The quadtree depth of the top adjacent block of index 2 The quadtree depth of the top adjacent block of index 3 The quadtree depth of the top adjacent block of index 4 The quadtree depth of the left adjacent block of index 1 The quadtree depth of the left adjacent block of index 2 The quadtree depth of the left adjacent block of index 3 And the quadtree depth of the left adjacent block of index 4 In this case, NF = .

[0080] Features of adjacent blocks other than the top / left adjacent blocks In previous sections, it was assumed that the encoding order of CTUs and CUs was from left to right and from top to bottom, as in H.265, H.266, and ECM. In hybrid block-based video codecs with altered encoding order, depending on the encoding parameterization, the current CU can be encoded from left to right or right to left and from top to bottom or bottom to top. In this case, in the third variant embodiment, different adjacent blocks can be considered.

[0081] For example, you can consider the top adjacent block, the left adjacent block, the right adjacent block, and the bottom adjacent block. Figure 9 An example of the top, left, right, and bottom neighbor blocks of a current W×H CU is shown. For each of these neighbor blocks, valuable data such as rate-distortion cost or quadtree depth can be used. For example, following Figure 9 The neighboring features included in vector V can be the rate-distortion cost of the top neighboring block. Rate distortion cost of the left adjacent block Rate distortion cost of adjacent blocks on the right Rate distortion cost of bottom adjacent blocks The quadtree depth of the top adjacent block The quadtree depth of the left adjacent block The quadtree depth of the right adjacent block The quadtree depth of the bottom adjacent block In this case, NF = .

[0082] Of these four adjacent blocks, if an adjacent block does not exist because it is outside the boundary of the current frame or has not yet been encoded, its associated data can be set to a default value. For example, for the current CU, if its right-hand adjacent block does not exist, then... It can be set to the maximum value of a 64-bit unsigned integer, that is . It can be set to -1. As another example, for the current CU, if its right-hand adjacent block does not exist, then... and It can be set to -2.

[0083] Any sub-implementation of the first variant embodiment can be directly adapted to the third variant embodiment. In particular, the fourth sub-implementation of the first variant embodiment can be transformed into the sub-implementations in the following sections.

[0084] Rate-distortion cost of adjacent blocks as leaves of the current segmentation tree In a sub-implementation of the third variant embodiment, Figure 9 The examples in the example are reused. Given the current state of the segmentation of the current frame when RDO is running on the current CU, the top neighbor block can correspond to a "leaf" CU containing the pixels above the pixel at the top left of the current CU. The rate-distortion cost of the top neighbor block can then correspond to its "no-splitting" cost. The left neighbor block can correspond to a "leaf" CU containing the pixels to the left of the pixel at the top left of the current CU. The rate-distortion cost of the left neighbor block can then correspond to its "no-splitting" cost. The right neighbor block can correspond to a "leaf" CU containing the pixels to the right of the pixel at the top right of the current CU. The rate-distortion cost of the right neighbor block can then correspond to its "no-splitting" cost. The bottom neighbor block can correspond to a "leaf" CU containing the pixels at the bottom of the pixel at the bottom left of the current CU. The rate-distortion cost of the bottom neighbor block can then correspond to its "no-splitting" cost.

[0085] Note that in this sub-implementation, any alternative criteria can be used to locate the top / left / right / bottom adjacent blocks of the current CU.

[0086] Rate distortion is used as the cost of replacing the top / left adjacent block with two eigenvalues. In the fourth variant embodiment, based on equation (1), the cost value of adjacent blocks can be considered as including two characteristic values: bitrate R and distortion D. In this embodiment, any of the sub-embodiments of the first and second variants can be directly adapted. As an example, Figure 8 The scenario illustrated in (a) can be adapted as follows: Vector V can contain the code rate of the middle and top adjacent blocks. Distortion of adjacent blocks in the middle and top The bit rate of the adjacent blocks on the left side of the middle Distortion of adjacent blocks on the left side of the middle In this case, Note that any one of the sub-examples can be directly adapted to this sub-example.

[0087] Parent cost (PC) During the application of Rate-Distortion Optimization (RDO) to coding units (CUs), this process generates sub-CUs. This implies that each CU has a parent CU. To maintain a comprehensive understanding of the overall segmentation structure, incorporating the cost of the splitting patterns evaluated on the parent CUs as informative data for the current sub-CUs may become crucial. For example, in H.266, the PC feature set could be... .

[0088] Whenever a split pattern is applied to the parent, its associated costs, normalized by the unsplit costs of the parent, can be populated into the parent cost list of the next state (CU), and arbitrary values ​​can be assigned to the remaining costs that have not yet been tested.

[0089] Current CU (BI) block information In addition to other information, block information can also include block size, QP value, and channel components. This allows for differentiation between CUs of different sizes, channels, prediction types, and frame types.

[0090] BI = {W, H, QP, Channel, Prediction Type, Frame Type} For example, in H.266, the characteristics of BI can be... · · • • •

[0091] In a variant embodiment, for the current CU belonging to an intra-frame slice, the BI includes channel components, while for the current CU belonging to an inter-frame slice, the BI does not include channel components.

[0092] In another variant embodiment, for the current CU, the BI always includes a channel component.

[0093] Spatial feature representation of all CU (SF) Convert texture information of various sizes by scaling it down / up and down into a fixed-size input. The target image size can be selected, for example, 12×12. Any size CU and its causal boundary can be scaled down / up to this target image size. The following sections on training database collection describe an example of database collection, where for each CU, a four-pixel-wide causal boundary is extracted using the actual CU. However, note that causal boundaries of different widths can be used without affecting the interpreted principle. Furthermore, note that the causal boundary of a CU can have different localizations with respect to that CU without affecting the interpreted principle.

[0094] For example, the target image size could be 12×12, and the causal boundary width could be 4. In this case, a CU with both height and width strictly greater than 8 and its causal boundary can be scaled down to 12×12. In this case, a CU of size 8×8 and its causal boundary can be neither scaled down nor scaled up. In this case, a CU with both height and width strictly less than 8 and its causal boundary can be scaled up to 12×12, see... Figure 10 In this case, one dimension Strictly greater than 8, another dimension CUs strictly less than 8, and their causal boundaries, can be shrunk along d0 and magnified along d1 to reach 12×12. In this case, one dimension Strictly greater than 8, another dimension Equals 8 ( The CUs of () and their causal boundaries can be exclusively reduced along d0 to reach 12×12. In this case, one dimension Strictly less than 8, another dimension Equals 8 ( The CUs of ) and their causal boundaries can be exclusively magnified along d0 to reach 12×12.

[0095] For example, the target image size could be 18×18, and the causal boundary width could be 2. In this case, CUs whose height and width are both strictly greater than 16 and their causal boundaries can be shrunk to 18×18. In this case, CUs of size 16×16 and their causal boundaries can be neither shrunk nor enlarged. In this case, CUs whose height and width are both strictly less than 16 and their causal boundaries can be enlarged to 18×18. In this case, one dimension Strictly greater than 16, another dimension Strictly less than 16 ( The CUs of ( ), and their causal boundaries, can be shrunk along d0 and magnified along d1 to reach 18×18. In this case, one dimension Strictly greater than 16, another dimension Equals 16 ( The CUs of () and their causal boundaries can be exclusively reduced along d0 to reach 18×18. In this case, a dimension Strictly less than 16, another dimension Equals 16 ( The CUs of ) and their causal boundaries can be exclusively magnified along d0 to reach 18×18.

[0096] In a variant embodiment, a nearest neighbor method can be used to shrink / enlarge a given CU and its causal boundaries. The nearest neighbor method assigns the value of the nearest pixel to the new location after resizing the input image, making the computational complexity negligible.

[0097] In another variant embodiment, linear interpolation can be used to reduce / enlarge a given CU and its causal boundaries.

[0098] Reduced / enlarged images used directly as spatial feature representations In a variant embodiment, an image generated by optional scaling down / scaling up of a given CU and its causal boundaries can be directly used as a spatial feature representation. For example, in Figure 10 In this context, it could mean that pixels in a 12×12 image are placed into the spatial feature vector V. SF middle.

[0099] In another variant embodiment, in addition to the image generated by the optional scaling down / scaling of a given CU, the spatial feature vector V SF It can include any other spatial features.

[0100] Images that have been reduced in size or enlarged undergo processing before being placed into spatial feature vectors. In a variant embodiment, the image generated by optional scaling down / upgrading of a given CU and its causal boundaries can be processed before being fed into a spatial feature vector. For example, the image generated by optional scaling down / upgrading of a given CU and its causal boundaries can be fed into a convolutional neural network, and the neural network output can be fed into a spatial feature vector, see [link to relevant documentation]. Figure 11 .

[0101] Traditional feature extraction transforms texture information of various sizes into a fixed-size input. For any size CU, instead of using scaling down / scaling to convert texture information into a fixed-size input, conventional features can be extracted from the CU and its causal boundaries to construct a fixed-size spatial feature vector. For example, the gray-level co-occurrence matrix (GLCM) and histogram of oriented gradients (HOG) can be selected as conventional features.

[0102] Gray-level co-occurrence matrix (GLCM) as a component of spatial features (SF) For a given component I (luminance, blueness, or redness) of a given CU, GLCM M It can have a size of 2 b ×2 b 'b' indicates the internal bit depth of the codec of interest. For example, b=10. As another example, b=8. M can be within a given distance range. and given direction Defined above. For example, As another example, .For example, As another example, Given a range of distance values ​​and a given direction value, It could be the number of times the pair appears at the spatial relation defined in I.

[0103] x and y are the coordinates of the current pixel. dx and dy define the spatial offset between the current pixel pairs. For example, if the distance range value is 1 and the direction value is... Then dx = 1 and dy = 1. As another example, if the distance range value is 1 and the direction value is... If , then dx = -1 and dy = 1.

[0104] Once M is computed, various statistical measures can be derived from M to form vectors. Spatial eigenvectors It can include .

[0105] Contrast, energy, homogeneity, correlation, and dissimilarity derived from GLCM In a variant embodiment, for a given component of a given CU, for a given range of distances and a given orientation, contrast, energy, homogeneity, correlation, and dissimilarity can be derived from the calculated M. For example, the formula for each of them can be as follows. Contrast = (7) Energy = (8) Homogeneity = (9) Correlation = (10) Dissimilarity = (11).

[0106] and These are the mean values ​​of the sums of the rows and columns of M, respectively. and These are their corresponding standard deviations. ={contrast, energy, homogeneity, correlation, dissimilarity}.

[0107] A subset of {contrast, energy, homogeneity, correlation, dissimilarity} In another variant embodiment, for a given component of a given CU, for a given range of distance values ​​and a given direction value, It can include any subset of {contrast, energy, homogeneity, correlation, dissimilarity}. For example, ={contrast, energy, homogeneity, dissimilarity}. As another example, ={contrast, homogeneity, dissimilarity}.

[0108] {Contrast, Energy, Homogeneity, Correlation, Dissimilarity} and any additional statistical measures In another variant embodiment, for a given component of a given CU, for a given range of distance values ​​and a given direction value, In addition to other statistical measures, it can also include {contrast, energy, homogeneity, correlation, dissimilarity}.

[0109] Cascade of statistical measures associated with each CU component In a variant embodiment, for those belonging to a given CU For each component compID, given a range of distance values ​​and a given direction value, GLCM can be calculated. And can be from The statistical metric set is derived from this. For example, if for a given CU, for a given range of distance values ​​and a given direction value, the statistical metric set is {contrast, energy, homogeneity, correlation, dissimilarity}, then... = , , .

[0110] As another example, if for a given CU, for a given range of distance values ​​and a given direction value, the statistical measure set is {contrast, correlation}, then = .

[0111] Multiple values ​​for distance range and direction In a variant embodiment, for a given component of a given CU, for each pair of a given range value and a given direction value... It can calculate GLCM And can be from The statistical measure set is derived from the given data. For example, if the statistical measure set is {contrast, energy, homogeneity, correlation, dissimilarity}, then for a given component of a given CU, the statistical measure set is derived from the given data. and against It can be calculated and . = .

[0112] As another example, if for a given component of a given CU, the statistical measure set is {contrast, correlation}, then using the pair and against It can be calculated and . ={ , }

[0113] Different definitions of statistical measures In a variant embodiment, the already defined statistical measure can be expressed differently. For example, homogeneity = .

[0114] Any combination of embodiments The different embodiments in the preceding chapters can be directly combined.

[0115] Histogram of Oriented Gradients (HOG) as a component of Spatial Features (SF) For a given CU, the HOG of that CU can be calculated and placed into the vector V. HOG In the middle. Spatial feature vector V SF It can include V HOG .

[0116] HOG given CU and its causal boundary In a variant embodiment, for a given CU, the HOG of the CU and its causal boundary can be calculated and placed into V. HOG middle.

[0117] HOG of a given CU and HOG of its causal boundary In a variant embodiment, for a given CU, the HOG of the CU and the HOG of its causal boundary can be calculated, and the cascade of these two HOGs can be placed into V. HOG middle.

[0118] HOG for each subdivision of a given CU and its causal boundary In a variant embodiment, a given CU and its causal boundary can be partitioned into parts. Any partitioning that constitutes a part can be applied. Then, for each of these parts, a HOG can be computed. Finally, a cascade of the resulting HOGs can be placed into V. HOG middle.

[0119] For example, let's consider a causal boundary (also called the "top boundary") that includes the four-pixel row above the current W×H CU, and a causal boundary (also called the "left boundary") that includes the four-pixel column to the left of the current W×H CU. Let's also consider the current CU and its causal boundaries as being divided into the following parts: {"top boundary", "left boundary", four non-overlapping sub-blocks of the same size to the current CU}, see [link to relevant documentation]. Figure 12 Then, the HOG labeled as the top HOG feature can be calculated at the "top boundary". The HOG labeled as the left HOG feature can be calculated at the "left boundary". The HOG labeled as HOG feature i can be calculated at index i ( Calculated on non-overlapping sub-blocks, = .

[0120] HOG calculation Any of the different definitions of HOG applies here.

[0121] As an example, all intervals of the HOG can be initialized to 0. Then, for each pixel (e.g., all pixels; e.g., one out of two pixels) in the pixel set belonging to the interval on which the HOG is computed (e.g., the causal boundary of the current CU; e.g., a sub-block in the current CU), the magnitude can be computed. and angle . in These can be the difference between the adjacent pixels to the right and left of the current pixel, and the difference between the adjacent pixels above and below the current pixel. i, j denote the coordinates of the current pixel within the region of interest I.

[0122] Then, amplitude and angle It can be used to find the index p of the HOG interval to be incremented. Finally, after completing the loop on each pixel, the resulting HOG can be normalized.

[0123] As another example, the magnitude can be defined differently. Besides... In addition, the previous examples can be reused.

[0124] As another example, It can be defined differently. Besides and In addition, the previous examples can be reused.

[0125] Note that any variation of the current embodiment can be directly combined with any of the other embodiments.

[0126] Spatial features (SF) combining GLCM and HOG In a variant embodiment, for a given CU and its causal boundary, GLCM and HOG can be placed into the associated spatial feature vector. For example, if the spatial feature vector of a given CU and its causal boundary exclusively includes GLCM and HOG, then .

[0127] The training process of the framework Training database collection To form a training database, many CUs can be collected by encoding many sequences at different resolutions (e.g., 22, 27, 32, 37) using several QPs (e.g., 22, 27, 32, 37).

[0128] Figure 13 An example of training database collection is depicted. When encoding the CU begins, its height and width can first be verified to be within the allowed size set. To form the state vector, the encoded block can be accompanied by a (W+4, H+4) patch, covering the block itself and a causal boundary four pixels wide. Furthermore, the associated cost of the splitting pattern of the current test can be extracted later against the reward signal. NF, BI, and PC features can also be extracted to form a total vector V representing the current state. V = {NF, PC, BI, block pixels}. At this stage, the block pixels may not yet have been processed to compute spatial features to form vector V. SF .

[0129] Reward function for incorporation cost and time complexity of agent-based action prediction The reward signal can be considered as the cost of the current action ( cost 动作 This is the cumulative penalty of the split-mode decision plus the time complexity term. Multiply To balance these two aspects. Reward (action, CU) = - cost 动作 - (17).

[0130] Theoretical RDO iteration as time complexity In a variant embodiment, time complexity can be considered as the theoretical number of times the RDO process is performed starting from a given CU.

[0131] The total number of intra / inter-frame test modes as a measure of time complexity. In a variant embodiment, when encoding the CU, the number of modes to be tested in different encoding components can be predetermined. For example, these components may include transform modes, intra-frame modes, and inter-frame modes. The total number of modes tested can be considered as the time complexity. Reward (action, CU) = - cost 动作 - (18) Indicates the number of test modes during the CU encoding process.

[0132] Adaptation to reinforcement learning algorithms for training neural networks In reinforcement learning, the agent can improve its performance by repeatedly applying the Q-value function over multiple iterations. Interactive learning of the Q-value function To make decisions in an environment. The environment can be defined using its state, action, and reward signals. In this description, the state can be the feature vector V of the current CU, the action can be a decision on splitting patterns, and the reward can be a combination of cost and time complexity as defined in equations (17-18).

[0133] Figure 14 An example of the training process is shown, in which the agent interacts with the environment and loops through each round. A round can represent the total segmentation tree of a 64×64 CU parent. The agent can take the parent as the first state S0, and at this stage, it can process the pixels of the current CU to compute the spatial feature vector and form the entire feature vector V. The next state can depend on the current action.

[0134] Characterizing the neural network segmentation pattern prediction used in the inference step The “Activate Prediction” flag allows the model to make predictions for the current CU within the RDO loop.

[0135] If the classic encoder heuristic is activated, the encoder can prepare a list of split patterns to test based on the heuristic. If the model is activated (activation prediction = 1) and the predicted action (the model's output) matches the "current test pattern" (the current split pattern being tested on the current CU), the encoder can test that specific split pattern on the CU. On the other hand, if the predicted action does not match the "current test pattern," the pattern can be skipped, and the encoder can proceed to the next configuration. Figure 15 An example of a flag used to control model predictions is provided. Figure 16 An example of the prediction process (PP) is given.

[0136] If the classical encoder heuristic is deactivated, and if the model is activated (activation prediction = 1), then the RDO process can test all split patterns that the agent must not skip.

[0137] Figure 17 An embodiment of method 1700 under the general aspects described herein is illustrated. The method begins at a start block 1701 and control proceeds to block 1710 for performing segmentation prediction of video blocks using a neural network. Control proceeds from block 1710 to block 1720 for segmenting video blocks based on the segmentation prediction. Control proceeds from block 1720 to block 1730 for encoding the segmented video blocks.

[0138] Figure 18 An embodiment of method 1800 under the general aspects described herein is illustrated. The method begins at a start block 1801 and control proceeds to block 1810 for performing segmentation prediction of video blocks using a neural network. Control proceeds from block 1810 to block 1820 for segmenting video blocks based on the segmentation prediction. Control proceeds from block 1820 to block 1830 for decoding the segmented video blocks.

[0139] Figure 19 An embodiment of an apparatus 1900 for encoding, decoding, compressing, decompressing, or filtering video data using the aforementioned methods is shown. The apparatus includes a processor 1910 and can be interconnected with a memory 1920 via at least one port. Both the processor 1910 and the memory 1920 may also have one or more additional interconnects to external connections.

[0140] The processor 1910 is also configured to insert or receive information in the bitstream and to compress, encode, or decode using any of the aspects described.

[0141] The embodiments described herein encompass a wide variety of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described, and often in a manner that may sound limiting, at least for the purpose of illustrating individual characteristics. However, this is for the purpose of clarity and does not limit the application or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide other aspects. Furthermore, the aspects described can also be combined and interchanged with those described in earlier applications.

[0142] The aspects described and envisioned in this application can be realized in many different forms. Figure 20 ,21 22 provides some embodiments, but other embodiments are contemplated, and... Figure 20 , 21 The discussion in section 22 does not limit the scope of implementation. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to the transmission of a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions thereon stored for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having bitstreams generated according to any of the described methods stored thereon.

[0143] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," and "frame." Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0144] This document describes various methods, and each of these methods includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined.

[0145] The various methods and other aspects described in this application can be used to modify, for example... Figure 20 and Figure 21 The modules of the video encoder 100 and decoder 200 shown are, for example, intra-frame prediction, entropy coding, and / or decoding modules (160, 260, 145, 230). Furthermore, this aspect is not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations (whether pre-existing or future-developed), and any extensions to such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this application may be used individually or in combination.

[0146] Various numerical values ​​are used in this application. Specific values ​​are for illustrative purposes, and the aspects described are not limited to these specific values.

[0147] Figure 20 The encoder 100 is illustrated. Variations of the encoder 100 are envisioned, but for clarity, the encoder 100 is described below without describing all the anticipated variations.

[0148] Before being encoded, the video sequence may undergo pre-coding (101), for example, applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing remapping of the input image components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization with one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0149] In encoder 100, the image is encoded by encoder elements as described below. The image to be encoded is segmented (102) and processed, for example, in units of CUs. Each unit is encoded using, for example, an intra-frame or inter-frame mode. When a unit is encoded in intra-frame mode, it performs intra-frame prediction (160). In inter-frame mode, motion estimation (175) and compensation (170) are performed. The encoder determines (105) which of the intra-frame or inter-frame modes to use to encode the unit and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (110) the prediction block from the original image block.

[0150] The predicted residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with the motion vector and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is directly encoded without applying either the transform or quantization process.

[0151] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inversely transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed image to perform, for example, deblocking / SAO (Sample Adaptive Shift) filtering, thereby reducing coding artifacts. The filtered image is stored at the reference image buffer (180).

[0152] Figure 21 A block diagram of a video decoder 200 is shown. In decoder 200, the bitstream is decoded by decoder elements as described below. Video decoder 200 typically performs decoding rounds in the reverse order of encoding rounds, such as... Figure 20 As described in [the document]. Encoder 100 typically also performs video decoding as part of the encoded video data.

[0153] Specifically, the input to the decoder includes a video bitstream, which can be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other encoded information. Image segmentation information indicates how the image is segmented. Therefore, the decoder can segment (235) the image based on the decoded image segmentation information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and prediction block are combined (255) to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion-compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored at the reference image buffer (280).

[0154] The decoded image can undergo further post-decoding processing (285), such as inverse color transformation (e.g., conversion from YcbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which is the inverse of the remapping process performed in the pre-encoding process (101). Post-decoding processing can utilize metadata derived in the pre-encoding process and signaled in the bitstream.

[0155] Figure 22 The diagram illustrates an example block diagram of a system implementing various aspects and embodiments. System 1000 may be embodied as a device including the various components described below and configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described in this document.

[0156] System 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing various aspects, such as those described in this document. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0157] System 1000 includes an encoder / decoder module 1030, which is configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents one or more modules that can be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both encoding and decoding modules. Furthermore, as is known to those skilled in the art, the encoder / decoder module 1030 may be implemented as a separate element of system 1000, or may be incorporated within processor 1010 as a combination of hardware and software.

[0158] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described herein may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0159] In some embodiments, the memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamically volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG stands for Moving Picture Experts Group; MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding; also known as H.265 and MPEG-H Part 2), or VVC (Universal Video Coding, a new standard developed by the Joint Video Experts Team JVET).

[0160] As indicated in box 1130, inputs can be provided to the components of system 1000 through various input devices. Such input devices include, but are not limited to, (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster, (ii) component (COMP) input terminals (or sets of COMP input terminals), (iii) universal serial bus (USB) input terminals, and / or (iv) high-definition multimedia interface (HDMI) input terminals. Figure 22 Other examples not shown include composite video.

[0161] In various embodiments, the input device of block 1130 has associated corresponding input processing elements, as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or limiting a signal band to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting the selected band again to select, for example, a signal band that may be referred to as a channel in a particular embodiment, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the aforementioned (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0162] Furthermore, USB and / or HDMI terminals may include corresponding interface processors for connecting System 1000 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, within a separate input processing IC or within processor 1010. Similarly, as needed, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within processor 1010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which operate in combination with memory and storage elements to process the data streams as needed for presentation on the output device.

[0163] Various components of the system 1000 can be provided within an integrated housing, in which the various components can be interconnected using a suitable connection arrangement and data can be transferred between them, for example, internal buses as known in the art, including inter-IC (I2C) buses, wiring and printed circuit boards.

[0164] System 1000 includes a communication interface 1050, which enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented in, for example, wired and / or wireless media.

[0165] In various embodiments, wireless networks such as Wi-Fi networks, such as IEEE 802.11 (IEEE stands for Institute of Electrical and Electronics Engineers), are used to stream or otherwise provide data to system 1000. In these embodiments, the Wi-Fi signal is received via a communication channel 1060 and a communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to system 1000, delivering data via an HDMI connection to input box 1130. Still other embodiments use an RF connection to input box 1130 to provide streaming data to system 1000. As indicated above, various embodiments provide data in a non-streaming manner. Furthermore, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0166] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110, and other peripheral devices 1120. Display 1100 in various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be used in a television, tablet computer, laptop computer, mobile phone, or other device. Display 1100 can also be integrated with other components (e.g., in a smartphone) or standalone (e.g., an external monitor for a laptop computer). In various embodiments, other peripheral devices 1120 include one or more of the following: a standalone digital video disc (or digital universal disc) (DVR, both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments utilize one or more peripheral devices 1120 that provide functionality based on the output of system 1000. For example, a disc player performs the function of playing the output of system 1000.

[0167] In various embodiments, signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention is used to transmit control signals between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120. Output devices can be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speaker 1110 can be integrated into a single unit with other components of system 1000 in electronic devices such as, for example, televisions. In various embodiments, display interface 1070 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0168] For example, if the RF portion of input 1130 is part of a standalone set-top box, then display 1100 and speaker 1110 can alternatively be separated from one or more of the other components. In various embodiments where display 1100 and speaker 1110 are external components, output signals can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0169] The embodiments can be implemented by computer software implemented by processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits. Memory 1020 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. As a non-limiting example, processor 1010 can be of any type suitable for the technical environment and can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0170] Various implementations involve decoding. As used herein, “decoding” can encompass, for example, all or part of the processing performed on a received encoded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also includes, or alternatively includes, processes performed by a decoder of the various implementations described herein.

[0171] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Given the context of the particular description, it will be clear, and is believed to be well understood by those skilled in the art, whether the expression "decoding process" is intended to specifically refer to a subset of operations or to refer to a broader decoding process.

[0172] Various implementations involve encoding. In a manner similar to the above discussion of “decoding,” the term “encoding,” as used herein, can encompass all or part of the processing performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder, such as segmentation, differential coding, transform, quantization, and entropy coding. In various embodiments, such a process also includes, or alternatively includes, processes performed by the encoders of the various implementations described herein.

[0173] As a further example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "encoding" refers only to differential encoding; and in yet another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. It will be clear, and is considered well understood by those skilled in the art, that the expression "encoding process" is intended to specifically refer to a subset of operations or to refer to a broader encoding process, given the context of the particular description.

[0174] Note that the grammatical elements used in this article are descriptive terms. Therefore, the use of other grammatical element names is not excluded.

[0175] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0176] Various implementations may involve parametric models or rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often with constraints on computational complexity. This can be measured by rate-distortion optimization (RDO), or by least mean square (LMS), average absolute error (MAE), or other such metrics. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. Different approaches exist to address the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all encoding options, including all considered modes or encoding parameter values, and a complete evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save encoding complexity, particularly by calculating approximate distortion based on predicting or predicting the residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, such as by using approximate distortion only for some of the possible encoding options and full distortion for others. Other methods evaluate only a subset of the possible encoding options. More generally, many methods employ any of a wide variety of techniques to perform optimization, but the optimization is not necessarily a complete evaluation of both coding cost and associated distortion.

[0177] The implementations and aspects described herein can be implemented, for example, in methods or processes, apparatuses, software programs, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., discussed only as a method), the implementation of the features in question can be implemented in other forms (e.g., apparatuses or programs). Apparatuses can be implemented, for example, in suitable hardware, software, and firmware. The methods can be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0178] References to "an embodiment" or "an embodiment" or "an implementation" or "an implementation," and other variations thereof, mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment. Therefore, the appearance of expressions "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation," and any other variations appearing throughout the application, do not necessarily refer to the same embodiment.

[0179] Furthermore, this application may involve "determining" various types of information. Determining information may include, for example, one or more of the following: estimation information, calculation information, prediction information, or information retrieved from memory.

[0180] Furthermore, this application may relate to "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or one or more of these.

[0181] Furthermore, this application may relate to "receiving" various types of information. Like "access," receiving is intended to be a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory) or one or more of it. Moreover, "receiving" is typically referred to in one way or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0182] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such expressions are intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to a large number of listed items.

[0183] Furthermore, as used herein, among other things, the term "signal" specifically refers to instructing the corresponding decoder to do something. For example, in a particular embodiment, the encoder signals a specific one of a plurality of transforms, encoding modes, or flags. In this way, in one embodiment, the same transform, parameter, or mode is used at both the encoder and decoder sides. Thus, for example, the encoder may transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling (implicit signaling) can be used without transmission to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the term "signal" has been used above, the term "signal" may also be used as a noun herein.

[0184] It will be apparent to those skilled in the art that implementations can generate a wide variety of signals, which are formatted to carry information, for example, that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry a bitstream of the described embodiment. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier wave using the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is well known that the signal can be transmitted via a wide variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0185] The preceding sections have described several embodiments spanning various claim classes and types. Features of these embodiments may be provided individually or in any combination. Furthermore, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination, spanning various claim classes and types: At least one embodiment includes using a neural network to determine segmentation predictions during the encoding and decoding process.

[0186] At least one embodiment includes encoding or decoding according to the above embodiments to achieve segmentation determination of video blocks.

[0187] At least one embodiment includes any of the above embodiments, wherein the neural network takes a vector including at least one of neighbor features, parent features, encoding unit information and spatial features as input.

[0188] At least one embodiment includes any of the above embodiments, wherein the neighbor features include at least one of the rate-distortion cost of the neighboring block and the quadtree depth of the neighboring block.

[0189] At least one embodiment includes any of the above embodiments, wherein the parent feature includes at least one of the costs of the splitting pattern normalized by the unsplit cost of the parent block.

[0190] At least one embodiment includes any of the above embodiments, wherein the coding unit information includes at least one of video block size, quantization parameters, and channel components.

[0191] At least one embodiment includes any of the above embodiments having spatial features, said spatial features including at least one of resized video blocks, processed resized video blocks, gray-level co-occurrence matrix, information derived from the gray-level co-occurrence matrix, contrast, energy, homogeneity, correlation, dissimilarity, histogram of directional gradients, and histogram of directional gradients of the causal boundaries of the video blocks.

[0192] At least one embodiment includes any of the above embodiments, wherein a cascade of gradient histograms is input into a neural network.

[0193] At least one embodiment includes any of the above embodiments, wherein information is signaled from the encoder to the decoder for determination by a neural network for segmentation of video blocks.

[0194] At least one embodiment includes any encoding or decoding operation based on the above operations.

[0195] At least one embodiment includes using the aforementioned method to encode or decode a sub-block.

[0196] At least one embodiment includes a bitstream or signal containing one or more of the described syntax elements or variations thereof.

[0197] At least one embodiment includes a bitstream or signal, which includes a syntax for conveying information generated according to any of the described embodiments.

[0198] At least one embodiment includes creation and / or transmission and / or reception and / or decoding according to any of the described embodiments.

[0199] At least one embodiment includes a method, process, apparatus, medium for storing instructions, medium for storing data, or signal according to any of the described embodiments.

[0200] At least one embodiment includes inserting in the signaling that enables the decoder to determine the syntax elements of the decoded information in a manner corresponding to that used by the encoder.

[0201] At least one embodiment includes creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0202] At least one embodiment includes a TV, a set-top box, a mobile phone, a tablet computer, or other electronic device that performs one or more transformation methods according to any of the described embodiments.

[0203] At least one embodiment includes a TV, set-top box, mobile phone, tablet computer, or other electronic device that performs one or more transformation methods according to any of the described embodiments to determine and display (e.g., using a monitor, screen, or other type of display) the resulting image.

[0204] At least one embodiment includes a TV, set-top box, mobile phone, tablet computer, or other electronic device that selects, band-limits, or tunes (e.g., using a tuner) a channel to receive a signal including an encoded image and performs one or more transformation methods according to any of the described embodiments.

[0205] At least one embodiment includes a TV, set-top box, mobile phone, tablet computer, or other electronic device that receives signals including encoded images over the air (e.g., using an antenna) and performs one or more transformation methods.

Claims

1. A method comprising: Use neural networks to perform video block segmentation prediction; Video blocks are segmented based on the segmentation prediction; as well as, Encode the segmented video blocks.

2. An apparatus comprising: Memory, and The processor is configured as follows: Use neural networks to perform video block segmentation prediction; Video blocks are segmented based on the segmentation prediction; as well as, Encode the segmented video blocks.

3. A method comprising: Use neural networks to perform video block segmentation prediction; Video blocks are segmented based on the segmentation prediction; as well as, Decode the segmented video blocks.

4. An apparatus comprising: Memory, and The processor is configured as follows: Use neural networks to perform video block segmentation prediction; Video blocks are segmented based on the segmentation prediction; as well as, Decode the segmented video blocks.

5. The method according to claim 1 or claim 3, or the apparatus according to claim 2 or claim 4, wherein the neural network receives an input comprising at least one of neighbor features, parent features, encoding unit information, and spatial features.

6. The method or apparatus of claim 5, wherein the neighbor features include at least one of the rate-distortion cost of the neighboring block and the quadtree depth of the neighboring block.

7. The method or apparatus of claim 5, wherein the parent feature includes at least one of the splitting pattern costs normalized by the unsplit cost of the parent block.

8. The method or apparatus according to claim 5, wherein the coding unit information includes at least one of the following: current video block size, quantization parameters and channel components, a prediction type and a frame type.

9. The method or apparatus of claim 5, wherein the spatial features include at least one of a resized video block, a processed resized video block, a gray-level co-occurrence matrix, information derived from the gray-level co-occurrence matrix, contrast, energy, homogeneity, correlation, dissimilarity, histogram of directional gradients, and histogram of directional gradients of the causal boundaries of the video block.

10. The method or apparatus of claim 9, wherein the spatial feature comprises using a cascade of two directional gradient histograms.

11. The method according to any one of claims 1, 3, 5 to 10, or the apparatus according to any one of claims 2, 4, or 5 to 10, wherein the signaling information is explicitly sent in the bitstream used for the neural network.

12. An apparatus comprising: The apparatus according to claim 2; and At least one of the following: (i) an antenna configured to receive a signal including a video block, (ii) a band limiter configured to limit the received signal to a band including the video block, and (iii) a display configured to display an output representing the video block.

13. A non-transitory computer-readable medium comprising data content generated by the method of any one of claims 1, 3, or 5 to 11, or by the apparatus of any one of claims 2, 4, or 5 to 11, for playback using a processor.

14. A signal comprising video data generated by the method of any one of claims 1, 3, 5 to 11, or the apparatus of any one of claims 2, 4, 5 to 11, for playback using a processor.

15. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of claim 1, claim 3, or any one of claims 5 to 11.

Images