Iterative training of neural networks for intra prediction

Abstract

The invention provides iterative training of neural networks for video encoding and decoding using intra prediction that finds a trade-off between extreme generality and extreme specialization for the codec of the trained neural networks. In a first iteration, a set of neural networks is trained following a partitioning approach. Then, for several iterations, the set of neural networks is inserted into the codec and pairs of blocks and their contexts are extracted from the partitions of the image by the codec with a single additional mode based on the neural networks, then the neural networks are retrained on these pairs. This way, the neural networks learn from the second iteration intra prediction that deviates from the intra prediction in the codec while still being valuable for the codec in terms of rate-distortion performance.

Description

Technical Field

[0001] At least one embodiment of this implementation typically relates to a method or apparatus for video encoding or decoding, compression or decompression. Background Technology

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transformation, including motion dimension prediction, to utilize spatial and temporal redundancy in video content. Generally, intra-frame or inter-frame prediction is used to leverage intra-frame or inter-frame correlations, and then the difference between the original image and the predicted image (usually represented as prediction error or prediction residual) is transformed, quantized, and entropy-coded. To reconstruct the video, the compressed data is decoded through the inverse process corresponding to entropy coding, quantization, transformation, and prediction. Summary of the Invention

[0003] At least one embodiment of this implementation generally relates to a method or apparatus for video encoding or decoding, and more specifically to a method or apparatus for simplifying encoding patterns based on a parameter model of neighbor-sample dependence.

[0004] According to a first aspect, a method is provided. The method includes the steps of training a set of neural networks for intra-frame prediction of a video block using pairs of partitioned portions of a video block and a surrounding region; extracting additional pairs of the video block and the surrounding region by iteratively using the set of neural networks as additional intra-frame coding modes for a codec; and retraining the set of neural networks using the extracted additional pairs to generate a set of neural networks for intra-frame prediction.

[0005] According to the first aspect, a method is provided. The method includes the aforementioned training step on an ensemble of neural networks for intra-frame prediction, and further includes performing encoding or decoding of video blocks using the generated ensemble of neural networks.

[0006] According to another aspect, an apparatus is provided. The apparatus includes a processor. The processor can be configured to encode video blocks or decode bitstreams by performing any of the methods described above.

[0007] According to another general aspect of at least one embodiment, an apparatus is provided comprising: means according to any embodiment of the decoding embodiment; and at least one of: (i) an antenna configured to receive a signal including a video block; (ii) a band limiter configured to limit the received signal to a band including the video block; and (iii) a display configured to display an output representing the video block.

[0008] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium containing data content generated according to any embodiment or variant of the described encoding embodiment or variant.

[0009] According to another general aspect of at least one embodiment, a signal is provided that includes video data generated according to any embodiment or variant of the described encoding embodiment or variant.

[0010] According to another general aspect of at least one implementation scheme, the bitstream is formatted to include data content generated according to any of the described encoding implementation schemes or variations.

[0011] According to another general aspect of at least one embodiment, a computer program product is provided, the computer program product including instructions that, when executed by a computer, cause the computer to perform any of the described coded embodiments or variations.

[0012] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments read in conjunction with the accompanying drawings. Attached Figure Description

[0013] Figure 1 The example shown is a reference sample used for intra-frame prediction in H.266 in the case of a square current block.

[0014] Figure 2 The intra-frame predicted orientation of a square block in H.266 is shown.

[0015] Figure 3 The top and left CU positions are shown for deriving the MPM list for different block shapes.

[0016] Figure 4 The diagram illustrates a decision tree for intra-frame prediction signaling for lumen in VTM-6.0.

[0017] Figure 5 An example decision tree illustrating intra-frame prediction signaling for chroma in VTM-6.0 is shown.

[0018] Figure 6 An example of the context surrounding the current square block to be predicted is shown.

[0019] Figure 7 An example of intra-frame prediction of square blocks from their context is shown using a fully connected neural network.

[0020] Figure 8An example of intra-frame prediction of square blocks from their context is shown using a convolutional neural network.

[0021] Figure 9 The diagram illustrates the extraction of W×W block Y from the original image I via H.265 and the extraction of its context X from the reconstruction of I.

[0022] Figure 10 The diagram illustrates the extraction of an additional W×W block Y from the original image I via H.265 and the extraction of its context X from the reconstruction of I.

[0023] Figure 11 An example of extraction via H.265 through the “extract_pair” of the W×W block Y from the original image I and the reconstructed context X from I.

[0024] Figure 12 An example of extraction via H.266 through the H×W block Y from the original image I and the “extract_pair” of its reconstructed context X from I.

[0025] Figure 13 An example is shown of extraction at spatial locations x, y given by image partitioning via an H×W block Y from the original image I and its context X, using H.266.

[0026] Figure 14 This demonstrates a standard, general-purpose video compression scheme.

[0027] Figure 15 This demonstrates a standard, general-purpose video decompression scheme.

[0028] Figure 16 A processor-based system for encoding / decoding is shown, based on aspects described in general.

[0029] Figure 17 An implementation of the method according to the aspects described in general is shown.

[0030] Figure 18 Another embodiment of the method according to the aspects described in general is shown.

[0031] Figure 19 An exemplary device based on the described aspects is shown. Detailed Implementation

[0032] Intra-frame prediction is a core coding tool in all video compression standards, such as H.264 / AVC, HEVC, and VVC. The basic idea is to leverage spatial correlations in an image frame sequence by predicting blocks of pixels based on already decoded causally adjacent pixels. The prediction residuals at the encoder are then transformed using a block transform, the transform coefficients are quantized, and then binary encoded. At the decoder, the blocks are reconstructed by adding the predictions to the decoded residuals, which is the reverse process of binary decoding, dequantization, and inverse transform.

[0033] For prediction purposes, the standard defines several models known as prediction modes. For example, HEVC defines 35 prediction modes, one of which is a planar mode, one is a DC mode, and the remaining 33 are angular modes. The planar and DC modes are designed to model regions of slowly and gradually changing intensity, while the angular modes are designed to model the orientation of different objects. On the other hand, VVC defines 67 regular intra-frame prediction modes, which include the 35 prediction modes from HEVC and an additional 32 angular modes. VVC also defines 28 wide-angle modes for use with rectangular coded blocks. The encoder prediction tool selects the optimal prediction mode in terms of rate distortion performance and signals it to the decoder using the mode coding scheme. The decoder prediction tool decodes the prediction mode and uses the decoded pixels from neighboring pixels to predict the current block using this mode.

[0034] The general aspects described in this paper address the problem of training neural networks for intra-frame prediction in video codecs such as H.265 / HEVC and H.266 / VVC. The term "context" is used to refer to the adjacent region of a block fed into the neural network, including several rows of decoded pixels above the block and several columns of pixels to the left of the block. In contrast, in H.265 / H.266, the term "reference sample" is always used to refer to the adjacent region of this block fed into the intra-frame prediction mode, including a row of decoded pixels above the block and a column of decoded pixels to the left of the block.

[0035] The neural network used for intra-frame prediction infers a prediction for the current block from the context or neighboring regions surrounding it. An ensemble of trained neural networks forms a single, additional intra-frame prediction pattern within the video codec of interest. In this additional single pattern, each neural network predicts blocks of a different size.

[0036] There are two ways to train a neural network within this set. The first method involves using an encoder-decoder pair of blocks at random spatial locations and their context from YC. b C r Images and their reconstructions are extracted, and then a neural network is trained on these pairs. More precisely, from YC at random spatial locations... b Cr Image patches are extracted, and their context is extracted from the image reconstruction at the same spatial location using an encoder-decoder. However, trained neural networks tend to provide fuzzy predictions because they are trained on multiple unrestricted pairs of patches and their contexts. Trained neural networks are said to be over-generalized. In the second approach, the YC is extracted via an encoder-decoder of interest. b C r Pairs of blocks and their contexts are extracted from image partitions, and then a neural network is trained on these pairs. More precisely, data collected by YC... b C r The image is partitioned by each block returned by the codec, and its context is extracted from the reconstruction of the image. However, the trained neural network primarily learns the codec's intra-frame predictions because the partitioning mechanism ensures that each returned block is predicted relatively well by the intra-frame prediction patterns in the codec from its reconstructed reference sample set. At this point, the trained neural network is considered overly "specialized" for the video codec.

[0037] To find a trade-off between extreme generalization and extreme specialization of the trained neural network's codec, an iterative training method for the neural network for intra-frame prediction is proposed. In the first iteration, the neural network ensemble is trained according to the second method described above. Then, for several iterations, (i) the neural network ensemble is inserted into the codec, and the signal is transmitted from YC through a codec with a single additional mode based on the neural network. b C r (ii) Extract pairs of blocks and their contexts from the partitions of the image, and retrain the neural network on these pairs. In this way, the neural network learns intra-prediction that deviates from the intra-prediction in the codec starting from the second iteration, while still being valuable to the codec in terms of rate distortion performance.

[0038] This section introduces intra-frame prediction components for video codecs. It focuses on the H.266 video codec because it is currently considered the best video codec in terms of compression performance, and it is an extension of H.265. Then, it presents neural network-based intra-frame prediction and two methods from the literature for training neural networks.

[0039] The intra-frame prediction process in H.266 consists of the following steps: collecting reference samples, processing these samples, deriving the actual predictions for the samples in the current block, and finally post-processing the predicted samples.

[0040] The reference sample generation process is in Figure 1As shown in the diagram, the "top" row of 2W samples is formed by previously reconstructed pixels located above the current block, where W represents the block width. Similarly, the "left" column of 2H samples is formed by reconstructed pixels located to the left of the current block, where H represents the block height. Corner pixels are also used to fill the gaps between the "top" row and the "left" column references. If some samples above and / or to the left of the current block are unavailable, because the corresponding coded block (CB) is not in the same slice, or the current CB is at a frame boundary, a method called reference sample replacement is performed, where the missing samples are copied from the available samples in a clockwise direction. The reference samples are then filtered using a specified filter according to the current CU size and prediction mode.

[0041] H.266 includes a family of prediction models derived from those in H.265. Planar and DC prediction models are used to predict smooth and gradually changing regions, while angular prediction models are used to capture different oriented structures. For each rectangular block shape, there are 65 directional prediction models organized in different ways. These prediction models correspond to, for example,... Figure 2 The different prediction directions are shown.

[0042] Intra-prediction is further extended using tools such as intra-prediction using multiple reference lines (MRL), intra-prediction using sub-partitions (ISP), and matrix intra-prediction (MIP). MIP is a set of intra-prediction modes, each of which infers the prediction of the current block from the reconstructed pixels through an affine transformation [4]. For 4×4 blocks, there are 35 modes. For 4×8, 8×4, and 8×8 blocks, there are 19 modes. For other blocks, 11 modes are used.

[0043] Intra-frame prediction signaling for lumen

[0044] The following paragraphs focus on signaling in planar mode, DC mode, and 65 directional modes, omitting signaling for MRL, ISP, and MIP. These last three will be described in detail in subsequent paragraphs.

[0045] Signaling in planar mode, DC mode, and 65 directional modes

[0046] On the encoder side, the optimal intra-prediction mode according to the rate distortion criterion is selected, and its index is transmitted from the encoder to the decoder. To execute the signaling of the selected mode index via entropy coding, a list of maximum possible modes (MPMs) is established.

[0047] In VTM-6.0, the MPM list contains six intra-prediction modes used to signal the intra-prediction mode for the current block. The MPM list is created from the prediction modes of the intra-coded CUs located above and to the left of the current CU, along with some default modes. The above and left CUs are located at the right and bottom edges of the current block, respectively. Figure 3 As shown.

[0048]

[0049]

[0050] Using circular adjacency in the range [2-66], it is equivalent to writing

[0051] ((L+offset)%mod)+2≡L–1

[0052] ((L+offset-1)%mod)+2≡L–2

[0053] ((L–1)%mod)+2≡L+1

[0054] ((L–0)%mod)+2≡L+2

[0055] Using the relationships described above, the MPM list derivation is shown in Table 1.

[0056] Table 1: MPM Derivation in VTM-6.0. A and L represent the prediction modes of the upper and left CUs, respectively. .

[0057]

[0058] If the selected intra-prediction mode used to predict the current block corresponds to one of the six MPM modes, this is signaled by: an mpmFlag with a value of 1, and subsequently by signaling candidate modes from the MPM list using the variable-length coding scheme shown in Table 2. Otherwise, the mpmFlag is equal to 0, and the candidate indices in the set of the remaining 61 modes are truncated into 5-bit or 6-bit binary encoding.

[0059] Table 2: MPM Signaling in VTM-6.0 .

[0060] Candidate Index code MPM[0] 0 MPM[1] 10 MPM[2] 110 MPM[3] 1110 MPM[4] 11110 MPM[5] 11111

[0061] Signaling of Multiple Reference Lines (MRLs)

[0062] For intra-frame prediction using MRL, the reference line used for prediction is signaled using the flag multiRefIdx. Valid values ​​for multiRefIdx are 0, 1, and 3, which signal the first, second, and fourth reference lines, respectively. When multiRefIdx is non-zero (meaning the second or fourth reference line is used), the prediction mode is always in the MPM list. Therefore, no mpmFlag is signaled. Furthermore, planes are excluded from the list. This means that when multiRefIdx is non-zero, only five prediction modes are available as possible candidate modes. When multiRefIdx is non-zero, the prediction modes are signaled as shown in Table 3.

[0063] Table 3: When multiRefIdx > 0 in VTM-6.0, a signal is sent to notify the MPM signaling. .

[0064] Candidate Index code MPM[1] 0 MPM[2] 10 MPM[3] 110 MPM[4] 1110 MPM[5] 1111

[0065] Signaling utilizing intra-fraction prediction (ISP) of sub-partitions

[0066] For the ISP, a flag called ispMode is used to signal the partition type used for the CU. ispMode is encoded only when multiRefIdx is equal to 0. Valid values ​​for ispMode are 0, 1, and 2, which signal no partition, horizontal partition, and vertical partition, respectively.

[0067] Signaling for Matrix Intra-Frame Prediction (MIP)

[0068] First, a flag called mipFlag is used to signal the MIP mode. A value of 1 means the MIP mode is used to predict the current block, and 0 means one of 67 intra-frame prediction modes is used. When mipFlag equals 1, multiRefIdx must equal 0, meaning the first reference line is used, and ispMode equals 0, indicating no target CU partition exists. Therefore, when mipFlag equals 1, multiRefIdx and ispMode are not written to the bitstream. If mipFlag equals 1, the index of the selected MIP mode is truncated in binary encoding, starting from VTM-6.0.

[0069] To handle cases where the intra-prediction mode used to predict the current block is one of 67 intra-prediction modes and the selected mode used to predict the upper CU or the selected mode used to predict the left CU is a MIP mode, a mapping between each MIP mode and one of the regular modes allows the MIP mode to be replaced with its mapped regular mode. Since VTM-6.0, any MIP mode is mapped to a plane.

[0070] Summary of Intra-Frame Prediction Signaling for Luminosity

[0071] By Figure 4 , the decision tree summarizes intra prediction signaling for luma. In Figure 4 , the light gray markers indicate that the value of the marker is derived from the value of the previous marker written into the bitstream at the encoder side and read from the bitstream at the decoder side. This means that the markers in light gray are not written into the bitstream at the encoder side; these markers are not read from the bitstream at the decoder side.

[0072] Intra-frame prediction signaling for chroma

[0073] For the two chroma channels, neither MRL, ISP nor MIP is used. However, two specific tools are used: Direct Mode and Cross-Component Linear Model (CCLM). Direct Mode corresponds to applying the selected mode for the collocated luma block used for prediction to the prediction of the current chroma block. If directFlag is equal to 1, Direct Mode is selected for predicting the current chroma block. Otherwise, when directFlag is equal to 0, one mode from the list L = [Planar, Vertical, Horizontal, DC] is selected. If a mode in L is equal to Direct Mode, this mode is replaced by the mode with index 66. In CCLM, a linear model predicts the current chroma block from reconstructed luma reference samples surrounding the collocated luma block. The parameters of the linear model are derived from the reconstructed reference samples. There are three CCLM modes, each associated with a different derivation of parameters. If cclmFlag is equal to 1, one of the three CCLM modes is selected. In this case, directFlag is not written into the bitstream. Otherwise, cclmFlag is equal to 0, and one of Direct Mode or a mode in L is selected. In Figure 5 , intra prediction signaling for chroma is presented.

[0074] Intra-frame prediction based on neural networks

[0075] The neural network for intra prediction infers the prediction of the current block to be predicted from the context surrounding the current block. The context X c is composed of reconstructed pixels located above and to the left of the current block Y, similar to the set of reconstructed reference samples for intra prediction in H.266. However, unlike this, the context X c extends further to the left and above, see Figure 6 . Due to this extension, the neural network can learn the relationship between the spatial correlation in its input context and the prediction it outputs. Note that the subscript "c" in X c indicates that the reconstructed pixels in the context have been preprocessed, as described in detail in the section "Signaling Neural Network-based Intra Prediction Mode inside a Video Codec".

[0076] If the neural network is fully connected, the context is typically flattened into vectors, and these vectors are fed into the neural network. The vectors provided by the neural network are then reshaped into the shape of the current block, thereby producing a prediction. See Figure 7 .Notice, The subscript "c" indicates that the predicted pixel has not yet been post-processed, which is explained in the chapter "Signaling Neural Network-Based Intra-Frame Prediction Modes Inside the Video Codec".

[0077] If the neural network is convolutional, the context can be divided into two parts. Each part is then fed into a stack of convolutional layers. The two stacks of feature maps at the outputs of the two convolutional layers are merged through full connectivity. Finally, the merged result is inserted into a stack of transposed convolutional layers to produce a prediction. See Figure 8 .

[0078] Create a neural network-based intra-frame prediction mode within the video codec.

[0079] In H.265 and H.266, images are divided into Coding Tree Units (CTUs). A CTU contains a Luminance Coding Tree Block (CTB), two Chroma CTBs, and syntax elements. Now, let's focus on the Luminance CTB for simplicity. One CTB is processed at a time in raster scan order. Each CTB can be further divided into Coding Blocks (CBs). CBs within a CTB are processed in Z-scan order. For example, in H.265, the size of the block to be predicted can be 64×64, 32×32, 16×16, 8×8, or 4×4. This means that five neural networks are needed, one for each size of the block to be predicted. Therefore, a neural network-based intra-frame prediction mode consists of five neural networks. In H.266, when the hierarchical segmentation is more complex, the block to be predicted can have the following sizes: 128×128, 64×64, 32×32, 16×16, 8×8, or 4×4. Alternatively, it can also be, for example, a rectangle of size 4×8. In this case, the solution is to allocate one neural network per block size to build neural network-based patterns.

[0080] Signaling is sent within the video codec to notify the neural network-based intra-frame prediction mode.

[0081] In different runs of integrating neural network-based intra-prediction modes into video codecs [1, 2, 3] (typically H.265), the neural network-based mode systematically competes with existing modes. For the current block to be predicted, a marker is written to the bitstream before all other markers used for intra-prediction. A value of 1 indicates that a neural network-based intra-prediction mode is selected for predicting the current block. In this case, other markers used for intra-prediction are not written to the bitstream. A value of 0 means that one of the regular intra-prediction modes is selected. In this case, the regular markers used for intra-prediction are then written to the bitstream.

[0082] Note that the signaling mentioned above has been implemented in H.265. No method has yet been proposed for H.266. It is worth noting that when selecting a neural network-based mode, it is unclear how the tags mipFlag, multiRefIdx, and ispMode will be handled.

[0083] Training a neural network for intra-frame prediction

[0084] In the literature, there are two main approaches to training neural networks for intra-frame prediction. These two approaches, referred to as training via “random” data extraction and training via “partitioned” data extraction, are described in the following two sections.

[0085] Training is performed using "random" data extraction.

[0086] In the first approach to training a neural network for intra-frame prediction, the codec of interest is used to determine the location of YC at a random spatial position. b C r Pairs of blocks and their contexts are extracted from the image and its reconstruction, and then a neural network is trained on these pairs. More specifically, we take the training of a neural network for predicting W×W blocks as an example. For YC b C r For each image I in the set of images, the image is encoded via a codec of interest, and several times, (i) a W×W block Y is extracted from I at a random spatial location (x,y), see [reference]. Figure 9 (ii) Reconstruction of I at (x,y) Extract its context X; (iii) preprocess the block and its context and add them to the training set of the neural network used to predict W×W blocks. Figure 9 The diagram illustrates the extraction of a W×W block Y from the original image I using H.265 with a quantization parameter (QP) of 37 at the same random spatial locations x, y, and the extraction of its context X from the reconstructed image I. Here, the image is in a 4:2:0 ratio, with W = 8, x = 8, and y = 16. Note that for x, y, the positions correspond to the pixels in the upper left of Y in I.

[0087] However, using neural networks trained through “random” data extraction, the trained networks often provide fuzzy predictions because they are trained on multiple unrestricted pairs of blocks and their contexts, and often many predictions of a block are possible given its context.

[0088] Training is performed using data extracted from "partitions".

[0089] In the second approach to training a neural network for intra-frame prediction, the codec is used to obtain data from YC. b C r Pairs of blocks and their contexts are extracted from image partitions, and then a neural network is trained on these pairs. Again, we will focus on an example of training a neural network for predicting W×W blocks. For each image I in the set of images, encoded via a codec of interest, and for each W×W block Y returned by the image partitioning, (i) at the spatial location (x) given by the image partitioning. Y ,y Y Extract Y from I at () (see) Figure 10 (ii) in (x) Y ,y Y Reconstruction of I at point ) (iii) Extract its context X; (iii) Preprocess the block and its context and add them to the training set of the neural network used to predict W×W blocks. Figure 10 The diagram illustrates the extraction of a W×W block Y from the original image I using H.265 with a QP of 37 at spatial locations xY, yY given by image partitioning, and the extraction of its context X from the reconstruction of I. Here, the image is in a 4:2:0 ratio, with W = 8, xY = 8, and yY = 24. Note that xY, yY correspond to the position of the pixel to the upper left of Y in I.

[0090] However, using training through “partitioned” data extraction, the trained neural network primarily learns intra-frame predictions for the codecs of interest.

[0091] The described aspect aims to train a neural network for intra-frame prediction such that it learns intra-frame predictions that deviate from the intra-frame predictions in the codec of interest, while still being valuable to the codec in terms of rate-distortion performance. An ensemble of neural networks is trained outside the encoder and decoder. A unique ensemble of neural networks may exist, and this set of neural networks is trained before actual encoding and decoding. The same trained ensemble of neural networks is then fed into both the encoder and decoder. The decoder does not require information to inform it of the selection of the neural network ensemble. Then, actual encoding and decoding can begin.

[0092] Iterative training of neural networks for intra-frame prediction

[0093] The first impetus for the described aspect is to avoid situations where the learned model gives fuzzy predictions, because the model is trained on multiple unrestricted pairs of blocks and their contexts. This is why YC is encoded via a codec. b C r A set of images Γ to generate a training set The reason, among which S H,W This includes pairs of blocks of size H×W provided by partitions of the image in Γ, and their contexts. Then, in S... H,W The training is done by θ H,W Each neural network f is parameterized H,W (.;θ H,W See Method 1. H It is the set of all possible block heights in the codec, where R W It is the set of all possible block widths in the codec.

[0094] It is important to note that the image partitioning in the codec of interest returns a Transform Block (TB). Within the codec, there may be coded block (CB) sizes that are not TB, as CBs of this size are forced to be segmented. A single additional intra-frame prediction mode based on a neural network in the codec has a neural network specifically designed to predict CBs of this size. Therefore, a neural network for predicting CBs of this size must be trained, but its training set cannot be generated using the methods described in the previous paragraphs. Instead, training via “random” data extraction, as explained in the section “Training via ‘Random’ Data Extraction,” can be used to train this neural network. As an example, in H.265, H = W, because the block is square, and W ∈ R. W ={4,8,16,32}. In fact, the maximum CB size of 64×64 is not the TB size because the 64×64CB is forced to be segmented during image partitioning.

[0095] Method 1: Iterative training of the neural network used for intra-frame prediction in the codec of interest.

[0096]

[0097]

[0098] During this phase of training, the learning model tends to reproduce intra-prediction within the codec of interest. This is because the image partitioning that generates the training blocks ensures that each training block is predicted relatively well from its reconstructed reference sample set by the intra-prediction mode in that codec. To allow the neural network to learn intra-prediction that gradually deviates from the predictions in the codec, while still being valuable to the codec, for l-1 iterations, (i) a training set is built as described in the last paragraph, but the codec is replaced with a codec having a single additional mode based on the neural network, and (ii) the neural network is retrained on these training sets, see Method 1.

[0099] The following functionality depends on the codec specificity: each image in Γ is encoded by the codec, and then pairs of H×W blocks and their contexts, H∈R, are extracted from the partitions of these images. H ,W∈R W This is referred to as "extract_from_partitioning" in Method 1. Similarly, the following function depends on the codec architecture: each image in Γ is encoded by a codec with a single additional intra-prediction mode based on a neural network, and then pairs of blocks of size H×W and their contexts provided by the partitions of these images are extracted, H∈R. H ,W∈R W In Method 1, this is called "extract_from_partitioning_nn". This is why "extract_from_partitioning" and "extract_from_partitioning_nn" are mentioned in the following two sections when iterative training is applied to a specific codec. In Method 1, L H,W Through neural network f H,W (.;θ H,W The parameter θ H,W The objective function to be minimized.

[0100] Iterative Training of Neural Networks for Intra-Frame Prediction in H.265

[0101] When the codec of interest is H.265, "extract_from_partitioning" can be precisely described, see Method 2. Image I in Γ is encoded via H.265, denoted as "h265", and a reconstruction of I is generated using H.265. And the feature set B of the blocks from partition I. The feature set of the block includes the position (x, y) of the top-left pixel of the block in I, the block width W, the number n0 of the bottom-left rows of the unreconstructed block, and the number n1 of the top-right columns of the unreconstructed block. Note that n0 and n1 are useful for filling pixels in the context of the unreconstructed blocks. Then, for each block, its features are used to extract block Y from I and its context X from I, see [link to relevant documentation]. Figure 11 . Figure 11 This example illustrates extraction at spatial locations x, y given by image partitioning, via H.265 QP of 37 through the "extract_pair" of a W×W block Y from the original image I and its reconstructed context X from I. The image is in a 4:2:0 configuration, with W = 8, x = 8, y = 16. Here, the lower left rows n0 = 8 of the block have not yet been reconstructed, but all rows in the upper right of the block have been reconstructed.

[0102] Finally, X and Y are preprocessed using the function "preprocessing" to produce the data to be added to the training set S. W Training pairs (X) c ,Y c ).

[0103] Method 2: "extract_from_partitioning" in the H.265 case

[0104]

[0105] “extract_from_partitioning_nn” follows the same description as “extract_from_partitioning”, but replaces H.265 with H.265 having a single additional intra-frame prediction mode based on a neural network, and is represented as “h265_nn” in Method 3.

[0106] Method 3: "extract_from_partitioning_nn" in the H.265 case

[0107]

[0108]

[0109] Iterative Training of Neural Networks for Intra-Frame Prediction in H.266

[0110] When the codec of interest is H.266, "extract_from_partitioning" shown in Method 2 and "extract_from_partitioning_nn" described in detail in Method 3 are modified in two ways. First, via H.266 by YC b C rThe partitioning of an image can return rectangular blocks. This means that the block height H is now added to the properties of each block in B. Furthermore, "extract_pair" is extended to rectangular blocks; see [link to relevant documentation]. Figure 12 . Figure 12 At the spatial location x, y given by the image partitioning, extraction is performed via H.266 QP of 37 through the "extract_pair" of the H×W block Y from the original image I and the context X of the reconstructed I from I. The image is in a 4:2:0 configuration with H=8, W=4, x=8, y=16. Here, the lower left n0=8 rows of the block have not yet been reconstructed, but all the upper right rows of the block have been reconstructed.

[0111] Secondly, H.265 was replaced by H.266, denoted as "h266". H.265 with a single additional intra-prediction mode based on a neural network was replaced by H.266 with a single additional mode based on a neural network, denoted as "h266". nn See Methods 4 and 5.

[0112] Method 4: "extract_from_partitioning" in the H.266 case

[0113]

[0114]

[0115] Method 5: "extract_from_partitioning_nn" in the H.266 case

[0116]

[0117] The proposed variant of iterative training

[0118] Any distribution from which quantization parameters are extracted

[0119] By default, in methods 2, 3, 4, and 5, for a given YC in Γ to be encoded by the codec of interest... b C r Image I uses quantization parameters (QPs) uniformly extracted from the set {22,27,32,37} for encoding. However, QPs can be extracted from any set; uniformity is not required.

[0120] Random initialization of the neural network during each iteration of training.

[0121] In method 1, when there is an exponent During iteration, at the start of minimization, the parameters of each neural network are initialized using the neural network parameters obtained at the end of the iteration with index i-1. Alternatively, during iteration with exponent i, the parameters of each neural network can be randomly initialized at the start of minimization.

[0122] Extracting the context of a given block from the original image

[0123] In previous chapters, for a given YC in Γ encoded by the codec of interest b C r Image I, generating its reconstruction Extract block Y from I, and from Extract its context X. Alternatively, the context can also be extracted from I. For example, in the case where the codec of interest is H.266, the current variant will... Figure 12 Transform into Figure 13 . Figure 13 This illustrates extraction at spatial locations x, y given by image partitioning, via an "extract_pair" of an H×W block Y from the original image I and its context X, where H.266 is used to encode I using a QP of 37. The image is in a 4:2:0 configuration with H=8, W=4, x=8, y=16. Here, the lower left rows (n0=8) of the block have not yet been reconstructed, but all rows in the upper right of the block have been reconstructed.

[0124] In the first iteration of training, "partitioned" data extraction was used instead of "random" data extraction.

[0125] In Method 1, the first step corresponds to “partitioned” data extraction, the principle of which has been explained previously. Conversely, the first step can correspond to “random” data extraction, the process of which has been described in detail in previous chapters. Note that in the latter case, at the end of the first iteration of iterative training, the trained neural network is an extremely “generalized” intra-frame predictor. Then, starting from the second iteration, the trained neural network is specialized for the codec of interest.

[0126] Eliminating "unpredictable" blocks from the training set individually from their context using a neural network.

[0127] In the iteration of iterative training When creating a training set At the same time, referring to Method 1, it can be achieved by a codec of interest with a single additional pattern based on a neural network. The image is partitioned to return blocks because the regular intra-frame prediction mode in this codec provides high prediction quality for this block. However, with a single additional mode based on the neural network, this block may be "unpredictable" from its context alone. To avoid retraining the neural network using previously trained models with blocks that are "unpredictable" from their context alone, methods 3 and 5 can be supplemented with conditions for detecting and removing these blocks, each paired with its context. Any condition can be used.

[0128] For example, consider two different conditions. Under both conditions, the training set contains only brightness patches. This means that in methods 2, 3, 4, and 5, for a given YC in Γ encoded by a codec of interest with a single additional pattern based on a neural network... b C r For image I, extract the luminance block Y from the luminance channel of I, and extract its luminance context X from the luminance channel of the reconstructed image I.

[0129] This first condition is divided into two cases. In the first case, via YC b C r The luminance blocks (TBs) returned by partitioning an image are not generated by dividing its luminance PB into different luminance TBs; that is, the TB and its PB are equivalent. The first case is indicated by the flag isSplit = false. In the second case, isSplit = true, and the luminance TB comes from at least one such partition. If isSplit = false, then t lowest “fast costs” are found across all intra-frame prediction modes on that luminance TB, along with the “fast cost” c of a single additional neural network-based mode on that luminance TB. nn ∈R + * The "fast cost" of intra-frame prediction modes on a block linearly combines the distortion between this block and mode prediction with an approximation of the cost of signaling the mode. In H.265, the distortion is the sum of absolute differences (SAD), while in H.266, the distortion is the minimum between SAD multiplied by 2 and the sum of absolute transform differences (SATD). If c nn Less than γ∈R + * Multiply by the t-th lowest "fast cost" c t ∈R + * If γ ∈ [0.90, 1.10], then the luminance TB is added to the training set. Otherwise, it is ignored. Typically, γ ∈ [0.90, 1.10] works well. t can take any value less than the number of regular intra-prediction modes in the codec of interest. For example, t ∈ {2, 3} works well. If isSplit = true, then the luminance TB is added to the training set if a single additional mode of the neural network based on index idxNN is selected for predicting this TB. For example, in this variant, method 5 becomes method 6.

[0130] Method 6: H.266 and the first conditional case in the above section "extract from partitioning_nn"

[0131]

[0132]

[0133] In method 6, m refers to the index of the intra-prediction mode, which is selected to predict the current TB returned by the image partition.

[0134] Under the second condition, the isSplit flag is no longer used; see method 7.

[0135] Method 7: H.266 and the second conditional case in the above section, "extract_from_partitioning_nn"

[0136]

[0137]

[0138] Replace "fast cost" with the distorted measure from the previous variant.

[0139] In the variant of the section “Eliminating Blocks That Are ‘Unpredictable’ Individually from Their Context’ in the Training Set Using Neural Networks,” each “fast cost” of the intra-prediction mode on the current luminance TB can be replaced with the distortion between the current luminance TB and the prediction provided by this mode. Any metric for distortion is valid. For example, if the sum of the squared differences between the current luminance TB and the prediction given by this mode (called the “predicted SSD”) is chosen as the metric for distortion, the variant is adjusted as in the previous section. If isSplit = false, then the t lowest predicted SSDs and the predicted SSDd of a single additional neural network-based mode on the luminance TB are found among all intra-prediction modes on that luminance TB. nn ∈R + * If d nn Less than γ multiplied by the t-th lowest prediction SSDd t ∈R + * If so, the brightness TB will be added to the training set.

[0140] Otherwise, ignore it. If isSplit=true, the brightness TB is added to the training set if a single additional neural network-based pattern of index idxNN is selected for predicting this TB. For example, Method 8 replaces the "fast cost" in Method 6 with the predicted SSD, as shown below.

[0141] Method 8: When "fast cost" is predicted to be replaced by SSD, "extract_from_" in the H.266 case partitioning_nn”

[0142]

[0143]

[0144] From each YC b C r Extract an equal number of training pairs from the image

[0145] From Method 2 to Method 8, if some images in Γ are larger than others, the training set is populated with more pairs of preprocessed patches extracted from the relatively larger images and their preprocessed context. This means that the various textures found in the training pairs are imbalanced compared to those present in the relatively larger images. To circumvent this, each method from Method 2 to Method 8 can be supplemented by the following criterion: this criterion limits the number of training pairs extracted from each image in Γ to s∈N. * Typically, when Γ contains more than 1 million YC b C r When images are displayed, It works well. To avoid only extracting from YC... b C r For each image, the blocks surrounding the top left corner are uniformly sampled from all training pairs extracted from that image, with the sum of s. For example, Method 9 supplements Method 5 with the above standard. Method 10 supplements Method 6 with the above standard.

[0146] Method 9: "extract_from_partitioning_nn" in the H.266 case, the standard is detailed in previous chapters. Detailed description

[0147]

[0148]

[0149] Method 10: H.266 and the section "Impurity of individual components from their context when removed from a training set by a neural network" The "extract_from_partitioning_nn" in the first conditional case of the "prediction" block is detailed in the preceding paragraphs. Detailed description

[0150]

[0151]

[0152] In methods 9 and 10, the function "shuffle" scrambles the elements of its input set. The element set of B contains the characteristics of the blocks returned by the image partitioning. Furthermore, the "break" statement breaks the innermost "for" loop, as in C.

[0153] Figure 17An embodiment of method 1700 according to the aspects described herein is illustrated. The method begins at a start box 1701 and commences at box 1710 to train an ensemble of neural networks for intra-frame prediction of a video patch using pairs of partitioned portions of a video patch and its surrounding region. The method proceeds from box 1710 to box 1720 to extract additional pairs of video patches and their surrounding regions by iteratively using the ensemble of neural networks as a single additional intra-frame coding mode for the codec. Control proceeds from box 1720 to box 1730 to retrain the ensemble of neural networks using the extracted additional pairs to generate an ensemble of neural networks for intra-frame prediction.

[0154] Figure 18 Another embodiment of method 1800 according to the aforementioned aspect is illustrated. The method begins at a start box 1801 and commences at box 1810 to train an ensemble of neural networks for intra-frame prediction of the video block using pairs of partitioned portions of the video block and surrounding regions. The method proceeds from box 1810 to box 1820 to extract additional pairs of the video block and surrounding regions by iteratively using the ensemble of neural networks as a single additional intra-frame coding mode for the codec. Control proceeds from box 1820 to box 1830 to retrain the ensemble of neural networks using the extracted additional pairs to generate an ensemble of neural networks for intra-frame prediction. Control proceeds from box 1830 to box 1840 to encode / decode the video block using the generated ensemble of neural networks for intra-frame prediction.

[0155] Figure 19 An embodiment of an apparatus 1900 for encoding, decoding, compressing, or decompressing video data using a simplified encoding mode based on a neighbor-sample dependency parameter model is shown. The apparatus includes a processor 1910 and can be interconnected to a memory 1920 via at least one port. Both the processor 1910 and the memory 1920 may also have one or more additional interconnects for external connection.

[0156] The processor 1910 is also configured to insert or receive information in the bitstream and to use any of the aspects described to perform compression, encoding, or decoding.

[0157] This document describes multiple aspects, including tools, features, implementation schemes, models, methods, etc. Many of these aspects are described in detail, and often in a way that may sound limited, at least to illustrate individual characteristics. However, this is for clarity and does not limit the application or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can also be combined and interchanged with those described in previous submissions.

[0158] The aspects described and envisioned in this document can be implemented in many different forms. The following... Figure 12 , Figure 13 and Figure 14 Some implementation schemes are provided, but other implementation schemes are envisioned, and Figure 12 , Figure 13 and Figure 14 The discussion does not limit the breadth of specific implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the methods stored thereon.

[0159] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," and "frame." Generally, but not necessarily, the term "reconstruction" is used at the encoding end, while "decoding" is used at the decoding end.

[0160] This document describes various methods, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or purpose of a particular step and / or action may be modified or combined.

[0161] The various methods and other aspects described in this document can be used to modify modules of the video encoder 100 and decoder 200 (e.g., intra-frame prediction, entropy coding, and / or decoding modules (160, 360, 145, 330)), such as Figure 12 and Figure 13 As shown. Furthermore, aspects of the invention are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this document may be used alone or in combination.

[0162] Various numerical values ​​are used in this document, such as {{1,0},{3,1},{1,1}}. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0163] Figure 12 Encoder 100 is illustrated. Variations of this encoder 100 are envisioned, but for clarity, encoder 100 is described below without describing all anticipated variations.

[0164] Before encoding, the video sequence may undergo pre-coding (101), for example, by applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or by performing remapping of the input image components to obtain a more resilient signal distribution to compression (e.g., histogram equalization using one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0165] In encoder 100, the image is encoded by encoder elements as described below. The image to be encoded is partitioned (102) and processed in units such as CUs. For example, each unit is encoded using either an intra-frame mode or an inter-frame mode. When a unit is encoded in intra-frame mode, it performs intra-frame prediction (160). In inter-frame mode, motion estimation (175) and compensation (170) are performed. The encoder determines (105) which of the intra-frame mode or inter-frame mode is used to encode the unit and indicates the intra-frame / inter-frame decision by, for example, a prediction mode label. For example, the prediction residual is calculated by subtracting (110) the prediction block from the original image block.

[0166] The predicted residual is then transformed (125) and quantized (130). The quantized transform coefficients, motion vectors, and other syntax elements are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0167] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. A loop filter (165) is applied to the reconstructed image to perform, for example, deblocking / SAO (sample adaptive offset) filtering, thereby reducing coded artifacts. The filtered image is stored in a reference image buffer (180).

[0168] Figure 13 A block diagram of a video decoder 200 is illustrated. In decoder 200, the bitstream is decoded by decoder elements, as described below. Video decoder 200 typically performs operations similar to... Figure 12 The encoding process is the opposite of the decoding process. Encoder 100 typically also performs video decoding as part of the video data encoding.

[0169] The decoder's input includes a video bitstream, which can be generated by the video encoder 100. First, the bitstream is entropy-decoded (230) to obtain transform coefficients, motion vectors, and other encoded information. Picture partitioning information indicates how the picture should be partitioned. Therefore, the decoder can partition (235) the picture based on the decoded picture partitioning information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (255) to reconstruct the image blocks. Prediction blocks (270) can be obtained from intra-frame prediction (260) or motion-compensated prediction (i.e., inter-frame prediction) (275). A loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0170] The decoded image can also undergo post-decoding processing (285), such as inverse color transformation (e.g., a transformation from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in the pre-encoding process (101). Post-decoding processing can utilize metadata derived in the pre-encoding process and signaled in the bitstream.

[0171] Figure 14 A block diagram illustrating examples of systems implementing various aspects and embodiments therein is provided. System 1000 may be embodied as a device including the various components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to other similar systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0172] System 1000 includes at least one processor 1010 configured to execute instructions loaded thereon for implementing various aspects, such as those described in this document. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 1040 that may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0173] System 1000 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both of an encoding module and a decoding module. Furthermore, the encoder / decoder module 1030 may be implemented as a standalone element of system 1000, or may be incorporated within processor 1010 as a combination of hardware and software known to those skilled in the art.

[0174] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described in this document may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more items from various projects during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0175] In several embodiments, the memory within processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 1010 or encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2, HEVC, or VVC (Video Coding).

[0176] Inputs to the components of system 1000 may be provided by various input devices as shown in box 1130. Such input devices include, but are not limited to: (i) an RF section that receives RF signals, for example, transmitted by a broadcaster via radio; (ii) a composite input terminal; (iii) a USB input terminal; and / or (iv) an HDMI input terminal.

[0177] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements necessary to: (i) select a desired frequency (also known as select a signal, or limit the signal band to a band), (ii) down-convert the selected signal, (iii) re-band limit the signal to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select the desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various functions among these functions, including, for example, down-converting received signals to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or to baseband. In one set-top box implementation, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to the desired frequency band. Various implementations rearrange the order of the aforementioned (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as inserting amplifiers and analog-to-digital converters. In various implementations, the RF section includes an antenna.

[0178] Furthermore, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or within the processor 1010. Similarly, aspects of USB or HDMI interface processing may be implemented as needed, either within a separate interface IC or within the processor 1010. Demodulated streams, error-corrected streams, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0179] Various components of system 1000 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data therebetween using a suitable connection arrangement 1140 (e.g., internal buses as known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0180] System 1000 includes a communication interface 1050 capable of communicating with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0181] In various implementations, data is streamed to system 1000 using a wireless network such as IEEE 802.11. For example, the wireless signals in these implementations are received via a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 in these implementations is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other cloud-based communications. Other implementations use a set-top box to provide streaming data to system 1000, delivering data via an HDMI connection to input block 1130. Still other implementations use an RF connection to input block 1130 to provide streaming data to system 1000.

[0182] System 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripheral devices 1120. In various embodiments, the other peripheral devices 1120 include one or more of the following: a standalone DVR, a disk player, a stereo system, a lighting system, and other devices that provide output functionality based on system 1000. In various embodiments, control signals are transmitted between system 1000 and the display 1100, speaker 1110, or other peripheral devices 1120 using signaling (such as AV.Link, CEC, or other communication protocols), which enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to system 1000 via communication interface 1050 using communication channel 1060. The display 1100 and speaker 1110 can be integrated into a single unit with other components of system 1000 in electronic devices (e.g., a television set). In various implementations, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0183] Alternatively, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may be separated from one or more other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0184] These implementations may be executed by processor 1010 or by computer software implemented in hardware or a combination of hardware and software. As a non-limiting example, these implementations may be implemented by one or more integrated circuits. As a non-limiting example, memory 1020 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 1010 may be of any type suitable for the technical environment and may encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0185] Various specific implementations involve decoding. As used in this application, "decoding" may include all or part of a process, for example, performing on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also or alternatively includes processes performed by a decoder of various embodiments described in this application, such as extracting weight indices to be used for various intra-frame prediction reference arrays.

[0186] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" specifically refers to a subset of operations or broadly refers to a wider decoding process will be clear based on the specific context of the description and is believed to be well understood by those skilled in the art.

[0187] Various specific implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding,” as used herein, can encompass all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various embodiments, such processes also or alternatively include processes performed by an encoder of the various embodiments described herein, such as weighting an intra-frame prediction reference array.

[0188] As a further example, in one implementation, "encoding" refers only to entropy encoding; in another implementation, "encoding" refers only to differential encoding; and in yet another implementation, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" specifically refers to a subset of operations or broadly refers to a wider encoding process will be clear based on the specific context of the description and is believed to be well understood by those skilled in the art.

[0189] Note that the grammatical elements used in this article are descriptive terms. Therefore, the use of other grammatical element names is not excluded.

[0190] When the accompanying drawings are presented as flowcharts, it should be understood that block diagrams of the corresponding devices are also provided. Similarly, when the accompanying drawings are presented as block diagrams, it should be understood that flowcharts of the corresponding methods / processes are also provided.

[0191] Various implementation schemes refer to rate-distortion computation or rate-distortion optimization. During the encoding process, a balance or trade-off between rate and distortion is typically considered, often taking into account computational complexity constraints. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. Different approaches exist to solve the rate-distortion optimization problem. For example, these methods may be based on extensive testing of all encoding options (including all considered modes or encoding parameter values) and a complete evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to reduce encoding complexity, particularly for the computation of approximate distortion based on prediction or prediction of the residual signal rather than the reconstructed residual signal. A hybrid of these two approaches can also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other methods evaluate only a subset of the possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both encoding costs and associated distortion.

[0192] The specific embodiments and aspects described herein may be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even if discussed only in the context of a single form of specific embodiment (e.g., discussed only as a method), specific embodiments of the discussed features may also be implemented in other forms (e.g., apparatus or program). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented, for example, in a processor that generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices, such as, for example, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0193] The reference to “an implementation plan” or “implementation plan” or “a specific implementation” or “specific implementation”, and their other variations, means that the specific features, structures, characteristics, etc., described in connection with the implementation plan are included in at least one implementation plan. Therefore, the appearance of the phrase “in an implementation plan” or “in an implementation plan” or “in a specific implementation” or “in a specific implementation”, and any other variations appearing throughout this document, do not necessarily refer to the same implementation plan.

[0194] In addition, this document may refer to the various pieces of information marked "determined". Determined information may include one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0195] Furthermore, this document may refer to "accessing" each piece of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or more of these.

[0196] Furthermore, this document may refer to "receiving" individual pieces of information. Like "access," "receiving" is intended as a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory). Moreover, "receiving" typically participates in one or more of the following operations: such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0197] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such phrases are intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as possible listed.

[0198] Moreover, as used herein, the term "signaling" refers to (among other things) instructing the corresponding decoder to do something. For example, in some implementations, the encoder signals a specific weight among several weights to be used in the intra-prediction reference array. Thus, in one implementation, the same parameters are used on both the encoder and decoder sides. Therefore, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters and others, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select specific parameters. Bit savings are achieved in various implementations by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various implementations, information is signaled to the corresponding decoder using one or more syntax elements, tags, etc. Although the verb form of the term "signal" has been used above, the term "signal" can also be used as a noun herein.

[0199] It will be apparent to those skilled in the art that embodiments can generate various signals formatted to carry, for example, storable or transmissible information. The information may include, for example, instructions for performing a method or data generated by one of the embodiments. For example, a signal may be formatted to carry a bitstream of the embodiment. Such signals may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of a spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating the carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is known that signals can be transmitted via various wired or wireless links. Signals may be stored on a processor-readable medium.

[0200] Implementations may include one or more of the following features or entities, individually or in combination, across a variety of different claim classes and types:

[0201] • Use a set of neural networks as an additional prediction mode in the codec. This mode can be an intra-frame prediction mode, and the neural networks can be of various sizes.

[0202] • Use a single additional intra-frame prediction mode that includes neural networks of different sizes.

[0203] • Use a single additional intra-prediction mode that includes neural networks of different sizes, wherein an initial training set of neural networks is constructed using an encoder and decoder with a single additional intra-prediction mode, and the neural networks including that set are retrained with the initial training set.

[0204] • The above single intra-frame prediction mode, in which a neural network corresponding to one of the sizes is trained by random data extraction.

[0205] • In the H.265 codec, a set of neural networks is used as additional intra-prediction modes. These modes can be intra-prediction modes, and the neural networks can be of various sizes.

[0206] • The above neural network set is trained in the H.265 codec as an additional intra-prediction mode, wherein for each block, the block and its context are extracted from the image using a feature set from the partitioned block, and the features and their context are preprocessed to produce training pairs to be added to the training set.

[0207] • Train at least one neural network for intra-frame prediction, wherein the neural network learns intra-frame predictions that deviate from the intra-frame predictions in the codec, while still being valuable to the codec in terms of rate distortion performance.

[0208] • Encode / decode the image set using a codec to produce a training set containing pairs of blocks and their contexts, the size of which is provided by partitioning at least one image in the image set, and use the training set to train the corresponding neural network.

[0209] • The above encoding / decoding, where the blocks are rectangular.

[0210] • Includes a bitstream or signal of one or more syntax elements or their variants from the described syntax elements.

[0211] • To create and / or transmit and / or receive and / or decode bit streams or signals that include one or more of the described syntax elements or their variants.

[0212] • A television, set-top box, mobile phone, tablet computer or other electronic device that performs loop filtering according to any of the embodiments described in the implementation plan.

[0213] • A television, set-top box, mobile phone, tablet computer, or other electronic device that performs loop filtering according to any of the described implementation schemes and displays the resulting image (e.g., using a monitor, screen, or other type of display).

[0214] A television, set-top box, mobile phone, tablet computer, or other electronic device that tunes (e.g., uses a tuner) a channel to receive signals including encoded images and performs loop filtering according to any of the described embodiments.

[0215] A television, set-top box, mobile phone, tablet computer, or other electronic device that receives signals including encoded images via radio (e.g., using an antenna) and performs loop filtering according to any of the described embodiments.

[0216] Various other general and specific inventions and claims are also supported and envisioned throughout this specification.

Claims

1. A method for video encoding, the method comprising: A set of neural networks for intra-frame prediction of the video blocks is trained using pairs of video blocks extracted from the partitions and their contexts. Additional pairs of the video blocks and their contexts are extracted by iteratively using the neural network ensemble as additional intra-coding modes for the codec, wherein training samples are encoded using the codec, wherein intra-frame prediction using neural networks is implemented in the codec using an initially trained network, wherein the parameters of each neural network in the neural network ensemble are initialized using neural network parameters obtained at the end of iterations of previous neural networks in the neural network ensemble, and wherein luminance blocks are extracted from the luminance channels of the image containing the video blocks, and their luminance context is extracted from the luminance channels of the reconstructed image of the image, wherein extracting the video blocks by iteratively using the neural network ensemble as additional intra-coding modes for the codec includes: for non-segmented blocks, only blocks whose prediction cost of the neural network mode is less than a threshold multiple of the lowest prediction cost among all intra-frame prediction modes; or for segmented blocks, blocks for which the neural network mode is selected; and, The extracted additional pairs are used to retrain the neural network ensemble to generate a neural network ensemble for intra-frame prediction, wherein the neural network ensemble is retrained on data from an encoder comprising a neural network for intra-frame prediction with fixed parameters; and, The retrained ensemble of neural networks is used to encode video blocks using intra-frame prediction, where... The set of neural networks trained for intra-frame prediction includes: For a fully connected neural network, the context surrounding the current block is flattened into vectors, the flattened vectors are fed into the neural network, and the flattened vectors are reshaped into the shape of the video block. For a convolutional neural network, the context surrounding the current block is divided into two parts, the two parts are fed into a stack of convolutional layers, the outputs of the convolutional layers are merged, and the result is inserted into a stack of transposed convolutional layers.

2. The method according to claim 1, wherein the partition portion is rectangular.

3. The method of claim 1, wherein the block height is added to the characteristics of the video block.

4. The method of claim 1, wherein blocks are extracted from its reconstruction using characteristics.

5. The method of claim 1, wherein the number of pairs of extracted partition portions is limited.

6. An apparatus for video encoding, the apparatus comprising: Processor, the processor being configured to: A set of neural networks for intra-frame prediction of the video blocks is trained using pairs of video blocks extracted from the partitions and their contexts. Additional pairs of the video blocks and their contexts are extracted by iteratively using the neural network ensemble as additional intra-coding modes for the codec, wherein training samples are encoded using the codec, wherein intra-frame prediction using neural networks is implemented in the codec using an initially trained network, wherein the parameters of each neural network in the neural network ensemble are initialized using neural network parameters obtained at the end of iterations of previous neural networks in the neural network ensemble, and wherein luminance blocks are extracted from the luminance channels of the image containing the video blocks, and their luminance context is extracted from the luminance channels of the reconstructed image of the image, wherein extracting the video blocks by iteratively using the neural network ensemble as additional intra-coding modes for the codec includes: for non-segmented blocks, only blocks whose prediction cost of the neural network mode is less than a threshold multiple of the lowest prediction cost among all intra-frame prediction modes; or for segmented blocks, blocks for which the neural network mode is selected; and, The extracted additional pairs are used to retrain the neural network ensemble to generate a neural network ensemble for intra-frame prediction, wherein the neural network ensemble is retrained on data from an encoder comprising a neural network for intra-frame prediction with fixed parameters; and, The retrained ensemble of neural networks is used to encode video blocks using intra-frame prediction, where... The set of neural networks trained for intra-frame prediction includes: For a fully connected neural network, the context surrounding the current block is flattened into vectors, the flattened vectors are fed into the neural network, and the flattened vectors are reshaped into the shape of the video block. For a convolutional neural network, the context surrounding the current block is divided into two parts, the two parts are fed into a stack of convolutional layers, the outputs of the convolutional layers are merged, and the result is inserted into a stack of transposed convolutional layers.

7. The apparatus according to claim 6, wherein the apparatus is further configured to: Video blocks are encoded using intra-frame prediction with the retrained neural network ensemble.

8. The apparatus according to claim 6, wherein the apparatus is further configured to: Video blocks are decoded using intra-frame prediction with the retrained neural network ensemble.

9. The apparatus according to any one of claims 6-8, wherein the partition portion is rectangular.

10. The apparatus of claim 6, wherein the block height is added to the characteristics of the video block.

11. The apparatus according to any one of claims 6-8, wherein the feature is used to extract blocks from its reconstruction.

12. The apparatus of claim 6, wherein the number of pairs of extracted partition portions is limited.

13. A method for video decoding, the method comprising: A set of neural networks for intra-frame prediction of the video blocks is trained using pairs of video blocks extracted from the partitions and their contexts. Additional pairs of the video blocks and their contexts are extracted by iteratively using the neural network ensemble as additional intra-coding modes for the codec, wherein training samples are decoded using the codec, wherein intra-frame prediction using neural networks is implemented in the codec using an initially trained network, wherein the parameters of each neural network in the neural network ensemble are initialized using neural network parameters obtained at the end of iterations of previous neural networks in the neural network ensemble, and wherein luminance blocks are extracted from the luminance channels of the image containing the video blocks, and their luminance context is extracted from the luminance channels of the reconstructed image of the image, wherein extracting the video blocks by iteratively using the neural network ensemble as additional intra-coding modes for the codec includes: for non-segmented blocks, only blocks whose prediction cost of the neural network mode is less than a threshold multiple of the lowest prediction cost among all intra-frame prediction modes; or for segmented blocks, blocks for which the neural network mode is selected; and, The extracted additional pairs are used to retrain the neural network set to generate a neural network set for intra-frame prediction, wherein the neural network set is retrained on data from the decoder, the decoder including a neural network for intra-frame prediction with fixed parameters. as well as, The video blocks are decoded using intra-frame prediction with the retrained ensemble of neural networks, wherein... The set of neural networks trained for intra-frame prediction includes: For a fully connected neural network, the context surrounding the current block is flattened into vectors, the flattened vectors are fed into the neural network, and the flattened vectors are reshaped into the shape of the video block. For a convolutional neural network, the context surrounding the current block is divided into two parts, the two parts are fed into a stack of convolutional layers, the outputs of the convolutional layers are merged, and the result is inserted into a stack of transposed convolutional layers.

14. The method of claim 13, wherein the partition portion is rectangular.

15. The method of claim 13, wherein the block height is added to the characteristics of the video block.

16. The method of claim 13, wherein blocks are extracted from its reconstruction using characteristics.

17. The method of claim 13, wherein the number of pairs of extracted partition portions is limited.

18. An apparatus for video decoding, the apparatus comprising a memory and a processor, configured to: A set of neural networks for intra-frame prediction of the video blocks is trained using pairs of video blocks extracted from the partitions and their contexts. Further pairs of the video block and its context are extracted by iteratively using the neural network set as an additional intra-frame coding mode for the codec, wherein training samples are decoded using the codec, wherein intra-frame prediction using the neural network is implemented in the codec using an initially trained network, wherein the parameters of each neural network in the neural network set are initialized using neural network parameters obtained at the end of iteration of the previous neural networks in the neural network set, and wherein a luminance block is extracted from the luminance channel of the image containing the video block, and its luminance context is extracted from the luminance channel of the reconstructed image of the image, wherein extracting the video block by iteratively using the neural network set as an additional intra-frame coding mode for the codec includes: For non-segmented blocks, only blocks whose prediction cost of the neural network pattern is less than a threshold multiple of the lowest prediction cost among all intra-frame prediction patterns; or for segmented blocks, blocks in which the neural network pattern is selected; and, The extracted additional pairs are used to retrain the neural network set to generate a neural network set for intra-frame prediction, wherein the neural network set is retrained on data from the decoder, the decoder including a neural network for intra-frame prediction with fixed parameters. as well as, The retrained ensemble of neural networks is used to decode video blocks using intra-frame prediction, where... The set of neural networks trained for intra-frame prediction includes: For a fully connected neural network, the context surrounding the current block is flattened into vectors, the flattened vectors are fed into the neural network, and the flattened vectors are reshaped into the shape of the video block. For a convolutional neural network, the context surrounding the current block is divided into two parts, the two parts are fed into a stack of convolutional layers, the outputs of the convolutional layers are merged, and the result is inserted into a stack of transposed convolutional layers.

19. The apparatus of claim 18, wherein the partition portion is rectangular.

20. The apparatus of claim 18, wherein the block height is added to the characteristics of the video block.

21. The apparatus of claim 18, wherein a block is extracted from its reconstruction using a characteristic.

22. The apparatus of claim 21, wherein the set B of characteristics of the blocks is generated by the partition.

23. The apparatus of claim 18, wherein the number of pairs of extracted partition portions is limited.

24. An apparatus, the apparatus comprising: The apparatus according to any one of claims 6-12; and At least one of the following: (i) an antenna configured to receive a signal including the video block; (ii) a bandwidth limiter configured to limit the received signal to a bandwidth including the video block; and (iii) a display configured to display an output representing the video block.

25. A non-transitory computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1-5 and 13-17.

26. An apparatus for transmitting or receiving a signal, the signal comprising video data for playback using a processor, the video data being generated by the method of any one of claims 1-5 and 13-17 or by the means of any one of claims 6-12 and 18-23.

27. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1-5 and 13-17.

Images