Depth Intra Predictor for Generating Edge Information

Abstract

The present invention proposes at least methods and apparatuses for efficiently encoding or decoding video. For example, at least one neural network using context from pixels including those surrounding an image block is used to determine intra prediction and side information of the image block. The side information allows a decoder to determine the intra prediction and is signaled for decoding.

Description

Technical Field

[0001] At least one of the present embodiments generally relates to a method or apparatus for video encoding or decoding, and more particularly to a method or apparatus that uses a neural network to obtain neural network intra prediction and side information from at least one input data; and encodes or decodes the side information and image blocks using the neural network intra prediction. Background Art

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction including motion dimensional prediction and transformation to exploit spatial and temporal redundancy in video content. Generally speaking, intra or inter prediction is used to exploit intra or inter correlation, and then the difference between the original image block and its prediction (usually expressed as prediction error or prediction residual) is transformed, quantized, and entropy encoded. To reconstruct the video, the compressed data is decoded through the inverse processes corresponding to entropy coding, quantization, transformation, and prediction.

[0003] Recent additions to video compression technology include various industry standards, versions of reference software, and / or documents, such as the Joint Exploration Model (JEM) developed by the JVET (Joint Video Exploration Team) group and the subsequent VTM (Versatile Video Coding (VVC) Test Model). The aim is to further improve the existing HEVC (High Efficiency Video Coding) standard.

[0004] Recent work has introduced deep neural networks in terms of bitrate savings to improve video compression efficiency. For example, a deep intra predictor infers the prediction of a block from the context surrounding the current block to be predicted. According to previous work related to the learning of deep intra predictors for image and video compression, when the learned deep intra predictor is inserted into a video codec, such as HEVC, it is always at least one additional intra prediction mode competing with the existing modes. In fact, due to "failure cases", the intra prediction component of the video codec cannot rely solely on the deep intra predictor. A "failure case" refers to a situation where the prediction quality of the learned deep intra predictor for inferring the prediction of the current block is relatively low compared to the prediction with the best quality among the predictions provided by the conventional intra prediction modes in the video codec of interest. Generally speaking, a "failure case" occurs when the information in the context is insufficient to infer that the prediction quality of the current block is good or, roughly speaking, when the context is not very relevant to the current block. Summary of the Invention

[0005] The deficiencies and drawbacks of the prior art are addressed and handled by the general aspects described herein, which relate to a depth intra predictor that generates side information. According to at least one embodiment, the depth intra predictor generates side information at the encoder side, the side information is written into the bitstream, and the depth intra prediction reads the side information at the decoder side. In this way, information can be transmitted from the encoder to the decoder to supplement the information contained in the context surrounding the current block to be predicted.

[0006] According to a first aspect, a method is provided. The method includes: for an encoded block, using a neural network from at least one input data to determine a neural network intra prediction and side information; encoding the block based on the neural network intra prediction; and encoding the side information.

[0007] According to another aspect, a second method is provided. The method includes for a decoded block, obtaining side information regarding a neural network intra prediction; for a decoded block, using a neural network applied to at least one input data and the side information to determine a neural network intra prediction; and decoding the block using the determined neural network intra prediction.

[0008] According to another aspect, an apparatus is provided. The apparatus includes one or more processors, wherein the one or more processors are configured to: for an encoded block, use a neural network to determine a neural network intra prediction and side information from at least one input data; encode the block based on the neural network intra prediction; and encode the side information.

[0009] According to another aspect, another apparatus is provided. The apparatus includes one or more processors, wherein the one or more processors are configured to: for a decoded block, obtain side information regarding a neural network intra prediction; for a decoded block, use a neural network applied to at least one input data and the side information to determine a neural network intra prediction; and decode the block using the determined neural network intra prediction.

[0010] According to another general aspect of at least one embodiment, the neural network is fully connected, and the side information Z is a vector whose coefficients belong to [0,1]. In a variant, the vector Z is transformed into a vector of bits by applying an element-wise threshold of 0.5 to the vector Z. In another variant, the element-wise threshold is different from 0.5.

[0011] According to another general aspect of at least one embodiment, the neural network is convolutional, and wherein the side information Z is a stack of feature maps whose coefficients belong to [0,1]. In a variant, the stack of feature maps Z is transformed into a stack of bit feature maps by applying an element-wise threshold of 0.5 to Z. In another variant, the element-wise threshold is different from 0.5.

[0012] In another general aspect according to at least one embodiment, the side information Z is a vector or a stack of feature maps whose coefficients belong to R, and the coefficients of the side information are quantized by scalar quantization or vector quantization. Then, the absolute value of each quantized coefficient is encoded by arithmetic coding. CABAC contexts can be used for the arithmetic coding of the absolute value of each quantized coefficient.

[0013] In another general aspect according to at least one embodiment, the side information Z is a vector or a stack of feature maps whose coefficients belong to [-1, 1], and the coefficients of the side information are mapped to {0, 1}.

[0014] In another general aspect according to at least one embodiment, for a luminance block Y, the encoded input data includes X c the context surrounding the current luminance block Y and the luminance block Y.

[0015] In another general aspect according to at least one embodiment, for a luminance block Y, the decoded input data includes the context X surrounding the current luminance block Y c and the decoded side information.

[0016] In another general aspect according to at least one embodiment, for a chrominance block Y CbCr , the encoded input data includes the context X surrounding the chrominance block Y CbC r, the context X surrounding the luminance block Y juxtaposed with the chrominance block c CbCr , the context X surrounding the luminance block Y juxtaposed with the chrominance block Y and the current chrominance block Y c Y . CbCr .

[0017] In another general aspect according to at least one embodiment, for a chrominance block Y CbCr , the encoded input data includes the context X surrounding the luminance block Y juxtaposed with the chrominance block Y and the current chrominance block Y c Y . CbCr .

[0018] In another general aspect according to at least one embodiment, for a chrominance block Y CbCr , the encoded input data includes the context X surrounding the chrominance block Y CbCr , the context X surrounding the luminance block Y juxtaposed with the chrominance block c CbCr , the context X surrounding the luminance block Y juxtaposed with the chrominance block Y , the current chrominance block Y c Y , the current chrominance block Y CbCr and the reconstructed luminance block

[0019] In another general aspect according to at least one embodiment, for chrominance block Y CbCr , the decoded input data includes context X CbCr surrounding chrominance block Y c CbCr , context X Y surrounding luminance block Y c Y adjacent to the chrominance block, and decoded side information.

[0020] In another general aspect according to at least one embodiment, for chrominance block Y CbCr , the decoded input data includes context X Y surrounding luminance block Y c Y adjacent to the chrominance block, and decoded side information.

[0021] In another general aspect according to at least one embodiment, for chrominance block Y CbCr , the decoded input data includes context X CbCr surrounding chrominance block Y c CbCr , context X Y surrounding luminance block Y c Y adjacent to the chrominance block, the reconstructed luminance block , and decoded side information.

[0022] In another general aspect according to at least one embodiment, the input data further includes the intra prediction mode L of the block located to the left of the encoded / decoded block and the intra prediction mode A of the block located above the encoded / decoded block.

[0023] In another general aspect according to at least one embodiment, for intra prediction of a luminance block, a syntax element DeepFlag (i.e., selecting an intra prediction mode based on a neural network to predict the current block) representing the intra prediction mode based on a neural network is encoded among the syntax elements of the intra prediction.

[0024] In another general aspect according to at least one embodiment, for intra prediction of a luminance block, the intra prediction mode based on a neural network is always selected. This means that all the conventional syntax elements of luminance intra prediction and DeepFlag do not exist.

[0025] According to another general aspect of at least one embodiment, for intra prediction of a luminance block, a syntax element DeepFlag representing a neural network-based intra prediction mode is encoded, and if DeepFlag is set to one, intra prediction is performed using the neural network-based intra prediction mode, otherwise if DeepFlag is set to zero, intra prediction of the luminance block is performed using the planar mode.

[0026] According to another general aspect of at least one embodiment, for intra prediction of a chrominance block, the neural network-based intra prediction mode is always selected.

[0027] According to another general aspect of at least one embodiment, for intra prediction of a chrominance block, a syntax element DeepFlag representing the neural network-based intra prediction mode is encoded among the syntax elements of intra prediction.

[0028] According to another general aspect of at least one embodiment, there is provided an apparatus comprising: means according to any one of the decoding embodiments; and at least one of the following: (i) an antenna configured to receive a signal including a video block; (ii) a band limiter configured to limit the received signal to a band including the video block; or (iii) a display configured to display an output representing the video block.

[0029] According to another general aspect of at least one embodiment, there is provided a non-transitory computer-readable medium containing data content generated according to any one of the described encoding embodiments or variants.

[0030] According to another general aspect of at least one embodiment, there is provided a signal including video data generated according to any one of the described encoding embodiments or variants.

[0031] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any one of the described encoding embodiments or variants.

[0032] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions that, when executed by a computer, cause the computer to perform any one of the described encoding embodiments or variants.

[0033] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of the exemplary embodiments when read in conjunction with the accompanying drawings. Description of the Drawings

[0034] In the accompanying drawings, examples of several embodiments are shown.

[0035] Figure 1 Examples of reference samples for intra prediction in VVC are shown.

[0036] Figure 2 Prediction directions for square blocks in VCC are shown.

[0037] Figure 3 Examples of top and left CU positions for deriving the intra MPM list for different block shapes in VVC are shown.

[0038] Figure 4 Represents a decision tree showing the intra prediction signaling for luminance in VVC.

[0039] Figure 5 Represents a decision tree showing the intra prediction signaling for chrominance in VVC.

[0040] Figure 6a Examples of the context of the current block to be predicted using a neural network are shown.

[0041] Figure 6b Examples of intra prediction by a fully connected neural network that can implement various aspects of the embodiments are shown.

[0042] Figure 6c Examples of intra prediction by a convolutional neural network that can implement various aspects of the embodiments are shown.

[0043] Figure 7 A general coding method according to at least one embodiment is shown.

[0044] Figure 8 A general decoding method according to at least one embodiment is shown.

[0045] Figure 9 、 Figure 10 、 Figure 12 、 Figure 13 、 Figure 14 、 Figure 15 、 Figure 16 、 Figure 17 、 Figure 18 Different variants of a neural network-based intra prediction method using side information in a video encoder and a video decoder according to at least one embodiment are shown.

[0046] Figure 11 Non-limiting examples of the conversion operation of stacking feature maps of bits into a bitstream according to at least one embodiment are shown.

[0047] Figure 19 and 20 shows different variants of a decision tree showing intra prediction signaling according to at least one embodiment.

[0048] Figure 21 shows a block diagram of an embodiment of a video encoder that can implement various aspects of the embodiments.

[0049] Figure 22 shows a block diagram of an embodiment of a video decoder that can implement various aspects of the embodiments.

[0050] Figure 23 shows a block diagram of an example apparatus in which various aspects and embodiments can be implemented. Detailed Description

[0051] We first introduce the intra prediction component of the video codec and then disclose various embodiments of the depth intra predictor implemented in the encoder or decoder according to the principles of the present invention. Finally, various embodiments of the encoder or decoder implementing the principles of the present invention are presented.

[0052] The principles of the present invention are described in the context of the video codec H.266 / VVC, as it is currently considered the best video codec in terms of compression performance. However, the principles of the present invention are compatible with any codec.

[0053] The intra prediction process in H.266 / VVC consists of: collecting reference samples, processing the reference samples, deriving the actual prediction of the samples of the current block, and finally post-processing the predicted samples.

[0054] Figure 1 Illustrates, in the case of a square current block, the reference samples used for intra prediction in H.266 / VVC (W = H = N), where the pixel value at coordinates (x, y) is Figure 1 denoted by P(x, y). An array of 2W samples for the top is formed from the previously reconstructed top and top-right pixels to the current block, where W represents the block width. Similarly, 2H samples for the left side are formed from the reconstructed left and bottom-left pixels, where H represents the block height. The corner pixel at the top-left position is also used to fill the gap between the top row and the left column of references. If some of the samples for the top or left side are not available, due to the corresponding coding unit (CU) not being in the same slice or the current CU being at the frame boundary, then a method called reference sample replacement is performed, where the missing samples are copied clockwise from the available samples. Then, the reference samples are filtered using a specified filter according to the current CU size and prediction mode.

[0055] H.266 / VVC includes the range of prediction models that are derived from those in H.265 / HEVC. Planar and DC prediction modes are used to predict smooth and gradually changing regions, while angular prediction modes are used to capture different directional structures. There are 65 directional prediction modes, which are organized differently for each rectangular block shape. Figure 2 Shows the prediction directions for square blocks in H.266 / VCC. The prediction modes correspond to different prediction directions, as Figure 2 shown.

[0056] Intra prediction is further extended using tools such as intra prediction with multiple reference lines (MRL), intra prediction with sub-partitions (ISP), and Matrix intra prediction (MIP). MIP is a set of intra prediction modes, and each intra prediction mode infers the prediction of the current block from the reconstructed pixels through a linear transformation. For 4×4 blocks, there are 35 modes. For 4×8 blocks, 8×4 blocks, and 8×8 blocks, there are 19 modes. For other blocks, 11 modes are used.

[0057] The intra prediction mode is signaled from the encoder to the decoder. For the luminance channel, the signaling of the planar mode, DC mode, and 65 directional modes is first described, omitting the signaling of MRL, ISP, and MIP. The signaling of these last three will then be described in detail.

[0058] On the encoder side, the best intra prediction mode belonging to a set including the planar mode, DC mode, and 65 directional modes is selected according to the rate-distortion criterion, and its index is transmitted from the encoder to the decoder. To perform the signaling of the selected mode index through entropy coding, a list of the most probable modes (MPM) is established.

[0059] In VTM, the MPM list contains 6 intra prediction modes for signaling the intra prediction mode of the current block. The MPM list is created from the prediction modes of the intra-coded CUs at the top and left of the current CU and some default modes. Figure 3 Shows the top and left CUs at the right and bottom edges of the current block for deriving the MPM list, where:

[0060] L is the prediction mode of the left CU (a value in the range [0 - 66])

[0061] A is the prediction mode of the upper CU (a value in the range [0 - 66]).

[0062] The derivation of the MPM list is as follows (where the 2 variables offset = 61 and mod = 64):

[0063] Initialization of the MPM list:

[0064]

[0065]

[0066] Using circular adjacency within the range [2 - 66] can be equivalently written as

[0067] ((L + offset) % mod) + 2 ≡ L - 1

[0068] ((L + offset - 1) % mod) + 2 ≡ L - 2

[0069] ((L - 1) % mod) + 2 ≡ L + 1

[0070] ((L - 0) % mod) + 2 ≡ L + 2

[0071] Using the above relationships, it can be shown that the MPM list derivation is the content in Table 1.

[0072] Table 1: MPM derivation in VTM. A and L represent the prediction modes above and to the left of the CU, respectively

[0073]

[0074] If the intra prediction for predicting the current block corresponds to one of the six MPM modes, this is signaled by an mpmFlag having a value of 1, and subsequently the candidate modes in the MPM list are signaled by the variable - length coding scheme shown in Table 2. Otherwise, the mpmFlag is equal to 0, and the candidate indices in the set of the remaining 61 modes are truncated binary coded using 5 bits or 6 bits.

[0075] Table 2: MPM signaling in VTM

[0076] Candidate Index Code MPM[0] 0 MPM[1] 10 MPM[2] 110 MPM[3] 1110 MPM[4] 11110 MPM[5] 11111

[0077] For intra prediction using MRL, the reference lines for prediction are signaled by the marker multiRefIdx. The valid values of multiRefIdx are 0, 1, and 3, which signal the first reference line, the second reference line, and the fourth reference line respectively. When multiRefIdx is non - zero (meaning the second reference line or the fourth reference line is used), the prediction mode always belongs to the MPM list. Therefore, the mpmFlag is not encoded. Additionally, the planar is excluded from the list. This means that when multiRefIdx is non - zero, only five prediction modes are available as possible candidates. Therefore, when multiRefIdx is non - zero, the prediction modes are signaled as shown in Table 3.

[0078] Table 3: MPM signaling when multiRefIdx > 0 in VTM

[0079] Candidate Index Code MPM[1] 0 MPM[2] 10 MPM[3] 110 MPM[4] 1110 MPM[5] 1111

[0080] For the ISP, the partitioning type for the CU is signaled using a flag called ispMode. The ispMode is coded only when multiRefIdx is equal to 0. The valid values of ispMode are 0, 1, and 2, signaling no partitioning, horizontal partitioning, and vertical partitioning, respectively.

[0081] Now regarding MIP, first, the MIP mode is signaled using a flag called mipFlag, where a value of 1 means the MIP mode is used for predicting the current block, and 0 means one of the 67 intra prediction modes is used. When mipFlag is equal to 1, multiRefIdx must be equal to 0, meaning the first reference line is used, and ispMode is equal to 0, i.e., there is no target CU partition. Therefore, when mipFlag is equal to 1, multiRefIdx and ispMode are not written to the bitstream. If mipFlag is equal to 1, the index of the selected MIP mode is truncated binary coded in the latest version of VTM.

[0082] To handle the case where the intra prediction mode used to predict the current block is one of the 67 intra prediction modes and the selected mode used to predict the top CU or the selected mode used to predict the left CU is the MIP mode, the mapping between each MIP mode and one of the regular modes enables this MIP mode to be replaced with its mapped regular mode. In the latest version of VTM, any MIP mode is mapped to planar.

[0083] Figure 4 A decision tree representing the intra prediction signaling for luminance in VVC is shown.

[0084] In Figure 4 , the lightly shaded flags indicate that the value of the flag is derived from the value of a previous flag that is written to the bitstream on the encoder side and read from the bitstream on the decoder side. This means that the lightly shaded flags are not written to the bitstream on the encoder side; these flags are not read from the bitstream on the decoder side.

[0085] Crucially, some flags are at the CU level while others are at the PU level. For CU-level flags, the value of the flag applies to the PUs within the CU. For PU-level flags, each PU within the CU has its own flag value. If the flag is at the PU level, in Figure 4 when the decision tree reaches the binary decision associated with that flag, the value of that flag for each PU is written to the bitstream before moving to the subsequent binary decision. Table 4 shows the level of each flag represented in Figure 4 .

[0086] Table 4: The levels of each flag involved in the intra prediction signaling in VTM

[0087] Flag Level mipFlag CU multiRefIndex PU ispMode CU mpmFlag PU

[0088] For the two chrominance channels, neither MRL, ISP nor MIP is used. Instead, two specific tools are used: direct mode and component - to - component linear model (CCLM).

[0089] Figure 5 Figure showing the decision tree for intra - prediction signaling for chrominance in VVC. The direct mode corresponds to applying the selected mode used to predict the neighboring luma block to the prediction of the current chrominance block. If directFlag is equal to 1, the direct mode is selected for predicting the current chrominance block. Otherwise, directFlag is equal to 0, and a mode from the list L = [Planar, Vertical, Horizontal, DC] is selected. If the mode in L is equal to the direct mode, this mode is replaced by the mode with index 66. In CCLM, a linear model predicts the current chrominance block from the reconstructed luma reference samples that surround the neighboring luma block. The parameters of the linear model are derived from the reconstructed reference samples. There are three CCLM modes, each associated with a different derivation of the parameters. If cclmFlag is equal to 1, one of the three CCLM modes is selected. In this case, directFlag is not written to the bitstream. Otherwise, cclmFlag is equal to 0, and either the direct mode or a mode from L is selected.

[0090] As previously explained, recent video codecs introduce neural - network - based intra - prediction. The deep intra - predictor infers the prediction from the context surrounding the current block to be predicted. Figure 6a Figure showing an example of the context surrounding the current block to be predicted using a neural network. Context X c is composed of the reconstructed pixels at the top and top - right, and to the left and bottom - left of the current block Y, similar to a set of reconstructed reference samples for intra - prediction in H.266 / VVC. However, different from it, context X c extends towards the left and the top, as Figure 6a shown. Due to this extension, the deep intra - predictor can learn the relationship between the spatial correlations in its input context and the prediction it gives. Note that the subscript "c" in X c indicates that the reconstructed pixels in the context have been pre - processed, as detailed later.

[0091] Figure 6bShows an example of intra prediction by a fully connected neural network that can implement various aspects of the embodiments. If the depth intra predictor is fully connected, the context is typically flattened into a vector, and the resulting vector is fed into the neural network. Then, the vector provided by the neural network is reshaped into the shape of the current block, thus producing a prediction As Figure 6b shown. Note that the subscript "c" in

[0092] Figure 6c indicates that the predicted pixel has not been post-processed, which will be explained later. During the preprocessing step, the context X c can be obtained by subtracting the average pixel intensity α from the original context X of the reconstructed pixels.

[0093] X c = X - α

[0094] Then, during the postprocessing step, the postprocessed prediction Y of the current block Y is calculated by adding the average pixel intensity and clipping to the prediction where b represents the pixel bit depth. As an alternative, α can also be the average pixel intensity over a large set of training images.

[0095]

[0096]

[0097] ​In video codecs such as H.265 / HEVC and H.266 / VVC, an image is divided into Coding Tree Units (CTUs). In raster scan order, one CTU is processed at a time. Each CTU can be hierarchically divided into Coding Units (CUs). The CUs within a CTU are processed in Z-scan order. Thus, for example, in H.265 / HEVC, the size of the block to be predicted can be 64×64, 32×32, 16×16, 8×8, or 4×4. This means that 5 neural networks are required, one for each block size to be predicted. Thus, the depth intra predictor mode consists of 5 neural networks. In H.266 / VVC, when the hierarchical separation is more complex, the sizes of the blocks to be predicted are 128×128, 64×64, 32×32, 16×16, 8×8, or 4×4. Additionally, the blocks can also be rectangular, e.g., of size 4×8. In this case, the solution is to allocate one neural network for each block size to establish the depth neural network mode.

[0098] In different works integrating the depth intra prediction mode into video codecs, typically for H.265 / HEVC, the depth intra prediction mode systematically competes with the existing modes. For the current block to be predicted, a flag is written to the bitstream before all other flags used for intra prediction. The value 1 indicates that the depth intra prediction mode is selected to predict the current block. In this case, no other flags used for intra prediction are written to the bitstream. The value 0 means that one of the conventional intra predictions is selected. In this case, the conventional flags used for intra prediction are then written to the bitstream.

[0099] Note that the signaling mentioned above has been implemented in H.265 / HEVC. No method has been proposed in H.266 / VVC. It is worth noting that it is not clear how to handle the flags mipFlag, multiRefIdx, and ispMode when the depth neural network mode is selected.

[0100] According to at least one general implementation of the principles of the present invention, instead of having a depth neural network mode that competes with the existing intra prediction modes, the depth neural network mode alone takes over the intra prediction component of the video codec. According to a specific feature, the depth neural network generates side information that is transmitted from the encoder to the decoder. According to another specific feature, on the decoder side, the depth intra predictor infers the prediction of the current block from the context surrounding the current block to be predicted and the side information.

[0101] Figure 7Shows a general coding method according to at least one embodiment. As various embodiments described in detail below, once the pictures of a video are partitioned into blocks for coding, the coding method 10 includes determining 12 neural network intra prediction and side information from at least one input data using a neural network. At the coding side, the neural network uses not only the context surrounding the block being coded as input, but also the block itself as input. Then, based on the neural network intra prediction, the block is coded 16, where as a non - limiting example of the steps, the coding further includes obtaining a residual block that is subsequently transformed and quantized. In step 14, the side information of the intra prediction neural network is also coded and transmitted to the decoder, thus advantageously enabling the decoder to avoid "failure cases" in the neural network intra prediction. Steps 14 and 16 are performed in any order or in parallel.

[0102] Figure 8 Shows a general decoding method according to at least one embodiment. As various embodiments described in detail below, the decoding method 20 includes, for a block decoded in a picture of a video, obtaining 22 the side information for neural network intra prediction. According to a non - limiting example, the side information is received and decoded using optionally transformed and quantized as block data. Then, in step 24, a neural network is used to determine the neural network intra prediction, which uses the side information together with other input data (such as the context surrounding the decoded block) as input, where the context includes available pixels previously reconstructed in the video picture. As with classical decoding methods, the block is reconstructed 26 based on the neural network intra prediction.

[0103] Below, different variants of a neural network - based intra prediction method based on side information in a video encoder and a video decoder according to at least one embodiment are presented. Thus, Figure 7 different variants of the coding method and Figure 8 the decoding method are thus shown below. The neural network architecture is divided into two parts: an encoder part and a decoder part. In the encoder part, the neural network can access not only the context X surrounding the current luma block Y c , but also Y. That is, the encoder architecture takes them as inputs to generate side information. The side information is written into the bitstream. In the decoder part, the neural network employs the side information read from the bitstream and the context X c to provide a prediction of the current luma block Figure 9 and Figure 10 Show respectively two non - limiting examples of the instantiation of the above - proposed for a fully - connected architecture and a convolutional architecture. In both cases, the last layer of the encoder architecture contains a sigmoid non - linearity.

[0104] Figure 9Illustrated is a depth intra predictor for predicting a luminance block by generating side information using a fully-connected architecture. The horizontal dashed line depicts the encoder part and the decoder part of the architecture. When context is available on both sides, the broken dashed line crosses the boundary between the encoder side and the decoder side. Thus, for the fully-connected architecture, the side information is a vector Z whose coefficients belong to [0, 1]. By applying a 0.5 element-wise threshold to Z, this vector is converted into a bit vector that is written to the bitstream.

[0105] Figure 10 Illustrated is a depth intra predictor for predicting a luminance block by generating side information using a convolutional architecture. The horizontal dashed line depicts the encoder part and the decoder part of the architecture. When context is available on both sides, the broken dashed line crosses the boundary between the encoder side and the decoder side. For the convolutional architecture, Z is a stack of feature maps whose coefficients belong to [0, 1]. The same threshold enables its conversion into a stack of feature maps of bits, and this stack of feature maps is written to the bitstream in raster scan order. Figure 11 Illustrated is a non-limiting example of the conversion operation of a stack of feature maps of bits to a bitstream, where these bits are written or read from the bitstream in raster scan order. In this case, on the decoder side, when reading the bitstream, a stack of feature maps of bits is reconstructed in raster scan order.

[0106] According to the first variant, the values of the output vector Z are unrestricted and belong to R. Figure 12 Illustrated is an example of a depth intra predictor that generates side information using a fully-connected architecture in the case of the first variant. Figure 13 Illustrated is an example of a depth intra predictor that generates side information using a convolutional architecture in the case of the first variant. In this variant, the sigmoid non-linearity is removed from the last layer of the encoder part of the depth intra predictor that generates side information. Since this layer has no non-linearity, the coefficients in its output Z belong to R. For the transform coefficients in H.265 / HEVC and H.266 / VVC, the coefficients in Z are quantized by uniform scalar quantization, as Figure 12 and 13 shown. Then, the sign of each quantized coefficient is losslessly encoded, and the absolute value is losslessly encoded by binary arithmetic coding. The CABAC context model can be used for the binary arithmetic coding of the absolute values of the quantized coefficients.

[0107] According to a second variant, the values Z of the output vector are binarized and belong to [-1, 1]. In this second variant, the sigmoid non-linearity in the last layer of the encoder part of the architecture is replaced by a tangent hyperbolic non-linearity. The coefficients in its output Z thus belong to [-1, 1]. According to yet another second variant, the output vector is converted into a bit vector by applying an element-wise threshold of 0.5 to the vector. In this case, an element-wise threshold of 0 at Z is applied, resulting in a vector in {-1, 1} for a fully-connected architecture and a stack of feature maps with coefficients belonging to {-1, 1} for a convolutional architecture. When writing to and reading from the bitstream, each coefficient equal to -1 is mapped to 0 and vice versa.

[0108] Figure 14 Shows the generation of side information using a fully-connected architecture for a depth intra-predictor for predicting a chrominance block. Figure 15 Shows the generation of side information using a convolutional architecture for a depth intra-predictor for predicting a chrominance block. The annotations are the same as in the previous embodiment. The encoder part of the deep neural network is fed the context X CbCr surrounding the current chrominance block Y c CbCr , the context X Y surrounding the luma block Y c Y adjacent to the current chrominance block Y CbCr , to generate side information. The decoder part of the deep neural network uses the side information read from the bitstream, X c CbCr and X c Y to provide a prediction of Y CbCr . Figure 14 Is shown in Figure 9 the adaptation for the case of a given chrominance block to be predicted in 4:2:0. Figure 15 Is depicted in Figure 10 the adaptation for the case of a given chrominance block to be predicted in 4:2:0.

[0109] According to a first variant of the prediction of the chrominance block, the prediction of the chrominance block depends only on the luma context. In the first variant, only the context X CbCr surrounding the luma block Y Y adjacent to the current chrominance block and the current chrominance block Y c Y is fed into the encoder part of the deep neural network. The decoder part of the deep neural network uses the side information read from the bitstream and X c Y to give a prediction of Y CbCr .

[0110] According to a second variant of the prediction of a chrominance block, the prediction of the chrominance block depends on the reconstructed luma block. When predicting the current chrominance block, the luma block juxtaposed with the current chrominance block has been encoded and decoded. Figure 16 An example is shown of generating side information for a depth intra predictor for predicting a current chrominance block using a fully-connected architecture when feeding the luma block juxtaposed with the current chrominance block to the depth intra predictor on the encoder side and the decoder side. In this second variant, the encoder part of the depth intra predictor takes a context X surrounding the current chrominance block c CbCr and a context X surrounding the luma block juxtaposed with the current chrominance block c CbCr along with the current chrominance block Y CbCr and the reconstruction of this luma block to generate side information. The decoder part of the deep neural network takes the side information read from the bitstream, X c CbCr and X c Y and to provide a prediction of Y CbCr

[0111] According to yet another variant, the intra prediction modes of the top and left CUs are also used as input data for predicting a luma block or a chrominance block. In this variant, the intra prediction mode (L) of the block located to the left of the block being encoded or decoded and the intra prediction mode (A) of the block located above the block being encoded or decoded are used as input data for the neural network. The intra prediction modes of the CU located to the left of the current CU and the CU located above the current CU (which are denoted as L and A in Figure 3 can be inserted into the architecture presented in Figure 9 and 10 .

[0112] Figure 17 An example is shown of generating side information for a depth intra predictor for predicting a chrominance block using a fully-connected architecture when feeding the intra prediction mode of the CU located to the left of the current CU and the intra prediction mode of the CU located above the current CU to the depth intra predictor.

[0113] Figure 18 An example is shown of generating side information for a depth intra predictor for predicting a chrominance block using a convolutional architecture when feeding the intra prediction mode of the CU located to the left of the current CU and the intra prediction mode of the CU located above the current CU to the depth intra predictor.

[0114] ​These two intra prediction modes L and A are fed into the architectures on the encoder side and the decoder side. In this way, on the encoder side, the deep neural network can generate more compressible side information by removing some redundancy between the side information it calculates and {A, L}.

[0115] According to yet another advantageous feature, the enabling / disabling of the deep intra predictor that generates side information is signaled from the encoder to the decoder. When the deep intra predictor that generates side information advantageously allows for separate processing of intra prediction, it replaces 67 intra prediction modes, MRL, ISP, and MIP. This means that on the encoder side, there are no more options to select the best intra prediction mode according to the rate-distortion criterion. In this variant, the prediction is always determined according to the deep intra prediction mode. In a given step of the hierarchical partitioning in H.266 / VVC, for a given luminance block to be predicted, the deep intra predictor is systematically selected for predicting the current luminance block. The signaling cost of intra prediction comes from the side information specifically generated by the deep intra predictor.

[0116] In the first variant of the above method, the deep intra prediction mode still competes with the intra prediction modes in H.266 / VVC. It will replace MIP in H.266 / VVC. This means that MIP is suppressed and mipFlag becomes deepFlag. Figure 19 A non-limiting example of a decision tree showing intra prediction signaling according to at least one embodiment is shown. If deepFlag is equal to 1, the deep intra predictor is selected for predicting the current luminance block. Otherwise, deepFlag is equal to 0, and one of the 67 intra prediction modes is used. When deepFlag is equal to 1, multiRefIdx must be equal to 0, meaning that the first reference line is used, and ispMode is equal to 0, i.e., there is no target CU partition. Therefore, when deepFlag is equal to 1, multiRefIdx and ispMode are not written into the bitstream. deepFlag has a CABAC context model that has two neighborhoods: the intra prediction mode selected for predicting the left PU and the intra prediction mode for selecting the above PU. Figure 19 Summarizes the decision sequence for intra prediction signaling through the decision tree. Here, deepFlag is at the CU level.

[0117] In the second variant of the above method, the intra prediction in H.266 / VVC is based on the deep intra predictor and is planar-unique. Figure 20Shows a non - restrictive example of a decision tree showing intra - frame prediction signaling according to this variant. At this time, rate - distortion optimization is maintained, but its candidates are limited to planar and depth intra - frame prediction modes. During rate - distortion optimization, fast - passing is suppressed, which means that two candidates are systematically tested for the current luminance block to be predicted. The deepFlag is written into the bitstream to indicate the selection of the depth intra - frame prediction mode for predicting the current luminance block. If deepFlag is equal to 1, the depth intra - frame predictor is selected. Then, the side information generated by it is written into the bitstream. Otherwise, as Figure 20 shown, the planar is selected. Note that in this second variant, MRL and ISP are removed. As in the first variant, deepFlag has a CABAC context model with two neighborhoods.

[0118] For luminance, when the depth intra - frame predictor that generates side information advantageously allows for separate processing of intra - frame prediction, it replaces CCLM, direct mode, and four other modes. In a given step of hierarchical partitioning in H.266 / VVC, for a given chrominance block to be predicted, the depth intra - frame predictor is systematically selected for predicting the current chrominance block. The signaling cost of intra - frame prediction comes from the side information specifically generated by the depth intra - frame predictor.

[0119] In the first variant, the depth intra - frame prediction mode still competes with other tools for chrominance intra - frame prediction in H.266 / VVC. The intra - frame prediction signaling for chrominance is Figure 5 one of. In the list of four modes for chrominance prediction L, the horizontal mode is replaced by the deep neural network mode. If the direct mode is equal to one of the four modes in L, that mode is replaced by the horizontal mode.

[0120] This application describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described,

[0121] and at least show individual characteristics, usually described in a way that may sound limited. However, this is for clarity of description and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with the aspects described in previous submissions.

[0122] The aspects described and contemplated in this patent application can be implemented in many different forms. The following Figure 21 , Figure 22 and Figure 23 provide some embodiments, but other embodiments are contemplated, and Figure 21 , Figure 22 and Figure 23The discussion is not limited to the breadth of specific implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting the generated or encoded bitstream. These and other aspects can be implemented as methods, apparatuses, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the methods stored thereon.

[0123] In this application, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture", and "frame" may be used interchangeably. Generally, but not necessarily, the term "reconstruction" is used at the encoding end, while "decoding" is used at the decoding end.

[0124] Various methods are described herein, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires steps or actions in a specific order, the order and / or use of specific steps and / or actions may be modified or combined.

[0125] The various methods and other aspects described in this patent application can be used to modify the modules of video encoder 100 and decoder 200 (e.g., the intra prediction, entropy encoding, and / or decoding modules (160, 360, 145, 330)), as Figure 21 and Figure 22 shown. Furthermore, the aspects of the present invention are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or future-developed) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0126] Various numerical values are used in this application, for example, the number of intra prediction modes, the parameters of the neural network (layers, output range, thresholds). The specific values are for illustrative purposes, and the aspects are not limited to these specific values.

[0127] Figure 21 Encoder 100 is shown. Variants of this encoder 100 are envisioned, but for clarity, encoder 100 is described below without describing all the expected variants.

[0128] Before encoding, the video sequence may undergo pre - encoding processing (101). For example, a color transformation is applied to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or remapping of the input picture components is performed to obtain a signal distribution that is more resilient to compression (e.g., histogram equalization of one of the color components in the color components). Metadata may be associated with the pre - processing and appended to the bitstream.

[0129] In the encoder 100, the picture is encoded by the encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units such as CUs, for example. Each unit is encoded using, for example, an intra - frame mode or an inter - frame mode. When a unit is encoded in the intra - frame mode, it performs intra - frame prediction (160). In the inter - frame mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of the intra - frame mode or the inter - frame mode is used to encode the unit, and indicates the intra - frame / inter - frame decision by, for example, a prediction mode flag. The prediction residual is calculated, for example, by subtracting (110) the prediction block from the original image block.

[0130] Then the prediction residual is transformed (125) and quantized (130). The quantized transform coefficients, motion vectors, and other syntax elements are entropy - encoded (145) to output a bitstream. The encoder may skip the transformation and directly apply quantization to the untransformed residual signal. The encoder may bypass both the transformation and quantization, that is, directly encode the residual without applying the transformation or quantization process.

[0131] The encoder decodes the encoded block to provide a reference for further prediction. The quantized transform coefficients are de - quantized (140) and inverse - transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. A loop filter (165) is applied to the reconstructed picture to perform, for example, de - blocking / SAO (Sample Adaptive Offset) filtering to reduce encoding artifacts. The filtered image is stored in the reference picture buffer (180).

[0132] Figure 22 A block diagram of the video decoder 200 is shown. In the decoder 200, the bitstream is decoded by the decoder elements as described below. The video decoder 200 generally performs a decoding process that is the reverse of the encoding process as Figure 21 described. The encoder 100 typically also performs video decoding as part of encoding the video data.

[0133] Specifically, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. First, the bitstream is entropy decoded (230) to obtain transform coefficients, motion vectors, and other encoded information. The picture partitioning information indicates how the picture is partitioned. Thus, the decoder can partition (235) the picture according to the decoded picture partitioning information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (255) to reconstruct the image block. The prediction block can be obtained (270) from intra prediction (260) or motion compensated prediction (i.e., inter prediction) (275). A loop filter is applied (265) to the reconstructed image. The filtered image is stored in the reference picture buffer (280).

[0134] The decoded picture may also be subject to post - decoding processing (285), e.g., an inverse color transformation (e.g., a transformation from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the remapping process performed in the pre - encoding processing (101). The post - decoding processing may use metadata derived in the pre - encoding processing and signaled in the bitstream.

[0135] Figure 23 A block diagram shows an example of a system in which various aspects and embodiments are implemented. System 1000 may be embodied as a device including the various components described below and is configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smart phones, tablets, digital multimedia set - top boxes, digital television receivers, personal video recording systems, connected household appliances, and servers. The elements of system 1000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0136] System 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing various aspects as described, for example, in this document. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 1000 includes at least one memory 1020 (e.g., volatile memory devices and / or non-volatile memory devices). System 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, the storage device 1040 may include internal storage devices, attached storage devices (including detachable and non-detachable storage devices), and / or network-accessible storage devices.

[0137] System 1000 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 1030 may be implemented as a stand-alone element of System 1000 or may be incorporated within the processor 1010 as a combination of hardware and software known to those skilled in the art.

[0138] The program code to be loaded onto the processor 1010 or encoder / decoder 1030 to execute the various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more of the various items during the execution of the processes described in this document. Such stored items may include but are not limited to input video, decoded video or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operation logic.

[0139] In some embodiments, the memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory can be the memory 1020 and / or the storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, fast external dynamic volatile memory such as RAM is used as the working memory for video encoding and decoding operations, such as MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Exploration Team (JVET)).

[0140] Inputs to the elements of the system 1000 can be provided through various input devices as shown in block 1130. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives, for example, RF signals transmitted over the air by a broadcaster; (ii) component (COMP) input terminals (or a set of COMP input terminals); (iii) universal serial bus (USB) input terminals; and / or (iv) high-definition multimedia interface (HDMI) input terminals. Figure 23 Other examples not shown include composite video.

[0141] In various embodiments, the input device of block 1130 has corresponding input processing elements associated therewith as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal band to one band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower band to select a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, filters, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various functions of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In one set-top box embodiment, the RF section and its associated input processing elements receive an RF signal transmitted through a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, e.g., inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF section includes an antenna.

[0142] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting system 1000 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, as needed, e.g., within a separate input processing IC or within processor 1010. Similarly, aspects of USB or HDMI interface processing may be implemented, as needed, within a separate interface IC or within processor 1010. The demodulated, error-corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data stream as needed for presentation on an output device.

[0143] The various elements of system 1000 may be disposed within an integrated housing. Within the integrated housing, the various elements may be interconnected using a suitable connection arrangement 1140 (e.g., an internal bus known in the art, including an inter-integrated circuit (I2C) bus, wiring, and a printed circuit board) and data may be transmitted therebetween.

[0144] System 1000 includes a communication interface 1050 capable of communicating with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0145] In various embodiments, data is streamed or otherwise provided to system 1000 using a wireless network such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signals of these embodiments are received via the communication channel 1060 and the communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or a router that provides access to an external network including the Internet for allowing streaming applications and other cloud-based communications. Other embodiments use a set-top box to provide streaming data to system 1000, and the set-top box delivers data via an HDMI connection of the input block 1130. Still other embodiments use an RF connection of the input block 1130 to provide streaming data to system 1000. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0146] System 1000 may provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 may be used for a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), or other devices. The display 1100 may also be integrated with other components (e.g., as in a smart phone) or be separate (e.g., an external monitor for a laptop computer). In various examples of the embodiments, other peripheral devices 1120 include one or more of an independent digital video disc (or digital versatile disc, both terms are DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functions based on the output of system 1000. For example, a disc player performs the function of playing the output of system 1000.

[0147] In various embodiments, the control signal is transmitted between the system 1000 and the display 1100, the speaker 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. The output devices may be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices may be connected to the system 1000 using the communication channel 1060 through the communication interface 1050. The display 1100 and the speaker 1110 may be integrated with other components of the system 1000 in a single unit in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver such as, for example, a timing controller (T Con) chip.

[0148] Alternatively, if the RF portion of the input 1130 is part of a separate set-top box, the display 1100 and the speaker 1110 may optionally be separate from one or more of the other components. In various embodiments where the display 1100 and the speaker 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0149] These embodiments may be implemented by the processor 1010 or by computer software implemented by hardware or a combination of hardware and software. As a non-limiting example, these embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, the processor 1010 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0150] Various embodiments are involved in decoding. As used in this application, "decoding" may cover, for example, all or part of a process performed on a received encoded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes may also or alternatively include processes performed by the decoders of the various embodiments described in this application, for example, for a current block, using a neural network to determine neural network intra prediction and side information from at least one input data, encoding the current block based on the neural network intra prediction; and encoding the side information.

[0151] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" specifically refers to a subset of operations or more generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0152] Various specific implementations are involved in encoding. In a manner similar to the discussion of "decoding" above, "encoding" as used in this application can cover, for example, all or part of the process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transformation, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by the encoders of the various embodiments described in this application, such as, for a current block, receiving information regarding side information for neural network intra prediction; using a neural network and the side information applied to at least one input data to determine a neural network intra prediction for the current block; and decoding the block using the determined neural network intra prediction.

[0153] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in yet another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" specifically refers to a subset of operations or more generally to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0154] Note that the grammatical elements (e.g., deepFlag) used herein are descriptive terms. Thus, they do not exclude the use of other grammatical element names.

[0155] When the drawings are presented as flowcharts, it should be understood that they also provide block diagrams of the corresponding apparatus. Similarly, when the drawings are presented as block diagrams, it should be understood that they also provide flowcharts of the corresponding method / process.

[0156] The various embodiments refer to rate - distortion optimization. Specifically, during the encoding process, a balance or trade - off between rate and distortion is typically considered, often taking into account the constraints of computational complexity. Rate - distortion optimization is generally formulated as minimizing a rate - distortion function, which is a weighted sum of rate and distortion. There are different ways to solve the rate - distortion optimization problem. For example, these methods can be based on extensive testing of all encoding options (including all considered modes or encoding parameter values), and fully evaluating their encoding costs as well as the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to reduce the encoding complexity, especially for the calculation of approximate distortion based on the predicted or prediction - residual signal rather than the reconstructed residual signal. A hybrid of these two methods can also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for other encoding options. Other methods only evaluate a subset of the possible encoding options. More generally, many methods employ any one of various techniques to perform the optimization, but the optimization does not necessarily involve a full evaluation of both the encoding cost and the associated distortion. However, according to at least one embodiment, when the deep intra - prediction mode is the only available mode for intra - prediction, rate - distortion optimization in encoding is removed.

[0157] The specific implementations and aspects described herein can be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of specific implementation (e.g., only as a method), the specific implementation of the discussed features can be implemented in other forms (e.g., a device or a program). A device can be implemented in, for example, appropriate hardware, software, and firmware. A method can be implemented in, for example, a processor that generally refers to a processing device, which includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as, for example, a computer, a mobile phone, a portable / personal digital assistant (“PDA”), and other devices that facilitate information communication among end - users.

[0158] Reference to “one embodiment” or “an embodiment” or “one specific implementation” or “specific implementation” and other variations thereof means that the particular features, structures, characteristics, etc. described in connection with the embodiment are included in at least one embodiment. Thus, the appearance of the phrases “in one embodiment” or “in an embodiment” or “in one specific implementation” or “in specific implementation” and any other variations that occur throughout this application do not necessarily all refer to the same embodiment.

[0159] Additionally, this application may relate to “determining” various information. Determining information can include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0160] In addition, the present application may relate to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, computing information, determining information, predicting information, or estimating information.

[0161] Additionally, the present application may relate to "receiving" various information. Like "accessing", receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Further, "receiving" is typically involved in one way or another during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, computing information, determining information, predicting information, or estimating information.

[0162] It should be understood that, for example, in the case of "A / B", "A and / or B", and "at least one of A and B", the use of any one of the following, namely " / ", "and / or", and "at least one", is intended to cover the selection of only the first-listed option (A), or only the second-listed option (B), or the selection of both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such phrases are intended to cover the selection of only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or the selection of the first-listed option and the second-listed option (A and B), or the selection of the first-listed option and the third-listed option (A and C), or the selection of the second-listed option and the third-listed option (B and C), or the selection of all three options (A and B and C). As will be apparent to those of ordinary skill in the art and related fields, this can be extended to as many items as are listed.

[0163] Moreover, as used herein, the term "signal" (as a verb) means, among other things, to indicate something to a corresponding decoder. For example, in some embodiments, an encoder signals a particular one of a plurality of parameters for intra-frame depth prediction. In this way, in one embodiment, the same parameters are used on both the encoder side and the decoder side. Thus, for example, the encoder may transmit (explicit signaling) a particular parameter to the decoder such that the decoder may use the same particular parameter. Conversely, if the decoder already has the particular parameter among others, signaling may be used without transmission (implicit signaling) simply to allow the decoder to know and select the particular parameter. By avoiding the transmission of any actual functionality, bit savings are achieved in various embodiments. It should be understood that signaling may be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. Although the foregoing relates to the verb form of the term "signal", the term "signal" (as a noun) is also used herein.

[0164] It will be apparent to those of ordinary skill in the art that a particular implementation may generate various signals formatted to carry information such as may be stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the particular implementations. For example, a signal may be formatted to carry the bitstream of the embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is known that signals may be transmitted over a variety of different wired or wireless links. A signal may be stored on a processor-readable medium.

[0165] We have described a number of embodiments. The features of these embodiments may be provided singly or in any combination in various claim categories and types. Additionally, embodiments may include one or more of the following features, devices, or aspects, singly or in any combination, across various claim categories and types:

[0166] · Modifying the intra-frame prediction process applied to the decoder and / or encoder.

[0167] · Using a neural network in the intra-frame prediction process applied to the decoder and / or encoder.

[0168] · Using a neural network to determine neural network intra-frame prediction and side information from at least one input data in the encoder;

[0169] · Using a neural network applied to at least one input data and side information received in the decoder to determine neural network intra-frame prediction;

[0170] ·Encode side information generated by an intra-depth predictor in an encoder;

[0171] ·Decode side information to be used by an intra-depth predictor in a decoder;

[0172] ·For a luma block, the input data of a neural network intra predictor in an encoder includes the context surrounding the luma block and the luma block;

[0173] ·For a luma block, the input data of a neural network intra predictor in a decoder includes the context surrounding the luma block and the decoded side information;

[0174] ·For a chroma block, the input data of a neural network intra predictor in an encoder includes the context surrounding the chroma block, the context surrounding the luma block juxtaposed with the chroma block, and the chroma block.

[0175] ·For a chroma block, the input data of a neural network intra predictor in a decoder includes the context surrounding the chroma block, the context surrounding the luma block juxtaposed with the chroma block, and the generated side information.

[0176] ·For a chroma block, the input data of a neural network intra predictor in an encoder includes the context surrounding the luma block juxtaposed with the chroma block and the chroma block.

[0177] ·For a chroma block, the input data of a neural network intra predictor in a decoder includes the context surrounding the luma block juxtaposed with the chroma block and the generated side information.

[0178] ·For a chroma block, the input data of a neural network intra predictor in an encoder includes the context surrounding the chroma block, the context surrounding the luma block juxtaposed with the chroma block, the chroma block, and the reconstructed luma block.

[0179] ·For a chroma block, the input data of a neural network intra predictor in a decoder includes the context surrounding the chroma block, the context surrounding the luma block juxtaposed with the chroma block, the reconstructed luma block, and the generated side information.

[0180] ·Any one of the variants of the input data for luma and chroma blocks further includes the intra prediction mode L of the block located to the left of the block and the intra prediction mode A of the block located above the block as input data.

[0181] ·Insert syntax elements in the signaling that enable the decoder to identify the intra prediction method to be used.

[0182] ·Select the intra prediction method to be applied at the decoder based on these syntax elements.

[0183] · Insert the syntax element DeepFlag representing the neural network-based intra prediction mode among the syntax elements of the intra prediction mode in the signaling.

[0184] · Enable only one of the neural network-based intra prediction mode and the planar intra prediction mode from the syntax elements for luma intra prediction.

[0185] · Enable only the neural network-based intra prediction mode.

[0186] · A bitstream or signal including one or more of the described syntax elements or their variants.

[0187] · A bitstream or signal that includes the syntax for transmitting the information generated according to any of the described embodiments.

[0188] · Insert in the signaling the syntax elements that enable the decoder to determine the depth intra prediction used in the encoder.

[0189] · Create and / or transmit and / or receive and / or decode a bitstream or signal including one or more of the described syntax elements or their variants.

[0190] · Create and / or transmit and / or receive and / or decode according to any of the described embodiments.

[0191] · A method, process, device, medium storing instructions, medium storing data, or signal according to any of the described embodiments.

[0192] · A television set, set-top box, mobile phone, tablet computer, or other electronic device that performs intra prediction according to any of the described embodiments.

[0193] · A television set, set-top box, mobile phone, tablet computer, or other electronic device that performs intra prediction according to any of the described embodiments and displays the resulting image (e.g., using a monitor, screen, or other type of display).

[0194] · A television set, set-top box, cellular phone, tablet, or other electronic device that selects a channel (e.g., using a tuner) to receive a signal including an encoded image and performs intra prediction according to any of the described embodiments.

[0195] · A television set, set-top box, mobile phone, tablet computer, or other electronic device that receives a signal including an encoded image via radio (e.g., using an antenna) and performs intra prediction according to any of the described embodiments.

Claims

1. A method for video decoding, the method comprising: - For a luminance block being decoded in a picture of a video, decoding side information for neural network intra-luminance prediction, the side information and the neural network intra-luminance prediction having been generated during encoding by applying a neural network to the luminance block and the context surrounding the luminance block, the side information being inferred during encoding from the context surrounding the luminance block and the luminance block; - For the luminance block being decoded, determining a neural network intra-luminance prediction by applying a neural network to the context surrounding the luminance block and the side information, wherein the side information guides the neural network intra-prediction during decoding; - Decoding the luminance block using the determined neural network intra-luminance prediction.

2. The method according to claim 1, further comprising: - For a chrominance block decoded in a picture of a video, decoding side information for neural network intra-chrominance prediction, the side information and the neural network intra-chrominance prediction having been generated during encoding, by applying a neural network to the context surrounding the chrominance block, the context surrounding the luminance block juxtaposed with the chrominance block, and the chrominance block; - For the chrominance block, determining a neural network intra-chrominance prediction by applying a neural network to the context surrounding the chrominance block, the context surrounding the luminance block juxtaposed with the chrominance block, and the side information; and - Decoding the chrominance block using the determined neural network intra-chrominance prediction.

3. The method according to claim 1, further comprising: - For a chrominance block decoded in a picture of a video, decoding side information for neural network intra-chrominance prediction, the side information and the neural network intra-chrominance prediction having been generated during encoding, by applying a neural network to the context surrounding the luminance block juxtaposed with the chrominance block; - For the chrominance block, determining a neural network intra-chrominance prediction by applying a neural network to the context surrounding the luminance block juxtaposed with the chrominance block and the side information; and - Decoding the chrominance block using the determined neural network intra-chrominance prediction.

4. The method according to claim 1, further comprising decoding a syntax element indicating that neural network-based intra-luminance prediction is used for intra-prediction of a luminance block.

5. A device for video decoding, comprising one or more processors, wherein the one or more processors are configured to: - For a luminance block being decoded in a picture of a video, decoding side information for neural network intra-luminance prediction, the side information and the neural network intra-luminance prediction having been generated during encoding by applying a neural network to the luminance block and the context surrounding the luminance block, the side information being inferred during encoding from the context surrounding the luminance block and the luminance block; - For the luminance block being decoded, determine neural network intra-luminance prediction by applying a neural network to the context surrounding the luminance block and the side information, where the side information guides the neural network intra prediction during decoding; - Decode the block using the determined neural network intra-luminance prediction.

6. The apparatus according to claim 5, wherein the one or more processors are configured to: - For a chrominance block decoded in a picture of a video, decode the side information for neural network intra-chrominance prediction, the side information already generated during encoding, and the neural network intra-chrominance prediction by applying a neural network to the context surrounding the chrominance block, the context surrounding the luminance block juxtaposed with the chrominance block, and the chrominance block; - For a chrominance block, determine neural network intra-chrominance prediction by applying a neural network to the context surrounding the chrominance block, the context surrounding the luminance block juxtaposed with the chrominance block, and the side information; and - Decode the chrominance block using the determined neural network intra-chrominance prediction.

7. The apparatus according to claim 5, wherein the one or more processors are configured to: - For a chrominance block decoded in a picture of a video, decode the side information for neural network intra-chrominance prediction, the side information already generated during encoding, and the neural network intra-chrominance prediction by applying a neural network to the context surrounding the luminance block juxtaposed with the chrominance block; - For a chrominance block, determine neural network intra-chrominance prediction by applying a neural network to the context surrounding the luminance block juxtaposed with the chrominance block and the side information; and - Decode the chrominance block using the determined neural network intra-chrominance prediction.

8. The apparatus according to claim 5, wherein for intra prediction of a luminance block, a syntax element indicating a neural network-based intra-luminance prediction mode for intra-luminance prediction of the luminance block is decoded.

9. A method for video coding, the method comprising: - Determine neural network intra-luminance prediction and side information guiding the neural network intra prediction during decoding from a luminance block encoded in a picture of a video by applying a neural network to the luminance block and the context surrounding the luminance block; - Encode the luminance block based on the neural network intra-luminance prediction; and - Encode the side information.

10. The method according to claim 9, further comprising: - For a chrominance block, determine neural network intra-chrominance prediction and side information by applying a neural network to the context surrounding the luminance block juxtaposed with the chrominance block and the chrominance block; and - Encode the chrominance block based on the neural network intra-chrominance prediction.

11. The method according to claim 9, further comprising: - For a chrominance block, determine neural network intra-chrominance prediction and side information by applying a neural network to the context surrounding the chrominance block, the context surrounding the luminance block juxtaposed with the chrominance block, and the chrominance block; and -Encode the chrominance block based on the neural network intra-chrominance prediction.

12. The method according to claim 9, further comprising: -For a chrominance block, determine a neural network intra-chrominance prediction and side information by applying a neural network to the context surrounding the luma block juxtaposed with the chrominance block; and -Encode the chrominance block based on the neural network intra-chrominance prediction.

13. The method according to claim 9, further comprising encoding a syntax element indicating that neural network-based intra prediction is used for intra prediction of a luma block.

14. An apparatus for video coding, comprising one or more processors, wherein the one or more processors are configured to: -Determine a neural network intra-luma prediction and side information for guiding the neural network intra prediction during decoding from a luma block encoded in a picture of a video by applying a neural network to the luma block and the context surrounding the luma block; -Encode the luma block based on the neural network intra-luma prediction; and -Encode the side information.

15. The apparatus according to claim 14, wherein the one or more processors are configured to: -For a chrominance block, determine a neural network intra-chrominance prediction and side information by applying a neural network to the context surrounding the luma block juxtaposed with the chrominance block and the chrominance block; and -Encode the chrominance block based on the neural network intra-chrominance prediction.

16. The apparatus according to claim 14, wherein the one or more processors are configured to: -For a chrominance block, determine a neural network intra-chrominance prediction and side information by applying a neural network to the context surrounding the chrominance block, the context surrounding the luma block juxtaposed with the chrominance block, and the chrominance block; and -Encode the chrominance block based on the neural network intra-chrominance prediction.

17. The apparatus according to claim 14, wherein the one or more processors are configured to: -For a chrominance block, determine a neural network intra-chrominance prediction and side information by applying a neural network to the context surrounding the luma block juxtaposed with the chrominance block; and -Encode the chrominance block based on the neural network intra-chrominance prediction.

18. The apparatus according to claim 14, wherein, Encode a syntax element indicating that neural network-based intra prediction is used for intra prediction of a luma block.

Images