Deep intra prediction of image blocks

Abstract

Provided is a method for performing intra-frame prediction on an image block using at least one neural network based on a context including pixels surrounding the image block. The intra-frame prediction further utilizes information to distinguish between available pixels and missing pixels in the context. Also provided are encoding / decoding methods and apparatus, devices, non-transitory computer-readable media, and computer program products.

Description

Technical Field

[0001] The technical field of one or more embodiments generally relates to video compression. At least some embodiments relate to improving compression efficiency compared to existing video compression systems such as HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2, which is described in "ITU-T H.265 ITU Telecommunication Standardization Sector (10 / 2014), Series H: Audiovisual and Multimedia Systems, Audiovisual Services Infrastructure - Coding of Mobile Video, High Efficiency Video Coding, ITU-T H.265 Recommendation") or compared to developing video compression systems such as VVC (Versatile Video Coding, a new standard being developed by the Joint Video Experts Team (JVET)).

[0002] In particular, at least some embodiments also relate to using a neural network to improve intra-frame prediction of an image block based on the context surrounding the image block. Summary of the Invention

[0003] The deep intra predictor uses at least one neural network to infer intra prediction for a current image block based on the context surrounding the current image block.

[0004] The context surrounding the current image block includes at least one pixel value in the causal neighborhood of the current image block. Causal means that the same neighborhood is available at both the encoding side and the decoding side. Typically, for intra prediction at both the encoding side and the decoding side, the causal neighborhood includes reconstructed pixel values. A pixel value in the context is available when its value is accessible (obtained). Conversely, a pixel value in the context is missing when its value is inaccessible.

[0005] For example, in HEVC and VVC, macroblocks (also called coding tree blocks (CTBs)) are hierarchically partitioned into sub-blocks, and different CTBs are scanned in raster scan order, while the sub-blocks within each CTB are scanned in zigzag scan order. This places two constraints on the context from which the deep intra predictor infers the intra prediction for the current image block. First, the context may only include pixels located above and to the left of the current image block. Second, depending on the position of the current image block in its parent CTB, pixels to the upper right and / or lower left of the current image block may be missing.

[0006] One of the challenges lies in distinguishing between available and missing pixels in the context used by the deep intra predictor.

[0007] This is addressed and processed by the general aspects described herein involving intra-prediction of a current image block using at least one neural network and information for distinguishing between available pixels and missing pixels in context.

[0008] According to a first aspect, a method is provided. The method comprises using at least one neural network to perform intra-frame prediction on an image block based on a context comprising pixels surrounding the image block. The intra-frame prediction further utilizes information to distinguish between available pixels and missing pixels in the context.

[0009] In one embodiment, the information is a mask value, wherein the method further comprises pre-processing the context before intra prediction, wherein the pre-processed context is obtained by assigning the mask value to missing pixels in the context.

[0010] In this way, for the deep intra predictor, any ambiguity between the unmasked parts of the context with meaningful information and the masked parts of the context with missing information is removed.

[0011] In another embodiment, the information is a binary vector indicating the position of the missing pixels in the context, and wherein the image block is intra predicted in response to the binary vector.

[0012] This way, the binary vector removes any ambiguity about where the missing pixels are in context.

[0013] According to another aspect, a second method is provided. The method includes determining, using at least one neural network, an intra-frame prediction for an image block to be encoded based on a context including pixels surrounding the image block, wherein the intra-frame prediction further uses information to distinguish between available pixels and missing pixels in the context, and encoding the image block based on the intra-frame prediction.

[0014] According to another aspect, a third method is provided that includes determining, using at least one neural network, an intra-frame prediction for an image block to be decoded based on a context including pixels surrounding the image block, wherein the intra-frame prediction further uses information to distinguish between available pixels and missing pixels in the context; and decoding the image block based on the intra-frame prediction.

[0015] According to another aspect, an apparatus is provided that includes one or more processors configured to, using at least one neural network, determine an intra-frame prediction for an image block to be encoded based on a context including pixels surrounding the image block, wherein the intra-frame prediction further uses information to distinguish between available pixels and missing pixels in the context, and encode the image block based on the intra-frame prediction.

[0016] According to another aspect, an apparatus is provided. The apparatus includes one or more processors, wherein the one or more processors are configured to use at least one neural network to determine an intra-frame prediction for an image block to be decoded based on a context including pixels surrounding the image block, wherein the intra-frame prediction further uses information to distinguish between available pixels and missing pixels in the context; and decode the image block based on the intra-frame prediction.

[0017] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium containing data content generated according to any of the described encoding embodiments or variations is provided.

[0018] According to another general aspect of at least one embodiment, a computer program product is provided that includes instructions that, when executed by a computer, cause the computer to perform any of the described coding embodiments or variations.

[0019] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In the accompanying drawings, examples of several embodiments are shown. The drawings show:

[0021] Figure 1 An example showing an image patch and its context;

[0022] Figure 2 and Figure 3 An example of the architecture of a neural network according to the prior art is shown;

[0023] Figure 4 Examples showing full and partial contexts;

[0024] Figure 5 A flowchart illustrating a method for intra-prediction of an image block according to at least one embodiment;

[0025] Figure 5a A flowchart illustrating a method for intra-prediction of an image block according to at least one embodiment;

[0026] Figure 6 illustrates an example of a one-to-one mapping between coefficients of a binary vector and the position of a pixel in a context according to at least one embodiment;

[0027] Figure 7 Another example of a one-to-one mapping between coefficients of a binary vector and the position of a pixel in a context is shown in accordance with at least one embodiment;

[0028] Figure 8 and Figure 9 illustrates an example of an architecture for a neural network according to at least one embodiment;

[0029] Figure 10 A block diagram illustrating an example of an encoder in which various aspects and embodiments are implemented;

[0030] Figure 11 A block diagram illustrating a video decoder in which various aspects and embodiments are implemented; and

[0031] Figure 12 A block diagram illustrating an example of a system in which various aspects and embodiments may be implemented is shown. DETAILED DESCRIPTION

[0032] Intra-frame prediction based on deep neural network

[0033] Using a neural network to perform intra-frame prediction for a current image block Y refers to a deep intra-frame predictor that provides a model of the dependency between the current image block and its surrounding context X. An example of a deep intra-frame predictor is described in the paper titled "Context-adaptive neural network-based prediction for image compression" (Thierry Dumas, Aline Roumy, Christine Guillemot, arXiv:1807.06244v2, August 2019). First, a set of neural network architectures (denoted as Prediction Neural Network Sets (PNNS)) based on both fully connected and / or convolutional neural networks are trained, typically using block pairs and their context pairs extracted from a set of training images. Next, one of the neural networks in the PNNS is selected to provide an intra-frame prediction for the current image block based on the surrounding context.

[0034] In this paper, the context X consists of pixels located above and to the left of the current image block Y, similar to the reconstructed reference sample set for intra prediction in VVC. However, the difference is that the context X extends to the left and top, for example, as in Figure 1 As shown in . Due to this extension, the deep intra predictor is able to learn the spatial correlation between its input context X and its intra prediction given the current image block Y The relationship between them.

[0035] (1) describes the m Parameterized fully connected neural network fW or by Parameterized convolutional neural network g m Convert context X to intra prediction for W×W block Y

[0036]

[0037] where α is a normalization value equal to the average pixel intensity calculated over all training images to center the pixel values ​​of the context X. The subscript c stands for centered.

[0038] Then, during the training phase, α is subtracted from each image block to be predicted and its context. During the testing phase, to perform intra prediction on the current image block, α is subtracted from the context X (see (1)) and added to the intra prediction (Output of the deep intra predictor).

[0039] Figure 2 and Figure 3 An example of the architecture of a neural network showing PNNS.

[0040] The first operation for both architectures consists of formatting the context to facilitate computation within the neural network.

[0041] like Figure 2 As shown, in the case of a fully connected neural network, the connection context X c Therefore, first we need to keep all the elements of context X in the 2D structure. c is vectorized (flattened), and fast vector-matrix arithmetic can be used.

[0042] For example, the fully connected architecture f W It consists of 4 fully connected layers L1, L2, L3 and L4. W Collect the weights and biases of the four fully connected layers. The first layer L1 calculates the context X c To arrive at the output coefficients, we use an overcomplete representation of the network. Overcompleteness is chosen because it has been empirically observed that overcomplete representations in early layers improve the performance of neural networks. The next two layers (L2 and L3) keep the number of coefficients constant, while the final layer, L4, reduces the number of coefficients to provide the predicted image patch.

[0043] like Figure 3 As shown, in the case of convolutional neural networks, fast computation of 2D filters requires maintaining the context X c 2D structure. In addition, for fast computation, the shape of the input to the convolution must be rectangular. This is why the context is split into two rectangles X0 and X1, which are then processed by different convolutions.

[0044] The first task of the convolutional architecture is to compute features that characterize the dependencies between elements in X0. X0 is therefore fed into a network consisting of Parameterized convolutional layers This will produce l∈N * Similarly, X1 is fed into a stack of feature maps Z0. Parameterized convolutional layers This results in a stack of l feature maps Z1. All elements in the context may be relevant to predicting any pixel in the image patch. This means that information associated with all spatial locations in the context must be combined. This is why the next layer in the convolutional architecture is constructed by using Parameterized functions Merge stack Z0 and stack Z1 spatially to obtain the merged stack More precisely, for i∈[1;l], all coefficients of the i-th feature map of Z0 and the i-th feature map of Z1 are merged via parametric affine combination.

[0045] The final task of the convolutional architecture is to The information of different feature maps is merged. is thus fed to the Parameterized transposed convolutional layer This generates intra predictions before post-processing.

[0046] The missing pixels in the context are fed into the neural network

[0047] In HEVC and VVC, images are partitioned into coding tree units (CTUs). A CTU contains a luma coding tree block (CTB), two chroma CTBs, and syntax elements. One CTB is processed at a time in raster scan order. Each CTB is hierarchically partitioned into coding blocks (CBs). CBs within a CTB are processed in zigzag scan order.

[0048] So, for example, in HEVC, the size of the image block to be predicted can be 64×64, 32×32, 16×16, 8×8, or 4×4. This means that 5 neural networks are needed, one for each size of the image block to be predicted. Therefore, according to this example, the deep intra predictor consists of 5 neural networks.

[0049] In VVC, due to the more complex hierarchical segmentation, the size of the image block to be predicted can be 128×128, 64×64, 32×32, 16×16, 8×8, or 4×4. In addition, it can also be rectangular, for example, with a size of 4×8. In this case, the solution is to assign a neural network to each image block size to construct a deep intra-frame predictor.

[0050] In HEVC and VVC, considering that image blocks are processed in a combination of raster scan order and zigzag scan order, when the depth intra predictor infers the prediction of the current image block based on its context, pixels to the upper right and lower left of the current image block may be missing, depending on the position of the current image block in its CTB.

[0051] To address this situation, one approach is to design several contexts for a given size of image block, and train a neural network for each context. More precisely, the first context is the complete context, corresponding to the case where no pixels are missing, while the second context is the partial context, corresponding to the case where the number of missing pixels is the largest. During prediction within HEVC, the first context is used if the current image block is located above and to the left of its parent image block CB. In any other case, the second context is used, e.g. Figure 4 However, this approach increases the number of models to be used inside HEVC and, therefore, the number of parameters to be stored in memory. Furthermore, in the case where pixels on the lower left side of the current image block are missing but pixels on the upper right side of the image block are available, the latter available pixels are not included in the second context, which wastes information used for neural network-based prediction.

[0052] In general, at least one of the embodiments is directed to a method that includes using at least one neural network to perform intra-frame prediction on a current image block Y based on a context X that includes pixels surrounding the current image block Y. The intra-frame prediction further uses information INF to distinguish between available pixels and missing pixels in the context.

[0053] In one embodiment of the method, the information INF is a mask value, such as Figure 5 Then, the method further includes preprocessing of intra prediction (step 510), wherein a preprocessed context X is obtained by assigning mask values ​​to the missing pixels in X. p (p stands for “preprocessing”). Then, the depth intra predictor is used to calculate the preprocessed context X p Get preprocessed intra prediction For example, Figures 1 to 3The mask value exceeds the range of values ​​that the available pixels in the context after pre-processing can take.

[0054] By setting the mask value to any value in ]-∞,-255]∪[255,+∞[, the mask value is necessarily outside the range of values ​​that the available pixels take after pre-processing to a bit depth of 8 available pixels.

[0055] Due to X p The mask value in exceeds X p The range of unmasked values ​​in , so the uninformative masked part of the context has X p There is no ambiguity between the unmasked parts of the neural network information. p This does not happen for any other mask value that is within the range of the unmasked values ​​in , such as the mean pixel intensity computed over all training images.

[0056] exist Figure 5 In a variant of , the mean (average) μ of the available pixels in the context X is subtracted from the available pixels. The method further comprises performing a pre-processed intra prediction Post-processing is performed (step 530), wherein the mean μ is added to the pre-processed intra prediction To obtain the intra-frame prediction of the current image block Y

[0057] This variant is used because during the training phase, when each coefficient of the input data (here, the context) arriving at the neural network is close to zero on average compared to the training data, it indicates that the minimization of the objective function of the neural network parameters has converged well, and the input data must be preprocessed identically during the training and testing phases.

[0058] Mathematically speaking, steps 510 to 530 can be expressed by formula (2).

[0059]

[0060] Among them, the function cover_missing covers the missing reconstructed pixels in its input context.

[0061] In the variant, the preprocessed context X p (Formula (2)) can be adapted to the bit depth of the current image block Y and the bit depth of the training images used to train the PNNS.

[0062] For example, scaling can be used to predict image patches with a bit depth of 10 using a PNNS trained on 8-bit training images. Therefore, Equation (2) is replaced by Equation (3):

[0063]

[0064] In another embodiment of the method, Figure 5a As shown, the information INF is a binary vector B∈{0,1} indicating the location of the missing pixel in the context X K The binary vector B is fed to the deep intra prediction step 520 together with the context X. K represents the binary vector element B k Therefore, the intra prediction of an image block responds to a binary vector.

[0065] In a variant, the binary vector B∈{0,1} K Indicative context X c The location of the missing pixel in the binary vector B and the context X c Feed together to the deep intra prediction step 520.

[0066] exist Figure 5a In a variant of , the mean (average) μ of the available pixels in the context X is subtracted from the available pixels. The method further comprises performing a pre-processed intra prediction Post-processing is performed (step 530), wherein the mean μ is added to the pre-processed intra prediction To obtain the intra-frame prediction of the current image block Y

[0067] This variant is used because during the training phase, when each coefficient of the input data (here, the context) arriving at the neural network is close to zero on average compared to the training data, it indicates that the minimization of the objective function of the neural network parameters has converged well, and the input data must be preprocessed identically during the training and testing phases.

[0068] Any one-to-one mapping between each coefficient of the binary vector and the location of a missing pixel or an available pixel in the context may be utilized.

[0069] In one embodiment of a one-to-one mapping, each binary vector coefficient B k Refers to the location of missing pixels or at least one consecutive row / column of available pixels in the context.

[0070] For example, each binary vector coefficient B representing the position of at least one consecutive row / column of missing pixels k Set to 1 and each binary vector coefficient B representing the row / column position of an available pixel k Set to 0.

[0071] In a variation of this embodiment, Figure 6 As shown, each binary vector coefficient Bk Refers to the location of a row / column of missing pixels or available pixels.

[0072] For example, the size of the current image block to be predicted is W×W, K=2W, and the first W vector coefficients B k Refers to the position of the missing pixel or available pixel in the upper right side of the current image block from left to right, and B k The last W vector coefficients in refer to the positions of the missing pixels or available pixels from top to bottom of the W rows on the lower left side of the current image block.

[0073] In a variant of this embodiment, each binary vector coefficient B k Refers to the locations of at least two consecutive rows / columns of missing pixels or available pixels.

[0074] Figure 7 An example of a one-to-one mapping between the coefficients of a binary vector and the positions of four consecutive rows / columns of missing or available pixels is shown.

[0075] For example, in HEVC and VVC, in image partitioning, each channel has a minimum image block size w u ×w u The missing pixels or available pixels in the context can be grouped together, and each group collects w u Similarly, consecutive rows of missing pixels or available pixels in the context can be grouped together, with each group containing w u Then, in the binary vector, we can Define.

[0076] For example, the size of the current block to be predicted is H×W, and the previous W / w u vector coefficients B k Refers to the W / w of missing pixels or available pixels on the upper right side of the current image block from left to right u The position of the group column, and the last H / w u vector coefficients B k Refers to the H / W of the missing pixels or available pixels on the lower left side of the current image block from top to bottom u The position of the group row.

[0077] Figure 6 and Figure 7 The example shown in FIG2 assumes that the number of columns of missing pixels or available pixels on the upper right side of the current image block and the number of rows of missing pixels or available pixels on the lower left side of the current image block are equal to W. However, this is not restrictive, as the present embodiment is applicable to any number of columns of missing pixels or available pixels on the upper right side of the current image block and any number of rows of missing pixels or available pixels on the lower left side of the current image block.

[0078] In the variant, Figure 8 In the case of a fully connected neural network as shown in , the binary vector B can be fed into its first layer L1 because all spatial information in the context is incorporated into this layer. The binary vector B provides the neural network with an explicit indication of which pixels in the context are missing and which are available. This way, it can learn how to ignore missing pixels during the training phase.

[0079] In the variant, Figure 9 In the case of the convolutional neural network shown in , the binary vector B can be fed into a layer containing full connectivity, because only in this layer is the full spatial information in the context combined.

[0080] According to another aspect, a method is provided that includes determining, using at least one neural network, an intra-frame prediction for an image block to be encoded based on a context including pixels surrounding the image block, wherein the intra-frame prediction further uses information to distinguish between available pixels and missing pixels in the context, and encoding the image block based on the intra-frame prediction.

[0081] In a variant, the method further comprises sending information related to intra prediction of the current image block, for example information for accessing a neural network for intra prediction or information defining access to a neural network for intra prediction.

[0082] According to another aspect, a method is provided that includes determining, using at least one neural network, an intra prediction for an image block to be decoded based on a context including pixels surrounding the image block, wherein the intra prediction further uses information to distinguish between available pixels and missing pixels in the context, and decoding the image block based on the intra prediction.

[0083] In a variant, the method further comprises receiving information related to intra prediction of the current image block, for example information for accessing a neural network for intra prediction or information defining access to a neural network for intra prediction.

[0084] For example, in HEVC, when the deep intra predictor competes with other intra predictors in terms of system, this information can be a flag written to the bitstream before all other flags for intra prediction. A value of 1 can indicate that the deep intra predictor is selected for use in predicting the current image block. In this case, no other flags for intra prediction are written to the bitstream. A value of 0 means that another intra predictor is selected.

[0085] This application describes various aspects, including tools, features, embodiments, models, methods, and the like. Many of these aspects are described in detail, and at least individual characteristics are shown, often in a manner that may sound limited. However, this is for clarity of description and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with aspects described in previous submissions.

[0086] The aspects described and contemplated in this patent application can be implemented in many different forms. Figure 10 、 Figure 11 and Figure 12 Some embodiments are provided, but other embodiments are contemplated, and Figure 10 、 Figure 11 and Figure 12 The discussion does not limit the breadth of implementations.

[0087] At least one of these aspects generally relates to video encoding and decoding. These and other aspects can be implemented as a method, apparatus, computer-readable medium having stored thereon instructions for encoding or decoding video data according to any of the above methods, or a computer program product comprising instructions that, when executed by a computer, cause the computer to perform any of the above methods.

[0088] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image," "picture," and "frame" are used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoding side, while "decoding" is used on the decoding side.

[0089] Various methods are described herein, and each method includes one or more steps or actions for achieving the method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0090] The various methods and other aspects described in this application can be used to modify modules, e.g. Figure 10 The intra prediction module 160 of the video encoder 100 and the Figure 11 and Figure 12 Intra prediction module 260 of decoder 200 is shown. In addition, aspects of the present invention are not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0091] Various numerical values ​​are used in this application, for example, the number of intra prediction modes, parameters of a neural network (layers, output ranges, thresholds), and the specific values ​​are for example purposes, and the aspects are not limited to these specific values.

[0092] Figure 10 Shown is a block diagram of an encoder 100. Variations of this encoder 100 are contemplated, but for clarity, the following describes the encoder 100 without describing all contemplated variations.

[0093] Before encoding, the video sequence may undergo a pre-encoding process (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-encoding process (101) and appended to the bitstream.

[0094] In an encoder 100, an image is encoded by encoder elements as described below. The image to be encoded is partitioned (102) and processed in units such as CUs. Each unit (typically an image block) is encoded, for example, using intra mode or inter mode. When a unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use for encoding the unit and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (110) the predicted unit from the original unit.

[0095] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0096] The encoder decodes the coding unit to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction unit are combined (155) to reconstruct the unit. A loop filter (165) is applied to the reconstructed unit to perform, for example, image deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered unit is stored in a reference picture buffer (180).

[0097] Figure 11Shown is a block diagram of a video decoder 200. Variations of this decoder 200 are contemplated, but for clarity, the following describes decoder 200 without describing all contemplated variations.

[0098] In decoder 200, the bitstream is decoded by decoder elements as described below.

[0099] The video decoder 200 generally performs the same Figure 10 The decoding process is the reverse of the encoding process described above. Decoder 200 also typically performs video decoding as part of encoding the video data.

[0100] Specifically, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other encoded information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can divide (235) the image according to the decoded picture partition information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction unit are combined (255) to reconstruct the unit (typically an image block). The prediction unit can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). A loop filter is applied to the reconstructed unit (265). The filtered unit is stored in a reference picture buffer (280).

[0101] The decoded image may also undergo post-decoding processing (285), such as an inverse color transform (e.g., a transform from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping performed in the pre-encoding process (101). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0102] Figure 12A block diagram illustrating an example of a system in which various aspects and embodiments are implemented is shown. System 12000 may be embodied as a device including the various components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 12000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 12000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 12000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 12000 is configured to implement one or more aspects described in this document.

[0103] System 12000 includes at least one processor 1010 configured to execute instructions loaded therein to implement, for example, various aspects described in this document. Processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 12000 includes at least one memory 12020 (e.g., a volatile memory device and / or a non-volatile memory device). System 12000 includes storage devices 12040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drives, and / or optical disk drives. As non-limiting examples, storage devices 12040 may include internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0104] System 12000 includes an encoder / decoder module 12030, which is configured to, for example, process data to provide encoded or decoded video, and may include its own processor and memory. Encoder / decoder module 12030 represents a module that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of an encoding module and a decoding module. Furthermore, encoder / decoder module 12030 may be implemented as a standalone component of system 12000, or may be incorporated into processor 12010 as a combination of hardware and software known to those skilled in the art.

[0105] Program code to be loaded onto the processor 12010 or the encoder / decoder 12030 to perform various aspects described in this document may be stored in the storage device 12040 and subsequently loaded onto the memory 12020 for execution by the processor 12010. According to various embodiments, one or more of the processor 12010, the memory 12020, the storage device 12040, and the encoder / decoder module 12030 may store one or more of the various items during execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0106] In some embodiments, memory internal to the processor 12010 and / or encoder / decoder module 12030 is used to store instructions and provide working memory for processing required during encoding or decoding.

[0107] However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 12010 or the encoder / decoder module 12030) is used for one or more of these functions. The external memory may be the memory 12020 and / or the storage device 12040, such as dynamic volatile memory and / or non-volatile flash memory.

[0108] In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television.

[0109] In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the Joint Video Experts Team (JVET)).

[0110] Inputs to the elements of system 12000 may be provided through various input devices, such as shown in block 12130. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal transmitted over the air, for example, by a broadcaster; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 12 Other examples not shown include composite video.

[0111] In various embodiments, the input device of module 12130 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting the desired frequency (also known as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal band that (for example) may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband.

[0112] In one set-top box implementation, the RF section and its associated input processing elements receive RF signals transmitted over a wired (eg, cable) medium and perform frequency selection by filtering, down-converting, and filtering again to the desired frequency band.

[0113] Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements can include inserting elements between existing elements, for example, inserting an amplifier and an analog-to-digital converter.

[0114] In various embodiments, the RF portion includes an antenna.

[0115] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 12000 to other electronic devices across the USB and / or HDMI connections.

[0116] It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within processor 12010, as desired.

[0117] Similarly, aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within processor 12010 as desired.

[0118] The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, a processor 12010 and an encoder / decoder 12030, which operate in conjunction with memory and storage elements to process the data stream as needed for presentation on an output device.

[0119] The various components of system 12000 can be disposed within an integrated housing. Within the integrated housing, the various components can be interconnected and data can be transferred between the components using a suitable connection arrangement 122140 (e.g., an internal bus known in the art, including an inter-IC (I2C) bus, wiring, and printed circuit boards).

[0120] System 12000 includes a communication interface 12050 capable of communicating with other devices via a communication channel 12060. Communication interface 12050 may include, but is not limited to, a transceiver configured to transmit and receive data through communication channel 12060. Communication interface 12050 may include, but is not limited to, a modem or a network card, and communication channel 12060 may be implemented, for example, within a wired and / or wireless medium.

[0121] In various embodiments, a wireless network such as a Wi-Fi network (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) is used to stream or otherwise provide data to system 12000. Wi-Fi signals for these embodiments are received via a communication channel 12060 suitable for Wi-Fi communications and communication interface 12050. Communication channel 12060 for these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, for allowing streaming applications and other on-cloud communications.

[0122] Other embodiments provide streaming data to the system 12000 using a set-top box that delivers data via the HDMI connection of the input image block 12130. Still other embodiments provide streaming data to the system 12000 using the RF connection of the input module 12130.

[0123] As described above, various embodiments provide data in a non-streaming manner.

[0124] Additionally, various embodiments utilize wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0125] System 12000 can provide output signals to various output devices, including a display 12100 , speakers 12110 , and other peripherals 12120 .

[0126] The display 12100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 12100 can be used in a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), or other devices. The display 12100 can also be integrated with other components (e.g., as in a smartphone) or be separate (e.g., an external monitor for a laptop computer).

[0127] In various examples of embodiments, other peripherals 12120 include one or more of a stand-alone digital video disc (or digital versatile disc, both terms are DVR), a disc player, a stereo system, and / or a lighting system.

[0128] Various embodiments utilize one or more peripheral devices 12120 that provide functionality based on the output of the system 12000. For example, a disc player performs the function of playing the output of the system 12000.

[0129] In various embodiments, control signals are transmitted between the system 12000 and the display 12100, speakers 12110, or other peripheral devices 12120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention.

[0130] Output devices can be communicatively coupled to system 12000 via dedicated connections through respective interfaces 12070 , 12080 , and 12090 .

[0131] Alternatively, output devices may be connected to system 12000 via communication interface 12050 using communication channel 12060 .

[0132] Display 12100 and speaker 12110 may be integrated into a single unit with other components of system 12000 in an electronic device such as, for example, a television.

[0133] In various embodiments, the display interface 12070 includes a display driver, such as, for example, a timing controller (TCon) chip.

[0134] For example, if the RF portion of the input 12130 is part of a separate set-top box, the display 12100 and speakers 12110 may alternatively be separate from one or more of the other components.

[0135] In various embodiments where the display 12100 and speakers 12110 are external components, the output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0136] These embodiments may be executed by the processor 12010 or by computer software implemented by hardware or by a combination of hardware and software.

[0137] By way of non-limiting example, these embodiments may be implemented by one or more integrated circuits.

[0138] As non-limiting examples, memory 12020 may be of any type suitable to the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory.

[0139] As non-limiting examples, processor 12010 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0140] Various implementations participate in decoding. As used in this application, "decoding" may encompass all or part of a process performed on a received coded sequence, for example, to produce a final output suitable for display.

[0141] In various implementations, such processes include one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding.

[0142] In various embodiments, such processes also include or alternatively include processes performed by the decoder of various embodiments described herein, such as receiving information related to intra-frame prediction of a current image block using a neural network; determining an intra-frame prediction for the current image block using the neural network applied to a context surrounding the current image block; and decoding the image block using the determined intra-frame prediction.

[0143] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" refers specifically to a subset of operations or broadly to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0144] Various implementations involved in coding.

[0145] In a similar manner to the discussion above regarding "decoding," "encoding" as used in this application may encompass all or part of a process performed on an input video sequence to produce an encoded bitstream, for example.

[0146] In various implementations, such processes include one or more processes typically performed by an encoder, such as partitioning, differential encoding, transforming, quantization, and entropy encoding.

[0147] In various embodiments, such processes also include or alternatively include processes performed by the encoder of the various embodiments described herein, such as determining an intra-frame prediction for the current image block using a neural network applied to a context surrounding the current image block; encoding the current image block based on the intra-frame prediction; and transmitting the encoded current image block.

[0148] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in yet another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" refers specifically to a subset of operations or broadly to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0149] When the figures are presented as flow charts, it should be understood that they also provide block diagrams of the corresponding apparatus. Similarly, when the figures are presented as block diagrams, it should be understood that they also provide flow charts of the corresponding methods / processes.

[0150] Various implementations are referred to as rate-distortion optimization.

[0151] Specifically, during the encoding process, a balance or trade-off between rate and distortion is typically considered, often taking into account computational complexity constraints. Rate-distortion optimization is often formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on extensive testing of all coding options (including all considered modes or coding parameter values) and a complete evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster approaches can also be used to reduce coding complexity, particularly by calculating approximate distortion based on predictions or prediction residual signals rather than reconstructed residual signals. A hybrid of these two approaches may also be used, such as by using approximate distortion for only some of the possible coding options and full distortion for others. Other approaches only evaluate a subset of the possible coding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization does not necessarily involve a complete evaluation of both coding costs and associated distortion. However, according to at least one embodiment, since deep intra prediction mode is the only available mode for intra prediction, rate-distortion optimization in encoding is removed.

[0152] The implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., an apparatus or program).

[0153] The means may be implemented in, for example, appropriate hardware, software, and firmware.

[0154] The methods may be implemented in, for example, a processor, which is generally referred to as a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device.

[0155] Processors also include communication devices such as, for example, computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the communication of information between end users.

[0156] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places throughout this application are not necessarily all referring to the same embodiment.

[0157] Additionally, this application may refer to “determining” various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0158] Furthermore, this application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0159] Additionally, this application may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves, in one way or another, an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0160] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selecting only the first-listed option (A), or only the second-listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such phrases are intended to encompass selecting only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or only the first-listed option and the second-listed option (A and B), or only the first-listed option and the third-listed option (A and C), or only the second-listed option and the third-listed option (B and C), or all three options (A, B, and C). As will be apparent to one of ordinary skill in this and related arts, this can be extended to as many items as listed.

[0161] Also, as used herein, the word "signaling" refers to, among other things, indicating something to a corresponding decoder. For example, in some embodiments, the encoder signals one particular parameter among multiple parameters for deep intra prediction.

[0162] Like this, in one embodiment, both the encoder side and the decoder side use the same parameters. Therefore, for example, the encoder can transmit specific parameters (explicit signaling) to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and others, signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding transmitting any actual function, bit saving is achieved in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, tags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signal" is mentioned above, the word "signal" can also be used as a noun in this article.

[0163] It will be apparent to one of ordinary skill in the art that a specific implementation may generate various signals formatted to carry, for example, storable or transmittable information. The information may include, for example, instructions for performing a method or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bit stream of the described embodiment. Such signals may be formatted, for example, as electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or as baseband signals. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is known that the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0164] We have described a number of embodiments. The features of these embodiments may be provided individually or in any combination in the various claim classes and types.

Claims

1. A method for performing intra-frame prediction on an image block, the method comprising: assigning a mask value to unavailable pixels in a context including pixels surrounding the image block, wherein the mask value exceeds a range of available pixels in the context; subtracting a mean of the available pixels in the context from the available pixels in the context; providing the context to at least one neural network; obtaining, from the at least one neural network, an output image block comprising an intra prediction of the image block; The output image block is post-processed by adding the mean value to the intra prediction of the image block.

2. The method according to claim 1, further comprising: The image block is encoded based on the intra-frame prediction.

3. The method according to claim 2, further comprising: Information for accessing a neural network for intra prediction or information for defining access to a neural network for intra prediction is transmitted.

4. The method according to claim 1, further comprising: The image block is decoded based on the intra prediction.

5. The method according to claim 4, further comprising: Information for accessing a neural network for intra prediction or information for defining access to a neural network for intra prediction is received.

6. The method according to claim 1, further comprising: The context values ​​are adapted from the bit depth of the image block to the bit depth of the training image by scaling the context values ​​prior to input to at least one neural network and inversely scaling the output image block after post-processing from the bit depth of the training image to the bit depth of the image block, wherein the training image is used to train the at least one neural network.

7. A device for performing intra-frame prediction on an image block, the device comprising one or more processors, wherein: The one or more processors are configured to determine an intra prediction for an image block, comprising: assigning a mask value to unavailable pixels in a context including pixels surrounding the image block, wherein the mask value exceeds a range of available pixels in the context; subtracting a mean of the available pixels in the context from the available pixels in the context; providing the context to at least one neural network; obtaining, from the at least one neural network, an output image block comprising an intra prediction of the image block; The output image block is post-processed by adding the mean value to the intra prediction of the image block.

8. The apparatus according to claim 7, comprising: - an encoder configured to encode the image block based on the intra prediction.

9. The apparatus of claim 8, wherein the one or more processors are further configured to: Information for accessing a neural network for intra prediction or information for defining access to a neural network for intra prediction is transmitted.

10. The apparatus according to claim 7, comprising: A decoder is configured to decode the image block based on the intra-frame prediction.

11. The apparatus of claim 10, wherein the one or more processors are configured to: Information for accessing a neural network for intra prediction or information for defining access to a neural network for intra prediction is received.

12. The apparatus of claim 7, wherein the one or more processors are configured to: The context values ​​are adapted from the bit depth of the image block to the bit depth of the training image by scaling the context values ​​prior to input to at least one neural network and inversely scaling the output image block after post-processing from the bit depth of the training image to the bit depth of the image block, wherein the training image is used to train the at least one neural network.

13. A non-transitory computer-readable medium comprising program code instructions, which, when executed by a computer, cause the computer to perform the method according to any one of claims 1, 2, or 4.

Images