Intra prediction method and apparatus

Abstract

This application provides an intra prediction method and apparatus. This application relates to the field of artificial intelligence (AI)-based video or image compression technology, particularly neural network-based video compression technology. The method includes: obtaining intra prediction modes or texture distributions for each of P reconstructed picture blocks within a surrounding area of ​​a current block; obtaining Q a priori candidate intra prediction modes and Q probability values ​​for the current block based on the intra prediction modes or texture distributions for each of the P reconstructed picture blocks; obtaining M weighting factors corresponding to the M a priori candidate intra prediction modes based on the M probability values ​​corresponding to the M a priori candidate intra prediction modes; performing intra prediction separately based on the M a priori candidate intra prediction modes to obtain M predicted values; and obtaining a predicted value for the current block based on a weighted sum of the M predicted values ​​and the corresponding M weighting factors. This application can improve the accuracy of intra prediction, reduce intra prediction errors, and improve the RDO efficiency of intra prediction.

Description

[Technical Field]

[0001] This application claims priority to Chinese Patent Application No. 202011043931.1, entitled "Intra Prediction Method and Apparatus," filed with the State Intellectual Property Office of China on September 28, 2020, which is hereby incorporated by reference in its entirety.

[0002] TECHNICAL FIELD Embodiments of this application relate to the field of artificial intelligence (AI)-based video or image compression technology, and in particular to an intra prediction method and apparatus. [Background technology]

[0003] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, DVD and Blu-ray discs, video content collection and editing systems, and video camera security applications.

[0004] The amount of video data required to depict even a short video can be considerable, which can pose challenges when transmitting or otherwise transmitting the data over networks with limited bandwidth capacity. Therefore, video data is typically compressed before transmission over modern telecommunications networks. Video size can also be an issue when the video is stored on a storage device, since memory resources may be limited. Video compression devices often use software and / or hardware to encode the video data at the source prior to transmission or storage, thereby reducing the amount of data required to represent a digital video picture. The compressed data is then received by a video decompressor at the destination. With limited network resources and ever-increasing demands for higher video quality, improved compression and decompression techniques that improve compression ratios with little to no sacrifice in picture quality are desirable.

[0005] In recent years, deep learning has gained popularity in the field of image and video encoding and decoding. Summary of the Invention

[0006] This application provides an intra prediction method and apparatus for improving the accuracy of intra prediction, reducing the error of intra prediction, and improving the RDO efficiency of intra prediction.

[0007] According to a first aspect, the application provides an intra prediction method, the method including: obtaining intra prediction modes or texture distributions of each of P reconstructed picture blocks in a surrounding region of a current block, the surrounding region including a spatial neighborhood of the current block; obtaining Q a priori candidate intra prediction modes of the current block and Q probability values ​​of the current block corresponding to the Q a priori candidate intra prediction modes based on the intra prediction modes or texture distributions of each of the P reconstructed picture blocks; obtaining M weighting factors corresponding to the M a priori candidate intra prediction modes based on the M probability values ​​corresponding to the M a priori candidate intra prediction modes; performing intra prediction separately based on the M a priori candidate intra prediction modes to obtain M predicted values; and obtaining a predicted value of the current block based on a weighted sum of the M predicted values ​​and the corresponding M weighting factors.

[0008] The surrounding area of ​​the current block includes the spatial neighborhood of the current block. The spatial neighborhood picture blocks may include a left candidate picture block located to the left of the current block and a top candidate picture block located above the current block. The reconstructed picture block may be a coded picture block coded at the encoder side, and the reconstructed picture block is obtained at the encoder side, or a decoded picture block decoded and reconstructed at the decoder side. The reconstructed picture block may also refer to a basic unit picture block of a predetermined size obtained by dividing a coded picture block or a decoded picture block into equal-sized blocks.

[0009] In solution 1, the intra-prediction mode of the reconstructed picture block may include (1) a plurality of a posteriori intra-prediction modes of the reconstructed picture block, which are determined based on the reconstruction values ​​of the reconstructed picture block and prediction values ​​corresponding to a plurality of a posteriori candidate intra-prediction modes, or (2) an optimal intra-prediction mode of the reconstructed picture block, which is the a posteriori intra-prediction mode with the largest probability value or the smallest prediction error value among the plurality of a posteriori intra-prediction modes.

[0010] The multiple a posteriori candidate intra prediction modes of the reconstructed picture block are obtained based on the multiple a priori candidate intra prediction modes of the reconstructed picture block. The multiple a posteriori candidate intra prediction modes may be the multiple a priori candidate intra prediction modes of the reconstructed picture block, or may be some intra prediction modes among the multiple a priori candidate intra prediction modes of the reconstructed picture block. All of the multiple a posteriori candidate intra prediction modes of each of the P reconstructed picture blocks mentioned above can be obtained based on this method, and they will not be listed here.

[0011] The multiple a posteriori intra prediction modes of the reconstructed picture block may refer to the multiple a posteriori candidate intra prediction modes, or may refer to some intra prediction modes among the multiple a posteriori candidate intra prediction modes, for example, multiple specified intra prediction modes selected from the multiple a posteriori candidate intra prediction modes. All of the multiple a posteriori intra prediction modes of each of the P reconstructed picture blocks mentioned above can be obtained based on this method, and they will not be listed here.

[0012] In solution 2, the texture distribution of the reconstructed picture block includes a horizontal texture distribution of the reconstructed picture block and a vertical texture distribution of the reconstructed picture block.

[0013] The texture of a picture is a visual feature that reflects the homogeneity phenomenon within a picture and reflects the organization and arrangement attributes of slowly or periodically changing surface structures on the surface of an object. Unlike picture features such as grayscale and color, texture is represented by the grayscale distribution of pixels and their surrounding spatial neighborhoods. Unlike color features, texture features are not sample-based features but need to be statistically calculated within an area containing multiple samples. The texture of a reconstructed picture block can be considered to include a large number of texture primitives. The texture distribution of a reconstructed picture block is analyzed based on the texture primitives. The texture representation depends on the different types, orientations, and number of texture primitives. The horizontal texture distribution of a reconstructed picture block can indicate horizontal texture features by using the types and number of texture primitives in the horizontal direction, and the vertical texture distribution can indicate vertical texture features by using the types and number of texture primitives in the vertical direction.

[0014] The intra-prediction modes or texture distributions of each of the P reconstructed picture blocks may be input to a neural network to obtain Q a priori candidate intra-prediction modes and Q probability values ​​for the current block corresponding to the Q a priori candidate intra-prediction modes. For details about the neural network, please refer to the description of the training engine 25. The details will not be described again here.

[0015] The Q a priori candidate intra prediction modes of the current block may refer to all of the remaining intra prediction modes after the multiple a posteriori intra prediction modes of each of the P reconstructed picture blocks have been de-duplicated, or may refer to a portion of all of the remaining intra prediction modes after the multiple a posteriori intra prediction modes of each of the P reconstructed picture blocks have been de-duplicated.

[0016] Optionally, M = Q. In this case, the M probability values refer to the Q probability values, and the M deductive candidate intra prediction modes refer to the Q deductive candidate intra prediction modes.

[0017] Optionally, M < Q. In this case, all of the M probability values are greater than the probability values other than the M probability values among the Q probability values, and the M deductive candidate intra prediction modes corresponding to those M probability values are selected from the Q deductive candidate intra prediction modes of the current block. That is, from the Q probability values of the current block corresponding to the Q deductive candidate intra prediction modes, the first M probability values with the largest probability values are selected, the M deductive candidate intra prediction modes corresponding to those M probability values are selected from the Q deductive candidate intra prediction modes of the current block, and based on those M probability values and those M deductive candidate intra prediction modes, a weight coefficient and a prediction value are calculated to obtain the prediction value of the current block. However, among the plurality of probability values of the current block corresponding to the plurality of deductive candidate intra prediction modes, the remaining probability values other than the M probability values can be ignored because of their small values. Thus, the amount of calculation can be reduced and the efficiency of intra prediction can be improved.

[0018] In particular, "corresponding" in "the M probability values corresponding to the M deductive candidate intra prediction modes" does not mean a one-to-one correspondence. For example, the current block has five deductive candidate intra prediction modes, and the plurality of probability values corresponding to the five deductive candidate intra prediction modes may be five probability values or may be less than five probability values.

[0019] When the sum of the M probability values ​​is 1, the probability value corresponding to the first a priori candidate intra-prediction mode is used as the weighting factor corresponding to the first a priori candidate intra-prediction mode. That is, the weighting factor of each of the M a priori candidate intra-prediction modes is the respective probability value of the M a priori candidate intra-prediction modes. Alternatively, when the sum of the M probability values ​​is not 1, a normalization process is performed on the M probability values, and the normalized value of the probability value corresponding to the first a priori candidate intra-prediction mode is used as the weighting factor corresponding to the first a priori candidate intra-prediction mode. That is, the weighting factor of each of the M a priori candidate intra-prediction modes is the normalized value of the respective probability value of the M a priori candidate intra-prediction modes. The term "first a priori candidate intra-prediction mode" is merely a noun used for ease of explanation and does not refer to a specific a priori candidate intra-prediction mode, but rather represents any one of the Q a priori candidate intra-prediction modes. It can be seen that the sum of the weighting factors corresponding to the M a priori candidate intra-prediction modes is one.

[0020] According to the principle of intra prediction, in a candidate intra prediction mode, a reference block can be found in the surrounding area of ​​the current block, and intra prediction is performed on the current block based on the reference block to obtain a predicted value corresponding to the candidate intra prediction mode.It can be seen that the predicted value of the current block corresponds to the candidate intra prediction mode.Therefore, intra prediction can be performed separately according to M a priori candidate intra prediction modes to obtain M predicted values ​​of the current block.

[0021] The predicted value of the current block is obtained based on a weighted sum of the M predicted values ​​and the corresponding M weighting factors. As mentioned above, the M predicted values ​​correspond to the M a priori candidate intra prediction modes, and the M weighting factors also correspond to the M a priori candidate intra prediction modes. Therefore, for the same a priori candidate intra prediction mode, a correspondence relationship between the predicted value and the weighting factor corresponding to the same a priori candidate intra prediction mode is also established, the weighting factor corresponding to the a priori candidate motion vector is multiplied by the predicted value corresponding to the same a priori candidate intra prediction mode, and the multiple products corresponding to the multiple a priori candidate intra prediction modes are added together to obtain the predicted value of the current block.

[0022] In this application, multiple weighting factors and multiple predicted values ​​of a current block are obtained based on the intra-prediction information of each of multiple reconstructed picture blocks in a surrounding area of ​​the current block, and a weighting factor corresponding to a priori candidate intra-prediction mode is multiplied by a predicted value corresponding to the same priori candidate intra-prediction mode, and then a predicted value of the current block is obtained by adding together multiple products corresponding to the multiple priori candidate intra-prediction modes. In this way, the predicted value of the current block is obtained by combining multiple priori candidate intra-prediction modes, which can better fit rich and variable textures in the real world, thereby improving the accuracy of intra-prediction, reducing intra-prediction errors, and improving the overall rate-distortion optimization (RDO) efficiency of intra-prediction.

[0023] In one possible implementation, in addition to the intra-prediction modes of each of the P reconstructed picture blocks, related information of each of the P reconstructed picture blocks may also be obtained. The related information of a reconstructed picture block may be a plurality of recursive intra-prediction modes of the reconstructed picture block and a plurality of prediction error values ​​corresponding to the plurality of recursive intra-prediction modes. The plurality of recursive intra-prediction modes and the plurality of prediction error values ​​corresponding to the plurality of recursive intra-prediction modes are determined based on the reconstructed values ​​of the reconstructed picture block and prediction values ​​corresponding to the plurality of candidate recursive intra-prediction modes.

[0024] Intra prediction is performed separately based on a plurality of a posteriori candidate intra prediction modes of the reconstructed picture block to obtain a plurality of predicted values, and the plurality of predicted values ​​correspond to the plurality of a posteriori candidate intra prediction modes.

[0025] A plurality of prediction values ​​are compared with the reconstructed values ​​of the reconstructed picture block to obtain a plurality of prediction error values, which correspond to the plurality of a posteriori candidate intra-prediction modes. In this application, the prediction error values ​​corresponding to the a posteriori candidate intra-prediction modes may be obtained using methods such as sum of absolute differences (SAD) or sum of squared differences (SSE).

[0026] When the multiple a posteriori intra-prediction modes of the reconstructed picture block refer to multiple a posteriori candidate intra-prediction modes, the multiple prediction error values ​​of the reconstructed picture block corresponding to the multiple a posteriori intra-prediction modes refer to multiple prediction error values ​​corresponding to the multiple a posteriori candidate intra-prediction modes. When the multiple a posteriori intra-prediction modes of the reconstructed picture block refer to some intra-prediction modes among the multiple a posteriori candidate intra-prediction modes, the multiple prediction error values ​​of the reconstructed picture block corresponding to the multiple a posteriori intra-prediction modes refer to prediction error values ​​corresponding to those intra-prediction modes selected from the multiple prediction error values ​​corresponding to the multiple a posteriori candidate intra-prediction modes.

[0027] Correspondingly, the input to the neural network includes a plurality of a posteriori intra-prediction modes for each of the P reconstructed picture blocks and a plurality of prediction error values ​​corresponding to the plurality of a posteriori intra-prediction modes.

[0028] In one possible implementation, in addition to the intra-prediction modes of each of the P reconstructed picture blocks, related information of each of the P reconstructed picture blocks may also be obtained. The related information of a reconstructed picture block may be a plurality of recursive intra-prediction modes of the reconstructed picture block and a plurality of probability values ​​corresponding to the plurality of recursive intra-prediction modes. The plurality of recursive intra-prediction modes and the plurality of probability values ​​corresponding to the plurality of recursive intra-prediction modes are determined based on the reconstructed values ​​of the reconstructed picture block and prediction values ​​corresponding to a plurality of candidate recursive intra-prediction modes.

[0029] The multiple probability values ​​of the reconstructed picture block corresponding to the multiple recursive intra prediction modes may be obtained based on the following two methods.

[0030] One method is to obtain multiple probability values ​​for the reconstructed picture block based on the multiple prediction error values ​​of the reconstructed picture block obtained by the above-mentioned method. For example, to obtain normalized values ​​of the multiple prediction error values, a normalization process may be performed on the multiple prediction error values ​​of the reconstructed picture block based on a method such as a normalized exponential function or a linear normalization method. The normalized values ​​of the multiple prediction error values ​​are the multiple probability values ​​of the reconstructed picture block. Based on the correspondence between the multiple prediction error values ​​of the reconstructed picture block and the multiple a posteriori intra prediction modes, the multiple probability values ​​of the reconstructed picture block also correspond to the multiple a posteriori intra prediction modes of the reconstructed picture block, and the probability value may represent the probability that the a posteriori intra prediction mode corresponding to the probability value is the optimal intra prediction mode of the reconstructed picture block.

[0031] The other method is to input the reconstructed value of the reconstructed picture block and multiple predicted values ​​of the reconstructed picture block obtained by the first method into a trained neural network to obtain multiple probability values ​​of the reconstructed picture block corresponding to multiple recursive intra-prediction modes. For details about the neural network, please refer to the description of the training engine 25. The details will not be described again here.

[0032] Correspondingly, the input to the neural network includes a plurality of recursive intra-prediction modes for each of the P reconstructed picture blocks and a plurality of probability values ​​corresponding to the plurality of recursive intra-prediction modes.

[0033] Therefore, after multiple prediction error values ​​or probability values ​​corresponding to multiple recursive intra-prediction modes are obtained based on the two methods described above, the optimal intra-prediction mode of the reconstructed picture block can be obtained based on the following two methods.

[0034] One is to use the recursive intra prediction mode corresponding to the smallest prediction error value among multiple prediction error values ​​corresponding to multiple recursive intra prediction modes as the optimal intra prediction mode for the reconstructed picture block.

[0035] The other is to use the recursive intra prediction mode corresponding to the maximum probability value among a plurality of probability values ​​corresponding to a plurality of recursive intra prediction modes as the optimal intra prediction mode for the reconstructed picture block.

[0036] In particular, the optimal intra prediction mode in this application is only the intra prediction mode obtained based on one of the two methods described above, and is one of multiple a posteriori intra prediction modes of the reconstructed picture block. However, the optimal intra prediction mode is not the inherent intra prediction mode used when inter prediction is performed on the reconstructed picture block.

[0037] In one possible implementation, after the reconstruction value of the current block is obtained, the intra prediction mode or texture distribution of the current block can be obtained directly. The obtaining method includes:

[0038] 1. Based on the reconstruction value of the current block and the predicted values ​​corresponding to the multiple a priori candidate intra prediction modes of the current block, multiple a priori intra prediction modes of the current block and multiple prediction error values ​​of the current block corresponding to the multiple a priori intra prediction modes are obtained, and the multiple a priori intra prediction modes of the current block are obtained based on the multiple a priori candidate intra prediction modes of the current block.

[0039] 2. Based on the reconstruction value of the current block and the prediction values ​​corresponding to the multiple a priori candidate intra prediction modes of the current block input into the neural network, obtain multiple a priori intra prediction modes of the current block and multiple probability values ​​of the current block corresponding to the multiple a priori intra prediction modes, where the multiple a priori intra prediction modes of the current block are obtained based on the multiple a priori candidate intra prediction modes of the current block, or obtain multiple probability values ​​corresponding to the multiple a priori intra prediction modes of the current block based on the multiple prediction error values ​​of the current block.

[0040] 3. The a posteriori intra prediction mode having the maximum probability value or the minimum prediction error value among the plurality of a posteriori intra prediction modes of the current block is determined as the optimal intra prediction mode of the current block.

[0041] 4. Obtain the horizontal texture distribution and vertical texture of the current block.

[0042] In one possible implementation, the training data set on which the training engine trains the neural network includes information about multiple groups of picture blocks, where the information about the picture blocks in each group includes multiple recursive intra-prediction modes for each of multiple reconstructed picture blocks, multiple probability values ​​corresponding to the multiple recursive intra-prediction modes, multiple recursive intra-prediction modes for a current block, and multiple probability values ​​for the current block corresponding to the multiple recursive intra-prediction modes. The multiple reconstructed picture blocks are picture blocks within a spatial neighborhood of the current block. The neural network is obtained through training based on the training data set.

[0043] In one possible implementation, the training data set based on which the training engine trains the neural network includes information about multiple groups of picture blocks, where the information about the picture blocks in each group includes multiple recursive intra-prediction modes for each of multiple reconstructed picture blocks, multiple prediction error values ​​corresponding to the multiple recursive intra-prediction modes, multiple recursive intra-prediction modes for a current block, and multiple probability values ​​for the current block corresponding to the multiple recursive intra-prediction modes. The multiple reconstructed picture blocks are picture blocks within a spatial neighborhood of the current block. The neural network is obtained through training based on the training data set.

[0044] In one possible implementation, the training engine trains the neural network based on a training data set that includes information about multiple groups of picture blocks, where the information about each group of picture blocks includes an optimal intra-prediction mode for each of multiple reconstructed picture blocks, multiple a posteriori intra-prediction modes for a current block, and multiple probability values ​​for the current block corresponding to the multiple a posteriori intra-prediction modes. The multiple reconstructed picture blocks are neighbors of the current block. The neural network is obtained through training based on the training data set.

[0045] In one possible implementation, the training engine trains the neural network based on a training data set that includes information about multiple groups of picture blocks, where the information about each group of picture blocks includes a horizontal texture distribution and a vertical texture distribution of each of multiple reconstructed picture blocks, multiple a posteriori intra-prediction modes of a current block, and multiple probability values ​​of the current block corresponding to the multiple a posteriori intra-prediction modes. The multiple reconstructed picture blocks are picture blocks within a spatial neighborhood of the current block. The neural network is obtained through training based on the training data set.

[0046] Optionally, the neural network includes at least a convolutional layer and an activation layer. The depth of the convolutional kernel of the convolutional layer is 2, 3, 4, 5, 6, 16, 24, 32, 48, 64, or 128, and the size of the convolutional kernel of the convolutional layer is 1×1, 3×3, 5×5, or 7×7. For example, the size of the convolutional layer is 3×3×2×10, where 3×3 represents the size of the convolutional kernel in the convolutional layer, 2 represents the depth of the convolutional kernel included in the convolutional layer, the number of data channels input to the convolutional layer is the same as the depth of the convolutional kernel included in the convolutional layer, i.e., the number of data channels input to the convolutional layer is also 2, and 10 represents the number of convolutional kernels included in the convolutional layer, and the number of data channels output from the convolutional layer is the same as the number of convolutional kernels included in the convolutional layer, i.e., the number of data channels output from the convolutional layer is also 10.

[0047] Optionally, the neural network comprises a convolutional neural network CNN, a deep neural network DNN, or a recurrent neural network RNN.

[0048] According to a second aspect, the application provides an encoder including a processing circuit configured to perform a method according to any one of the first aspects.

[0049] According to a third aspect, the application provides a decoder including processing circuitry configured to perform a method according to any one of the first aspects.

[0050] According to a fourth aspect, the application provides a computer program product including program code, the computer program product being configured to perform a method according to any one of the first aspects when the computer program product is run on a computer or processor.

[0051] According to a fifth aspect, the application provides an encoder comprising one or more processors and a non-transitory computer-readable storage medium, coupled to the processors, storing a program for execution by the processors, the program, when executed by the processors, enabling the encoder to perform a method according to any one of the first aspects.

[0052] According to a sixth aspect, the application provides a decoder comprising one or more processors and a non-transitory computer-readable storage medium, coupled to the processors, storing a program for execution by the processors, the program, when executed by the processors, enabling the decoder to perform a method according to any one of the first aspects.

[0053] According to a seventh aspect, the application provides a non-transitory computer readable storage medium comprising program code, the program code being configured to perform a method according to any one of the first aspects when executed by a computing device.

[0054] According to an eighth aspect, the present invention relates to a decoding device. For beneficial effects, please refer to the description of the first aspect. Details will not be described again here. The decoding device has a function of performing the operations of the method embodiments of the first aspect. The functions may be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the decoding device includes an intra prediction module configured to perform the method according to any one of the first aspects. These modules may implement corresponding functions in the method examples of the first aspect. For details, please refer to the detailed description of the method examples. Details will not be described again here.

[0055] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0056] [Figure 1a] 1 is an example block diagram of a coding system 10 according to an embodiment of the present application. [Figure 1b] 1 is an example block diagram of a video coding system 40 according to an embodiment of the present application. [Figure 2] 2 is an example block diagram of a video encoder 20 according to an embodiment of the present application. [Figure 3] 2 is an example block diagram of a video decoder 30 according to an embodiment of the present application. [Figure 4] 4 is an example block diagram of a video coding system 400 according to an embodiment of the present application. [Figure 5] 5 is an example of a block diagram of an apparatus 500 according to an embodiment of the present application. [Figure 6a] 6a to 6e show some example architectures of neural networks for intra prediction according to one embodiment of this application. [Figure 6b] 6a to 6e show some example architectures of neural networks for intra prediction according to one embodiment of this application. [Figure 6c] 6a to 6e show some example architectures of neural networks for intra prediction according to one embodiment of this application. [Figure 6d] 6a to 6e show some example architectures of neural networks for intra prediction according to one embodiment of this application. [Figure 6e] 6a to 6e show some example architectures of neural networks for intra prediction according to one embodiment of this application. [Figure 7] 7 is a flowchart of a process 700 of an intra prediction method according to an embodiment of the present application. [Figure 8] 8 is a flowchart of a process 800 of an intra prediction method according to an embodiment of the present application. [Figure 9] 1 is an example of a schematic diagram of a reconstructed picture block in a surrounding region according to an embodiment of the present application; [Figure 10] 10 is a flowchart of a process 1000 of an intra prediction method according to an embodiment of the present application. [Figure 11] 11 is a flowchart of a process 1100 of an intra prediction method according to an embodiment of the present application. [Figure 12] 12 is a flowchart of a process 1200 of an intra prediction method according to an embodiment of the present application. [Figure 13] 13 is a schematic diagram of a configuration of a decoding device 1300 according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0057] Embodiments of the present application provide an AI-based video compression technique, in particular a neural network-based video compression technique, and more particularly a neural network (NN)-based intra prediction technique, for improving conventional hybrid video encoding and decoding systems.

[0058] Video coding typically refers to the processing of a series of pictures, which form a video or video sequence. In the field of video coding, the terms "picture," "frame," or "image" are sometimes used synonymously. Video coding (or, in general, coding) includes two parts: video encoding and video decoding. Video encoding is performed on the source side and typically involves processing the original video picture (e.g., by compression) to reduce the amount of data needed to represent the video picture (for more efficient storage and / or transmission). Video decoding is performed on the destination side and typically involves the reverse process to the encoder to reconstruct the video picture. Embodiments referring to "coding" a video picture (or, in general, a picture) shall be understood to relate to "encoding" or "decoding" a video picture or respective video sequence. The combination of the encoding and decoding parts is also referred to as CODEC (Coding and Decoding).

[0059] In the case of lossless video coding, the original video picture can be reconstructed. In other words, the reconstructed video picture has the same quality as the original video picture (assuming no transmission or other data loss occurs during storage or transmission). In the case of lossy video coding, further compression is performed, for example through quantization, to reduce the amount of data required to represent the video picture, and the video picture cannot be perfectly reconstructed at the decoder side. In other words, the quality of the reconstructed video picture is inferior to the quality of the original video picture.

[0060] Some video coding standards use "lossy hybrid video coding" (i.e., spatial and temporal prediction in the sample domain are combined with 2D transform coding that applies quantization in the transform domain). Each picture in a video sequence is typically divided into a set of non-overlapping blocks, and coding is typically performed at the block level. Specifically, at the encoder side, video is usually processed, i.e., encoded, at the block (video block) level. For example, a predictive block is generated through spatial (intra) prediction and temporal (inter) prediction, the predictive block is subtracted from a current block (being processed or to be processed) to obtain a residual block, and the residual block is transformed and quantized in the transform domain to reduce the amount of data to be transmitted (compression). At the decoder side, the encoded or compressed block is subjected to the inverse processing steps of the encoder to reconstruct the current block for representation. Furthermore, the encoder replicates the decoder processing loop, so that both generate the same predictions (e.g., intra-prediction and inter-prediction) and / or reconstructions for processing, i.e., coding, subsequent blocks.

[0061] In the following embodiments of coding system 10, video encoder 20 and video decoder 30 are described with reference to FIGS. 1a-3.

[0062] 1a is an example block diagram of a coding system 10 according to an embodiment of this application, e.g., video coding system 10 (or coding system 10 for short), which may employ techniques of this application. Video encoder 20 (or encoder 20 for short) and video decoder 30 (or decoder 30 for short) of video coding system 10 represent example devices that may be configured to perform techniques according to various examples described in this application.

[0063] As shown in Figure 1a, coding system 10 includes a source device 12. Source device 12 is configured to provide coded picture data 21, e.g., coded pictures, to a destination device 14, which decodes the coded picture data 21.

[0064] The source device 12 includes an encoder 20 and may additionally, i.e., optionally, include a picture source 16 , a pre-processor (or pre-processing unit) 18 , such as a picture pre-processor 18 , and a communication interface (or communication unit) 22 .

[0065] Picture source 16 may include or be any type of picture capture device, such as a camera for capturing real-world pictures, and / or any type of picture generation device, such as a computer graphics processor for generating computer-animated pictures, or any type of other device for obtaining and / or providing real-world pictures, computer-generated pictures (e.g., screen content or virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Picture source may also be any type of memory or storage for storing any of the above-mentioned pictures.

[0066] In distinction to the processing performed by the preprocessor (or preprocessing unit) 18, the picture (or picture data) 17 may also be referred to as a raw picture or raw picture data 17.

[0067] The pre-processor 18 is configured to receive the raw picture data 17 and perform pre-processing on the raw picture data 17 to obtain a pre-processed picture (or pre-processed picture data) 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be appreciated that the pre-processing unit 18 may be an optional component.

[0068] Video encoder (or encoder) 20 is configured to receive pre-processed picture data 19 and to provide coded picture data 21 (as described in further detail below, eg, with reference to FIG. 2).

[0069] The communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) over the communication channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0070] The destination device 14 includes a decoder 30 and may additionally, ie optionally, include a communications interface (or communications unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0071] The communications interface 28 of the destination device 14 is configured to receive the coded picture data 21 (or any further processed version thereof) directly from the source device 12 or from any other source device, for example a storage device, for example the storage device being a coded picture data storage device, and to provide the coded picture data 21 to the decoder 30.

[0072] The communication interface 22 and the communication interface 28 may be configured to transmit or receive encoded picture data (or encoded data) 21 via a direct communication link between the source device 12 and the destination device 14, for example a direct wired or wireless connection, or via any type of network, for example a wired or wireless network or any combination thereof, or any type of private network, any type of public network, or any type of combination thereof.

[0073] The communications interface 22 may be configured, for example, to package the coded picture data 21 into a suitable format, such as packets, and / or process the coded picture data using any type of transmission coding or processing for transmission over a communications link or network.

[0074] The communications interface 28 forms a counterpart to the communications interface 22 and may be configured, for example, to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or de-packaging to obtain encoded picture data 21.

[0075] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow corresponding to communication channel 13 in FIG. 1a pointing from source device 12 to destination device 14, or may be configured as bidirectional communication interfaces, and may be configured to send and receive messages, for example, to set up a communication link and / or connection to acknowledge and exchange other information related to data transmission, such as the transmission of coded picture data.

[0076] Video decoder (or decoder) 30 is configured to receive encoded picture data 21 and to provide decoded picture data (or decoded picture data) 31 (as described in further detail below, for example with reference to FIG. 3).

[0077] The post-processor 32 is configured to perform post-processing on the decoded picture data 31 (also called reconstructed video data), e.g., decoded picture 31, to obtain post-processed picture data 33, e.g., post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing, e.g., to prepare the decoded picture data 31 for display by a display device 34.

[0078] The display device 34 is configured to receive the post-processed picture data 33 for displaying the picture, e.g., to a user or viewer. The display device 34 may be or include any type of display for presenting the reconstructed picture, e.g., an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display.

[0079] Coding system 10 further includes a training engine 25. Training engine 25 is configured to train encoder 20 (particularly, an intra-prediction unit within encoder 20) or decoder 30 (particularly, an intra-prediction unit within decoder 30) to process input pictures, picture regions, or picture blocks and generate predictions for the input pictures, picture regions, or picture blocks.

[0080] Optionally, in an embodiment of this application, the training dataset includes information about multiple groups of picture blocks. The information about the picture blocks of each group includes multiple a posteriori intra-prediction modes for each of multiple reconstructed picture blocks and multiple probability values ​​corresponding to the multiple a posteriori intra-prediction modes, and multiple a posteriori intra-prediction modes for a current block and multiple probability values ​​for the current block corresponding to the multiple a posteriori intra-prediction modes. The multiple reconstructed picture blocks are picture blocks in a spatial neighborhood of the current block. A neural network is obtained through training based on the training dataset. The input to the neural network is multiple a posteriori intra-prediction modes for each of multiple reconstructed picture blocks in a surrounding area of ​​the current block and multiple probability values ​​corresponding to the multiple a posteriori intra-prediction modes, and the output from the neural network is multiple a posteriori candidate intra-prediction modes for the current block and multiple probability values ​​for the current block corresponding to the multiple a posteriori candidate intra-prediction modes.

[0081] Optionally, the training dataset in the embodiment of this application includes information about multiple groups of picture blocks. The information about the picture blocks of each group includes multiple a posteriori intra-prediction modes for each of multiple reconstructed picture blocks, multiple prediction error values ​​corresponding to the multiple a posteriori intra-prediction modes, multiple a posteriori intra-prediction modes for a current block, and multiple probability values ​​for the current block corresponding to the multiple a posteriori intra-prediction modes. The multiple reconstructed picture blocks are picture blocks in a spatial neighborhood of the current block. A neural network is obtained through training based on the training dataset. The inputs to the neural network are multiple a posteriori intra-prediction modes for each of multiple reconstructed picture blocks in a surrounding area of ​​the current block and multiple prediction error values ​​corresponding to the multiple a posteriori intra-prediction modes, and the outputs from the neural network are multiple a posteriori candidate intra-prediction modes for the current block and multiple probability values ​​for the current block corresponding to the multiple a posteriori candidate intra-prediction modes.

[0082] Optionally, in an embodiment of this application, the training dataset includes information about multiple groups of picture blocks. The information about the picture blocks of each group includes an optimal intra-prediction mode for each of multiple reconstructed picture blocks, multiple a priori candidate intra-prediction modes for a current block, and multiple probability values ​​for the current block corresponding to the multiple a priori candidate intra-prediction modes. The multiple reconstructed picture blocks are picture blocks in a spatial neighborhood of the current block. A neural network is obtained through training based on the training dataset. The input to the neural network is the optimal intra-prediction mode for each of multiple reconstructed picture blocks in a surrounding area of ​​the current block, and the output from the neural network is multiple a priori candidate intra-prediction modes for the current block and multiple probability values ​​for the current block corresponding to the multiple a priori candidate intra-prediction modes.

[0083] Optionally, in an embodiment of this application, the training dataset includes information about multiple groups of picture blocks. The information about the picture blocks of each group includes a horizontal texture distribution and a vertical texture distribution of each of multiple reconstructed picture blocks, a plurality of a priori candidate intra-prediction modes for a current block, and a plurality of probability values ​​for the current block corresponding to the plurality of a priori candidate intra-prediction modes. The plurality of reconstructed picture blocks are picture blocks in a spatial neighborhood of the current block. A neural network is obtained through training based on the training dataset. The input to the neural network is the horizontal texture distribution and the vertical texture distribution of each of multiple reconstructed picture blocks in a surrounding region of the current block, and the output from the neural network is a plurality of a priori candidate intra-prediction modes for the current block and a plurality of probability values ​​for the current block corresponding to the plurality of a priori candidate intra-prediction modes.

[0084] Optionally, in an embodiment of this application, the training dataset includes information about multiple groups of picture blocks, where the information about the picture blocks in each group includes a reconstructed value of the picture block, a predicted value corresponding to multiple candidate a posteriori intra-prediction modes, multiple a posteriori intra-prediction modes of the picture block, and multiple probability values ​​of the picture block corresponding to the multiple candidate a posteriori intra-prediction modes. A neural network is obtained through training based on the training dataset. The inputs to the neural network are the reconstructed value of the current block and the predicted values ​​corresponding to the multiple candidate a posteriori intra-prediction modes, and the outputs from the neural network are the multiple a posteriori intra-prediction modes of the current block and multiple probability values ​​of the current block corresponding to the multiple a posteriori intra-prediction modes.

[0085] During the neural network training process by the training engine 25, the output a priori candidate intra-prediction modes of the current block approximate the a posteriori intra-prediction modes of the current block, and the output probability values ​​corresponding to the a priori candidate intra-prediction modes approximate the probability values ​​corresponding to the a posteriori intra-prediction modes. Each training process may be performed with a stride of 10 using a small batch size of 64 pictures and an initial learning rate of 1e-4. The information about the groups of picture blocks may be data generated when the encoder performs intra-coding on the current blocks. The neural network can implement the intra-prediction method provided in the embodiments of this application. Specifically, the intra-prediction modes and related information of the reconstructed picture blocks in the surrounding area of ​​the current block are input to the neural network to obtain a priori candidate intra-prediction modes of the current block and a corresponding probability value of the current block. The neural network will be described in detail below with reference to Figures 6a to 6e.

[0086] The training data in the embodiment of this application may be stored in a database (not shown). The training engine 25 obtains a target model (which may be, for example, a neural network for intra-picture prediction) through training based on the training data. In particular, the source of the training data is not limited in the embodiment of this application. For example, the training data may be obtained from a cloud or another location for model training.

[0087] The target model in the embodiment of this application may specifically be an intra-prediction network. The target model will be described in detail below with reference to Figures 6a to 6e.

[0088] The target model obtained by the training engine 25 through training is applied to the coding system 10, for example, to the source device 12 (e.g., the encoder 20) or the destination device 14 (e.g., the decoder 30) shown in FIG. 1a. The training engine 25 may obtain the target model through training on the cloud, and the coding system 10 may download and use the target model from the cloud. Alternatively, the training engine 25 may obtain the target model through training on the cloud and use the target model, and the coding system 10 may obtain the processing result directly from the cloud. For example, the training engine 25 obtains a target model with intra-prediction capabilities through training. The coding system 10 downloads the target model from the cloud. Then, the intra-prediction unit 254 in the encoder 20 or the intra-prediction unit 354 in the decoder 30 may perform intra-prediction on an input picture or picture block based on the target model to obtain a prediction of the picture or picture block. In another example, the training engine 25 performs training to obtain a target model with intra-prediction capabilities. The coding system 10 does not need to download the target model from the cloud. The encoder 20 or decoder 30 sends a picture or picture block to the cloud, and the cloud performs intra prediction on the picture or picture block using the target model to obtain a prediction of the picture or picture block and sends the prediction to the encoder 20 or decoder 30.

[0089] 1a depicts source device 12 and destination device 14 as separate devices, an embodiment of the device may also include both source device 12 and destination device 14, or the functionality of both source device 12 and destination device 14, i.e., both source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, by separate hardware and / or software, or any combination thereof.

[0090] As will be clear to those skilled in the art based on the description, the presence and (exact) division of functions within the source device 12 and / or destination device 14 as shown in FIG. 1a may vary depending on the actual device and application.

[0091] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30, may be implemented by processing circuitry such as that shown in FIG. 1b, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding processors, or any combination thereof. Encoder 20 may be implemented by processing circuitry 46 to include various modules such as those described with reference to encoder 20 of FIG. 2 and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented by processing circuitry 46 to include various modules such as those described with reference to decoder 30 of FIG. 3 and / or any other decoder system or subsystem described herein. The processing circuitry may be configured to perform various operations, as described below. If any techniques are implemented partially in software, as shown in Figure 5, an apparatus may perform the techniques of this application by storing software instructions on a suitable non-transitory computer-readable storage medium and executing those instructions in hardware using one or more processors. Either video encoder 20 or video decoder 30 may be integrated as part of a combined encoder / decoder (CODEC) within a single apparatus, as shown in Figure 1b, for example.

[0092] Source device 12 and destination device 14 may include any of a wide range of devices, including any type of handheld or fixed device, such as, for example, a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or a content distribution server), a broadcast receiver device, a broadcast transmitter device, or the like, and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0093] In some cases, the video coding system 10 shown in FIG. 1a is merely an example, and the techniques provided herein may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily involve any data communication between an encoder and a decoder. In other examples, data may be retrieved from local memory and streamed over a network, etc. A video encoder may encode data and store the encoded data in memory, and / or a video decoder may retrieve data from memory and decode the data. In some examples, encoding and decoding are performed by devices that simply encode data to memory and / or retrieve data from memory and decode it without communicating with each other.

[0094] 1b is an example block diagram of a video coding system 40 according to an embodiment of the present application. As shown in FIG. 1b, the video coding system 40 may include an image capture device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0095] 1b, image capture device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 may be in communication with one another. Video coding system 40 may include only video encoder 20 or only video decoder 30 in different examples.

[0096] In some examples, antenna 42 may be configured to transmit or receive an encoded bitstream of video data. Furthermore, in some examples, display device 45 may be configured to present the video data. Processing circuitry 46 may include application-specific integrated circuit (ASIC) logic, a graphics processing unit, a general-purpose processor, or the like. Video coding system 40 may also include optional processor 43. Optional processor 43 may also include application-specific integrated circuit (ASIC) logic, a graphics processing unit, a general-purpose processor, or the like. Furthermore, memory 44 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM) or dynamic random access memory (DRAM)) or non-volatile memory (e.g., flash memory). In one non-limiting example, memory 44 may be implemented by a cache memory. In another example, processing circuitry 46 may include memory (e.g., a cache) for implementing a picture buffer.

[0097] In some examples, video encoder 20 implemented by logic circuitry may include a picture buffer (which may be implemented, for example, by processing circuitry 46 or memory 44) and a graphics processing unit (which may be implemented, for example, by processing circuitry 46). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit may include video encoder 20 implemented by processing circuitry 46 to embody various modules described with reference to FIG. 2 and / or any other encoder system or subsystem described herein. The logic circuitry may be configured to perform various operations described herein.

[0098] In some examples, video decoder 30 may be implemented by processing circuitry 46 in a similar manner to embody the various modules described with reference to video decoder 30 of FIG. 3 and / or any other decoder system or subsystem described herein. In some examples, video decoder 30 implemented by logic circuitry may include a picture buffer (which may be implemented by processing circuitry 46 or memory 44) and a graphics processing unit (which may be implemented, for example, by processing circuitry 46). The graphics processing unit may be communicatively coupled to the picture buffer. The graphics processing unit may include video decoder 30 implemented by logic circuitry 46 to embody the various modules described with reference to FIG. 3 and / or any other decoder system or subsystem described herein.

[0099] In some examples, antenna 42 may be configured to receive an encoded bitstream of video data. As described, the encoded bitstream may include data related to the video frame coding described herein, such as data related to a coding partition (e.g., transform coefficients or quantized transform coefficients, optional indicators (to be described), and / or data defining the coding partition), indicators, index values, mode selection data, or the like. Video coding system 40 may further include a video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0100] It should be understood that, in embodiments of this application, with respect to the examples described with reference to video encoder 20, video decoder 30 may be configured to perform an inverse process. With respect to signaling syntax elements, video decoder 30 may be configured to receive and parse such syntax elements and correspondingly decode the associated video data. In some examples, video encoder 20 may entropy encode the syntax elements into the encoded video bitstream. In such examples, video decoder 30 may parse such syntax elements and correspondingly decode the associated video data.

[0101] For ease of explanation, embodiments of this application are described herein with reference to the versatile video coding (VVC) reference software or high-efficiency video coding (HEVC) developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of this application are not limited to HEVC or VVC.

[0102] Encoder and encoding method FIG. 2 is an example block diagram of a video encoder 20 according to an embodiment of the present application. As shown in FIG. 2, the video encoder 20 includes an input end (or input interface) 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output end (or output interface) 272. The mode selection unit 260 may include an inter prediction unit 244, an intra prediction unit 254, and a division unit 262. The inter prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in FIG. 2 may also be referred to as a hybrid video encoder or a video encoder based on a hybrid video codec.

[0103] Referring to Figure 2, the intra prediction unit is a trained target model (also referred to as a neural network). The neural network is configured to process an input picture, picture region, or picture block to generate a prediction value for the input picture block. For example, a neural network for intra prediction is configured to receive an input picture, picture region, or picture block and generate a prediction value for the input picture, picture region, or picture block. Below, a neural network for intra prediction is described in detail with reference to Figures 6a to 6e.

[0104] The residual calculation unit 204, the transform processing unit 206, the quantization unit 208, and the mode selection unit 260 may form a forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction unit 244, and the intra prediction unit 254 may form a backward signal path of the video encoder 20. The backward signal path of the video encoder 20 corresponds to the signal path of a decoder (see video decoder 30 of FIG. 3). The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction unit 244, and the intra prediction unit 254 also form a “built-in decoder” of the video encoder 20.

[0105] Picture & Picture Split (Picture & Block) The encoder 20 may be configured to receive, e.g. via input 201, a picture (or picture data) 17, e.g. a picture in a series of pictures forming a video or a video sequence. The received picture or picture data may also be a preprocessed picture (or preprocessed picture data) 19. For simplicity, the following description uses picture 17. Picture 17 may also be referred to as a current picture or a picture to be coded (particularly in video coding, to distinguish the current picture from other pictures, e.g. previously coded and / or decoded pictures of the same video sequence, i.e. the video sequence that also contains the current picture).

[0106] A (digital) picture is, or can be considered as, a two-dimensional array or matrix of samples with intensity values. The samples in the array may also be referred to as pixels (or pels) (short for picture element). The number of samples in the horizontal and vertical directions (or axes) of the array or picture determines the size and / or resolution of the picture. For color representation, three color components are usually used; specifically, a picture may be represented as or contain three sample arrays. In an RGB format or color space, a picture contains corresponding red, green, and blue sample arrays. However, in video coding, each pixel is typically represented in a luminance and chrominance format or color space, such as YCbCr, which has a luminance component denoted by Y (although L may alternatively be used) and two chrominance components denoted by Cb and Cr. The luminance (luma) component Y represents brightness or gray-level intensity (e.g., as in a grayscale picture), and the two chrominance (abbreviated as chrominance) components Cb and Cr represent chromaticity or color information components. Thus, a picture in YCbCr format includes a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). A picture in RGB format can be converted to YCbCr format, or vice versa; this process is also known as color transformation. If a picture is monochrome, it may include only a luminance sample array. Thus, a picture can be, for example, an array of luma samples in a monochrome format, or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0107] Embodiments of video encoder 20 may include a picture division unit (not shown in FIG. 2 ) configured to divide picture 17 into multiple (typically non-overlapping) picture blocks 203. These blocks are sometimes referred to as root blocks, macroblocks (H.264 / AVC), coding tree blocks (CTBs), or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. The picture division unit may be configured to divide each picture into corresponding blocks using the same block size for all pictures in a video sequence and a corresponding grid that defines the block size, or varying the block size between pictures or between subsets or groups of pictures.

[0108] In further embodiments, the video encoder may be configured to directly receive blocks 203 of picture 17, such as one, some, or all of the blocks that form picture 17. Picture blocks 203 may also be referred to as current picture blocks or to-be-coded picture blocks.

[0109] Like picture 17, picture block 203 is also considered, or can be considered, as a two-dimensional array or matrix of samples having intensity values ​​(sample values), but of smaller dimensions than picture 17. In other words, block 203 may include, for example, one sample array (e.g., a luma array in the case of a monochrome picture 17, or a luma array or a chroma array in the case of a color picture), or three sample arrays (e.g., a luma array and two chroma arrays in the case of a color picture 17), or some other number and / or type of array depending on the color format applied. The number of samples in the horizontal and vertical directions (or axes) of block 203 determine the size of block 203. Thus, a block may be, for example, an M×N (M columns by N rows) array of samples, or an M×N array of transform coefficients.

[0110] In one embodiment, the video encoder 20 shown in FIG. 2 may be configured to encode the picture 17 on a block-by-block basis, eg, encoding and prediction is performed on a block-by-block basis.

[0111] In one embodiment, the video encoder 20 shown in FIG. 2 may be further configured to divide and / or encode a picture by using slices (also referred to as video slices). A picture may be divided into or encoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., coding tree units CTUs) or one or more groups of blocks (e.g., tiles in the H.265 / HEVC / VVC standard or bricks in the VVC standard).

[0112] In one embodiment, video encoder 20 shown in FIG. 2 may be further configured to divide and / or encode a picture by using slices / tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles). A picture may be divided into or encoded using one or more slices / tile groups (typically non-overlapping), and each slice / tile group may include, for example, one or more blocks (e.g., CTUs) or one or more tiles. Each tile may be, for example, rectangular in shape, and may include one or more blocks (e.g., CTUs), e.g., full or partial blocks.

[0113] Residual calculation The residual calculation unit 204 is configured to calculate the residual block 205 based on the picture block (or row block) 203 and the prediction block 265 (the prediction block 265 will be described in detail later), for example, by subtracting sample values ​​of the prediction block 265 from sample values ​​of the picture block 203 on a sample-by-sample (pixel-by-pixel) basis to obtain the residual block 205 in the pixel domain.

[0114] conversion The transform processing unit 206 may be configured to apply a transform, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), to the sample values ​​of the residual block 205 to obtain transform coefficients 207 in a transform domain. The transform coefficients 207, which may also be referred to as transform residual coefficients, may represent the residual block 205 in the transform domain.

[0115] The transform processing unit 206 may be configured to apply an integer approximation of a DCT / DST, such as the transform specified in HEVC / H.265. Compared to an orthogonal DCT transform, such an integer approximation is typically scaled based on a factor. To preserve the norm of the residual block processed through the forward and inverse transforms, an additional scale factor is applied as part of the transform process. The scale factor is typically selected based on several constraints, such as the scale factor being a power of two due to shift operations, the bit depth of the transform coefficients, and a trade-off between accuracy and implementation cost. Specific scale factors may be specified, for example, for the inverse transform by the inverse transform processing unit 212 on the encoder side 20 (and for the corresponding inverse transform by the inverse transform processing unit 312 on the decoder side 30), and corresponding scale factors for the forward transform by the transform processing unit 206 on the encoder side 20 may also be specified.

[0116] An embodiment of the video encoder 20 (and correspondingly, the transform processing unit 206) may be configured to output transform parameters, e.g., one or more transform types, via the entropy coding unit 270, e.g., directly or in an encoded or compressed form, so that, for example, the video decoder 30 may receive and use the transform parameters for decoding.

[0117] Quantization The quantization unit 208 may be configured to quantize the transform coefficients 207, for example by applying scalar quantization or vector quantization, to obtain quantized coefficients 209. The quantized transform coefficients 209 may also be referred to as quantized transform coefficients 209.

[0118] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, an n-bit transform coefficient may be rounded to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization may be changed by adjusting a quantization parameter (QP). For example, in the case of scalar quantization, different scales may be applied to achieve finer or coarser quantization. A smaller quantization stride corresponds to finer quantization, and a larger quantization stride corresponds to coarser quantization. An appropriate quantization stride may be indicated by the quantization parameter (QP). For example, the quantization parameter may be an index into a predetermined set of appropriate quantization strides. For example, a smaller quantization parameter may correspond to finer quantization (smaller quantization stride), and a larger quantization parameter may correspond to coarser quantization (larger quantization stride), or vice versa. Quantization may include division by the quantization stride, and corresponding or inverse dequantization, e.g., by the inverse quantization unit 210, may include multiplication by the quantization stride. Some standards, such as HEVC, may be configured to determine the quantization stride using a quantization parameter. In general, the quantization stride may be calculated based on the quantization parameter by using a fixed-point approximation of a formula involving division. Additional scale factors may be introduced for quantization and dequantization to restore the norm of the residual block, and the norm of the residual block may be modified due to the scale used in the fixed-point approximation of the formula for the quantization stride and the quantization parameter. In one implementation, scaling of the inverse transform and dequantization may be combined. Alternatively, customized quantization tables may be used and signaled from the encoder to the decoder, e.g., in the bitstream. Quantization is a lossy operation, and loss increases with increasing quantization stride.

[0119] An embodiment of video encoder 20 (and correspondingly, quantization unit 208) may be configured to output a quantization parameter (QP), e.g., directly or encoded, via entropy coding unit 270, so that, for example, video decoder 30 may receive and apply the quantization parameter for decoding.

[0120] inverse quantization Inverse quantization unit 210 is configured to apply the inverse quantization of quantization unit 208 to the quantized coefficients, e.g., by applying the inverse of the quantization scheme applied by quantization unit 208, based on or using the same quantization stride as quantization unit 208, to obtain dequantized coefficients 211. The dequantized coefficients 211, which are sometimes referred to as dequantized residual coefficients 211, correspond to the transform coefficients 207, but are typically not the same as the transform coefficients due to loss due to quantization.

[0121] Inverse transformation The inverse transform processing unit 212 is configured to apply an inverse transform of the transform applied by the transform processing unit 206, such as an inverse discrete cosine transform (DCT), an inverse discrete sine transform (DST), or other inverse transform, to obtain a reconstructed residual block 213 (or corresponding dequantized coefficients 213) in the sample domain. The reconstructed residual block 213 is sometimes referred to as a transform block 213.

[0122] Reconstruction The reconstruction unit 214 (e.g., adder or summer 214) is configured to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265, for example by adding the sample values ​​of the reconstructed residual block 213 and the sample values ​​of the prediction block 265 sample by sample, to obtain a reconstructed block 215 in the sample domain.

[0123] filtering The loop filter unit 220 (or "loop filter" 220 for short) is configured to filter the reconstructed block 215 to obtain a filtered block 221, or generally, to filter reconstructed samples to obtain filtered sample values. The loop filter unit is configured, for example, to smooth pixel transitions or otherwise improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a bilateral filter, an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. In one example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The filtering process order may be a deblocking filter, an SAO, and an ALF. In another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. In another example, the deblocking filter process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although loop filter unit 220 is shown as an in-loop filter in FIG. 2, in other configurations, loop filter unit 220 may be implemented as a post-loop filter. Filtered block 221 may also be referred to as filtered reconstruction block 221.

[0124] In one embodiment, video encoder 20 (and correspondingly, loop filter unit 220) may be configured to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters, etc.) via entropy coding unit 270, e.g., directly or in an encoded form, so that, for example, decoder 30 may receive and apply the same loop filter parameters or a different loop filter for decoding.

[0125] Decoded Picture Buffer The decoded picture buffer (DPB) 230 may be a memory that stores reference pictures, or reference picture data in general, for encoding video data by the video encoder 20. The DPB 230 may be formed by any of a variety of memory devices, such as, for example, dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer 230 may be configured to store one or more filtered blocks 221. The decoded picture buffer 230 may also be configured to store other previous filtered blocks, such as previously reconstructed and filtered blocks 221, of the same current picture or of a different picture, such as a previous reconstructed picture, and may provide a complete previous reconstructed or decoded picture (and corresponding reference blocks and samples) and / or a partially reconstructed current picture (and corresponding reference blocks and samples), e.g., for inter-prediction. The decoded picture buffer 230 may also be configured to store one or more unfiltered reconstructed blocks 215, for example if the reconstructed blocks 215 are not filtered by the loop filter unit 220, or in general, unfiltered reconstructed samples, or may be configured to store any other further processed version of the reconstructed blocks or reconstructed samples.

[0126] Mode selection (segmentation and prediction) The mode selection unit 260 includes a partitioning unit 262, an inter prediction unit 244, and an intra prediction unit 254, and is configured to receive or obtain raw picture data, such as raw block 203 (current block 203 of current picture 17), and reconstructed picture data, such as filtered and / or unfiltered reconstructed samples or blocks of the same (current) picture and / or from one or more previous decoded pictures, for example from a decoded picture buffer 230 or another buffer (e.g., a line buffer, not shown). The reconstructed picture data is used as reference picture data for prediction, e.g., inter prediction or intra prediction, to obtain a prediction block 265 or a prediction value 265.

[0127] The mode selection unit 260 may be configured to determine or select a type of partitioning (including no partitioning) and a prediction mode (e.g., intra or inter prediction mode) for the current block and generate a corresponding prediction block 265 for calculation of the residual block 205 and for reconstruction of the reconstruction block 215.

[0128] In one embodiment, the mode selection unit 260 may be configured to select a partitioning and prediction mode (e.g., from those supported by or available to the mode selection unit 260) that provides the best match, or in other words, the smallest residual (smallest residual means better compression for transmission or storage), or the smallest signaling overhead (smallest signaling overhead means better compression for transmission or storage), or that considers or balances both. The mode selection unit 260 may also be configured to determine the partitioning and prediction mode based on rate distortion optimization (RDO), i.e., to select the prediction mode that provides the smallest rate distortion. Terms such as “best,” “lowest,” and “optimal” in this specification do not necessarily mean “best,” “lowest,” and “optimal” in general, but may refer to cases where termination criteria or selection criteria are met. For example, values ​​above or below a threshold or other constraint may result in a “suboptimal selection,” but with reduced complexity and processing time.

[0129] In other words, the division unit 262 may be configured to divide a picture from a video sequence into a series of coding tree units (CTUs), which may be further divided into smaller block partitions or sub-blocks (which also form blocks), e.g., using quad-tree partitioning (QT), binary-tree partitioning (BT), or triple-tree partitioning (TT), or any combination thereof, iteratively, and the division unit 262 may be configured to perform prediction on each of the block partitions or sub-blocks, e.g., The mode selection includes selecting a tree structure for the divided block 203, and a prediction mode is applied to each of the block partitions or sub-blocks.

[0130] The partitioning (eg, performed by partitioning unit 262) and prediction processes (eg, performed by inter prediction unit 244 and intra prediction unit 254) performed by video encoder 20 are described in detail below.

[0131] Split The division unit 262 can divide (or separate) a picture block (or CTU) 203 into smaller parts, such as square or rectangular shaped sub-blocks. In a picture with three sample arrays, a CTU contains N×N blocks of luma samples along with two corresponding blocks of chroma samples. The maximum allowable size of a luma block in a CTU is specified as 128×128 in the developing Versatile Video Coding (VVC) standard, but may be specified as a value different from 128×128 in the future, such as 256×256. CTUs of a picture can be clustered / grouped as slices / tile groups, tiles, or bricks. A tile covers a rectangular area of ​​the picture, and a tile can be divided into one or more bricks. A brick contains multiple CTU rows within the tile. A tile that is not divided into multiple bricks can be referred to as a brick. However, a brick is a true subset of a tile and is not referred to as a tile. Two modes of tile groups are supported in VVC: raster scan slice / tile group mode and rectangular slice mode. In raster scan tile group mode, a slice / tile group contains a series of tiles in a tile raster scan of a picture. In rectangular slice mode, a slice contains multiple bricks of a picture that collectively form a rectangular region of the picture. The bricks in a rectangular slice are in the order of the brick raster scan of the slice. These smaller blocks (sometimes referred to as sub-blocks) may be further divided into even smaller partitions. This is also called tree partitioning or hierarchical tree partitioning, where a root block, for example at root tree level 0 (hierarchical level 0, depth 0), can be recursively partitioned into two or more blocks at the next lower tree level, for example a node at tree level 1 (hierarchical level 1, depth 1).These blocks may be split again into two or more blocks at the next lower level, for example tree level 2 (hierarchical level 2, depth 2), and so on, until the splitting terminates (e.g., because a termination criterion is met, such as reaching a maximum tree depth or a minimum block size). Blocks that are not further split are also called leaf blocks or leaf nodes of the tree. A tree that is split into two partitions is called a binary-tree (BT), a tree that is split into three partitions is called a ternary-tree (TT), and a tree that is split into four partitions is called a quad-tree (QT).

[0132] For example, a coding tree unit (CTU) can be or include a CTB of luma samples of a picture having three sample arrays, a CTB of two corresponding chroma samples, a CTB of samples of a monochrome picture, or a CTB of samples of a picture coded using three separate color planes and a syntax structure (for coding the samples). Correspondingly, a coding tree block (CTB) can be an N×N block of samples with some value N, and the division of a component into multiple CTBs is a partition. A coding unit (CU) can be or include a coding block of luma samples of a picture having three sample arrays, a coding block of two corresponding chroma samples, a coding block of samples of a monochrome picture, or a coding block of samples of a picture coded using three separate color planes and a syntax structure (for coding the samples). Correspondingly, a coding block (CB) can be an M×N block of samples with some values ​​M and N, and the division of a CTB into multiple coding blocks is a partition.

[0133] In an embodiment, for example, according to HEVC, a coding tree unit (CTU) can be split into multiple CUs by using a quadtree structure referred to as a coding tree. A decision as to whether to code a picture region using inter (temporal) prediction or intra (spatial) prediction is made at the leaf CU level. Each leaf CU can be further split into one, two, or four PUs according to a PU split type. Within one PU, the same prediction process is applied, and related information is transmitted to the decoder on a PU-by-PU basis. After obtaining a residual block by applying a prediction process based on the PU split type, the leaf CU can be split into transform units (TUs) according to another quadtree structure similar to the coding tree for a CU.

[0134] In an embodiment, for example, according to the latest video coding standard currently under development (called Versatile Video Coding (VVC)), a combined quadtree nested multi-type tree (e.g., binary tree / ternary tree) splits the segmentation structure for dividing coding blocks. In the coding tree structure within a coding tree unit, a CU can have either a square or rectangular shape. For example, a coding tree unit (CTU) is first split by a quadtree. Then, the quadtree leaf node can be further split by a multi-type tree structure. There are four split types for the multi-type tree structure: vertical bisection (SPLIT_BT_VER), horizontal bisection (SPLIT_BT_HOR), vertical trisection (SPLIT_TT_VER), and horizontal trisection (SPLIT_TT_HOR). The multi-type tree leaf node is called a coding unit (CU), and this segmentation is used for prediction and transform processing without further splitting unless the CU is too large for the maximum transform length. This means that in a quadtree with a nested multi-type tree coding block structure, in most cases, CUs, PUs, and TUs have the same block size. An exception occurs when the maximum supported transform length is smaller than the width or height of a color component of a CU. VVC has developed a unique signaling mechanism for partitioning information in a quadtree with a nested multi-type tree coding structure. In this signaling mechanism, a coding tree unit (CTU) is treated as the root of the quadtree and is first partitioned by the quadtree structure. Then, each quadtree leaf node (when large enough to allow it) is further partitioned by the multi-type tree structure.In a multi-type tree structure, a first flag (mtt_split_cu_flag) is signaled to indicate whether a node is further split; when the node is further split, a second flag (mtt_split_cu_vertical_flag) is signaled to indicate the split direction; and a third flag (mtt_split_cu_binary_flag) is signaled to indicate whether the split is bifurcated or trifurcated. Based on the values ​​of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the multi-type tree slit mode (MttSplitMode) of the CU can be derived by the decoder based on a predetermined rule or table. Note that in a specific design, such as a 64x64 luma block and 32x32 chroma pipelined design in a VVC hardware decoder, when either the width or height of the luma coding block is greater than 64, TT split is prohibited, as shown in Figure 6. TT splits are also prohibited when either the width or height of a chroma coding block is greater than 32. This pipelined design divides a picture into multiple virtual pipeline data units (VPDUs), and all VPDUs are defined as non-overlapping units within a picture. In a hardware decoder, consecutive VPDUs are processed simultaneously by multiple pipeline stages. VPDU size is roughly proportional to the buffer size in most pipeline stages; therefore, it is important to keep the VPDU size small. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, ternary tree (TT) and binary tree (BT) partitioning can lead to an increase in the VPDU size.

[0135] Also, in particular, when part of a tree node block exceeds the bottom or right picture boundary, the tree node block is forced to be split until all samples of all coded CUs are located inside the picture boundary.

[0136] As an example, an intra sub-partition (ISP) tool may divide a luma intra-prediction block into two or four sub-partitions vertically or horizontally depending on the block size.

[0137] In one example, mode select unit 260 of video encoder 20 may be configured to perform any combination of the partitioning techniques described herein.

[0138] As described above, video encoder 20 is configured to determine or select a best or optimal prediction mode from a (e.g., predetermined) set of prediction modes, which may include, for example, multiple intra-prediction modes and / or multiple inter-prediction modes.

[0139] Intra prediction The set of intra prediction modes may include 35 different intra prediction modes, such as non-directional modes such as DC (or average) mode and planar mode, or directional modes such as those specified in HEVC, or may include 67 different intra prediction modes, such as non-directional modes such as DC (or average) mode and planar mode, or directional modes such as those specified in VVC. For example, some conventional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes for non-square blocks, such as those specified in VVC. In another example, to avoid division operations in DC prediction, only the longer side is used to calculate the average of non-square blocks. In addition, the intra prediction results of planar modes may be further improved by position-dependent intra prediction combination (PDPC) methods.

[0140] The intra prediction unit 254 is configured to generate the intra prediction block 265 according to an intra prediction mode within a set of intra prediction modes using reconstructed samples of neighboring blocks of the same current picture.

[0141] The intra prediction unit 254 (or, generally, the mode selection unit 260) is further configured to output intra prediction parameters (or, generally, information indicating the intra prediction mode selected for the block) to the entropy coding unit 270 in the form of syntax elements 266 for inclusion in the coded picture data 21, so that, for example, the video decoder 30 can receive and use the prediction parameters for decoding.

[0142] Intra prediction modes in HEVC include DC prediction mode, planar prediction mode, and 33 angular prediction modes, for a total of 35 candidate prediction modes. Figure 3 is a schematic diagram of HEVC intra prediction directions. As shown in Figure 3, a current block may perform intra prediction using pixels in the reconstructed picture blocks to the left and above as references. Picture blocks in the surrounding area of ​​the current block that perform intra prediction on the current block are called reference blocks, and pixels in the reference blocks are called reference pixels. Among the 35 candidate prediction modes, DC prediction mode is applicable to areas in the current block with monotonous texture, and all pixels in the area use the average value of reference pixels in the reference block as prediction. Planar prediction mode is applicable to picture blocks with smooth texture changes, and a current block that satisfies the conditions performs bilinear interpolation using reference pixels in the reference block as prediction for all pixels in the current block. In the angular prediction mode, by using the feature that the texture of the current block is related to the texture height of the adjacent reconstructed picture block, the value of the reference pixel in the corresponding reference block is copied along the angle and used as the prediction for all pixels in the current block.

[0143] An HEVC encoder selects an optimal intra prediction mode for a current block from the 35 candidate prediction modes shown in FIG. 3 and writes the optimal intra prediction mode into a video bitstream. To improve coding efficiency of intra prediction, the encoder / decoder uses intra prediction in surrounding regions to derive three most probable modes from each optimal intra prediction mode of a reconstructed picture block. If the optimal intra prediction mode selected for the current block is one of the three most probable modes, a first index is coded to indicate that the selected optimal intra prediction mode is one of the three most probable modes. If the selected optimal intra prediction mode is not one of the three most probable modes, a second index is coded to indicate that the selected optimal intra prediction mode is one of the remaining 32 modes (modes other than the three most probable modes among the 35 candidate prediction modes). In the HEVC standard, a 5-bit fixed-length code is used as the second index.

[0144] The method by which the HEVC encoder derives the three most probable modes includes selecting the optimal intra prediction mode of the left-neighboring picture block of the current block and the optimal intra prediction mode of the above-neighboring picture block of the current block as a set, and if the two optimal intra prediction modes are the same, keeping only one of the two optimal intra prediction modes in the set. If the two optimal intra prediction modes are the same and both are angular prediction modes, the two angularly adjacent angular prediction modes are selected and added to the set; otherwise, the planar prediction mode, the DC mode, and the vertical prediction mode are selected in order and added to the set until the number of modes in the set reaches three.

[0145] After performing entropy decoding on the bitstream, the HEVC decoder obtains mode information of the current block, which includes an indication identifier indicating whether the optimal intra-prediction mode of the current block is among the three most probable modes, and an index of the optimal intra-prediction mode of the current block in the three most probable modes or an index of the optimal intra-prediction mode of the current block in the other 32 modes.

[0146] Inter Prediction In one possible implementation, the set of inter prediction modes (or possible inter prediction modes) depends on available reference pictures (i.e., previous at least partially decoded pictures, e.g., stored in DBP 230) and other inter prediction parameters, such as whether the entire reference picture or only a portion of it, e.g., a search window region around the area of ​​the current block, is used to search for the best matching reference block, and / or whether pixel interpolation, e.g., half / semi-pel, 1 / 4-pel, and / or 1 / 16-pel interpolation, is applied.

[0147] In addition to the above prediction modes, skip mode, direct mode, and / or other inter prediction modes may be applied.

[0148] For example, in enhanced merge prediction, the merge candidate list for such a mode is constructed by sequentially including the following five types of candidates: spatial MVPs from spatially neighboring CUs, temporal MVPs from co-located CUs, history-based MVPs from a FIFO table, pairwise average MVPs, and zero MVs. The accuracy of the MVs in the merge mode is enhanced through decoder-side intra-prediction mode refinement (DMVR) based on bilateral matching. The merge mode with MVD (MMVD) is derived from the merge mode with intra-prediction mode difference. To specify whether the MMVD mode is used for a CU, an MMVD flag is signaled immediately after sending the skip flag and merge flag. An adaptive intra-prediction mode resolution (AMVR) scheme at the CU level can be used. AMVR allows the MVD of a CU to be coded with different precisions. The MVD of the current CU can be adaptively selected depending on the prediction mode for the current CU. When a CU is coded in merge mode, a combined inter / intra prediction (CIIP) mode can be applied to the current CU. To obtain a CIIP prediction, a weighted average of the inter prediction signal and the intra prediction signal is performed. In affine motion compensation prediction, motion information of two control points (four parameters) or three control points (six parameters) in intra prediction mode describes the affine motion field of a block. Sub-block-based intra prediction mode prediction (SbTMVP) is similar to temporal motion vector prediction (TMVP) in HEVC, but the intra prediction mode of a sub-CU within the current CU is predicted.Bi-directional optical flow (BDOF), formerly called BIO, is a simpler version that requires much less computation, especially in terms of the number of multiplications and the magnitude of the multipliers. In triangular partition mode, the CU is evenly split into two triangular-shaped partitions using either a diagonal or anti-diagonal split. Additionally, bi-prediction mode has been extended beyond simple averaging to allow for a weighted average of the two prediction signals.

[0149] The inter prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (neither of which are shown in FIG. 2 ). The motion estimation unit may be configured to receive or obtain, for motion estimation, a picture block 203 (current picture block 203 in current picture 17) and a decoded picture 231 or at least one or more previous reconstructed blocks, such as reconstructed blocks of one or more other / different previous decoded pictures 231. For example, a video sequence may include the current picture and the previous decoded picture 231; in other words, the current picture and the previous decoded picture 231 may be part of or form a series of pictures that form a video sequence.

[0150] The encoder 20 may be configured to, for example, select a reference block from multiple reference blocks of the same picture or multiple different pictures among multiple other pictures, and provide the reference picture (or reference picture index) and / or an offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as an inter-prediction parameter to the motion estimation unit, which offset is also referred to as an intra-prediction mode (motion vector, MV).

[0151] The motion compensation unit is configured to obtain, for example, receive, inter prediction parameters and perform inter prediction based on or using the inter prediction parameters to obtain an inter prediction block 246. The motion compensation performed by the motion compensation unit may include fetching or generating a prediction block based on a motion / block vector determined by motion estimation, possibly performing interpolation to sub-pixel accuracy. Interpolation filtering may generate additional pixel samples from known pixel samples, potentially increasing the number of candidate prediction blocks that may be used to code the picture block. Upon receiving a motion vector corresponding to the PU of the current picture block, the motion compensation unit may locate the prediction block pointed to by the intra prediction mode within one of the reference picture lists.

[0152] The motion compensation unit may also generate syntax elements associated with the blocks and video slices for use by video decoder 30 in decoding picture blocks of the video slices. In addition to, or instead of, slices and their respective syntax elements, tile groups and / or tiles and their respective syntax elements may be generated or used.

[0153] Entropy Coding The entropy coding unit 270 is configured to apply, for example, an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC scheme (CAVLC), an arithmetic coding scheme, a binarization algorithm, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding method or technique) to the quantized coefficients 209, inter-prediction parameters, intra-prediction parameters, loop filter parameters, and / or other syntax elements to obtain coded picture data 21, e.g., in the form of a coded bitstream 21, which can be output via an output 272 so that, for example, the video decoder 30 can receive and use these parameters for decoding. Encoded bitstream 21 may be transmitted to video decoder 30 or may be stored in memory for later transmission or retrieval by video decoder 30 .

[0154] Other configuration variations of the video encoder 20 can be used to encode the video stream. For example, a non-transform-based encoder 20 may quantize the residual signal directly for some blocks or frames without using the transform processing unit 206. In another implementation, the encoder 20 can have the quantization unit 208 and the inverse quantization unit 210 combined into a single unit.

[0155] Decoder and decoding method 3 is an example block diagram of a video decoder 30 according to an embodiment of this application. The video decoder 30 is configured to receive coded picture data 21 (e.g., coded bitstream 21), e.g., coded by encoder 20, to obtain a decoded picture 331. The coded picture data or bitstream includes information for decoding the coded picture data, e.g., data representing picture blocks of coded video slices (and / or tile groups or tiles) and associated syntax elements.

[0156] 3, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., adder 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter prediction unit 344, and an intra prediction unit 354. Inter prediction unit 344 may be or include a motion compensation unit. Video decoder 30 may, in some examples, perform a decoding pass that is generally inverse to the encoding pass described with respect to video encoder 100 of FIG. 2.

[0157] Referring to Figure 3, the intra prediction unit includes a trained target model (also referred to as a neural network). The neural network is configured to process an input picture, picture region, or picture block to generate a prediction value for the input picture block. For example, a neural network for intra prediction is configured to receive an input picture, picture region, or picture block and generate a prediction value for the input picture, picture region, or picture block. Below, a neural network for intra prediction is described in detail with reference to Figures 6a to 6e.

[0158] As described with respect to encoder 20, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer (DPB) 230, inter prediction unit 344, and intra prediction unit 354 also form a “built-in decoder” of video encoder 20. Accordingly, inverse quantization unit 310 may be identical in function to inverse quantization unit 210, inverse transform processing unit 312 may be identical in function to inverse transform processing unit 212, reconstruction unit 314 may be identical in function to reconstruction unit 214, loop filter 320 may be identical in function to loop filter 220, and decoded picture buffer 330 may be identical in function to decoded picture buffer 230. Accordingly, descriptions of the respective units and functions of video encoder 20 apply correspondingly to the respective units and functions of video decoder 30.

[0159] Entropy Decoding The entropy decoding unit 304 is configured to parse the bitstream 21 (or, generally, the coded picture data 21), e.g., perform entropy decoding on the coded picture data 21 to obtain quantized coefficients 309 and / or decoded coding parameters (not shown in FIG. 3 ), e.g., any or all of inter-prediction parameters (e.g., reference picture indices and intra-prediction modes), intra-prediction parameters (e.g., intra-prediction modes or indices), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 may be configured to apply a decoding algorithm or scheme corresponding to the encoding schemes described with respect to the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 may be further configured to provide the inter-prediction parameters, intra-prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice level and / or the video block level. In addition to or instead of slices and their respective syntax elements, tile groups and / or tiles and their respective syntax elements may be received and / or used.

[0160] inverse quantization Inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or, generally, information regarding inverse quantization) and quantized coefficients from coded picture data 21 (e.g., by parsing and / or decoding, e.g., by entropy decoding unit 304), and apply inverse quantization to the decoded quantized coefficients 309 based on the quantization parameter to obtain dequantized coefficients 311. The dequantized coefficients 311 may also be referred to as transform coefficients 311. The inverse quantization process may include using the quantization parameter determined by video encoder 20 for each video block within a video slice (or tile or tile group) to determine the degree of quantization, and similarly, the degree of inverse quantization to be applied.

[0161] Inverse transformation The inverse transform processing unit 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain the reconstructed residual block 213 in the sample domain. The reconstructed residual block 213 may also be referred to as a transform block 313. This transform may be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may further be configured to receive transform parameters or corresponding information from the coded picture data 21 (e.g., by analyzing and / or decoding, e.g., by the entropy decoding unit 304) and determine the transform to apply to the dequantized coefficients 311.

[0162] Reconstruction The reconstruction unit 314 (e.g., adder or summer 314) is configured to add the reconstructed residual block 313 to the prediction block 365, for example by adding the sample values ​​of the reconstructed residual block 313 and the sample values ​​of the prediction block 365 sample by sample, to obtain the reconstructed block 315 in the sample domain.

[0163] filtering The loop filter unit 320 (either in the coding loop or after the coding loop) is configured to filter the reconstruction block 315 to obtain a filtered block 321, e.g., to smooth pixel transitions or otherwise improve image quality. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a bilateral filter, an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. In one example, the loop filter unit 320 may include a deblocking filter, an SAO filter, and an ALF filter. The filtering process order may be deblocking filter, SAO, and ALF. In another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. In another example, the deblocking filter process may also be applied to interior sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although loop filter unit 320 is shown in FIG. 3 as an in-loop filter, in other configurations, loop filter unit 320 may be implemented as a post-loop filter.

[0164] Decoded Picture Buffer The decoded video blocks 321 of the picture are then stored in a decoded picture buffer 330. The decoded picture buffer 330 stores the decoded picture 331 as a reference picture for motion compensation after other pictures and / or for output for respective display. The decoder 30 is arranged to output the decoded pictures 331, for example via an output 332, for presentation or display to a user.

[0165] prediction In functionality, the inter prediction unit 344 may be identical to the inter prediction unit 244 (in particular, a motion compensation unit), and the intra prediction unit 354 may be identical to the intra prediction unit 254, and performs split or partition decision and prediction based on the split and / or prediction parameters or respective information received (e.g., by analyzing and / or decoding by the entropy decoding unit 304) from the coded picture data 21. The mode application unit 360 may be configured to perform prediction (intra prediction or inter prediction) for each block based on the reconstructed picture, reconstructed block, or corresponding samples (filtered or unfiltered) to obtain a prediction block 365.

[0166] When a video slice is coded as an intra-coded (I) slice, intra prediction unit 354 of mode application unit 360 is configured to generate prediction blocks 365 for picture blocks of the current video slice based on the signaled intra prediction mode and data from previously decoded blocks of the current picture. When a video picture is coded as an inter-coded (i.e., B or P) slice, inter prediction unit 344 (e.g., a motion compensation unit) of mode application unit 360 is configured to generate prediction blocks 365 for picture blocks of the current video slice based on the intra prediction mode and other syntax elements received from entropy decoding unit 304. In inter prediction, a predictive block may be generated from one of the reference pictures in one of the reference picture lists 365. In inter prediction, a predictive block may be generated from one of the reference pictures in one of the reference picture lists. Video decoder 30 may construct the reference frame lists List 0 and List 1 using a default construction technique based on the reference pictures stored in DPB 330. The same or similar may apply to or with embodiments that use tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles) in addition to or instead of slices (e.g., video slices); for example, video may be coded using I, P, or B tile groups and / or tiles.

[0167] Mode application unit 360 is configured to determine prediction information for video blocks of the current video slice by analyzing intra-prediction modes and other syntax elements, and use the prediction information to generate a prediction block for the current video block being decoded. For example, mode application unit 360 uses some of the received syntax elements to determine a prediction mode (e.g., intra-prediction or inter-prediction) for coding video blocks of the video slice, an inter-prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more of a reference picture list for the slice, an intra-prediction mode for each inter-coded video block of the slice, an inter-prediction status for each inter-coded video block of the slice, and other information for decoding video blocks in the current video slice. The same or similar may also apply to or with embodiments that use tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles) in addition to or instead of slices (e.g., video slices), e.g., video may be coded using I, P, or B tile groups and / or tiles.

[0168] In one embodiment, video decoder 30 of Figure 3 may be further configured to divide and / or decode pictures by using slices (also referred to as video slices). A picture may be divided into or decoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., tiles in the H.265 / HEVC / VVC standard or bricks in the VVC standard).

[0169] In one embodiment, the embodiment of video decoder 30 shown in FIG. 3 may be further configured to divide and / or decode a picture by using slices / tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles). A picture may be divided into or decoded using one or more slices / tile groups (typically non-overlapping), and each slice / tile group may include, for example, one or more blocks (e.g., CTUs) or one or more tiles. Each tile may be, for example, rectangular in shape, and may include one or more blocks (e.g., CTUs), e.g., full or partial blocks.

[0170] Other variations of video decoder 30 may decode coded picture data 21. For example, decoder 30 may generate an output video stream without using loop filter unit 320. For example, a non-transform-based decoder 30 may dequantize the residual signal directly for some blocks or frames without using inverse transform processing unit 312. In another implementation, video decoder 30 may have inverse quantization unit 310 and inverse transform processing unit 312 combined into a single unit.

[0171] It should be understood that in the encoder 20 and the decoder 30, the processing result of the current step may be further processed before being output to the next step. For example, after interpolation filtering, intra-prediction mode derivation, or loop filtering, further operations such as clipping or shifting may be performed on the processing result of the interpolation filtering, intra-prediction mode derivation, or loop filtering.

[0172] In particular, further operations may be applied to the derived intra-prediction mode of the current block (including, but not limited to, control point intra-prediction mode in affine mode, sub-block intra-prediction mode in affine, planar, and ATMVP modes, temporal intra-prediction mode, and the like). For example, the value of the intra-prediction mode is restricted to a predetermined range according to the representation bit of the intra-prediction mode. If the representation bit of the intra-prediction mode is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents the power. For example, if bitDepth is set to 16, the range is -32768 to 32767, and if bitDepth is set to 18, the range is -131072 to 131071. For example, the derived values ​​of intra prediction modes (e.g., the MVs of four 4x4 sub-blocks in an 8x8 block) are constrained such that the maximum difference between the integer parts of the MVs of those four 4x4 sub-blocks is no more than N pixels, e.g., no more than 1 pixel. Two methods are provided for constraining intra prediction modes based on bitDepth.

[0173] Although the above-described embodiments primarily describe video coding, in particular, embodiments of coding system 10, encoder 20, and decoder 30, as well as other embodiments described herein, may also be configured for still image processing or coding, i.e., processing or coding of individual pictures independent of preceding or subsequent pictures, as in video coding. In general, inter prediction unit 244 (encoder) and inter prediction unit 344 (decoder) may not be available when picture processing coding is limited to only a single picture 17. For example, all other functions (also referred to as tools or techniques) of video encoder 20 and video decoder 30, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, partitioning 262 / 362, intra prediction 254 / 354, and / or loop filtering 220 / 320, and entropy encoding 270 and entropy decoding 304, may be equally used for still image processing.

[0174] 4 is an example block diagram of a video coding device 400 according to an embodiment of the present application. The video coding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video coding device 400 may be a decoder, such as the video decoder 30 of FIG. 1a, or an encoder, such as the video encoder 20 of FIG. 1a.

[0175] The video coding device 400 includes an ingress port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 for processing the data, a transmitter unit (Tx) 440 and an egress port 450 (or output port 450) for transmitting the data, and a memory 460 for storing the data, where the processor 430 may be, for example, a neural network processing unit 430. The video coding device 400 may further include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to the ingress port 410, the receiver unit 420, the transmitter unit 440, and the egress port 450 for the egress or ingress of optical or electrical signals.

[0176] The processor 430 is implemented in hardware and software. The processor 430 may be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 is in communication with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470 (e.g., a neural network-based coding module 470). The coding module 470 implements the above-disclosed embodiments. For example, the coding module 470 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the coding module 470 provides substantial improvements to the functionality of the video coding device 400 and enables the video coding device 400 to switch to different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0177] Memory 460 may include one or more disks, tape drives, and solid-state drives, and may also be used as an overflow data storage device to store programs when they are selected for execution, as well as to store instructions and data read during program execution. Memory 460 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0178] 5 is an example block diagram of an apparatus 500 according to an embodiment of the present application. The apparatus 500 can be used as either or both of the source device 12 and the destination device 14 of FIG. 1a.

[0179] The processor 502 in the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices, now existing or later developed, capable of manipulating or processing information. While the disclosed implementations may be implemented with a single processor, such as processor 502, as shown, two or more processors may be used to achieve advantages in speed and efficiency.

[0180] The memory 504 in the apparatus 500, in one implementation, may be a read-only memory (ROM) device or a random-access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 that is accessed by the processor 502 via a bus 512. The memory 504 may further include an operating system 508 and application programs 510. The application programs 510 include at least one program that enables the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1 through N, which may further include a video coding application that performs the methods described herein.

[0181] The device 500 may further include one or more output devices, such as a display 518. The display 518, in one example, may be a touch-sensitive display that combines a display with touch-sensitive elements operable to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0182] Although depicted here as a single bus, bus 512 of device 500 may include multiple buses. Additionally, secondary storage may be directly coupled to other components of device 500, accessed over a network, and may include a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Accordingly, device 500 may be implemented in a wide variety of configurations.

[0183] The embodiments of this application relate to the application of neural networks. To facilitate understanding, some terms used in the embodiments of this application are first explained below, and these terms are also used as part of the content of the present invention.

[0184] (1) Neural Networks A neural network (NN) is a machine learning model. A neural network can contain neurons. A neuron takes x as input. s and an intercept of 1, and the output of the calculation unit may be:

number

[0185] (2) Deep Neural Networks Deep neural networks (DNNs), also known as multi-layer neural networks, can be understood as neural networks with multiple hidden layers. There is no specific standard for "multiple" here. DNNs are divided based on the location of various layers, and neural networks in DNNs can be divided into three types: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the middle layer is the hidden layer. The layers are fully connected. Specifically, every neuron in the i-th layer is connected to any neuron in the (i+1)-th layer without exception. Although DNNs appear complex, DNNs are actually not complex in terms of the work at each layer, but simply rely on the following linear relationship:

number

number

number

number

[0186] (3) Convolutional Neural Networks A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. In a deep learning architecture, multi-layer learning is performed at different levels of abstraction according to a machine learning algorithm. As a deep learning architecture, a CNN is a feed-forward artificial neural network. Neurons in a feed-forward artificial neural network can respond to an input picture. A convolutional neural network includes a feature extractor composed of a convolutional layer and a pooling layer. The feature extractor can be considered as a filter. The convolution process can be thought of as performing convolution on an input picture or a convolutional feature plane (feature map) using a trainable filter.

[0187] A convolutional layer is a neuron layer in a convolutional neural network where a convolutional operation is performed on an input signal. A convolutional layer may include multiple convolution operators. A convolution operator is also called a kernel. In picture processing, a convolution operator functions as a filter that extracts specific information from an input picture matrix. A convolutional operator can basically be considered as a weight matrix, which is usually predefined. In the process of performing a convolution operation on a picture, the weight matrix is ​​usually used to process pixels at a granularity level of one pixel (or two pixels, depending on the stride value) horizontally on the input picture to extract specific features from the picture. The size of the weight matrix should be related to the size of the picture. In particular, the depth dimension of the weight matrix is ​​the same as the depth dimension of the input picture. In a convolution operation, the weight matrix extends to the entire depth of the input picture. Therefore, convolution with a single weight matrix generates a convolution output with a single depth dimension. However, in most cases, instead of a single weight matrix, multiple weight matrices of the same size (rows by columns), i.e., multiple homogeneous matrices, are used. The outputs of the multiple weight matrices are stacked to form a convolved picture with a certain depth dimension. The dimension here can be understood to be determined based on the above term "multiple." Multiple different weight matrices can be used to extract multiple different features from a picture. For example, one weight matrix is ​​used to extract edge information of the picture, another weight matrix is ​​used to extract a specific color of the picture, and yet another weight matrix is ​​used to blur unwanted noise in the picture. The multiple weight matrices have the same size (rows by columns). The feature maps extracted from the multiple weight matrices with the same size also have the same size, and then the multiple extracted feature maps with the same size are combined to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications.Each weight matrix may contain weight values ​​obtained through training to extract information from the input picture so that the convolutional neural network can make accurate predictions. When a convolutional neural network has multiple convolutional layers, a large number of general features are usually extracted in the first convolutional layer. The general features are sometimes called low-level features. As the depth of the convolutional neural network increases, the features extracted in subsequent convolutional layers become more complex, e.g., high-level semantic features. The higher-level features are more applicable to the problem to be solved.

[0188] It is often necessary to reduce the number of training parameters. Therefore, it is often necessary to periodically introduce pooling layers after convolutional layers. One convolutional layer may be followed by one pooling layer, or multiple convolutional layers may be followed by one or more pooling layers. In picture processing, pooling layers simply reduce the spatial size of a picture. Pooling layers may include average pooling operators and / or max pooling operators to sample the input picture to obtain a smaller-sized picture. The average pooling operator may calculate pixel values ​​in a picture within a specific range to generate an average value. The average value is used as the average pooling result. The max pooling operator may select the pixel with the maximum value within a specific range as the max pooling result. Also, just as the size of the weight matrix in a convolutional layer needs to be related to the size of the picture, the operators in a pooling layer also need to be related to the size of the picture. The size of the processed picture output from the pooling layer may be smaller than the size of the picture input to the pooling layer. Each sample in the picture output from the pooling layer represents the average or maximum value of the corresponding sub-region of the picture input to the pooling layer.

[0189] After the processing performed in the convolutional / pooling layers, the convolutional neural network is not yet ready to output the required output information. This is because, as described above, the convolutional / pooling layers only extract features and reduce the parameters obtained from the input picture. However, to generate the final output information (required class information or other related information), the convolutional neural network needs to use neural network layers to generate an output of one required class or a group of required classes. Therefore, the convolutional neural network may include multiple hidden layers. The parameters included in the multiple hidden layers may be obtained through pre-training based on training data related to a specific task type. For example, task types may include picture recognition, picture classification, and super-resolution picture reconstruction.

[0190] Optionally, in the neural network layer, multiple hidden layers are followed by an output layer of the entire convolutional neural network. The output layer has a loss function similar to categorical cross-entropy, which is used specifically to calculate prediction errors. Once forward propagation through the entire convolutional neural network is completed, backpropagation begins to update the weight values ​​and deviations of each of the above layers in order to reduce the loss of the convolutional neural network and the error between the results output by the convolutional neural network and the ideal results using the output layer.

[0191] (4) Recurrent Neural Networks Recurrent neural networks (RNNs) process sequence data. Traditional neural network models begin with an input layer, then a hidden layer, and then an output layer. While layers are fully connected, nodes within each layer are not. While this conventional neural network solves many problems, it is still insufficient for many. For example, when predicting the next word in a sentence, it is usually necessary to use previous words, because words within a sentence are not unrelated. RNNs are called recurrent neural networks because the current output of a sequence is related to previous outputs of that sequence. Expressed more clearly, the network remembers previous information and applies it to the calculation of the current output. Specifically, nodes in the hidden layer are connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer from previous points in time. In theory, RNNs can process sequence data of any length. Training an RNN is the same as training a traditional CNN or DNN. The backpropagation algorithm is also used, but with the difference that when an RNN is extended, parameters such as W of the RNN are shared. This differs from the traditional neural network described in the example above. Also, when using the gradient descent algorithm, the output at each step depends not only on the network state at the current step, but also on the network state at several previous steps. The learning algorithm is called the backpropagation through time (BPTT) algorithm.

[0192] Why are recurrent neural networks still needed when convolutional neural networks are available? The reason is simple. Convolutional neural networks assume that elements, such as cats and dogs, are independent of each other and that inputs and outputs are also independent. However, in the real world, multiple elements are interconnected. For example, stocks change over time. Another example is when a person says, "I love traveling, and my favorite place is Yunnan. If I have the opportunity in the future, I plan to go there ( )." A human can tell that the person intends to go to "Yunnan" because humans can make inferences from context. So, how does a machine do this? This is where RNNs come in. RNNs are intended to enable machines to memorize, like humans. Therefore, the output of an RNN must depend on the current input information and the information stored as history.

[0193] (5) Loss function In the process of training a deep neural network, since the output of the deep neural network is expected to be as close as possible to the actual expected prediction, the current network prediction value can be compared with the actual expected target value, and the weight vector of each layer of the neural network is updated based on the difference between the prediction value and the target value (it is true that there is usually an initialization process before the first update, specifically, parameters are preset for all layers of the deep neural network). For example, if the network prediction value is large, the weight vector is adjusted to reduce the prediction value, and adjustments are made continuously until the deep neural network can predict the actual expected target value or a value very close to the actual expected target value. Therefore, it is necessary to predetermine how to obtain the difference between the prediction value and the target value through comparison. This is the loss function or objective function. Loss functions and objective functions are important formulas that measure the difference between the prediction value and the target value. We will use the loss function as an example. A higher output value (loss) of the loss function indicates a larger difference. Therefore, training a deep neural network is a process of minimizing the loss as much as possible.

[0194] (6) Backpropagation Algorithm During the training process, convolutional neural networks use the backpropagation (BP) algorithm to correct the parameter values ​​of the initial super-resolution model so that the error loss in reconstructing the super-resolution model is reduced. Specifically, the input signal is forward propagated until an error loss occurs at the output, and the parameters of the initial super-resolution model are updated based on the backpropagated error loss information to converge the error loss. The backpropagation algorithm is an error-loss-centric backpropagation operation intended to obtain parameters, such as the weight matrix, of the optimal super-resolution model.

[0195] (7) Generative Adversarial Networks A generative adversarial network (GAN) is a deep learning model. The model includes at least two modules: one is a generative model and the other is a discriminative model. Mutual competitive learning is performed between these two modules to generate better outputs. Both the generative model and the discriminative model can be neural networks, specifically, deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: Using a GAN for generating images as an example, assume there are two networks, G (generator) and D (discriminator). G is a network for generating images. G receives random noise z and generates an image by using the noise, which is denoted as G(z). D is a discriminator network used to determine whether an image is "real." The input parameter of D is x, where x represents an image, and the output D(x) represents the probability that x is a real image. When D(x) has a value of 1, it indicates that the image is 100% authentic. When D(x) has a value of 0, it indicates that the image cannot be authentic. In the process of training a generative adversarial network, the goal of the generative network G is to generate as realistic an image as possible to fool the discriminative network D, and the goal of the discriminative network D is to distinguish the image generated by G from the authentic image as much as possible. Thus, between G and D, there is a dynamic "gaming" process in the "generative adversarial network," specifically, an "adversary." The final gaming result is that, ideally, G can generate an image G(z) that is difficult to distinguish from the authentic image, and D has difficulty determining whether the image generated by G is authentic, specifically, D(G(z)) = 0.5. Thus, an excellent generative model G is obtained and can be used to generate images.

[0196] In the following, the target model (also referred to as a neural network) for intra prediction will be described in detail with reference to Figures 6a to 6e, which show some example architectures of neural networks for intra prediction according to one embodiment of this application.

[0197] As shown in Figure 6a, based on the processing sequence, the neural network includes a 3x3 convolution layer (3x3Conv), an activation layer (Relu), a block processing layer (Res-Block), ..., a block processing layer, a 3x3 convolution layer, an activation layer, and a 3x3 convolution layer in that order. After the original matrix of the input neural network is processed by the above layers, the resulting matrix is ​​added to the original matrix to obtain the final output matrix.

[0198] As shown in Figure 6b, the neural network includes two 3x3 convolutional layers and an activation layer, one block processing layer, ..., a block processing layer, a 3x3 convolutional layer, an activation layer, and a 3x3 convolutional layer in this order based on the processing sequence. The first matrix passes through one 3x3 convolutional layer and an activation layer, and the second matrix passes through the other 3x3 convolutional layer and activation layer. The two processed matrices are combined and then processed by the block processing layer, ..., a block processing layer, a 3x3 convolutional layer, an activation layer, and a 3x3 convolutional layer, and then added to the first matrix to obtain the final output matrix.

[0199] As shown in Figure 6c, the neural network includes, in order, two 3x3 convolutional layers and an activation layer, one block processing layer, ..., a block processing layer, a 3x3 convolutional layer, an activation layer, and a 3x3 convolutional layer based on a processing sequence. Before the first and second matrices are input to the neural network, the first matrix is ​​multiplied by the second matrix. The first matrix then passes through one 3x3 convolutional layer and an activation layer, and the resulting matrix passes through the other 3x3 convolutional layer and activation layer. After the two processed matrices are added together, the two matrices are processed by the block processing layer, ..., a block processing layer, a 3x3 convolutional layer, an activation layer, and a 3x3 convolutional layer, and then added to the first matrix to obtain the final output matrix.

[0200] As shown in Figure 6d, the block processing layer includes a 3x3 convolution layer, an activation layer, and a 3x3 convolution layer in order based on the processing sequence. After the input matrix is ​​processed by these three layers, the processed matrix is ​​added to the original input matrix to obtain the output matrix. As shown in Figure 6e, the block processing layer includes a 3x3 convolution layer, an activation layer, a 3x3 convolution layer, and an activation layer in order based on the processing sequence. After the input matrix is ​​processed by the 3x3 convolution layer, the activation layer, and a 3x3 convolution layer, the processed matrix is ​​added to the original input matrix, and then the sum is processed by the activation layer to obtain the output matrix.

[0201] 6a to 6e merely illustrate some example architectures of neural networks for intra prediction in embodiments of this application, and do not constitute limitations on the architecture of the neural network. The number of layers included in the neural network, the layer structure, and processes such as addition, multiplication, or combination, as well as the number and size of input matrices and / or output matrices, can be determined based on actual situations. These are not particularly limited in this application.

[0202] 7 is a flowchart of a process 700 of an intra prediction method according to one embodiment of the present application. The process 700 may be performed by the video encoder 20 or the video decoder 30, and specifically, may be performed by the intra prediction unit 254 or 354 of the video encoder 20 or the video decoder 30. The process 700 is described as a series of steps or operations. It should be understood that the steps or operations of the process 700 may be performed in various orders and / or simultaneously, and are not limited to the order of execution shown in FIG. 7. Assume that the video encoder or the video decoder is used to perform the process 700, including the following steps, for a video data stream having multiple picture frames to perform intra prediction on a picture or a picture block. The process 700 may include the following steps:

[0203] Step 701: Obtain the intra prediction mode or texture distribution of each of a plurality of reconstructed picture blocks in a surrounding area of ​​the current block.

[0204] The surrounding area of ​​the current block includes spatial neighbors of the current block. The spatial neighbors may include a left candidate picture block located to the left of the current block and a top candidate picture block located above the current block. A reconstructed picture block may be a coded picture block coded at the encoder side, where the reconstructed picture block is obtained at the encoder side, or a decoded picture block decoded and reconstructed at the decoder side. A reconstructed picture block may also refer to a basic unit picture block of a predetermined size obtained by dividing a coded picture block or a decoded picture block into equal-sized blocks. For example, FIG. 9 is a schematic diagram of a reconstructed picture block in a surrounding area according to an embodiment of this application. As shown in FIG. 9, the size of the coded picture block or the decoded picture block may be, for example, 16×16, 64×64, or 32×16, and the size of the basic unit picture block may be, for example, 4×4 or 8×8.

[0205] In the following, a reconstructed picture block is used as an example for explanation, and the reconstructed picture block can be any one of multiple reconstructed picture blocks in the surrounding area. For other reconstructed picture blocks, please refer to the corresponding method.

[0206] In solution 1, the intra-prediction mode of the reconstructed picture block may include (1) a plurality of recursive intra-prediction modes of the reconstructed picture block, which are determined based on the reconstruction values ​​of the reconstructed picture block and prediction values ​​corresponding to a plurality of recursive candidate intra-prediction modes, or (2) an optimal intra-prediction mode of the reconstructed picture block, which is the recursive intra-prediction mode having the largest probability value or the smallest prediction error value among the plurality of recursive intra-prediction modes.

[0207] The multiple a posteriori candidate intra-prediction modes are obtained based on the multiple a priori candidate intra-prediction modes of the reconstructed picture block, and the multiple a posteriori candidate intra-prediction modes may refer to the multiple a priori candidate intra-prediction modes or to some of the multiple a priori candidate intra-prediction modes.

[0208] The multiple a posteriori intra prediction modes of a reconstructed picture block may refer to the multiple a posteriori candidate intra prediction modes, or may refer to some of the multiple a posteriori candidate intra prediction modes, for example, multiple specified intra prediction modes selected from the multiple a posteriori candidate intra prediction modes.

[0209] For probability values ​​or prediction error values ​​of multiple a posteriori intra-prediction modes, see the description below.

[0210] In one possible implementation, in addition to the intra-prediction mode of the reconstructed picture block, related information of the reconstructed picture block may also be obtained. The related information and the method for obtaining the related information are as follows:

[0211] 1. A plurality of prediction error values ​​of a reconstructed picture block corresponding to a plurality of recursive intra-prediction modes, wherein the plurality of prediction error values ​​are also determined based on reconstructed values ​​of the reconstructed picture block and prediction values ​​corresponding to a plurality of recursive candidate intra-prediction modes.

[0212] Intra prediction may be performed separately based on the multiple a posteriori candidate intra-prediction modes to obtain multiple predicted values, where the multiple predicted values ​​correspond to the multiple a posteriori candidate intra-prediction modes.

[0213] A plurality of prediction values ​​are compared with the reconstructed values ​​of the reconstructed picture block to obtain a plurality of prediction error values, which correspond to the plurality of a posteriori candidate intra-prediction modes. In this application, the prediction error values ​​corresponding to the a posteriori candidate intra-prediction modes may be obtained using methods such as sum of absolute differences (SAD) or sum of squared differences (SSE).

[0214] When the multiple a posteriori intra-prediction modes of the reconstructed picture block refer to multiple a posteriori candidate intra-prediction modes, the multiple prediction error values ​​of the reconstructed picture block corresponding to the multiple a posteriori intra-prediction modes refer to multiple prediction error values ​​corresponding to the multiple a posteriori candidate intra-prediction modes. When the multiple a posteriori intra-prediction modes of the reconstructed picture block refer to some intra-prediction modes among the multiple a posteriori candidate intra-prediction modes, the multiple prediction error values ​​of the reconstructed picture block corresponding to the multiple a posteriori intra-prediction modes refer to prediction error values ​​corresponding to those intra-prediction modes selected from the multiple prediction error values ​​corresponding to the multiple a posteriori candidate intra-prediction modes.

[0215] 2. A plurality of probability values ​​of a reconstructed picture block corresponding to a plurality of recursive intra-prediction modes, wherein the plurality of probability values ​​are also determined based on the reconstructed values ​​of the reconstructed picture block and predicted values ​​corresponding to a plurality of recursive candidate intra-prediction modes.

[0216] The multiple probability values ​​of the reconstructed picture block corresponding to the multiple recursive intra prediction modes may be obtained based on the following two methods.

[0217] One method is to obtain multiple probability values ​​for the reconstructed picture block based on the multiple prediction error values ​​of the reconstructed picture block obtained by the first method. For example, to obtain normalized values ​​of the multiple prediction error values, a normalization process may be performed on the multiple prediction error values ​​of the reconstructed picture block based on a method such as a normalized exponential function or a linear normalization method. The normalized values ​​of the multiple prediction error values ​​are the multiple probability values ​​of the reconstructed picture block. Based on the correspondence between the multiple prediction error values ​​of the reconstructed picture block and the multiple a posteriori intra prediction modes, the multiple probability values ​​of the reconstructed picture block also correspond to the multiple a posteriori intra prediction modes of the reconstructed picture block, and the probability value may represent a probability that the a posteriori intra prediction mode corresponding to the probability value is the optimal intra prediction mode of the reconstructed picture block.

[0218] The other method is to input the reconstructed value of the reconstructed picture block and multiple predicted values ​​of the reconstructed picture block obtained by the first method into a trained neural network to obtain multiple probability values ​​of the reconstructed picture block corresponding to multiple recursive intra-prediction modes. For details about the neural network, please refer to the description of the training engine 25. The details will not be described again here.

[0219] Therefore, after multiple prediction error values ​​or probability values ​​corresponding to multiple recursive intra-prediction modes are obtained based on the two methods described above, the optimal intra-prediction mode of the reconstructed picture block can be obtained based on the following two methods.

[0220] One is to use the recursive intra prediction mode corresponding to the smallest prediction error value among multiple prediction error values ​​corresponding to multiple recursive intra prediction modes as the optimal intra prediction mode for the reconstructed picture block.

[0221] The other is to use the recursive intra prediction mode corresponding to the maximum probability value among a plurality of probability values ​​corresponding to a plurality of recursive intra prediction modes as the optimal intra prediction mode for the reconstructed picture block.

[0222] In this application, the memory can be directly read to obtain the intra-prediction mode or the intra-prediction mode and related information of the reconstructed picture block. After the reconstructed picture block is coded or decoded, the intra-prediction mode or the intra-prediction mode and related information of the reconstructed picture block can be immediately obtained based on the above-mentioned method, and the intra-prediction mode or the intra-prediction mode and related information are stored. When intra-prediction is performed on a subsequent picture block (current block), the intra-prediction mode or the intra-prediction mode and related information can be directly read from the corresponding location in the memory. Thus, the efficiency of intra-prediction for the current block can be improved.

[0223] In this application, the intra-prediction mode or the intra-prediction mode and related information of a reconstructed picture block can be calculated only when intra-prediction is performed on a current block. That is, when intra-prediction is performed on a current block, the intra-prediction mode or the intra-prediction mode and related information of the reconstructed picture block are obtained according to the above-mentioned method. Thus, calculation is performed only when it is determined that the reconstructed picture block needs to be used, thereby saving storage space.

[0224] When intra prediction is used in all of the encoding or decoding processes of the plurality of reconstructed picture blocks, the intra prediction modes or the intra prediction modes and related information of the plurality of reconstructed picture blocks can be obtained based on the above-mentioned method. When intra prediction is not used in the encoding or decoding processes of some picture blocks among the plurality of reconstructed picture blocks, the intra prediction modes or the intra prediction modes and related information of the some picture blocks can also be obtained based on any of the methods described in the above three cases.

[0225] When a reconstructed picture block includes multiple basic unit picture blocks, the intra prediction mode or the intra prediction mode and related information of the reconstructed picture block may be used as the intra prediction mode or the intra prediction mode and related information of all the basic unit picture blocks included in the reconstructed picture block, and the intra prediction mode or the intra prediction mode and related information of the reconstructed picture block may be used as the intra prediction mode or the intra prediction mode and related information of all the pixels included in the reconstructed picture block.

[0226] In solution 2, the texture distribution of the reconstructed picture block includes a horizontal texture distribution of the reconstructed picture block and a vertical texture distribution of the reconstructed picture block.

[0227] The texture of a picture is a visual feature that reflects the homogeneity phenomenon within a picture and reflects the organization and arrangement attributes of slowly or periodically changing surface structures on the surface of an object. Unlike picture features such as grayscale and color, texture is represented by the grayscale distribution of pixels and their surrounding spatial neighborhoods. Unlike color features, texture features are not sample-based features but need to be statistically calculated within an area containing multiple samples. The texture of a reconstructed picture block can be considered to include a large number of texture primitives. The texture distribution of a reconstructed picture block is analyzed based on the texture primitives. The texture representation depends on the different types, orientations, and number of texture primitives. The horizontal texture distribution of a reconstructed picture block can indicate horizontal texture features by using the types and number of texture primitives in the horizontal direction, and the vertical texture distribution can indicate vertical texture features by using the types and number of texture primitives in the vertical direction.

[0228] Step 702: Based on the intra prediction modes or texture distributions of each of the multiple reconstructed picture blocks, obtain multiple a priori candidate intra prediction modes for the current block and multiple probability values ​​for the current block corresponding to the multiple a priori candidate intra prediction modes.

[0229] The multiple a priori candidate intra prediction modes of the current block may refer to all of the remaining intra prediction modes after the multiple a posteriori intra prediction modes of each of the multiple reconstructed picture blocks have been de-duplicated, or may refer to a portion of all of the remaining intra prediction modes after the multiple a posteriori intra prediction modes of each of the multiple reconstructed picture blocks have been de-duplicated.

[0230] The intra-prediction modes or texture distributions of each of the multiple reconstructed picture blocks may be input to the neural network to obtain multiple a priori candidate intra-prediction modes for the current block and multiple probability values ​​for the current block corresponding to the multiple a priori candidate intra-prediction modes. For details about the neural network, please refer to the description of the training engine 25. Details will not be described again here.

[0231] Optionally, a plurality of recursive intra-prediction modes for each of a plurality of reconstructed picture blocks and a plurality of prediction error values ​​corresponding to the plurality of recursive intra-prediction modes may be input into a trained neural network to obtain a plurality of a priori candidate intra-prediction modes for the current block and a plurality of probability values ​​for the current block corresponding to the plurality of a priori candidate intra-prediction modes.

[0232] Optionally, a plurality of recursive intra-prediction modes for each of a plurality of reconstructed picture blocks and a plurality of probability values ​​corresponding to the plurality of recursive intra-prediction modes may be input into a trained neural network to obtain a plurality of a priori candidate intra-prediction modes for the current block and a plurality of probability values ​​for the current block corresponding to the plurality of a priori candidate intra-prediction modes.

[0233] Optionally, optimal intra-prediction modes of multiple reconstructed picture blocks may be input to a neural network to obtain multiple a priori candidate intra-prediction modes for the current block and multiple probability values ​​for the current block corresponding to the multiple a priori candidate intra-prediction modes.

[0234] Optionally, the horizontal texture distribution and vertical texture distribution of multiple reconstructed picture blocks may be input to a neural network to obtain multiple a priori candidate intra-prediction modes for the current block and multiple probability values ​​for the current block corresponding to the multiple a priori candidate intra-prediction modes.

[0235] Step 703: Obtain a plurality of weighting factors corresponding to a plurality of a priori candidate intra-prediction modes based on a plurality of probability values ​​corresponding to the plurality of a priori candidate intra-prediction modes.

[0236] When the sum of the plurality of probability values ​​is 1, the probability value corresponding to the first a priori candidate intra-prediction mode is used as the weighting factor corresponding to the first a priori candidate intra-prediction mode. That is, the weighting factor of each of the M a priori candidate intra-prediction modes is the respective probability value of the plurality of a priori candidate intra-prediction modes. Alternatively, when the sum of the plurality of probability values ​​is not 1, a normalization process is performed on the plurality of probability values, and the normalized value of the probability value corresponding to the first a priori candidate intra-prediction mode is used as the weighting factor corresponding to the first a priori candidate intra-prediction mode. That is, the weighting factor of each of the plurality of a priori candidate intra-prediction modes is the normalized value of the respective probability value of the plurality of a priori candidate intra-prediction modes. The first a priori candidate intra-prediction mode is any one of the plurality of a priori candidate intra-prediction modes. It is understood that the sum of the plurality of weighting factors corresponding to the plurality of a priori candidate intra-prediction modes is 1.

[0237] Step 704: Perform intra prediction separately based on the multiple a priori candidate intra prediction modes to obtain multiple predicted values.

[0238] According to the principle of intra prediction, in a candidate intra prediction mode, a reference block can be found in the surrounding area of ​​the current block, and intra prediction is performed on the current block based on the reference block to obtain a predicted value corresponding to the candidate intra prediction mode. It can be seen that the predicted value of the current block corresponds to the candidate intra prediction mode. Therefore, intra prediction can be performed separately based on multiple a priori candidate intra prediction modes to obtain multiple predicted values ​​of the current block.

[0239] Step 705: Obtain a predicted value of the current block based on a weighted sum of the plurality of weighting factors and the plurality of predicted values.

[0240] A weighting factor corresponding to an a priori candidate motion vector is multiplied by a predicted value corresponding to the same a priori candidate intra-prediction mode, and multiple products corresponding to multiple a priori candidate intra-prediction modes are added together to obtain a predicted value for the current block.

[0241] In this application, multiple weighting factors and multiple predicted values ​​of a current block are obtained based on the intra-prediction information of each of multiple reconstructed picture blocks in a surrounding area of ​​the current block, and a weighting factor corresponding to a priori candidate intra-prediction mode is multiplied by a predicted value corresponding to the same priori candidate intra-prediction mode, and then a predicted value of the current block is obtained by adding together multiple products corresponding to the multiple priori candidate intra-prediction modes. In this way, the predicted value of the current block is obtained by combining multiple priori candidate intra-prediction modes, which can better fit rich and variable textures in the real world, thereby improving the accuracy of intra-prediction, reducing intra-prediction errors, and improving the overall rate-distortion optimization (RDO) efficiency of intra-prediction.

[0242] In one possible implementation, after the reconstruction value of the current block is obtained, the intra prediction mode or texture distribution of the current block can be obtained directly. For the intra prediction mode or texture distribution, see step 701. The obtaining method includes:

[0243] 1. Based on the reconstruction value of the current block and the predicted values ​​corresponding to the multiple a priori candidate intra prediction modes of the current block, multiple a priori intra prediction modes of the current block and multiple prediction error values ​​of the current block corresponding to the multiple a priori intra prediction modes are obtained, and the multiple a priori intra prediction modes of the current block are obtained based on the multiple a priori candidate intra prediction modes of the current block.

[0244] 2. Based on the reconstruction value of the current block and the prediction values ​​corresponding to the multiple a priori candidate intra prediction modes of the current block input into the neural network, obtain multiple a priori intra prediction modes of the current block and multiple probability values ​​of the current block corresponding to the multiple a priori intra prediction modes, where the multiple a priori intra prediction modes of the current block are obtained based on the multiple a priori candidate intra prediction modes of the current block, or obtain multiple probability values ​​corresponding to the multiple a priori intra prediction modes of the current block based on the multiple prediction error values ​​of the current block.

[0245] 3. The a posteriori intra prediction mode having the maximum probability value or the minimum prediction error value among the plurality of a posteriori intra prediction modes of the current block is determined as the optimal intra prediction mode of the current block.

[0246] 4. Obtain the horizontal texture distribution and vertical texture of the current block.

[0247] In one possible implementation, the plurality of probability values ​​for the current block includes M probability values, all of which are greater than the remaining M probability values. Therefore, M a priori candidate intra-prediction modes corresponding to the M probability values ​​may be selected from the plurality of a priori candidate intra-prediction modes for the current block. Then, M weighting factors are obtained based on the M probability values. Intra prediction is separately performed based on the M a priori candidate intra-prediction modes to obtain M predicted values ​​for the current block. Finally, a weighted sum is performed based on the M weighting factors and the M predicted values ​​to obtain a predicted value for the current block. That is, the first M probability values ​​with the largest probability values ​​are selected from the plurality of probability values ​​of the current block corresponding to the plurality of a priori candidate intra-prediction modes, and M a priori candidate intra-prediction modes corresponding to the M probability values ​​are selected from the plurality of a priori candidate intra-prediction modes of the current block. Then, weighting factors and predicted values ​​are calculated based on the M probability values ​​and the M a priori candidate intra-prediction modes to obtain a predicted value of the current block. However, among the plurality of probability values ​​of the current block corresponding to the plurality of a priori candidate intra-prediction modes, the remaining probability values ​​excluding the M probability values ​​may be ignored due to their small values. This reduces the amount of calculation and improves the efficiency of intra-prediction.

[0248] Hereinafter, the technical solution of the method embodiment shown in FIG. 7 will be described in detail using several specific embodiments.

[0249] Embodiment 1 In this embodiment, a plurality of a priori candidate intra-prediction modes of the current block and a plurality of prediction error values ​​of the current block corresponding to the plurality of a priori candidate intra-prediction modes are determined based on a plurality of recursive intra-prediction modes of each of a plurality of reconstructed picture blocks in the surrounding area and a plurality of probability values ​​corresponding to the plurality of recursive intra-prediction modes.

[0250] 8 is a flowchart of a process 800 of an intra prediction method according to one embodiment of the present application. The process 800 may be performed by the video encoder 20 or the video decoder 30, and specifically, may be performed by the intra prediction unit 254 or 354 of the video encoder 20 or the video decoder 30. The process 800 is described as a series of steps or operations. It should be understood that the steps or operations of the process 800 may be performed in various orders and / or simultaneously, and are not limited to the order of execution shown in FIG. 8. Assume that the video encoder or the video decoder is used to perform the process 800, including the following steps, for a video data stream having multiple picture frames to perform intra prediction on a picture or a picture block. The process 800 may include the following steps:

[0251] Step 801: Obtain a plurality of recursive intra-prediction modes of a plurality of reconstructed picture blocks in a surrounding region, and a plurality of prediction error values ​​corresponding to the plurality of recursive intra-prediction modes.

[0252] In the following, one reconstructed picture block is used as an example for explanation, and the reconstructed picture block can be any one of multiple reconstructed picture blocks in the surrounding area. For other reconstructed picture blocks, multiple a posteriori intra-prediction modes and multiple prediction error values ​​corresponding to the multiple a posteriori intra-prediction modes can be obtained by referring to the method.

[0253] There are N4 a posteriori candidate intra prediction modes for the reconstructed picture block, and the N4 a posteriori candidate intra prediction modes are obtained based on the multiple a posteriori candidate intra prediction modes for the reconstructed picture block. For the obtaining method, see the description of step 701. Intra prediction is performed separately based on the N4 a posteriori candidate intra prediction modes to obtain N4 predicted values ​​for the reconstructed picture block. The N4 predicted values ​​correspond to the N4 a posteriori candidate intra prediction modes. That is, intra prediction is performed on the reconstructed picture block based on a reference block corresponding to one a posteriori candidate intra prediction mode to obtain a predicted value for the reconstructed picture block. The N4 predicted values ​​are compared with the reconstructed value for the reconstructed picture block to obtain N4 prediction error values ​​for the reconstructed picture block. The N4 prediction error values ​​correspond to the N4 a posteriori candidate intra prediction modes. In this application, the prediction error values ​​of the reconstructed picture block corresponding to the a posteriori candidate intra prediction modes may be obtained based on a method such as SAD or SSE.

[0254] The N2 a posteriori intra prediction modes of the reconstructed picture block may refer to the N4 a posteriori candidate intra prediction modes, or may refer to some of the N4 a posteriori candidate intra prediction modes, for example, multiple specified intra prediction modes selected from the N4 a posteriori candidate intra prediction modes.

[0255] Correspondingly, the number of prediction error values ​​of the reconstructed picture block corresponding to the N2 recursive intra prediction modes is also N2.

[0256] All the a posteriori intra-prediction modes of the plurality of reconstructed picture blocks can be represented as an N2 × Q two-dimensional matrix, where N2 is the number of the plurality of a posteriori intra-prediction modes, Q is the number of reconstructed picture blocks, and the elements therein are M2 k nwhere k=0, 1, ..., Q-1 represents the index of the reconstructed picture block, and n=0, 1, ..., N2-1 represents the index of the recursive intra-prediction mode, which means the recursive intra-prediction mode indicated by n of the reconstructed picture block indicated by k.

[0257] All prediction error values ​​of multiple reconstructed picture blocks can also be represented as an N2 × Q two-dimensional matrix, the elements of which are E k nb where k=0, 1, ..., Q-1 represents the index of the reconstructed picture block, and n=0, 1, ..., N2-1 represents the index of the recursive intra-prediction mode, which means the prediction error value corresponding to the recursive intra-prediction mode indicated by n of the reconstructed picture block indicated by k.

[0258] Step 802: Based on a plurality of a priori intra-prediction modes of each of a plurality of reconstructed picture blocks and a plurality of prediction error values ​​corresponding to the plurality of a priori intra-prediction modes, obtain a plurality of a priori candidate intra-prediction modes of the current block and a plurality of probability values ​​of the current block corresponding to the plurality of a priori candidate intra-prediction modes.

[0259] In this application, all prediction error values ​​and all a posteriori intra-prediction modes of multiple reconstructed picture blocks, i.e., the two N2×Q two-dimensional matrices mentioned above, can be input to a trained neural network, and the neural network outputs multiple a priori candidate intra-prediction modes of the current block and multiple prediction error values ​​of the current block corresponding to the multiple a priori candidate intra-prediction modes. For details about the neural network, please refer to the description of the training engine 25. The details will not be described again here.

[0260] The multiple a priori candidate intra-prediction modes of the current block can be represented as a two-dimensional matrix of N1 × S, where N1 is the number of a priori candidate intra-prediction modes of the current block, and S is the number of basic unit picture blocks or pixels included in the current block. If the current block is not further divided, S = 1. The elements of the matrix are M1 l n where l=0, 1, ..., S-1 represents the index of a basic unit picture block or pixel, and n=0, 1, ..., N1-1 represents the index of a priori candidate intra prediction mode, which means the a priori candidate intra prediction mode indicated by n of the basic unit picture block or pixel indicated by l.

[0261] The multiple prediction error values ​​of the current block corresponding to the multiple a priori candidate intra-prediction modes can also be represented as a two-dimensional matrix of N1 × S. The elements of the matrix are P l nc where l=0, 1, ..., S-1 represents the index of a basic unit picture block or pixel, and n=0, 1, ..., N1-1 represents the index of a priori candidate intra prediction mode, which means the probability that the a priori candidate intra prediction mode indicated by n of the basic unit picture block or pixel indicated by l will be the optimal intra prediction mode for that basic unit picture block or pixel.

[0262] Optionally, when l remains unchanged, i.e., the sum of the N1 probability values ​​corresponding to the N1 a priori candidate intra-prediction modes of the elementary unit picture block or pixel indicated by l is 1; or

number

number

[0263] Step 803: Obtain a plurality of weighting factors corresponding to a plurality of a priori candidate intra-prediction modes based on a plurality of prediction error values ​​of the current block corresponding to the plurality of a priori candidate intra-prediction modes.

[0264] The weighting factors of the current block corresponding to the multiple a priori candidate intra-prediction modes can also be expressed as a two-dimensional matrix of N1 × S. The elements of the matrix are W l n where l=0, 1, ..., S-1 represents the index of a basic unit picture block or pixel, and n=0, 1, ..., N1-1 represents the index of a priori candidate intra prediction mode, which means the a priori candidate intra prediction mode indicated by n of the basic unit picture block or pixel indicated by l.

[0265] When the normalization process is performed on the N1 probability values ​​of the basic unit picture block or pixel denoted by l in the current block corresponding to the N1 a priori candidate intra prediction modes, i.e.,

number

number

[0266] Step 804: Perform intra prediction separately based on the multiple a priori candidate intra prediction modes to obtain multiple predicted values.

[0267] For illustrative purposes, one a priori candidate intra-prediction mode is used as an example, and the a priori candidate intra-prediction mode is any one of the multiple a priori candidate intra-prediction modes. For all other a priori candidate intra-prediction modes, the method can be referenced.

[0268] Intra prediction is performed based on the a priori candidate intra prediction modes to obtain a predicted value of the current block, and thus N1 predicted values ​​can be obtained for the N1 a priori candidate intra prediction modes.

[0269] The multiple predicted values ​​of the current block can be expressed as a three-dimensional matrix of BH×WH×S, where BH×WH represents the size of the basic unit picture block included in the current block, and S is the number of basic unit picture blocks or pixels included in the current block. If the current block is not further divided, S=1. The elements of the matrix are Pred l n(i,j), where l = 0, 1, ..., S-1 represents the index of the basic unit picture block or pixel, and n = 0, 1, ..., N1-1 represents the index of the a priori candidate intra-prediction mode, which means the predicted value of the pixel in the i-th row and j-th column in the basic unit picture block indicated by l corresponding to the a priori candidate intra-prediction mode indicated by n.

[0270] Step 805: Obtain a predicted value of the current block based on a weighted sum of the plurality of weighting factors and the plurality of predicted values.

[0271] The weighting factor corresponding to each a priori candidate intra-prediction mode is multiplied by the predicted value corresponding to the same a priori candidate intra-prediction mode, and the multiple products corresponding to the multiple a priori candidate intra-prediction modes are summed to obtain the predicted value of the current block. In the current block, the predicted value of the pixel at the ith row and jth column in the basic unit picture block denoted by l is:

number

[0272] Embodiment 2 In this embodiment, a plurality of a priori candidate intra-prediction modes of the current block and a plurality of probability values ​​of the current block corresponding to the plurality of a priori candidate intra-prediction modes are determined based on a plurality of recursive intra-prediction modes of each of a plurality of reconstructed picture blocks in the surrounding area and a plurality of probability values ​​corresponding to the plurality of recursive intra-prediction modes.

[0273] 10 is a flowchart of a process 1000 of an intra prediction method according to one embodiment of the present application. The process 1000 may be performed by the video encoder 20 or the video decoder 30, and specifically, may be performed by the intra prediction unit 254 or 354 of the video encoder 20 or the video decoder 30. The process 1000 is described as a series of steps or operations. It should be understood that the steps or operations of the process 1000 may be performed in various orders and / or simultaneously, and are not limited to the order of execution shown in FIG. 10. Assume that the video encoder or the video decoder is used to perform the process 1000, including the following steps, for a video data stream having multiple picture frames to perform intra prediction on a picture or a picture block. The process 1000 may include the following steps:

[0274] Step 1001: Obtain a plurality of recursive intra-prediction modes for each of a plurality of reconstructed picture blocks in a surrounding region, and a plurality of prediction probability values ​​corresponding to the plurality of recursive intra-prediction modes.

[0275] Step 1001 in this embodiment differs from step 801 in embodiment 1 in that multiple prediction error values ​​corresponding to multiple recursive intra-prediction modes are changed to multiple probability values ​​corresponding to multiple recursive intra-prediction modes.

[0276] In the following, one reconstructed picture block is used as an example for explanation, and the reconstructed picture block can be any one of multiple reconstructed picture blocks in the surrounding area. For other reconstructed picture blocks, multiple a posteriori intra-prediction modes and multiple probability values ​​corresponding to the multiple a posteriori intra-prediction modes can be obtained by referring to the method.

[0277] The N2 a posteriori intra prediction modes of the reconstructed picture block can be obtained with reference to the method in step 801, and the details will not be described again here.

[0278] The N2 probability values ​​of the reconstructed picture block corresponding to the N2 a posteriori intra prediction modes may be obtained based on the following two methods.

[0279] One is to obtain N2 probability values ​​of the reconstructed picture block based on the N2 prediction error values ​​of the reconstructed picture block obtained according to the first embodiment.

[0280] The N2 prediction error values ​​of a reconstructed picture block correspond to one N2-dimensional vector of all prediction error values ​​of the multiple reconstructed picture blocks, the elements of which are E k1 nb where k1 is the index of the reconstructed picture block, n=0, 1, ..., N2-1 represents the index of the recursive intra-prediction mode, and the N2 probability values ​​of the reconstructed picture block may be calculated based on the N2 prediction error values ​​of the reconstructed picture block. The N2 probability values ​​of the reconstructed picture block may also be represented as an N2-dimensional vector, whose elements are P k1 nb where k1 is the index of the reconstructed picture block, and n=0, 1, ..., N2-1 represents the index of the recursive intra-prediction mode, which means the probability that the recursive intra-prediction mode indicated by n of the reconstructed picture block indicated by k1 will be the optimal intra-prediction mode of that reconstructed picture block.

[0281] Optionally, E k1 nb is the normalized exponential function:

number

[0282] In another example, E k1 nb is based on the linear normalization methodk1 nb can be converted into

[0283] Therefore, when k remains constant,

number

[0284] The other is to input the reconstructed value of the first reconstructed picture block and the N2 predicted values ​​corresponding to the N2 recursive intra-prediction modes into a trained neural network to obtain N2 probability values ​​of the reconstructed picture block corresponding to the N2 recursive intra-prediction modes. For the neural network, please refer to the description of the training engine 25. The details will not be described again here.

[0285] The reconstructed value of the reconstructed picture block can be obtained after the reconstructed picture block is coded. For the N2 predicted values ​​of the reconstructed picture block corresponding to the N2 a posteriori intra prediction modes, please refer to the method in step 801 of embodiment 1. Details will not be described again here.

[0286] All the recursive intra-prediction modes of multiple reconstructed picture blocks can be represented as an N2 × Q two-dimensional matrix, where N2 is the number of recursive intra-prediction modes and Q is the number of reconstructed picture blocks, and the elements therein are M2 k n where k=0, 1, ..., Q-1 represents the index of the reconstructed picture block, and n=0, 1, ..., N2-1 represents the index of the recursive intra-prediction mode, which means the recursive intra-prediction mode indicated by n of the reconstructed picture block indicated by k.

[0287] All probability values ​​of multiple reconstructed picture blocks can be represented as an N2 × Q two-dimensional matrix, where N2 is the number of recursive intra-prediction modes and Q is the number of reconstructed picture blocks, and the elements therein are P k nb where k=0, 1, ..., Q-1 represents the index of the reconstructed picture block, and n=0, 1, ..., N2-1 represents the index of the recursive intra-prediction mode, which means the recursive intra-prediction mode indicated by n of the reconstructed picture block indicated by k.

[0288] Step 1002: Based on the respective multiple a priori intra prediction modes of the multiple reconstructed picture blocks and the multiple probability values ​​corresponding to the multiple a priori intra prediction modes, obtain multiple a priori candidate intra prediction modes of the current block and multiple probability values ​​of the current block corresponding to the multiple a priori candidate intra prediction modes.

[0289] Step 1002 in this embodiment differs from step 802 in embodiment 1 in that the multiple prediction error values ​​input to the neural network corresponding to the multiple recursive intra prediction modes are changed to multiple probability values ​​corresponding to the multiple recursive intra prediction modes.

[0290] Step 1003: Obtain a plurality of weighting factors corresponding to a plurality of a priori candidate intra-prediction modes based on a plurality of probability values ​​of the current block corresponding to the plurality of a priori candidate intra-prediction modes.

[0291] Step 1004: Perform intra prediction separately based on the multiple a priori candidate intra prediction modes to obtain multiple predicted values.

[0292] Step 1005: Obtain a predicted value of the current block based on a weighted sum of the plurality of weighting factors and the plurality of predicted values.

[0293] For Steps 1003 to 1005 in this embodiment, please refer to Steps 803 to 805 in Embodiment 1, and the details will not be described again here.

[0294] Embodiment 3 In this embodiment, a plurality of a priori candidate intra-prediction modes for the current block and a plurality of probability values ​​for the current block corresponding to the plurality of a priori candidate intra-prediction modes are determined based on the respective optimal intra-prediction modes of a plurality of reconstructed picture blocks in the surrounding area.

[0295] 11 is a flowchart of a process 1100 of an intra prediction method according to one embodiment of the present application. The process 1100 may be performed by the video encoder 20 or the video decoder 30, and specifically, may be performed by the intra prediction unit 254 or 354 of the video encoder 20 or the video decoder 30. The process 1100 is described as a series of steps or operations. It should be understood that the steps or operations of the process 1100 may be performed in various orders and / or simultaneously and are not limited to the order of execution shown in FIG. 11. Assume that the video encoder or the video decoder is used to perform the process 1100, including the following steps, for a video data stream having multiple picture frames to perform intra prediction on a picture or a picture block. The process 1100 may include the following steps:

[0296] Step 1101: Obtain an optimal intra prediction mode for each of a plurality of reconstructed picture blocks in a surrounding region.

[0297] Step 1101 in this embodiment differs from step 801 in embodiment 1 in that a plurality of recursive intra-prediction modes and a plurality of prediction error values ​​corresponding to the plurality of recursive intra-prediction modes are changed to an optimal intra-prediction mode.

[0298] In the following, one reconstructed picture block is used as an example for explanation, and the reconstructed picture block can be any one of multiple reconstructed picture blocks in the surrounding area, and the optimal intra prediction modes of all other reconstructed picture blocks can be obtained by referring to this method.

[0299] The optimal intra prediction mode of a reconstructed picture block may be obtained based on the following two methods.

[0300] One is to obtain the optimal intra prediction mode of the reconstructed picture block based on the N2 prediction error values ​​of the reconstructed picture block obtained in embodiment 1, i.e., to use the recursive intra prediction mode corresponding to the smallest prediction error value among the N2 prediction error values ​​of the reconstructed picture block as the optimal intra prediction mode of the reconstructed picture block.

[0301] The other is to obtain the optimal intra prediction mode of the reconstructed picture block based on the N2 probability values ​​of the reconstructed picture block obtained in embodiment 2, i.e., to use the recursive intra prediction mode corresponding to the maximum probability value among the N2 probability values ​​of the reconstructed picture block as the optimal intra prediction mode of the reconstructed picture block.

[0302] Step 1102: Based on the respective optimal intra prediction modes of the multiple reconstructed picture blocks, obtain multiple a priori candidate intra prediction modes of the current block and multiple probability values ​​of the current block corresponding to the multiple a priori candidate intra prediction modes.

[0303] Step 1102 in this embodiment differs from step 802 in embodiment 1 in that the multiple recursive intra-prediction modes input to the neural network corresponding to the multiple recursive intra-prediction modes and the multiple prediction error values ​​corresponding to the multiple recursive intra-prediction modes are changed to optimal intra-prediction modes for the multiple reconstructed picture blocks.

[0304] Step 1103: Obtain a plurality of weighting factors corresponding to a plurality of a priori candidate intra-prediction modes based on a plurality of probability values ​​of the current block corresponding to the plurality of a priori candidate intra-prediction modes.

[0305] Step 1104: Perform intra prediction separately based on the multiple a priori candidate intra prediction modes to obtain multiple predicted values.

[0306] Step 1105: Obtain a predicted value of the current block based on a weighted sum of the plurality of weighting coefficients and the plurality of predicted values.

[0307] For Steps 1103 to 1105 in this embodiment, please refer to Steps 803 to 805 in Embodiment 1, and the details will not be described again here.

[0308] Embodiment 4 In this embodiment, a plurality of a priori candidate intra-prediction modes for the current block and a plurality of probability values ​​for the current block corresponding to the plurality of a priori candidate intra-prediction modes are determined based on the respective horizontal texture distributions and respective vertical texture distributions of a plurality of reconstructed picture blocks in the surrounding region.

[0309] 12 is a flowchart of a process 1200 of an intra prediction method according to an embodiment of the present application. The process 1200 may be performed by the video encoder 20 or the video decoder 30, and specifically, may be performed by the intra prediction unit 254 or 354 of the video encoder 20 or the video decoder 30. The process 1200 is described as a series of steps or operations. It should be understood that the steps or operations of the process 1200 may be performed in various orders and / or simultaneously and are not limited to the order of execution shown in FIG. 12. Assume that the video encoder or the video decoder is used to perform the process 1200, including the following steps, for a video data stream having multiple picture frames to perform intra prediction on a picture or a picture block. The process 1200 may include the following steps:

[0310] Step 1201: Obtain each horizontal texture distribution and each vertical texture distribution of a plurality of reconstructed picture blocks in the surrounding region.

[0311] Step 1201 in this embodiment differs from step 801 in embodiment 1 in that the multiple recursive intra-prediction modes and the multiple prediction error values ​​corresponding to the multiple recursive intra-prediction modes are changed to horizontal texture distribution and vertical texture distribution.

[0312] The texture of a picture is a visual feature that reflects the homogeneity phenomenon within a picture, and reflects the organization and arrangement attributes of slowly or periodically changing surface structures on the surface of an object. Unlike picture features such as grayscale and color, texture is represented by the grayscale distribution of pixels and surrounding spatial neighborhoods. Unlike color features, texture features are not sample-based features, but need to be statistically calculated within an area containing multiple samples. The texture of a reconstructed picture block can be considered to include a large number of texture primitives. The texture distribution of a reconstructed picture block is analyzed based on the texture primitives. The representation of texture depends on different types, orientations, and numbers of texture primitives.

[0313] Step 1202: Based on the respective horizontal texture distributions and respective vertical texture distributions of the plurality of reconstructed picture blocks, obtain a plurality of a priori candidate intra-prediction modes of the current block and a plurality of probability values ​​of the current block corresponding to the plurality of a priori candidate intra-prediction modes.

[0314] Step 1202 in this embodiment differs from step 802 in embodiment 1 in that the multiple recursive intra prediction modes input to the neural network and the multiple prediction error values ​​corresponding to the multiple recursive intra prediction modes are changed into horizontal texture distributions and vertical texture distributions of multiple reconstructed picture blocks.

[0315] Step 1203: Obtain a plurality of weighting factors corresponding to a plurality of a priori candidate intra-prediction modes based on a plurality of probability values ​​of the current block corresponding to the plurality of a priori candidate intra-prediction modes.

[0316] Step 1204: Perform intra prediction separately based on the multiple a priori candidate intra prediction modes to obtain multiple predicted values.

[0317] Step 1205: Obtain a predicted value of the current block based on a weighted sum of the plurality of weighting factors and the plurality of predicted values.

[0318] For Steps 1203 to 1205 in this embodiment, please refer to Steps 803 to 805 in Embodiment 1, and the details will not be described again here.

[0319] 13 is a schematic diagram of a configuration of a decoding device 1300 according to an embodiment of the present application. The decoding device 1300 may correspond to the video encoder 20 or the video decoder 30. The decoding device 1300 includes an intra prediction module 1301 configured to implement the method embodiments shown in any one of FIG. 7 to FIG. 13. In one example, the intra prediction module 1301 may correspond to the intra prediction unit 254 of FIG. 2 or the intra prediction unit 354 of FIG. 3. It should be understood that the coding device 1300 may include other units related to the intra prediction unit 254 or the intra prediction unit 354, and the details will not be described again here.

[0320] In the implementation process, the steps in the above-described method embodiments may be implemented by hardware integrated logic circuits in a processor or by instructions in the form of software. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor or the like. The steps of the methods disclosed in the embodiments of this application may be directly presented as being performed and completed by a hardware encoding processor, or may be performed and completed by a combination of hardware and software modules in the encoding processor. The software modules may be located in a storage medium well-established in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is placed in the memory, and the processor reads the information in the memory and completes the steps in the above-described method in combination with the processor hardware.

[0321] The memory in the above embodiments may be volatile or nonvolatile memory, or may include both volatile and nonvolatile memory. Nonvolatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM) used as an external cache. By way of example and not limitation, numerous forms of RAM may be used, such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchlink dynamic random access memory (synchlink DRAM, SLDRAM), and direct rambus dynamic random access memory (direct rambus RAM, DR RAM). In particular, the memory of the systems and methods described herein includes, but is not limited to, these and any other suitable types of memory.

[0322] Those skilled in the art can recognize that, in combination with the examples described in the embodiments disclosed in this application, the units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether a function is performed by hardware or computer software depends on the specific application and design constraints of these technical solutions. Those skilled in the art may implement the described functions using different methods for different specific applications, but these implementations should not be considered to go beyond the scope of this application.

[0323] It can be clearly understood by those skilled in the art that for the sake of convenience and conciseness, the detailed operation processes of the above-mentioned systems, devices, and units can be referred to the corresponding processes in the foregoing method embodiments, and the details will not be described again here.

[0324] In some embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described are merely examples. For example, the division into units is merely a logical functional division, and in actual implementation, other divisions may be used. For example, multiple units or components may be combined or integrated into other systems, or some features may be ignored or not implemented. Furthermore, the illustrated or described mutual couplings, direct couplings, or communication connections may be implemented via some interface. Indirect couplings or communication connections between devices or units may be implemented in electrical, mechanical, or other forms.

[0325] Units described as separate parts may or may not be physically separated, and parts illustrated as units may or may not be physical units, located in one location, or distributed over multiple network units, and some or all of the units may be selected based on actual requirements to achieve the objectives of the solution of the embodiment.

[0326] Additionally, multiple functional units in the embodiments of this application may be integrated into a single processing unit, each of which may exist physically alone, or two or more units may be integrated into a single unit.

[0327] When functions are implemented in the form of software functional units and sold or used as an independent product, the functions may be stored in a computer-readable storage medium. Based on this understanding, the technical solution in this application may essentially be implemented, or a portion of the technical solution may be implemented in the form of a software product. A computer software product is stored in a storage medium and includes several instructions for instructing a computer device (such as a personal computer, a server, a network device, or the like) to execute all or part of the steps of the method in the embodiments of this application. The storage medium includes any medium capable of storing program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or a compact disk.

[0328] The above description is merely a specific implementation of this application and is not intended to limit the scope of protection of this application. Any modifications or replacements that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application shall fall within the scope of protection of this application. Therefore, the scope of protection of this application shall be governed by the scope of protection of the claims.

Claims

1. 1. An intra prediction method, comprising: obtaining intra prediction modes for each of P reconstructed picture blocks in a surrounding region of a current block, the surrounding region comprising spatial neighborhoods of the current block; obtaining Q a priori candidate intra-prediction modes of the current block based on the respective intra-prediction modes of the P reconstructed picture blocks, and Q probability values ​​of the current block corresponding to the Q a priori candidate intra-prediction modes; obtaining M weighting factors corresponding to M a priori candidate intra prediction modes among the Q a priori candidate intra prediction modes based on M probability values ​​corresponding to the M a priori candidate intra prediction modes, where M, P, and Q are positive integers and M is less than or equal to Q; performing intra prediction separately based on the M a priori candidate intra prediction modes to obtain M predicted values; obtaining a predicted value of the current block based on a weighted sum of the M predicted values ​​and the corresponding M weighting factors; and M is equal to Q, the M a priori candidate intra-prediction modes are the Q a priori candidate intra-prediction modes, and the M probability values ​​are the Q probability values; or M is less than Q, and M a priori candidate intra-prediction modes are selected from the Q a priori candidate intra-prediction modes such that all of the M corresponding probability values ​​are greater than probability values ​​other than the M probability values ​​among the Q probability values. method.

2. The step of obtaining Q a priori candidate intra-prediction modes of the current block and Q probability values ​​of the current block corresponding to the Q a priori candidate intra-prediction modes based on the respective intra-prediction modes of the P reconstructed picture blocks includes: inputting the respective intra-prediction modes of the P reconstructed picture blocks into a trained neural network to obtain the Q a priori candidate intra-prediction modes and the Q probability values ​​of the current block corresponding to the Q a priori candidate intra-prediction modes; 2. The method of claim 1, comprising:

3. The step of obtaining M weighting factors corresponding to M a priori candidate intra-prediction modes based on M probability values ​​corresponding to the M a priori candidate intra-prediction modes comprises: using the probability value corresponding to the first a priori candidate intra-prediction mode as the weighting factor corresponding to the first a priori candidate intra-prediction mode when the sum of the M probability values ​​is 1; or when the sum of the M probability values ​​is not 1, performing a normalization process on the M probability values ​​and using the normalized value of the probability value corresponding to the first a priori candidate intra-prediction mode as a weighting factor corresponding to the first a priori candidate intra-prediction mode; and the first a priori candidate intra-prediction mode is any one of the M a priori candidate intra-prediction modes.

3. The method according to claim 1 or 2.

4. The step of obtaining Q a priori candidate intra-prediction modes of the current block and Q probability values ​​of the current block corresponding to the Q a priori candidate intra-prediction modes based on the respective intra-prediction modes of the P reconstructed picture blocks includes: inputting a plurality of a priori intra prediction modes of each of the P reconstructed picture blocks and a plurality of probability values ​​corresponding to the plurality of a priori intra prediction modes into a trained neural network to obtain Q a priori candidate intra prediction modes of the current block and Q probability values ​​of the current block corresponding to the Q a priori candidate intra prediction modes, wherein the plurality of a priori intra prediction modes of the reconstructed picture block and the plurality of probability values ​​corresponding to the plurality of a priori intra prediction modes are determined based on the reconstructed values ​​of the reconstructed picture block and the predicted values ​​corresponding to the plurality of a priori intra prediction modes, and the reconstructed picture block is any one of the P reconstructed picture blocks; 4. The method according to claim 1, wherein the

5. The step of obtaining Q a priori candidate intra-prediction modes of the current block and Q probability values ​​of the current block corresponding to the Q a priori candidate intra-prediction modes based on the respective intra-prediction modes of the P reconstructed picture blocks includes: inputting a plurality of a priori intra prediction modes of each of the P reconstructed picture blocks and a plurality of prediction error values ​​corresponding to the plurality of a priori intra prediction modes into a trained neural network to obtain Q a priori candidate intra prediction modes of the current block and Q probability values ​​of the current block corresponding to the Q a priori candidate intra prediction modes, wherein the plurality of a priori intra prediction modes of the reconstructed picture block and the plurality of prediction error values ​​corresponding to the plurality of a priori intra prediction modes are determined based on the reconstructed values ​​of the reconstructed picture block and prediction values ​​corresponding to the plurality of a priori intra prediction modes, and the reconstructed picture block is any one of the P reconstructed picture blocks; 4. The method according to claim 1, wherein the

6. The step of obtaining Q a priori candidate intra-prediction modes of the current block and Q probability values ​​of the current block corresponding to the Q a priori candidate intra-prediction modes based on the respective intra-prediction modes of the P reconstructed picture blocks includes: inputting the optimal intra prediction modes of the P reconstructed picture blocks into a trained neural network to obtain Q a priori candidate intra prediction modes of the current block and Q probability values ​​of the current block corresponding to the Q a priori candidate intra prediction modes, wherein the optimal intra prediction mode of the reconstructed picture block is the a priori intra prediction mode with the largest probability value or the smallest prediction error value among a plurality of a priori intra prediction modes of the reconstructed picture block, and the reconstructed picture block is any one of the P reconstructed picture blocks; and the plurality of recursive intra-prediction modes of the reconstructed picture block correspond to a plurality of probability values, and the plurality of recursive intra-prediction modes and the plurality of probability values ​​corresponding to the plurality of recursive intra-prediction modes are determined based on reconstructed values ​​of the reconstructed picture block and prediction values ​​corresponding to the plurality of recursive intra-prediction modes; or the plurality of recursive intra-prediction modes of the reconstructed picture block correspond to a plurality of prediction error values, and the plurality of recursive intra-prediction modes and the plurality of prediction error values ​​corresponding to the plurality of recursive intra-prediction modes are determined based on the reconstructed values ​​of the reconstructed picture block and prediction values ​​corresponding to the plurality of recursive intra-prediction modes.

4. The method according to any one of claims 1 to 3.

7. obtaining a training dataset, the training dataset having information about a plurality of groups of picture blocks, the information about the picture blocks of each group having a plurality of recursive intra-prediction modes of a plurality of reconstructed picture blocks, a plurality of probability values ​​corresponding to the plurality of recursive intra-prediction modes, a plurality of recursive intra-prediction modes of a current block, and a plurality of probability values ​​of the current block corresponding to the plurality of recursive intra-prediction modes, the plurality of reconstructed picture blocks being picture blocks in a spatial neighborhood of the current block; obtaining the neural network through training based on the training data set; The method of claim 4 further comprising:

8. obtaining a training dataset, the training dataset having information about a plurality of groups of picture blocks, the information about the picture blocks of each group having a plurality of recursive intra-prediction modes for each of a plurality of reconstructed picture blocks, a plurality of prediction error values ​​corresponding to the plurality of recursive intra-prediction modes, a plurality of recursive intra-prediction modes for a current block, and a plurality of probability values ​​for the current block corresponding to the plurality of recursive intra-prediction modes, the plurality of reconstructed picture blocks being picture blocks in a spatial neighborhood of the current block; obtaining the neural network through training based on the training data set; The method of claim 5 further comprising:

9. obtaining a training dataset, the training dataset having information about a plurality of groups of picture blocks, the information about each group of picture blocks having an optimal intra-prediction mode for each of a plurality of reconstructed picture blocks, a plurality of a posteriori intra-prediction modes for a current block, and a plurality of probability values ​​for the current block corresponding to the plurality of a posteriori intra-prediction modes, the plurality of reconstructed picture blocks being picture blocks in a spatial neighborhood of the current block; obtaining the neural network through training based on the training data set; The method of claim 6 further comprising:

10. 10. The method of claim 2, wherein the neural network has at least a convolutional layer and an activation layer.

11. 11. The method of claim 10, wherein a depth of a convolution kernel of the convolution layer is 2, 3, 4, 5, 6, 16, 24, 32, 48, 64, or 128, and a size of the convolution kernel of the convolution layer is 1x1, 3x3, 5x5, or 7x7.

12. 10. The method according to claim 2, wherein the neural network comprises a convolutional neural network CNN, a deep neural network DNN, or a recurrent neural network RNN.

13. An encoder having processing circuitry configured to perform the method of any one of claims 1 to 12.

14. A decoder comprising processing circuitry configured to perform the method of any one of claims 1 to 12.

15. A computer program having a program code for performing the method of any one of claims 1 to 12 when the program code is run on a computer or processor.

16. 1. An encoder comprising: one or more processors; a non-transitory computer readable storage medium coupled to the processor and storing a program for execution by the processor, the program, when executed by the processor, enabling the encoder to perform the method of any one of claims 1 to 12; and An encoder having:

17. A decoder comprising: one or more processors; a non-transitory computer readable storage medium coupled to the processor and storing a program for execution by the processor, the program, when executed by the processor, enabling the decoder to perform the method of any one of claims 1 to 12; and A decoder having:

18. 13. A non-transitory computer readable storage medium having program code thereon, the program code being for performing the method of any one of claims 1 to 12 when executed by a computing device.

Images