Coding method, decoding method and device

Abstract

The present application discloses an encoding method, a decoding method and a device, which relate to the field of video or image compression technology based on artificial intelligence (AI), and specifically to an intra-frame prediction mode encoding or decoding method based on a neural network, so as to reduce the hardware implementation complexity of entropy coding of syntax elements related to the intra-frame prediction mode. The method comprises: determining the index of the intra-frame prediction mode syntax element value set of the current image block; performing probability estimation processing on the input data representing the characteristics of the current image block through a neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, and the probability distribution represents the probability value of each of the multiple candidate intra-frame prediction mode syntax element value sets; according to the probability distribution of the multiple candidate intra-frame prediction mode syntax element value sets, entropy coding is performed on the probability value related to the index of the intra-frame prediction mode syntax element value set of the current image block to obtain a bitstream.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video processing, and in particular to an encoding method, a decoding method, and a device. Background Art

[0002] Video coding (video encoding and / or decoding) is widely used in digital video applications such as broadcast digital television, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, Digital Versatile Disc (DVD) and Blu-ray Disc, video content acquisition and editing systems, and security applications in camcorders.

[0003] Even in the case of relatively short films, large amounts of video data are required to describe them, which can create difficulties when the data is sent or otherwise transmitted across networks with limited bandwidth capacity. Therefore, video data is often compressed before being transmitted across modern telecommunications networks. Since memory resources may be limited, the size of the video can also be an issue when storing the video on a storage device. Video compression equipment typically uses software and / or hardware at the source to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received by a video decompression device at the destination. With limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase compression rates with little to no impact on image quality.

[0004] The industry has proposed a variety of video coding standards to guide video encoding and decoding, such as H.264 / AVC, high-efficiency video coding (HEVC), and versatile video coding (VVC). These standards all employ a hybrid architecture encoding and decoding solution based on block partitioning and transform quantization, and continuously iterate on technical modules such as block partitioning, prediction, transform, entropy coding / decoding, and loop filtering to continuously improve compression efficiency.

[0005] In video coding and decoding, the prediction mode information used to indicate the prediction mode used in the image block prediction operation will be written into the bitstream through entropy coding. When the syntax elements related to the intra-frame prediction mode include multiple pieces of information, entropy coding encodes the syntax elements related to the intra-frame prediction mode in sequence. For example, the syntax elements related to the intra-frame prediction mode may include a matrix-based intra prediction (MIP) flag, a MIP mode, a multi-row reference index, an intra sub-partitions (ISP) mode, a most probable mode (MPM) flag, an MPM index and an intra-frame prediction mode index, etc. When entropy coding is performed on the syntax elements related to the intra-frame prediction mode, such as Figure 1a The VVC intra-frame prediction mode encoding method shown in the figure first encodes the MIP flag. If the MIP flag is true, the MIP mode is encoded. If the MIP flag is false, the multi-row reference index, ISP mode, and MPM flag are encoded in sequence. If the MPM flag is true, the MPM index is encoded. If the MPM flag is false, the intra-frame prediction mode is encoded. However, entropy coding sequentially encodes the syntax elements related to the intra-frame prediction mode, which makes hardware implementation complex.

[0006] Therefore, how to reduce the hardware implementation complexity of entropy coding of syntax elements related to intra-frame prediction modes is the key to implementing entropy coding or entropy decoding. Summary of the Invention

[0007] The present application provides an encoding method, a decoding method, and a device to reduce the hardware implementation complexity of entropy encoding or decoding of syntax elements related to intra-frame prediction modes.

[0008] To achieve the above objectives, this application adopts the following technical solutions:

[0009] In a first aspect, a coding method is provided, which may include: determining an index of a set of syntax element values ​​for an intra-frame prediction mode of a current image block, the syntax element value set including values ​​of multiple syntax elements involved in the intra-frame prediction mode of the current image block, the values ​​of the multiple syntax elements including the index of the intra-frame prediction mode adopted by the current image block; then, performing probability estimation processing on input data through a neural network to obtain a probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, the probability distribution representing the probability values ​​of each of the multiple candidate intra-frame prediction mode syntax element value sets; the input data is used to characterize features of the current image block (e.g., texture features of the current image block); based on the probability distribution of the multiple candidate intra-frame prediction mode syntax element value sets, entropy coding is performed on the probability values ​​related to the index of the intra-frame prediction mode syntax element value set of the current image block to obtain a bitstream. It should be understood that the neural network is a trained neural network with probability estimation function.

[0010] Through the encoding method provided by the embodiments of the present application, the values ​​of multiple (for example, all) syntax elements related to the intra-frame prediction mode of the current image block are used as a set of intra-frame prediction mode syntax element values. The intra-frame prediction mode syntax element value set corresponds to an index, and the index of the intra-frame prediction mode syntax element value set is understood as a syntax element, which is equivalent to converting the values ​​of multiple entropy-coded syntax elements into the value of one entropy-coded syntax element. The syntax element can represent the intra-frame prediction mode state of the current image block (for example, it can indicate the values ​​of multiple syntax elements involved in the intra-frame prediction mode actually adopted by the current image block). During entropy coding, only the index of the intra-frame prediction mode syntax element value set needs to be entropy coded once to replace the entropy coding of many intra-frame prediction mode-related syntax element values, thereby reducing the complexity of hardware implementation and improving coding performance.

[0011] In one possible implementation, the values ​​of the multiple syntax elements included in the intra-prediction mode syntax element value set for the current image block are the values ​​of the multiple syntax elements related to the intra-prediction mode actually adopted by the current image block. The values ​​of the multiple syntax elements are dependent on each other and do not exist discretely. For example, if the current image block actually adopts intra-prediction mode a, which belongs to type A intra-prediction mode, the intra-prediction mode syntax element value set for the current image block may include a type A intra-prediction mode flag bit set to true and an intra-prediction mode index of a.

[0012] In another possible implementation, an intra-frame prediction mode syntax element value set includes values ​​of multiple syntax elements involved in an intra-frame prediction mode, and an intra-frame prediction mode syntax element value set can represent or reflect the dependency relationship between the multiple syntax elements it includes. The values ​​of the multiple syntax elements included in each candidate intra-frame prediction mode syntax element value set in multiple candidate intra-frame prediction mode syntax element value sets refer to the values ​​of multiple syntax elements involved in the intra-frame prediction mode that can be used when performing intra-frame prediction on the current image block. The values ​​of the multiple syntax elements included in a candidate intra-frame prediction mode syntax element value set are dependent on each other. The intra-frame prediction mode syntax element value set actually used by the current image block can be one of the multiple candidate intra-frame prediction mode syntax element value sets.

[0013] For example, when the intra-frame prediction mode information encoding process is represented using a tree diagram structure, the tree diagram includes multiple chain structure examples from top to bottom, where each chain structure example corresponds to a certain set of candidate intra-frame prediction mode syntax element values, and different chain structure examples correspond to different sets of candidate intra-frame prediction mode syntax element values. It should be understood that each chain structure example represents or reflects a dependency relationship between multiple syntax elements.

[0014] In another possible implementation, determining the index of the intra-frame prediction mode syntax element value set of the current image block includes: determining the index of the intra-frame prediction mode syntax element value set of the current image block according to the index of the intra-frame prediction mode adopted by the current image block.

[0015] In another possible implementation, determining the index of the set of syntax element values ​​of the intra-frame prediction mode of the current image block based on the index of the intra-frame prediction mode used by the current image block includes: determining the sum of the index of the intra-frame prediction mode used by the current image block and the additional value of the first type of intra-frame prediction mode as the index of the set of syntax element values ​​of the intra-frame prediction mode of the current image block. The intra-frame prediction mode used by the current image block belongs to the first type of intra-frame prediction mode; and the intervals formed by the sum of the minimum index value and the corresponding additional value and the sum of the maximum index value and the corresponding additional value of intra-frame prediction modes of different types do not intersect. Through the correspondence between different types of intra-frame prediction modes and different additional values, and the fact that there is no intersection between the intervals formed by the sum of the minimum index value and the corresponding additional value and the sum of the maximum index value and the corresponding additional value of different types of intra-frame prediction modes, the additional values ​​corresponding to different types of intra-frame prediction modes can be reasonably configured, so that after the index entropy encoding of the set of syntax element values ​​of the intra-frame prediction mode of the current image block determined according to the additional value is performed, the index of the intra-frame prediction mode adopted by the current image block can be quickly and effectively determined at the decoding end according to the value range of the index of the set of syntax element values ​​of the intra-frame prediction mode of the current image block.

[0016] For example, the intra prediction mode of type A includes five intra prediction modes, the index range of which is 1 to 5, and the additional value corresponding to the intra prediction mode of type A is X. The intra prediction mode of type B includes three intra prediction modes, the index range of which is 1 to 3, and the additional value corresponding to the intra prediction mode of type B is Y. Therefore, the interval [1+X, 5+X] does not intersect with the interval [1+Y, 3+Y]. Here, "[" and "]" are closed interval symbols.

[0017] In another possible implementation, the additional value of the Nth intra-frame prediction mode includes: the sum of the label values ​​of each intra-frame prediction mode before the Nth intra-frame prediction mode in a predetermined order. The Nth intra-frame prediction mode is any intra-frame prediction mode.

[0018] The label value of the intra prediction mode may be a parameter configured for the intra prediction mode. Optionally, the specific value of the label value of the intra prediction mode may be configured according to actual needs, and the label values ​​of different types of intra prediction modes may be the same or different.

[0019] In another possible implementation, the label value of each type of intra-frame prediction mode may include: the total number of multiple intra-frame prediction modes in each type of intra-frame prediction mode.

[0020] In another possible implementation, the intra-frame prediction mode adopted by the current image block belongs to the first type of intra-frame prediction mode, and the values ​​of multiple syntax elements included in the intra-frame prediction mode syntax element value set also include a flag bit of the first type of intra-frame prediction mode is true, a flag bit of the second type of intra-frame prediction mode is false, and a flag bit of the third type of intra-frame prediction mode is false; the total number of multiple intra-frame prediction modes under the second type of intra-frame prediction mode is B, and the total number of multiple intra-frame prediction modes under the third type of intra-frame prediction mode is C; the predetermined order is the second type of intra-frame prediction mode, the third type of intra-frame prediction mode, and the first type of intra-frame prediction mode, and determining the sum of the index of the intra-frame prediction mode adopted by the current image block and the additional value of the first type of intra-frame prediction mode is the index of the intra-frame prediction mode syntax element value set of the current image block, including: determining the index of the intra-frame prediction mode adopted by the current image block, the sum of B and C is the index of the intra-frame prediction mode syntax element value set of the current image block.

[0021] In another possible implementation, before performing probability estimation processing on the input data through a neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, the encoding method provided by the present application may also include: determining a neural network suitable for the current image block based on network structure distinguishing information, the network structure distinguishing information includes the size of the current image block, or the network structure distinguishing information depends on the size of the current image block. Accordingly, performing probability estimation processing on the input data through a neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets includes: performing probability estimation processing on the input data through a determined neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets. In this way, different neural networks can be designed for different image blocks (for example, different sizes) to maximize the encoding performance.

[0022] In another possible implementation, the input data may include information about neighboring image blocks of the current image block, or may include preprocessed information about neighboring image blocks of the current image block, where the neighboring image blocks include image blocks adjacent to the upper left, adjacent above, and adjacent to the left; the information about the neighboring image blocks includes reconstructed values ​​of the neighboring image blocks. By fully leveraging the correlation between the neighboring image blocks of the current image block and the current image block, and because the neighboring image blocks have already been reconstructed, the information about the neighboring image blocks can characterize the characteristics of the current image block. The probability distribution obtained by inputting the information about the neighboring image blocks into the neural network has a higher accuracy.

[0023] In another possible implementation, the information of the neighboring image blocks of the preprocessed current image block may include: an M-channel data block formed by splicing the information of the neighboring image blocks and the information of the current image block; M is greater than or equal to 1.

[0024] For example, M may be less than or equal to the type of information included in the information of the image block.

[0025] In a possible implementation, the pre-processed information of the neighboring image blocks of the current image block may include a one-dimensional vector formed by connecting vectors converted from the information of the neighboring image blocks and the information of the current image block.

[0026] The vector of the information conversion of the image block may be a one-dimensional valid One-Hot vector.

[0027] It should be noted that the information of the current image block in the input data may be a default value for filling, and the embodiment of the present application does not limit the content of the default value.

[0028] In another possible implementation, the information about neighboring image blocks may further include one or more of the following: a prediction value, a residual value, an index to a set of intra-prediction mode syntax element values, and partitioning information. This allows full utilization of available information about neighboring image blocks, allowing for the use of multiple different combinations of information as input to the neural network, improving the accuracy of the probability distribution output by the neural network and maximizing the use of available coding information.

[0029] In another possible implementation, the neural network may include a convolutional network, a fully connected layer, and a normalized exponential function Softmax layer. The convolutional network includes multiple densely connected blocks and pooling layers between the densely connected blocks. The densely connected blocks include a convolutional layer, an excitation ReLU layer, and a batch normalization (BN) layer. Accordingly, the input data is subjected to probability estimation processing by the neural network to obtain a probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, including: extracting feature information from the input data through the convolutional network; performing probability estimation on the feature information extracted by the convolutional network through the fully connected layer and the Softmax layer, and outputting a probability distribution of multiple candidate intra-frame prediction mode syntax element value sets.

[0030] Exemplarily, the neural network can be used to perform probability estimation processing on input data of a 4×4 coding block to obtain probability distribution of multiple candidate intra-frame prediction mode syntax element value sets of the 4×4 coding block.

[0031] In another possible implementation, the neural network may include a fully connected network, a convolutional network, a concat layer, a first fully connected layer, and a Softmax layer. The fully connected network includes multiple second fully connected layers and activated PReLU layers connected one-to-one to the second fully connected layers. The convolutional network includes multiple densely connected blocks and pooling layers between the densely connected blocks. The densely connected blocks include convolutional layers, excitation ReLU layers, and BN layers. Accordingly, the input data is subjected to probability estimation processing by the neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, including: extracting feature information of the first category of information in the input data through the convolutional network; extracting feature information of information other than the first category of information in the input data through the fully connected network; concatenating the feature information extracted by the fully connected network and the feature information extracted by the convolutional network through the concat layer, and probabilistically estimating the feature information after concatenation in the concat layer through the first fully connected layer and the Softmax layer, and outputting the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets. The first category of information is pixel-level information of the image block.

[0032] In a possible implementation, the image block pixel-level information may refer to information related to the pixel values ​​of the image block, such as a reconstructed value, a predicted value, and a residual value.

[0033] The first type of information in the input data may be multi-channel data obtained by splicing; and information other than the first type of information in the input data may be a one-dimensional vector formed by connecting the converted vectors.

[0034] Exemplarily, the neural network can be used to perform probability estimation processing on input data of coding blocks of sizes other than 4×4 coding blocks to obtain probability distributions of multiple candidate intra-frame prediction mode syntax element value sets of the 4×4 coding blocks.

[0035] Exemplarily, the above-mentioned feature information may be texture topology information.

[0036] In another possible implementation, the neural network may be a trained neural network with a probability estimation function.

[0037] In another possible implementation, the neural network is used to receive a training data set; and to train the neural network based on the training data set.

[0038] In another possible implementation, the training data set may be information of multiple adjacent image blocks of an image block.

[0039] In another possible implementation, the probability value related to the index of the set of syntax element values ​​of the intra-frame prediction mode of the current image block can be any probability value in the probability interval corresponding to the index of the set of syntax element values ​​of the intra-frame prediction mode of the current image block in multiple probability intervals constructed according to arithmetic coding based on the above-mentioned probability distribution.

[0040] In another possible implementation, the entropy coding described in this application may include: binary arithmetic coding, or multi-level arithmetic coding.

[0041] In a second aspect, a decoding method is provided, which may include: performing probability estimation processing on input data through a neural network to obtain a probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, the probability distribution representing the probability values ​​of each of the multiple candidate intra-frame prediction mode syntax element value sets, and the input data is used to characterize the characteristics of the current image block (such as the texture characteristics of the current image block); according to the probability distribution, entropy decoding is performed on the received code stream to obtain the index of the intra-frame prediction mode syntax element value set of the current image block; according to the index of the intra-frame prediction mode syntax element value set of the current image block, the intra-frame prediction mode syntax element value set of the current image block is determined, the syntax element value set including the values ​​of multiple syntax elements involved in the intra-frame prediction mode of the current image block, the multiple syntax elements including the index of the intra-frame prediction mode adopted by the current image block; and performing intra-frame prediction processing based on the values ​​of the multiple syntax elements. It should be understood that the neural network is a trained neural network with probability estimation function.

[0042] Through the decoding method provided by the embodiments of the present application, the values ​​of multiple (for example, all) syntax elements related to the intra-frame prediction mode of the current image block are used as a set of intra-frame prediction mode syntax element values. The intra-frame prediction mode syntax element value set corresponds to an index, and the index of the intra-frame prediction mode syntax element value set is understood as a syntax element, which is equivalent to converting the values ​​of multiple entropy-decoded syntax elements into the value of one entropy-decoded syntax element. The syntax element can represent the intra-frame prediction mode coding state of the current image block (for example, it can indicate the values ​​of multiple syntax elements involved in the intra-frame prediction mode actually adopted by the current image block). During entropy decoding, only one entropy decoding of the index of the intra-frame prediction mode syntax element value set is required to replace the entropy decoding of many intra-frame prediction mode-related syntax element values, thereby reducing the complexity of hardware implementation and improving decoding performance.

[0043] In one possible implementation, the intra-frame prediction mode syntax element value set of the current image block is determined according to the index of the intra-frame prediction mode syntax element value set of the current image block. Specifically, it can be implemented as follows: the intra-frame prediction mode syntax element value set corresponding to the first numerical interval to which the index of the intra-frame prediction mode syntax element value set of the current image block belongs is determined as the intra-frame prediction mode syntax element value set of the current image block. The starting value of the first numerical interval is the sum of the minimum index value of the first type of intra-frame prediction mode and the additional value of the first type of intra-frame prediction mode, and the ending value of the first numerical interval is the sum of the maximum index value of the first type of intra-frame prediction mode and the additional value of the first type of intra-frame prediction mode. The intra-frame prediction mode adopted by the current image block belongs to the first type of intra-frame prediction mode; the intervals formed by the sum of the minimum index value and the corresponding additional value and the sum of the maximum index value and the corresponding additional value of different types of intra-frame prediction modes do not intersect. This implementation method corresponds to the encoding end determining the index of the intra-frame prediction mode syntax element value set of the current image block based on the index of the intra-frame prediction mode adopted by the current image block, and the decoding end can simply and quickly determine the index of the intra-frame prediction mode adopted by the current image block based on the value range of the index of the intra-frame prediction mode syntax element value set of the current image block.

[0044] In another possible implementation, the additional value of the Nth type of intra-frame prediction mode includes: the sum of the label values ​​of each type of intra-frame prediction mode that is located before the Nth type of intra-frame prediction mode in a predetermined order.

[0045] The label value of the intra prediction mode may be a parameter configured for the intra prediction mode. Optionally, the specific value of the label value of the intra prediction mode may be configured according to actual needs, and the label values ​​of different types of intra prediction modes may be the same or different.

[0046] In another possible implementation, the label value of each type of intra-frame prediction mode may include: the total number of multiple intra-frame prediction modes in each type of intra-frame prediction mode.

[0047] In another possible implementation, if the predetermined order is the second type of intra-frame prediction mode, the third type of intra-frame prediction mode, and the first type of intra-frame prediction mode; the total number of multiple intra-frame prediction modes in the first type of intra-frame prediction mode is A, the total number of multiple intra-frame prediction modes in the second type of intra-frame prediction mode is B, and the total number of multiple intra-frame prediction modes in the first type of intra-frame prediction mode is C, the syntax element value set corresponding to the first numerical interval is determined, which is the intra-frame prediction mode syntax element value set of the current image block, including: if the first numerical interval is [0, B), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the second type of intra-frame prediction mode is true, the index of the intra-frame prediction mode adopted by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block, the flag bit of the first type of intra-frame prediction mode is false, and the flag bit of the third type of intra-frame prediction mode is false. If the first numerical interval is [B, B+C), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the third type of intra-frame prediction mode is true, the index of the intra-frame prediction mode adopted by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block minus B, the flag bit of the first type of intra-frame prediction mode is false, and the flag bit of the second type of intra-frame prediction mode is false. If the first numerical interval is [B+C, A+B+C), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the first type of intra-frame prediction mode is true, the index of the intra-frame prediction mode adopted by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block minus (B+C), the flag bit of the second type of intra-frame prediction mode is false, and the flag bit of the third type of intra-frame prediction mode is false.

[0048] In another possible implementation, before performing probability estimation processing on the input data through a neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, the encoding method provided by the present application may also include: determining a neural network suitable for the current image block based on network structure distinguishing information, the network structure distinguishing information includes the size of the current image block, or the network structure distinguishing information depends on the size of the current image block. Accordingly, performing probability estimation processing on the input data through a neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets includes: performing probability estimation processing on the input data through a determined neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets. In this way, different neural networks can be designed for different image blocks (for example, different sizes) to maximize the encoding performance.

[0049] In another possible implementation, the input data may include information about neighboring image blocks of the current image block, or may include preprocessed information about neighboring image blocks of the current image block, where the neighboring image blocks include image blocks adjacent to the upper left, above, and to the left; the information about the neighboring image blocks includes reconstructed values ​​of the neighboring image blocks. By fully leveraging the correlation between the neighboring image blocks of the current image block and the current image block, and since the neighboring image blocks have already been reconstructed, they can represent the characteristics of the current image block. The probability distribution obtained by inputting the information about the neighboring image blocks into the neural network has a higher accuracy.

[0050] In another possible implementation, the information of the neighboring image blocks of the preprocessed current image block may include: an M-channel data block formed by splicing the information of the neighboring image blocks and the information of the current image block; M is greater than or equal to 1.

[0051] For example, M may be less than or equal to the type of information included in the information of the image block.

[0052] In a possible implementation, the pre-processed information of the neighboring image blocks of the current image block may include a one-dimensional vector formed by connecting vectors converted from the information of the neighboring image blocks and the information of the current image block.

[0053] The vector of the information conversion of the image block may be a one-dimensional valid One-Hot vector.

[0054] It should be noted that the information of the current image block in the input data may be a default value for filling, and the embodiment of the present application does not limit the content of the default value.

[0055] In another possible implementation, the information about neighboring image blocks may further include one or more of the following: a prediction value, a residual value, an index to a set of intra-prediction mode syntax element values, and partitioning information. This allows full utilization of available information about neighboring image blocks, allowing for the use of multiple different combinations of information as input to the neural network, improving the accuracy of the probability distribution output by the neural network and maximizing the use of available coding information.

[0056] In another possible implementation, the neural network may include a convolutional network, a fully connected layer, and a normalized exponential function Softmax layer. The convolutional network includes multiple densely connected blocks and pooling layers between the densely connected blocks. The densely connected blocks include a convolutional layer, an excitation ReLU layer, and a BN layer. Accordingly, the input data is subjected to probability estimation processing by the neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, including: extracting feature information from the input data through the convolutional network; performing probability estimation on the feature information extracted by the convolutional network through the fully connected layer and the Softmax layer, and outputting the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets.

[0057] Exemplarily, the neural network can be used to perform probability estimation processing on input data of a 4×4 coding block to obtain probability distribution of multiple candidate intra-frame prediction mode syntax element value sets of the 4×4 coding block.

[0058] In another possible implementation, the neural network may include a fully connected network, a convolutional network, a concat layer, a first fully connected layer, and a Softmax layer. The fully connected network includes multiple second fully connected layers and activated PReLU layers connected one-to-one to the second fully connected layers. The convolutional network includes multiple densely connected blocks and pooling layers between the densely connected blocks. The densely connected blocks include convolutional layers, excitation ReLU layers, and BN layers. Accordingly, the input data is subjected to probability estimation processing by the neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, including: extracting feature information of the first category of information in the input data through the convolutional network; extracting feature information of information other than the first category of information in the input data through the fully connected network; concatenating the feature information extracted by the fully connected network and the feature information extracted by the convolutional network through the concat layer, and probabilistically estimating the feature information after concatenation in the concat layer through the first fully connected layer and the Softmax layer, and outputting the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets. The first category of information is pixel-level information of the image block.

[0059] In a possible implementation, the image block pixel-level information may refer to information related to the pixel values ​​of the image block, such as a reconstructed value, a predicted value, and a residual value.

[0060] The first type of information in the input data may be multi-channel data obtained by splicing; and information other than the first type of information in the input data may be a one-dimensional vector formed by connecting the converted vectors.

[0061] Exemplarily, the neural network can be used to perform probability estimation processing on input data of coding blocks of sizes other than 4×4 coding blocks to obtain probability distributions of multiple candidate intra-frame prediction mode syntax element value sets of the 4×4 coding blocks.

[0062] The above-mentioned feature information may be texture topology information.

[0063] In another possible implementation, the neural network may be a trained neural network with a probability estimation function.

[0064] In another possible implementation, the neural network is used to receive a training data set; and to train the neural network based on the training data set.

[0065] In another possible implementation, the training data set may be information of multiple adjacent image blocks of an image block.

[0066] In another possible implementation, the entropy decoding described in this application may include: binary arithmetic coding, or multi-level arithmetic coding.

[0067] In a third aspect, an encoder is provided, comprising an intra-frame prediction module and an entropy coding module, wherein:

[0068] An intra-frame prediction module is used to determine the index of a set of syntax element values ​​of the intra-frame prediction mode of the current image block, where the syntax element value set includes the values ​​of multiple syntax elements involved in the intra-frame prediction mode of the current image block, and the values ​​of the multiple syntax elements include the index of the intra-frame prediction mode adopted by the current image block.

[0069] The entropy coding module is configured to perform probability estimation processing on input data using a neural network to obtain a probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, where the probability distribution represents the probability value of each of the multiple candidate intra-frame prediction mode syntax element value sets. The input data is used to characterize the characteristics of the current image block. Based on the probability distribution of the multiple candidate intra-frame prediction mode syntax element value sets, entropy coding is performed on the probability values ​​associated with the index of the intra-frame prediction mode syntax element value set of the current image block to obtain a bitstream. It should be understood that the neural network is a trained neural network with probability estimation capabilities.

[0070] It should be noted that the encoder provided in the third aspect is used to execute the method described in any one of the first aspects above, and can achieve the same effect as the solution described in the first aspect above, and its specific implementation will not be repeated here.

[0071] In a fourth aspect, a decoder is provided, comprising an entropy decoding module and an intra-frame prediction module.

[0072] The entropy decoding module is configured to perform probability estimation processing on input data using a neural network to obtain a probability distribution of multiple candidate intra-frame prediction mode syntax element value sets. The probability distribution represents the probability values ​​of each of the multiple candidate intra-frame prediction mode syntax element value sets. The input data is used to characterize the characteristics of the current image block. Based on the probability distribution, the module performs entropy decoding on the received bitstream to obtain an index of the intra-frame prediction mode syntax element value set for the current image block. It should be understood that the neural network is a trained neural network with probability estimation capabilities.

[0073] An intra-frame prediction module is used to determine a set of syntax element values ​​for the intra-frame prediction mode of the current image block based on an index of the set of syntax element values ​​for the intra-frame prediction mode of the current image block, where the set of syntax element values ​​includes values ​​of multiple syntax elements involved in the intra-frame prediction mode of the current image block, where the multiple syntax elements include an index of the intra-frame prediction mode adopted by the current image block; and perform intra-frame prediction processing based on the values ​​of the multiple syntax elements.

[0074] It should be noted that the decoder provided in the fourth aspect is used to execute the method described in any one of the second aspects above, and can achieve the same effect as the solution described in the second aspect above, and its specific implementation will not be repeated here.

[0075] In a fifth aspect, the present application provides an encoder comprising a processing circuit for executing the method according to any one of the above-mentioned first aspects.

[0076] In a sixth aspect, the present application provides a decoder comprising a processing circuit for executing the method described in any one of the second aspects above.

[0077] In a seventh aspect, the present application provides a computer program product, comprising program code, which, when executed on a computer or processor, is used to execute the method described in any one of the first or second aspects above.

[0078] In an eighth aspect, the present application provides an encoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the encoder executes the method described in any one of the first aspects above.

[0079] In the ninth aspect, the present application provides a decoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the decoder executes the method described in any one of the second aspects above.

[0080] In a tenth aspect, the present application provides a non-transitory computer-readable storage medium comprising program code, which, when executed by a computer device, is used to execute the method described in any one of the first or second aspects above.

[0081] It should be noted that various possible implementations of any of the above aspects can be combined under the premise that the solutions are not contradictory. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1a A flowchart of a universal video coding (VVC) intra-frame prediction mode encoding method provided in an embodiment of the present application;

[0083] Figure 1b An exemplary block diagram of a decoding system provided in an embodiment of the present application;

[0084] Figure 1c A schematic diagram of another decoding system provided in an embodiment of the present application;

[0085] Figure 2 A schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0086] Figure 3 A schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0087] Figure 4 An exemplary block diagram of a video decoding device provided in an embodiment of the present application;

[0088] Figure 5 A schematic diagram of the structure of a video decoding device provided in an embodiment of the present application;

[0089] Figure 6 A schematic diagram of the network structure of a convolutional neural network provided in an embodiment of the present application;

[0090] Figure 7 A schematic diagram of a video transmission system architecture provided in an embodiment of the present application;

[0091] Figure 8 A schematic diagram of the structure of a neural network for predicting or estimating the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets provided in an embodiment of the present application;

[0092] Figure 9 A schematic diagram of the structure of another neural network for predicting or estimating the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets provided in an embodiment of the present application;

[0093] Figure 10A schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0094] Figure 11 A flowchart of another encoding method provided in an embodiment of the present application;

[0095] Figure 12 A schematic diagram of a tree structure provided in an embodiment of the present application;

[0096] Figure 13 A flowchart of another encoding method provided in an embodiment of the present application;

[0097] Figure 14 A flowchart of a decoding method provided in an embodiment of the present application;

[0098] Figure 15 A flowchart of another decoding method provided in an embodiment of the present application;

[0099] Figure 16 A schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0100] Figure 17 A schematic diagram of the structure of a decoder provided in an embodiment of the present application. DETAILED DESCRIPTION

[0101] The embodiments of the present application provide an AI-based video compression technology, in particular, a neural network-based video compression technology, and specifically provide a coding technology to improve the hybrid video coding and decoding system.

[0102] Video decoding generally refers to the processing of a sequence of images that form a video or video sequence. In the field of video decoding, the terms "picture", "frame" or "image" can be used as synonyms. Video decoding (or commonly referred to as CODEC) includes two parts: video encoding and video decoding. Video encoding is performed on the source side and generally includes processing (for example, compressing) the original video image to reduce the amount of data required to represent the video image (thereby making it more efficient to store and / or transmit). Video decoding is performed on the destination side and generally includes inverse processing relative to the encoder to reconstruct the video image. The "decoding" of the video image (or commonly referred to as image) involved in the embodiment should be understood as the "encoding" or "decoding" of the video image or video sequence. The encoding part and the decoding part are also collectively referred to as codec (encoding and decoding, CODEC).

[0103] In the case of lossless video decoding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (assuming there is no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed through quantization and the like to reduce the amount of data required to represent the video image, but the decoder side cannot fully reconstruct the video image, that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0104] Several video coding standards fall under the category of "lossy hybrid video coding" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is typically divided into a set of non-overlapping blocks, which are typically coded at the block level. In other words, the encoder typically processes, or encodes, the video at the block level (video blocks, also known as image blocks or coding blocks). For example, it generates a prediction block through spatial (intra-frame) and temporal (inter-frame) prediction; subtracts the prediction block from the current block (the block currently being processed / to be processed) to produce a residual block; transforms and quantizes the residual block in the transform domain to reduce the amount of data to be transmitted (compressed). The decoder then applies the inverse of the encoder's processing to the coded or compressed block to reconstruct the current block for representation. Furthermore, the encoder is required to repeat the decoder's processing steps so that the encoder and decoder generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstructed pixels for processing (i.e., encoding) subsequent blocks.

[0105] In the following embodiment of the decoding system 10, the encoder 20 and the decoder 30 are based on Figures 1b to 3 Provide a description.

[0106] Figure 1b FIG1 is an exemplary block diagram of a decoding system 10 according to an embodiment of the present application, for example, a video decoding system 10 (or simply decoding system 10) that can utilize the techniques of the present application. The video encoder 20 (or simply encoder 20) and the video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices that can be used to perform various techniques according to various examples described in this application.

[0107] like Figure 1b As shown, the decoding system 10 includes a source device 12 for providing encoded image data 21 such as an encoded image to a destination device 14 for decoding the encoded image data 21.

[0108] The source device 12 includes an encoder 20 , and optionally, may include an image source 16 , a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22 .

[0109] Image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). Image source 16 may be any type of memory or storage for storing any of the aforementioned images.

[0110] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data) 17 may also be referred to as a raw image (or raw image data) 17 .

[0111] The preprocessor 18 is configured to receive raw image data 17 and preprocess the raw image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color grading, or denoising. It will be appreciated that the preprocessor 18 may be an optional component.

[0112] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide the encoded image data 21 (hereinafter referred to as Figure 2 etc. for further description).

[0113] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0114] The destination device 14 includes a decoder 30 and, in addition or alternatively, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0115] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0116] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0117] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0118] The communication interface 28 corresponds to the communication interface 22 , and can be used, for example, to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0119] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1b The unidirectional communication interface or the bidirectional communication interface indicated by the arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0120] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as Figure 3 etc. for further description).

[0121] The post-processor 32 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image, to obtain post-processed image data 33, such as the post-processed image. The post-processing performed by the post-processor 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 31 for display on a display device 34, etc.

[0122] The display device 34 is configured to receive the post-processed image data 33 and display the image to a user or viewer. The display device 34 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0123] The decoding system 10 further includes a training engine 25 for training the encoder 20 (particularly a neural network in an entropy coding unit of the encoder 20 for predicting or estimating a probability distribution of multiple candidate intra-prediction mode syntax element value sets, or a neural network invoked by the entropy coding unit for predicting or estimating a probability distribution of multiple candidate intra-prediction mode syntax element value sets) or the decoder 30 (particularly a neural network in an entropy decoding unit of the decoder 30 for predicting or estimating a probability distribution of multiple candidate intra-prediction mode syntax element value sets, or a neural network invoked by the entropy decoding unit for predicting or estimating a probability distribution of multiple candidate intra-prediction mode syntax element value sets) to predict or estimate a probability distribution of multiple candidate intra-prediction mode syntax element value sets or intra-coding states for a current input image, a current image region, or a current image block, thereby performing entropy coding based on the probability distribution. The intra-coding states may be used to indicate multiple candidate intra-prediction mode syntax element value sets.

[0124] Among them, an intra-frame prediction mode syntax element value set includes the values ​​of multiple syntax elements involved in an intra-frame prediction mode, and an intra-frame prediction mode syntax element value set can represent or reflect the dependency relationship between the multiple syntax elements it includes. The values ​​of the multiple syntax elements included in each candidate intra-frame prediction mode syntax element value set in multiple candidate intra-frame prediction mode syntax element value sets refer to the values ​​of multiple syntax elements involved in the intra-frame prediction mode that can be used when performing intra-frame prediction on the current image block. The values ​​of the multiple syntax elements included in a candidate intra-frame prediction mode syntax element value set are dependent on each other. The intra-frame prediction mode syntax element value set actually used by the current image block can be one of the multiple candidate intra-frame prediction mode syntax element value sets.

[0125] For example, when the intra-frame prediction mode information encoding process is represented using a tree diagram structure, the tree diagram includes multiple chain structure examples from top to bottom, where each chain structure example corresponds to a certain set of candidate intra-frame prediction mode syntax element values, and different chain structure examples correspond to different sets of candidate intra-frame prediction mode syntax element values. It should be understood that each chain structure example represents or reflects a dependency relationship between multiple syntax elements.

[0126] For example, the current image block actually adopts intra-frame prediction mode a, which belongs to type A intra-frame prediction mode. The intra-frame prediction mode syntax element value set of the current image block may include type A intra-frame prediction mode flag bit being true and intra-frame prediction mode index being a.

[0127] For example, in VCC, the syntax elements related to the intra prediction mode include the MIP flag, MIP mode, multi-line reference index, ISP mode, MPM flag, MPM index, and intra prediction mode. For an image block using intra prediction, the syntax elements related to its intra prediction mode include the values ​​of the MIP flag, MIP mode, multi-line reference index, ISP mode, MPM flag, MPM index, and intra prediction mode, which is the set of syntax element values ​​for the intra prediction mode of the image block.

[0128] The training engine 25 may employ a training data set to train a neural network for predicting or estimating the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets.

[0129] Optionally, in an embodiment of the present application, the training data set may include: information about neighboring image blocks, wherein the information about neighboring image blocks includes, but is not limited to, one or more of the following information: reconstruction values, prediction values, residual values, indexes of intra-frame prediction mode syntax element value sets, and partitioning information of respective neighboring reconstructed image blocks. A neural network is obtained by training the training data set, wherein the input data of the neural network includes at least information about neighboring image blocks of the current image block, and the output data includes a probability distribution of the current image block adopting a plurality of candidate intra-frame prediction mode syntax element value sets, wherein the probability distribution represents the probability value of each of the plurality of candidate intra-frame prediction mode syntax element value sets.

[0130] The training engine 25 trains the neural network so that the probability value of the intra-frame prediction mode syntax element value set actually adopted by the current image block is as large as possible in the probability distribution of the multiple candidate intra-frame prediction mode syntax element value sets output by the current image block. Figure 8 、 Figure 9 A detailed introduction to neural networks.

[0131] The training data set in the embodiment of the present application can be stored in a database (not shown in the figure), and the training engine 25 trains a target model based on the training data set (for example, a neural network for predicting or estimating the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets). It should be noted that the embodiment of the present application does not limit the source of the training data set. For example, the training data set can be obtained from the cloud or other places for model training.

[0132] The target model trained by the training engine 25 can be applied to the decoding system 10, for example, Figure 1b The source device 12 (e.g., encoder 20) or the destination device 14 (e.g., decoder 30) is shown. The training engine 25 can train to obtain a target model in the cloud, and then the decoding system 10 downloads and uses the target model from the cloud; alternatively, the training engine 25 can train to obtain a target model in the cloud and use the target model, and the decoding system 10 directly obtains the processing results from the cloud. For example, the training engine 25 trains to obtain a target model with the function of predicting or estimating the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, and the decoding system 10 downloads the target model from the cloud. Then, the entropy encoding unit 270 in the encoder 20 or the entropy decoding unit 304 in the decoder 30 can perform entropy encoding or entropy decoding on the intra-frame prediction syntax elements of the input image or image block based on the output result of the target model. For another example, the training engine 25 trains a target model that has the function of predicting or estimating the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets. The decoding system 10 does not need to download the target model from the cloud. The encoder 20 or the decoder 30 transmits the image or image block to the cloud, and the cloud predicts or estimates the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets for the image or image block through the target model, and transmits it to the encoder 20 or the decoder 30. Then, the entropy encoding unit 270 in the encoder 20 or the entropy decoding unit 304 in the decoder 30 can entropy encode or entropy decode the intra-frame prediction syntax elements of the input image or image block based on the probability distribution.

[0133] although Figure 1b The source device 12 and the destination device 14 are shown as independent devices, but the device embodiments may also include both the source device 12 and the destination device 14 or the functions of both the source device 12 and the destination device 14, that is, include both the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions. In these embodiments, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0134] According to the description, Figure 1bIt will be apparent to the skilled person that the presence and (exact) division of the different units or functionalities in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application.

[0135] The encoder 20 (eg, video encoder 20) or the decoder 30 (eg, video decoder 30), or both, may be configured to generate a video signal by: Figure 1c The processing circuitry shown may be implemented, for example, by one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding processors, or any combination thereof. The encoder 20 may be implemented by processing circuitry 46 to include reference to Figure 2 The various modules discussed in connection with encoder 20 and / or any other encoder systems or subsystems described herein. Decoder 30 may be implemented by processing circuitry 46 to include reference to Figure 3 The processing circuitry 46 may be used to perform the various operations discussed below. Figure 5 As shown, if part of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium and use one or more processors to execute the instructions in hardware to perform the technology of the present application. One of the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as Figure 1c shown.

[0136] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook computer or laptop, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or content distribution server), a broadcast receiving device, a broadcast transmitting device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0137] In some cases, Figure 1bThe video decoding system 10 shown is merely exemplary, and the techniques provided herein may be applicable to video encoding devices (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding device and the decoding device. In other examples, data is retrieved from local memory, sent over a network, and so on. The video encoding device may encode the data and store the data in memory, and / or the video decoding device may retrieve the data from memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to memory and / or retrieve and decode data from memory.

[0138] Figure 1c FIG. 4 is an exemplary block diagram of a video decoding system 40 according to an embodiment of the present application. Figure 1c As shown, the video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by a processing circuit 46), an antenna 42, one or more processors 43, one or more memory storages 44 and / or a display device 45.

[0139] like Figure 1c As shown, imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory storage 44, and / or display device 45 can communicate with each other. In different embodiments, video decoding system 40 can include only video encoder 20 or only video decoder 30.

[0140] In some instances, antenna 42 can be used to transmit or receive an encoded bitstream of video data. Additionally, in some instances, display device 45 can be used to present the video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Video decoding system 40 can also include an optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Furthermore, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory). In a non-limiting example, memory storage 44 can be implemented as cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache memory, etc.) for implementing an image buffer, etc.

[0141] In some examples, video encoder 20 implemented by logic circuitry may include an image buffer (e.g., implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuitry 46 to implement reference Figure 2 and / or any other encoder system or subsystem described herein. Logic circuits may be used to perform the various operations discussed herein.

[0142] In some examples, video decoder 30 may be implemented in a similar manner by processing circuitry 46 to implement reference Figure 3 The various modules discussed herein may be used with reference to the video decoder 30 and / or any other decoder systems or subsystems described herein. In some examples, the video decoder 30 implemented by logic circuitry may include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented by processing circuitry 46 to implement the video decoder 30 as described above. Figure 3 and / or the various modules discussed with respect to any other decoder system or subsystem described herein.

[0143] In some examples, antenna 42 can be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream can include an index into a set of intra-prediction mode syntax element values ​​for an image block, as discussed herein. Optionally, the encoded bitstream can also include data, indicators, and the like related to the encoded video frame. Video decoding system 40 can also include a video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0144] It should be understood that for the examples described with reference to video encoder 20 in the embodiments of the present application, video decoder 30 can be used to perform the reverse process. With respect to signaling syntax elements, video decoder 30 can be used to receive and parse such syntax elements and decode the associated video data accordingly. In some examples, video encoder 20 can entropy encode syntax elements (e.g., indices into the set of intra-prediction mode syntax element values ​​for image blocks discussed herein) into the encoded video bitstream. In such examples, video decoder 30 can parse such syntax elements and decode the associated video data accordingly.

[0145] For ease of description, the embodiments of the present application are described with reference to the universal video coding (VVC) reference software or the high-efficiency video coding (HEVC) developed by the joint collaboration team on video coding (JCT-VC) of the ITU-T video coding experts group (VCEG) and the ISO / IEC motion picture experts group (MPEG). Those skilled in the art will understand that the embodiments of the present application are not limited to HEVC or VVC.

[0146] Encoders and encoding methods

[0147] Figure 2 FIG. 2 is an exemplary block diagram of a video encoder 20 according to an embodiment of the present application. Figure 2As shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The illustrated video encoder 20 may also be referred to as a hybrid video encoder or a hybrid video codec-based video encoder.

[0148] See also Figure 2 The inter-frame prediction unit 244 is configured to process an input image, image region, or image block to generate a prediction value for the input image block. For example, the inter-frame prediction unit 244 is configured to receive an input image, image region, or image block and generate a prediction value for the input image, image region, or image block.

[0149] The residual calculation unit 204, the transform processing unit 206, the quantization unit 208, and the mode selection unit 260 constitute the forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244, and the intra-frame prediction unit 254 constitute the backward signal path of the encoder, where the backward signal path of the encoder 20 corresponds to the signal path of the decoder (see Figure 3 The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer 230, the inter-frame prediction unit 244, and the intra-frame prediction unit 254 also constitute the "built-in decoder" of the video encoder 20.

[0150] Images and image segmentation (images and blocks)

[0151] Encoder 20 is operable to receive, via input 201 or the like, an image (or image data) 17, for example, an image from a sequence of images forming a video or video sequence. The received image or image data may also be a pre-processed image (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 may also be referred to as a current image or image to be encoded (particularly when distinguishing the current image from other images in video encoding, such as previously encoded and / or decoded images in the same video sequence, i.e., a video sequence that also includes the current image).

[0152] A (digital) image is, or can be considered to be, a two-dimensional array or matrix of pixels with intensity values. The pixels in the array are also referred to as pixels (or pels, short for picture elements). The number of pixels in the array or image in the horizontal and vertical directions (or axes) determines the image size and / or resolution. To represent color, three color components are typically used, meaning that an image can be represented as or include three pixel arrays. In the RBG format or color space, an image includes corresponding arrays of red, green, and blue pixels. However, in video coding, each pixel is typically represented in a luma / chroma format or color space, such as YCbCr, which includes a luma component indicated by Y (sometimes also indicated by L) and two chroma components, indicated by Cb and Cr. The luma component Y represents the brightness or grayscale level intensity (for example, in grayscale images, both are the same), while the two chroma components (abbreviated as chroma) Cb and Cr represent the chroma or color information components. Accordingly, an image in YCbCr format includes a luma pixel array of luma pixel values ​​(Y) and two chroma pixel arrays of chroma values ​​(Cb and Cr). An image in RGB format can be converted or transformed into YCbCr format, and vice versa, a process also known as color conversion or transformation. If the image is black and white, the image may include only a luma pixel array. Accordingly, the image may be, for example, a luma pixel array in monochrome format or a luma pixel array and two corresponding chroma pixel arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0153] In one embodiment, an embodiment of the video encoder 20 may include an image segmentation unit ( Figure 2), is used to partition the image 17 into a plurality of (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC) or coding tree blocks (CTBs), or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. The partitioning unit may be used to use the same block size for all images in a video sequence and a corresponding grid of defined block sizes, or to vary the block size between images or subsets or groups of images, and to partition each image into corresponding blocks.

[0154] In other embodiments, the video encoder may be configured to directly receive an image block 203 of the image 17, for example, one, several or all image blocks constituting the image 17. The image block 203 may also be referred to as a current image block or an image block to be encoded.

[0155] Like image 17, image block 203 is also or can be considered to be a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 203 is smaller than image 17. In other words, image block 203 may include one pixel array (e.g., a luminance array in the case of monochrome image 17, or a luminance array or chrominance array in the case of a color image), or three pixel arrays (e.g., one luminance array and two chrominance arrays in the case of color image 17), or any other number and / or type of arrays depending on the color format used. The number of pixels in the horizontal and vertical directions (or axes) of image block 203 defines the size (also referred to as the dimensions) of image block 203. Accordingly, the block can be an M×N (M columns × N rows) pixel array, or an M×N transform coefficient array, etc.

[0156] In one embodiment, Figure 2 The video encoder 20 shown is used to encode the image 17 block by block, for example, performing encoding and prediction on each image block 203.

[0157] In one embodiment, Figure 2 The illustrated video encoder 20 may also be configured to partition and / or encode an image using slices (also referred to as video slices), where an image may be partitioned or encoded using one or more (typically non-overlapping) slices. Each slice may include one or more blocks (e.g., coding tree units (CTUs)) or one or more groups of blocks (e.g., coding tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0158] In one embodiment, Figure 2The video encoder 20 shown can also be used to partition and / or encode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), where an image can be partitioned or encoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTUs) or one or more coding blocks, etc., where each coding block can be rectangular or other shaped and may include one or more complete or partial blocks (e.g., CTUs).

[0159] Residual calculation

[0160] The residual calculation unit 204 is used to calculate the residual block 205 (the prediction block 265 is described in detail later) based on the image block (or original block) 203 and the prediction block 265 in the following manner: for example, the pixel value of the prediction block 265 is subtracted from the pixel value of the image block 203 pixel by pixel (pixel by pixel) to obtain the residual block 205 in the pixel domain.

[0161] Transform

[0162] The transform processing unit 206 is configured to perform a discrete cosine transform (DCT) or a discrete sine transform (DST) on the pixel values ​​of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be referred to as transform residual coefficients, representing the residual block 205 in the transform domain.

[0163] The transform processing unit 206 may be used to apply an integerized approximation of the DCT / DST, such as the transform specified for H.265 / HEVC. This integerized approximation is typically scaled by a factor compared to the orthogonal DCT transform. In order to maintain the norm of the residual block after the forward and inverse transforms, other scaling factors are used as part of the transform process. The scaling factor is typically selected based on certain constraints, such as whether the scaling factor is a power of 2 for the shift operation, the bit depth of the transform coefficients, a trade-off between accuracy and implementation cost, etc. For example, a specific scaling factor is specified for the inverse transform on the encoder 20 side by the inverse transform processing unit 212 (and for the corresponding inverse transform on the decoder 30 side by, for example, the inverse transform processing unit 312), and correspondingly, a corresponding scaling factor may be specified for the forward transform on the encoder 20 side by the transform processing unit 206.

[0164] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206) may be configured to output transform parameters such as one or more transform types, for example, directly output or output after being encoded or compressed by the entropy coding unit 270, such that the video decoder 30 may receive and use the transform parameters for decoding.

[0165] Quantification

[0166] The quantization unit 208 is configured to quantize the transform coefficients 207 by, for example, scalar quantization or vector quantization to obtain quantized transform coefficients 209 . The quantized transform coefficients 209 may also be referred to as quantized residual coefficients 209 .

[0167] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, during quantization, an n-bit transform coefficient may be rounded down to an m-bit transform coefficient, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, varying degrees of scaling may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to finer quantization, while a larger quantization step size corresponds to coarser quantization. The appropriate quantization step size may be indicated by a quantization parameter (QP). For example, the quantization parameter may be an index into a predefined set of appropriate quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (a smaller quantization step size), while a larger quantization parameter may correspond to coarse quantization (a larger quantization step size), or vice versa. Quantization may include dividing by the quantization step size, while the corresponding or inverse dequantization performed by the inverse quantization unit 210, etc., may include multiplying by the quantization step size. Embodiments according to some standards, such as HEVC, may be used to determine the quantization step size using the quantization parameter. Generally, the quantization step size may be calculated based on the quantization parameter using a fixed-point approximation of an equation involving division. Other scaling factors can be introduced for quantization and dequantization to restore the norm of the residual block that may have been modified by the scaling used in the fixed-point approximation of the equations for the quantization step size and the quantization parameter. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where larger quantization step sizes result in greater losses.

[0168] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) may be configured to output a quantization parameter (QP), for example, directly or after being encoded or compressed by the entropy coding unit 270, such that the video decoder 30 may receive and use the quantization parameter for decoding.

[0169] Dequantization

[0170] The inverse quantization unit 210 is configured to perform inverse quantization performed by the quantization unit 208 on the quantized coefficients to obtain dequantized coefficients 211. For example, the inverse quantization scheme performed by the quantization unit 208 may be performed according to or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 may also be referred to as dequantized residual coefficients 211, and correspond to the transform coefficients 207. However, due to the loss caused by quantization, the dequantized coefficients 211 are generally not identical to the transform coefficients.

[0171] Inverse transform

[0172] The inverse transform processing unit 212 is configured to perform the inverse transform of the transform performed by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to obtain a reconstructed residual block 213 (or corresponding dequantized coefficients 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 213.

[0173] reconstruction

[0174] The reconstruction unit 214 (e.g., the summer 214) is used to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 213 and the pixel point values ​​of the prediction block 265.

[0175] Filtering

[0176] The loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain a filter block 221, or generally to filter the reconstructed pixels to obtain filtered pixel values. For example, the loop filter unit is used to smoothly perform pixel transitions or improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process can be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 2 2. The loop filter unit 220 is shown as a loop filter in FIG. 2, but in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. The filtering block 221 can also be referred to as a filtering and reconstruction block 221.

[0177] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) may be configured to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly or after being entropy-encoded by the entropy coding unit 270, such that the decoder 30 may receive and use the same or different loop filter parameters for decoding.

[0178] Decoded Image Buffer

[0179] The decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use by the video encoder 20 when encoding video data. The DPB 230 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer 230 may be used to store one or more filter blocks 221. The decoded picture buffer 230 may also be used to store other previously filtered blocks, such as previously reconstructed and filtered blocks 221, for the same current picture or a different picture, such as a previously reconstructed picture, and may provide a complete previously reconstructed, or decoded, picture (and corresponding reference blocks and pixels) and / or a partially reconstructed current picture (and corresponding reference blocks and pixels), for example, for inter-frame prediction. The decoded image buffer 230 may also be used to store one or more unfiltered reconstructed blocks 215, or generally to store unfiltered reconstructed pixels, for example, reconstructed blocks 215 that have not been filtered by the loop filtering unit 220, or reconstructed blocks or reconstructed pixels that have not undergone any other processing.

[0180] Mode selection (segmentation and prediction)

[0181] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, and is configured to receive or obtain original image data, such as an original block 203 (current block 203 of the current image 17) and reconstructed image data, such as filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images, from the decoded image buffer 230 or other buffer (e.g., a column buffer, not shown). The reconstructed image data is used as reference image data for prediction, such as inter-frame prediction or intra-frame prediction, to obtain a prediction block 265 or a prediction value 265.

[0182] The mode selection unit 260 may be used to determine or select a partitioning for the current block (including no partitioning) and prediction mode (eg, intra-frame or inter-frame prediction mode), generate a corresponding prediction block 265 , and calculate the residual block 205 and reconstruct the reconstruction block 215 .

[0183] In one embodiment, the mode selection unit 260 may be configured to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260) that provides the best match or minimum residual (minimum residual means better compression during transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression during transmission or storage), or simultaneously considers or balances both. The mode selection unit 260 may be configured to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., select the prediction mode that provides minimum rate distortion optimization. Terms such as "best," "lowest," and "optimal" herein do not necessarily refer to "best," "lowest," or "optimal" overall, but may also refer to situations where termination or selection criteria are met, e.g., values ​​exceeding or falling below a threshold or other limit may result in a "suboptimal selection" but reduce complexity and processing time.

[0184] In other words, the partitioning unit 262 may be configured to partition an image in a video sequence into a sequence of coding tree units (CTUs), the CTU 203 being further partitioned into smaller block portions or sub-blocks (again forming blocks), e.g., by iteratively using quad-tree partitioning (QT), binary-tree partitioning (BT), or triple-tree partitioning (TT), or any combination thereof, and for, e.g., performing prediction on each of the block portions or sub-blocks, wherein the mode selection comprises selecting a tree structure for partitioning the block 203 and selecting a prediction mode to be applied to each of the block portions or sub-blocks.

[0185] The segmentation (eg, performed by segmentation unit 262) and prediction processes (eg, performed by inter-prediction unit 244 and intra-prediction unit 254) performed by video encoder 20 are described in detail below.

[0186] segmentation

[0187] The partitioning unit 262 can partition (or divide) an image block (or CTU) 203 into smaller parts, such as square or rectangular blocks. For an image with three pixel arrays, a CTU consists of N×N luminance pixel blocks and two corresponding chrominance pixel blocks. The maximum allowed size of a luminance block in a CTU is specified as 128×128 in the developing Universal Video Coding (VVC) standard, but may be specified to a value other than 128×128, such as 256×256, in the future. The CTUs of an image can be grouped / collected into slices / coding block groups, coding blocks, or bricks. A coding block covers a rectangular area of ​​an image and can be divided into one or more bricks. A brick consists of multiple CTU rows within a coding block. A coding block that is not partitioned into multiple bricks can be called a brick. However, a brick is a true subset of a coding block and is therefore not called a coding block. VVC supports two coding block group modes: raster scan slice / coding block group mode and rectangular slice mode. In raster scan CBG mode, a slice / CBG contains a sequence of CBs from a raster scan of the CBs of an image. In rectangular slice mode, a slice contains multiple bricks of an image that together form a rectangular region of the image. The bricks within a rectangular slice are arranged in the slice's brick raster scan order. These smaller blocks (also called sub-blocks) can be further split into smaller parts. This is also known as tree partitioning or hierarchical tree partitioning, where a root block at, for example, root tree level 0 (hierarchy level 0, depth 0) can be recursively split into two or more blocks at the next lower tree level, such as nodes at tree level 1 (hierarchy level 1, depth 1). These blocks can in turn be split into two or more blocks at the next lower level, such as tree level 2 (hierarchy level 2, depth 2), and so on, until the partitioning is complete (because an end criterion is met, such as reaching a maximum tree depth or minimum block size). Blocks that are not further split are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary tree (BT), a tree divided into three parts is called a ternary tree (TT), and a tree divided into four parts is called a quadtree (QT).

[0188] For example, a coding tree unit (CTU) may be or include a CTB for luma pixels, two corresponding CTBs for chroma pixels of an image with a three-pixel array, or a CTB for pixels of a monochrome image, or a CTB for pixels of an image encoded using three independent color planes and syntax structures for encoding pixels. Accordingly, a coding tree block (CTB) may be an N×N block of pixels, where N may be set to a value such that a component is divided into CTBs, which is known as partitioning. A coding unit (CU) may be or include a coding block of luma pixels, two corresponding coding blocks for chroma pixels of an image with a three-pixel array, or a coding block of pixels of a monochrome image, or a coding block of pixels of an image encoded using three independent color planes and syntax structures for encoding pixels. Accordingly, a coding block (CB) may be an M×N block of pixels, where M and N may be set to a value such that a CTB is divided into coding blocks, which is known as partitioning.

[0189] For example, in an embodiment, according to HEVC, a coding tree unit (CTU) can be divided into multiple CUs using a quadtree structure represented as a coding tree. A decision is made at the leaf-CU level whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode an image area. Each leaf-CU can be further divided into one, two, or four PUs according to the PU partition type. The same prediction process is used within a PU, and relevant information is transmitted to the decoder in units of PUs. After applying the prediction process according to the PU partition type to obtain a residual block, the leaf-CU can be divided into transform units (TUs) according to other quadtree structures similar to the coding tree for the CU.

[0190] For example, in an embodiment, according to the latest video coding standard currently under development (called Versatile Video Coding (VVC), a combined quadtree of nested multi-type trees (such as binary trees and ternary trees) is used to divide the segment structure for partitioning the coding tree unit. In the coding tree structure within the coding tree unit, the CU can be square or rectangular. For example, the coding tree unit (CTU) is first partitioned by the quadtree structure. The quadtree leaf nodes are further partitioned by the multi-type tree structure. The multi-type tree structure has four partition types: vertical binary tree partition (SPLIT_BT_VER), horizontal binary tree partition (SPLIT_BT_HOR), vertical ternary tree partition The multi-type tree leaf nodes are called coding units (CUs), unless the CU is too large for the maximum transform length, such segmentation is used for prediction and transform processing without any other splitting. In most cases, this means that the block sizes of CUs, PUs, and TUs in the coding block structure of the quadtree nested multi-type tree are the same. This exception occurs when the maximum supported transform length is less than the width or height of the color components of the CU. VVC has developed a unique signaling mechanism for the split partitioning information in the coding structure with quadtree nested multi-type trees. In the signaling mechanism, the coding The tree unit (CTU) as the root of the quadtree is first split by the quadtree structure. Then each quadtree leaf node (when large enough) is further split into a multi-type tree structure. In the multi-type tree structure, the first flag (mtt_split_cu_flag) is used to indicate whether the node is further split. When the node is further split, the second flag (mtt_split_cu_vertical_flag) is used to indicate the division direction, and the third flag (mtt_split_cu_binary_flag) is used to indicate whether the division is a binary tree division or a ternary tree division. According to mtt_split_c The values ​​of u_vertical_flag and mtt_split_cu_binary_flag allow the decoder to derive the multi-type tree split mode (MttSplitMode) of the CU based on predefined rules or tables. It should be noted that for certain designs, such as the 64×64 luma block and 32×32 chroma pipeline design in the VVC hardware decoder, TT splitting is not allowed when the width or height of the luma coding block is greater than 64. TT splitting is also not allowed when the width or height of the chroma coding block is greater than 32. The pipeline design divides the image into multiple virtual pipeline data units (VPDUs), each of which is defined as a non-overlapping unit in the image. In the hardware decoder, consecutive VPDUs are processed simultaneously in multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it is desirable to keep the VPDU small.In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, the ternary tree (TT) and binary tree (BT) partitioning may increase the VPDU size.

[0191] In addition, it should be noted that when a part of the tree node block exceeds the bottom or the right boundary of the image, the tree node block is forcibly divided until all pixels of each coding CU are located within the image boundary.

[0192] For example, the intra sub-partitions (ISP) tool may divide the luma intra prediction block into two or four sub-partitions vertically or horizontally according to the block size.

[0193] In one example, mode select unit 260 of video encoder 20 may be used to perform any combination of the segmentation techniques described above.

[0194] As described above, the video encoder 20 is configured to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra-frame prediction modes and / or inter-frame prediction modes.

[0195] Intra-frame prediction

[0196] The intra prediction mode set may include 35 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in HEVC, or may include 67 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in VVC. For example, several traditional angle intra prediction modes are adaptively replaced with wide-angle intra prediction modes for non-square blocks defined in VVC. For another example, in order to avoid the division operation of DC prediction, only the longer side is used to calculate the average value of the non-square block. In addition, the intra prediction result of the planar mode can also be modified using the position-dependent intra prediction combination (PDPC) method.

[0197] The intra prediction unit 254 is configured to generate an intra prediction block 265 using reconstructed pixels of adjacent blocks of the same current image according to an intra prediction mode in the intra prediction mode set.

[0198] The intra-frame prediction unit 254 (or generally the mode selection unit 260) is also used to output intra-frame prediction parameters (or generally information indicating the selected intra-frame prediction mode of the block, or an index into a set of intra-frame prediction mode syntax element values ​​as described herein) in the form of syntax elements 266 to be sent to the entropy coding unit 270 for inclusion in the encoded image data 21, so that the video decoder 30 can perform operations such as receiving and using the prediction parameters for decoding.

[0199] Among them, the intra-frame prediction unit 254 can execute the solution provided by the present application to determine the index of the intra-frame prediction mode syntax element value set corresponding to the information of the selected intra-frame prediction mode (in the form of syntax element 266). Its specific implementation is detailed in the introduction to the method embodiment of the present application.

[0200] It should be noted that determining the index of the set of intra-frame prediction mode syntax element values ​​corresponding to the information of the selected intra-frame prediction mode can be performed by the intra-frame prediction unit 254 or by a unit deployed separately from the intra-frame prediction unit 254, and this embodiment of the present application is not limited to this.

[0201] The intra prediction modes in HEVC include DC prediction mode, plane prediction mode and 33 angular prediction modes, with a total of 35 candidate prediction modes. The current block can use the pixels of the reconstructed image blocks on the left and above as references for intra prediction. The image blocks in the surrounding area of ​​the current block used for intra prediction of the current block are called reference blocks, and the pixels in the reference blocks are called reference pixels. Among the 35 candidate prediction modes, the DC prediction mode is applicable to areas with flat textures in the current block. All pixels in this area use the average value of the reference pixels in the reference block as prediction; the plane prediction mode is applicable to image blocks with smoothly changing textures. The current block that meets this condition uses the reference pixels in the reference block for bilinear interpolation as the prediction of all pixels in the current block; the angular prediction mode uses the characteristic that the texture of the current block is highly correlated with the texture of the adjacent reconstructed image blocks, and copies the values ​​of the reference pixels in the corresponding reference block along a certain angle as the prediction of all pixels in the current block.

[0202] The HEVC encoder selects an optimal intra-frame prediction mode for the current block from 35 candidate prediction modes and writes this optimal intra-frame prediction mode into the video bitstream. To improve the coding efficiency of intra-frame prediction, the encoder / decoder derives three most probable modes from the optimal intra-frame prediction modes of the reconstructed image blocks in the surrounding area using intra-frame prediction. If the optimal intra-frame prediction mode selected for the current block is one of these three most probable modes, a first index is encoded to indicate that the selected optimal intra-frame prediction mode is one of these three most probable modes; if the selected optimal intra-frame prediction mode is not one of these three most probable modes, a second index is encoded to indicate that the selected optimal intra-frame prediction mode is one of the other 32 modes (other than the aforementioned three most probable modes among the 35 candidate prediction modes). The HEVC standard uses a 5-bit fixed-length code as the aforementioned second index.

[0203] The HEVC encoder derives the three most probable modes by selecting the optimal intra-frame prediction modes of the left and upper neighboring image blocks of the current block and adding them to a set. If these two optimal intra-frame prediction modes are the same, only one is retained in the set. If these two optimal intra-frame prediction modes are the same and both are angular prediction modes, two angular prediction modes adjacent to the angular direction are selected and added to the set. Otherwise, the planar prediction mode, the DC mode, and the vertical prediction mode are selected and added to the set in sequence until the number of modes in the set reaches three.

[0204] After the HEVC decoder performs entropy decoding on the bitstream, it obtains the mode information of the current block, which includes an indicator indicating whether the optimal intra-frame prediction mode of the current block is among the three most probable modes, and the index of the optimal intra-frame prediction mode of the current block among the three most probable modes or the index of the optimal intra-frame prediction mode of the current block among the other 32 modes.

[0205] Inter-frame prediction

[0206] In a possible implementation, the set of inter-prediction modes depends on the available reference picture (i.e., at least part of the previously decoded picture stored in the DBP 230 as mentioned above) and other inter-prediction parameters, e.g., on whether the entire reference picture is used or only a part of the reference picture is used, e.g., a search window area around the area of ​​the current block, to search for the best matching reference block, and / or on whether pixel interpolation such as half-pixel, quarter-pixel and / or 1 / 16 interpolation is performed, for example.

[0207] In addition to the above prediction modes, skip mode and / or direct mode may also be employed.

[0208] For example, in extended merge prediction, the merge candidate list of this mode consists of the following five candidate types in order: spatial MVP from spatially adjacent CUs, temporal MVP from collocated CUs, history-based MVP from the FIFO table, pairwise average MVP, and zero MV. Decoder-side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of the MV in merge mode. Merge mode with MVD (MMVD) is derived from merge mode with motion vector difference. The MMVD flag is sent immediately after the skip flag and merge flag to specify whether the CU uses MMVD mode. A CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding the CU's MVD with different precisions. The MVD of the current CU is adaptively selected based on the prediction mode of the current CU. When the CU is encoded in merge mode, the combined inter / intra prediction (CIIP) mode can be applied to the current CU. The CIIP prediction is obtained by weighted averaging the inter and intra prediction signals. For affine motion compensation prediction, the affine motion field of the block is described by the motion information of 2 control points (4 parameters) or 3 control points (6 parameters) motion vectors. Subblock-based temporal motion vector prediction (SbTMVP) is similar to the temporal motion vector prediction (TMVP) in HEVC, but it predicts the motion vector of the sub-CU within the current CU. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces calculations, especially in terms of the number of multiplications and the size of the multipliers. In the triangle partitioning mode, the CU is evenly divided into two triangular parts using diagonal partitioning and anti-diagonal partitioning. In addition, the bidirectional prediction mode is extended based on simple averaging to support weighted averaging of two prediction signals.

[0209] The inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in Figure 2(not shown in the figure). The motion estimation unit may be configured to receive or obtain an image block 203 (the current image block 203 of the current image 17) and a decoded image 231, or at least one or more previously reconstructed blocks, e.g., reconstructed blocks of one or more other / different previously decoded images 231, for motion estimation. For example, a video sequence may include the current image and the previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or form a sequence of images forming the video sequence.

[0210] For example, the encoder 20 may be configured to select a reference block from a plurality of reference blocks of the same or different images in a plurality of other images, and provide the reference image (or reference image index) and / or the offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. This offset is also referred to as a motion vector (MV).

[0211] The motion compensation unit is configured to obtain, for example, receive, inter-frame prediction parameters and perform inter-frame prediction based on or using the inter-frame prediction parameters to obtain an inter-frame prediction block 246. The motion compensation performed by the motion compensation unit may include extracting or generating a prediction block based on a motion / block vector determined by motion estimation, and may also include performing interpolation with sub-pixel accuracy. Interpolation filtering can generate pixel points of other pixels from pixel points of known pixels, thereby potentially increasing the number of candidate prediction blocks that can be used to encode the image block. Upon receiving a motion vector corresponding to a PU of the current image block, the motion compensation unit may locate the prediction block pointed to by the motion vector in one of the reference picture lists.

[0212] The motion compensation unit may also generate syntax elements associated with blocks and video slices for use by video decoder 30 when decoding image blocks of a video slice. In addition to, or in lieu of, slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be generated or used.

[0213] In the process of obtaining a candidate motion vector list in an advanced motion vector prediction (AMVP) mode, motion vectors (MVs) that can be added to the candidate motion vector list as alternatives include MVs of spatially adjacent and temporally adjacent image blocks of the current block, wherein the MVs of the spatially adjacent image blocks may include the MV of a left candidate image block located on the left side of the current block and the MV of an upper candidate image block located above the current block.

[0214] After obtaining the candidate motion vector list, the optimal MV is determined from the candidate motion vector list using the rate distortion cost (RDcost). The candidate motion vector with the smallest RD cost is used as the motion vector predictor (MVP) for the current block. The rate distortion cost is calculated using the following formula:

[0215] J=SAD+λR

[0216] Wherein, J represents RD cost, SAD is the sum of absolute differences (SAD) between the pixel values ​​of the predicted block obtained after motion estimation using the candidate motion vector and the pixel values ​​of the current block, R represents the bit rate, and λ represents the Lagrange multiplier.

[0217] The encoder passes the index of the determined MVP in the candidate motion vector list to the decoder. Furthermore, a motion search can be performed within a neighborhood centered on the MVP to obtain the actual motion vector of the current block. The encoder calculates the motion vector difference (MVD) between the MVP and the actual motion vector and also passes the MVD to the decoder. The decoder parses the index, finds the corresponding MVP in the candidate motion vector list based on the index, parses the MVD, and adds the MVD to the MVP to obtain the actual motion vector of the current block.

[0218] In the process of obtaining the candidate motion information list in the Merge mode, the motion information that can be added to the candidate motion information list as an alternative includes the motion information of the spatially adjacent or temporally adjacent image blocks of the current block. If the spatially adjacent block is not available or is intra-frame predicted, its motion information is not added to the candidate motion information list. The temporal candidate motion information of the current block is obtained by scaling the MV of the corresponding position block in the reference frame according to the picture order count (POC) of the reference frame and the current frame. First, it is determined whether the block at position T in the reference frame is available. If not, the block at position C is selected. After obtaining the above candidate motion information list, the optimal motion information is determined from the candidate motion information list through the RD cost as the motion information of the current block. The encoder passes the index value of the position of the optimal motion information in the candidate motion information list (denoted as merge index) to the decoder.

[0219] Entropy Coding

[0220] The entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (for example, a variable length coding (VLC) scheme, a context adaptive VLC scheme (CALVC), an arithmetic coding scheme, a binarization algorithm, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters (or generally information indicating the selected intra-frame prediction mode of the block, or an index to the set of intra-frame prediction mode syntax element values ​​described in the present application), loop filter parameters and / or other syntax elements to obtain coded image data 21 that can be output through the output terminal 272 in the form of a coded bitstream 21, etc., so that the video decoder 30, etc. can receive and use the parameters for decoding. The encoded bitstream 21 may be transmitted to the video decoder 30 or stored in memory for later transmission or retrieval by the video decoder 30 .

[0221] For example, compared to the previous generation international video coding standard HEVC, VVC introduces some new intra-frame prediction technologies, including matrix-based intra prediction (MIP), multiple reference lines (MRL), and intra sub-block partitioning (ISP). In addition, VVC expands the number of intra-frame prediction directions from 33 in HEVC to 65, and the number of most probable modes (MPM) from 3 in HEVC to 6.

[0222] In VVC, a context-adaptive binary arithmetic coding (CABAC) is used to entropy encode multiple syntax element values. The encoding process can be divided into three steps: binarization, context modeling, and arithmetic coding. If the value of a syntax element is not binary, it is first binarized. Then, for each bit, there are two coding modes: normal coding mode and bypass coding mode. In normal coding mode, the probability of each bit is estimated according to its context model. The bit value and the estimated probability distribution are then fed into the arithmetic encoder, which then updates the context model of the bit. In bypass coding mode, each bit is encoded with equal probability to speed up the encoding process.

[0223] In VCC, the set of syntax element values ​​related to intra prediction mode includes MIP flag, MIP mode, multi-line reference index, ISP mode, MPM flag, MPM index and intra prediction mode. Figure 1a This article presents a flowchart of a VVC intra-frame prediction mode encoding method. For a prediction block, the MIP flag is first encoded (using conventional coding mode). If the MIP flag is true, the MIP mode is encoded (bypass coding mode). Otherwise, the multi-row reference index (conventional coding mode), ISP mode (conventional coding mode), and MPM flag (conventional coding mode) are encoded in sequence. If the MPM flag is true, the MPM index is encoded (bypass coding mode). Otherwise, the intra-frame prediction mode is encoded (bypass coding mode).

[0224] In an embodiment of the present application, taking VVC intra-frame prediction mode encoding as an example, the entropy encoding unit 270 can perform entropy encoding on the index (provided by the intra-frame prediction unit 254) of a set of syntax element values ​​related to the intra-frame prediction mode (including MIP flag, MIP mode, multi-line reference index, ISP mode, MPM flag, MPM index and intra-frame prediction mode, etc.).

[0225] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform based encoder 20 may directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another implementation, the encoder 20 may have the quantization unit 208 and the inverse quantization unit 210 combined into a single unit.

[0226] Decoder and decoding method

[0227] Figure 3This is an exemplary block diagram of a video decoder 30 according to an embodiment of the present application. The video decoder 30 is configured to receive, for example, encoded image data 21 (e.g., an encoded bitstream 21) encoded by an encoder 20, and obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks of an encoded video slice (and / or coding block group or coding block) and related syntax elements (e.g., indices of the intra-frame prediction mode syntax element value set described in this application).

[0228] exist Figure 3 In the example of FIG. 3 , decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. Inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same motion compensation as reference video. Figure 2 The decoding process of the video encoder 20 is the opposite of the encoding process described.

[0229] See also Figure 3 The inter-frame prediction unit 344 is configured to process an input image, image region, or image block to generate a prediction value for the input image block. For example, the inter-frame prediction unit 344 is configured to receive an input image, image region, or image block and generate a prediction value for the input image, image region, or image block.

[0230] As described with respect to encoder 20, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer DPB 230, inter prediction unit 244, and intra prediction unit 254 further constitute the "built-in encoder" of video encoder 20. Correspondingly, inverse quantization unit 310, inverse transform processing unit 312, reconstruction unit 314, loop filter 320, decoded picture buffer 330, inter prediction unit 344, and intra prediction unit 354 further constitute the "built-in decoder" of video decoder 30. Inverse quantization unit 310 may be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 may be functionally identical to inverse transform processing unit 212, reconstruction unit 314 may be functionally identical to reconstruction unit 214, loop filter 320 may be functionally identical to loop filter 220, and decoded picture buffer 330 may be functionally identical to decoded picture buffer 230. Therefore, the explanations of the corresponding units and functions of video encoder 20 apply correspondingly to the corresponding units and functions of video decoder 30 .

[0231] Entropy decoding

[0232] The entropy decoding unit 304 is used to parse the bit stream 21 (or generally the coded image data 21) and perform entropy decoding on the coded image data 21 to obtain the quantization coefficients 309 and / or the decoded coding parameters ( Figure 3 , such as any or all of inter-frame prediction parameters (e.g., reference picture index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index, or index into the intra-frame prediction mode syntax element value set described herein), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. Entropy decoding unit 304 may be configured to apply a decoding algorithm or scheme corresponding to the encoding scheme of entropy encoding unit 270 of encoder 20. Entropy decoding unit 304 may also be configured to provide inter-frame prediction parameters, intra-frame prediction parameters (e.g., index into the intra-frame prediction mode syntax element value set described herein), and / or other syntax elements to mode application unit 360, as well as to provide other parameters to other units of decoder 30. Video decoder 30 may receive syntax elements at the video slice and / or video block level. In addition to, or in lieu of, slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be received or used.

[0233] Furthermore, the intra-frame prediction parameters (such as the index of the intra-frame prediction mode syntax element value set described in the present application) provided by the entropy decoding unit 304 to the mode application unit 360 can be used by the intra-frame prediction unit 354 to execute the scheme of the present application, determine the intra-frame prediction mode syntax element value set according to the index of the intra-frame prediction mode syntax element value set, and then perform inter-frame prediction. For details of its specific implementation, please refer to the introduction of the method embodiment of the present application.

[0234] It should be noted that determining the intra-frame prediction mode syntax element value set based on the index of the intra-frame prediction mode syntax element value set can be performed by the intra-frame prediction unit 354 or by a unit deployed separately from the intra-frame prediction unit 354, and this is not limited to the embodiments of the present application.

[0235] Dequantization

[0236] The inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or generally information related to inverse quantization) and quantization coefficients from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and inverse quantize the decoded quantization coefficients 309 based on the quantization parameter to obtain inverse quantization coefficients 311, which may also be referred to as transform coefficients 311. The inverse quantization process may include using the quantization parameter calculated by the video encoder 20 for each video block in the video slice to determine a degree of quantization, and thus a degree of inverse quantization to be performed.

[0237] Inverse transform

[0238] The inverse transform processing unit 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain a reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 313. The transform may be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may also be configured to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform to be applied to the dequantized coefficients 311.

[0239] reconstruction

[0240] The reconstruction unit 314 (eg, summer 314 ) is configured to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the pixel domain, eg, by adding the pixel values ​​of the reconstructed residual block 313 and the pixel values ​​of the prediction block 365 .

[0241] Filtering

[0242] The loop filter unit 320 (in or after the encoding loop) is used to filter the reconstructed block 315 to obtain a filter block 321, thereby smoothly performing pixel conversion or improving video quality. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process can be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 3 3. Although shown as a loop filter in FIG. 3, in other configurations, the loop filter unit 320 may be implemented as a post-loop filter.

[0243] Decoded Image Buffer

[0244] The decoded video blocks 321 of one picture are then stored in a decoded picture buffer 330, which stores the decoded picture 331 as a reference picture for subsequent motion compensation of other pictures and / or for respective output displays.

[0245] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., for display to the user or for the user to view.

[0246] predict

[0247] The inter-frame prediction unit 344 may be functionally identical to the inter-frame prediction unit 244 (particularly the motion compensation unit), and the intra-frame prediction unit 354 may be functionally identical to the inter-frame prediction unit 254 and may determine the partitioning or segmentation and perform prediction based on the segmentation and / or prediction parameters or corresponding information received from the coded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304). The mode application unit 360 may be configured to perform prediction (intra-frame or inter-frame prediction) for each block based on the reconstructed image, block, or corresponding pixel point (filtered or unfiltered), resulting in a prediction block 365.

[0248] When the video slice is encoded as an intra-coded (I) slice, the intra-prediction unit 354 in the mode application unit 360 is configured to generate a prediction block 365 for the image block of the current video slice based on the indicated intra-prediction mode and data from previously decoded blocks of the current image. When the video image is encoded as an inter-coded (i.e., B or P) slice, the inter-prediction unit 344 (e.g., a motion compensation unit) in the mode application unit 360 is configured to generate a prediction block 365 for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 304. For inter-prediction, these prediction blocks can be generated from one of the reference pictures in one of the reference picture lists. Video decoder 30 can construct reference frame list 0 and list 1 using a default construction technique based on the reference pictures stored in DPB 330. The same or similar processes may be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks) in addition to or instead of slices (e.g., video slices), e.g., video may be encoded using I, P, or B coding block groups and / or coding blocks.

[0249] Mode application unit 360 is configured to determine prediction information for video blocks of a current video slice by parsing motion vectors and other syntax elements, and to use the prediction information to generate a prediction block for the current video block being decoded. For example, mode application unit 360 uses received syntax elements to determine the prediction mode (e.g., intra prediction or inter prediction) used to encode the video blocks of the video slice, the inter prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more reference picture lists for the slice, motion vectors for each inter-coded video block in the slice, inter prediction status for each inter-coded video block in the slice, and other information to decode the video blocks within the current video slice. In addition to or in lieu of slices (e.g., video slices), the same or similar processes may be applied to embodiments involving coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), for example, where video may be encoded using I, P, or B coding block groups and / or coding blocks.

[0250] In one embodiment, Figure 3 The video encoder 30 may also be configured to partition and / or decode an image using slices (also referred to as video slices), where an image may be partitioned or decoded using one or more (typically non-overlapping) slices. Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coding blocks in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0251] In one embodiment, Figure 3 The video decoder 30 shown can also be used to segment and / or decode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), where an image can be segmented or decoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTUs) or one or more coding blocks, etc., where each coding block can be rectangular or other shaped and may include one or more complete or partial blocks (e.g., CTUs).

[0252] Other variations of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, a non-transform-based decoder 30 may directly inverse quantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, the video decoder 30 may have the inverse quantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0253] It should be understood that the processing result of the current step can be further processed in the encoder 20 and the decoder 30 and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, the processing result of interpolation filtering, motion vector derivation, or loop filtering can be further operated on, such as clipping or shifting operations.

[0254] It should be noted that further operations can be performed on the derived motion vector of the current block (including but not limited to the control point motion vector of the affine mode, the sub-block motion vector of the affine, planar, ATMVP mode, the temporal motion vector, etc.). For example, the value of the motion vector is limited to a predefined range based on the representation bit of the motion vector. If the representation bit of the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents a power. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of four 4×4 sub-blocks in an 8×8 block) is limited so that the maximum difference between the integer parts of the four 4×4 sub-block MVs does not exceed N pixels, for example, not more than 1 pixel. Two methods of limiting the motion vector based on bitDepth are provided here.

[0255] It should be noted that both the encoder and decoder process each image block in a pre-agreed order, so the encoder does not need to transmit additional information about the processing order of the coded blocks to the decoder.

[0256] Although the above embodiments primarily describe video coding, it should be noted that embodiments of the decoding system 10, encoder 20, and decoder 30, as well as other embodiments described herein, may also be used for still image processing or coding, i.e., processing or coding a single image in a video codec that is independent of any previous or subsequent images. In general, if image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and the inter-frame prediction unit 344 (decoder) may not be available. All other functionalities (also referred to as tools or techniques) of the video encoder 20 and video decoder 30 are also applicable to still image processing, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354, and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.

[0257] Figure 4FIG4 is an exemplary block diagram of a video decoding device 400 according to an embodiment of the present application. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder, for example, Figure 1b The video decoder 30 in , may also be an encoder, such as Figure 1b The video encoder 20 in .

[0258] Video decoding device 400 includes: an input port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; for example, processor 430 may be a neural network processor 430; a transmitter unit (Tx) 440 and an output port 450 (or output port 450) for transmitting data; and a memory 460 for storing data. Video decoding device 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to input port 410, receiver unit 420, transmitter unit 440, and output port 450 for outputting or transmitting optical or electrical signals.

[0259] The processor 430 is implemented in hardware and software. The processor 430 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the input port 410, the receiving unit 420, the transmitting unit 440, the output port 450, and the memory 460. The processor 430 includes a decoding module 470 (e.g., a neural network-based decoding module 470). The decoding module 470 implements the embodiments disclosed above. For example, the decoding module 470 performs, processes, prepares, or provides various encoding operations. Therefore, the decoding module 470 provides substantial improvements to the functionality of the video decoding device 400 and affects the switching of the video decoding device 400 to different states. Alternatively, the decoding module 470 is implemented by instructions stored in the memory 460 and executed by the processor 430.

[0260] Memory 460 includes one or more disks, tape drives, and solid-state drives and can be used as overflow data storage for storing programs when such programs are selected for execution, and for storing instructions and data read during program execution. Memory 460 can be volatile and / or non-volatile and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0261] Figure 5 This is an exemplary block diagram of an apparatus 500 according to an embodiment of the present application. The apparatus 500 may be used as Figure 1b Either or both of the source device 12 and the destination device 14 in .

[0262] The processor 502 in the apparatus 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices, now available or developed in the future, capable of manipulating or processing information. While the disclosed implementations may be implemented using a single processor, such as the processor 502 shown, using more than one processor may provide greater speed and efficiency.

[0263] In one implementation, the memory 504 in the apparatus 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 accessed by the processor 502 via a bus 512. The memory 504 may also include an operating system 508 and application programs 510, which include at least one program that allows the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1 through N, as well as a video encoding application and a video decoding application that perform the methods described herein.

[0264] The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with touch-sensitive elements that can be used to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0265] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. Furthermore, secondary storage may be directly coupled to other components of device 500 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 500 may have a variety of configurations.

[0266] Since the embodiments of the present application involve the application of neural networks, in order to facilitate understanding, some nouns or terms used in the embodiments of the present application are explained below, and these nouns or terms are also considered part of the content of the invention.

[0267] (1) Neural Network

[0268] A neural network (NN) is a machine learning model. A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0269]

[0270] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0271] (2) Deep Neural Networks

[0272] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. The "many" here does not have a specific metric. Based on the position of different layers in a DNN, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, Output vector, b is the offset vector, W is the weight matrix (also called coefficient), α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the number of coefficients W and offset vectors b is also large. The definitions of these parameters in DNN are as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0273] (3) Convolutional Neural Networks

[0274] A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture involves multiple levels of learning at different levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which individual neurons respond to an input image. A convolutional neural network consists of a feature extractor consisting of convolutional and pooling layers. The feature extractor can be thought of as a filter, and the convolution process can be thought of as convolving an input image or feature map with a trainable filter.

[0275] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution on the input signal. A convolutional layer can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from the input image matrix. A convolution operator is essentially a weight matrix, which is usually predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), to extract specific features from the image. The size of this weight matrix should be proportional to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during the convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolved output with a single depth dimension. However, in most cases, multiple weight matrices of the same size (row × column) are applied instead. This is known as multiple homogeneous matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. The dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors in the image, and yet another weight matrix is ​​used to blur unwanted noise in the image. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation. In practical applications, the weight values ​​in these weight matrices require extensive training. The weight matrices formed by the trained weight values ​​can be used to extract information from the input image, allowing the convolutional neural network to make accurate predictions. When a convolutional neural network has multiple convolutional layers, the initial convolutional layers often extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by subsequent convolutional layers become increasingly complex, such as high-level semantic features. Features with higher semantics are more applicable to the problem being solved.

[0276] Because it's often necessary to reduce the number of trainable parameters, pooling layers are often periodically introduced after convolutional layers. This can be done in a single convolutional layer followed by a pooling layer, or in a multi-layered system followed by one or more pooling layers. In image processing, the sole purpose of a pooling layer is to reduce the spatial size of an image. Pooling layers can include average pooling and / or max pooling operators, which are used to downsample the input image to produce a smaller image. The average pooling operator calculates the average value of pixel values ​​within a specific range, producing the average pooling result. The max pooling operator takes the pixel with the largest value within a specific range as the max pooling result. Furthermore, just as the size of the weight matrix used in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The output image size after processing by a pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding subregion of the image input to the pooling layer.

[0277] After being processed by the convolution layer / pooling layer, the convolutional neural network is still not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers can be pre-trained based on relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0278] Optionally, after the multiple hidden layers in the neural network layer, an output layer of the entire convolutional neural network is also included. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0279] (4) Loss function

[0280] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0281] (5) Backpropagation algorithm

[0282] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the optimal super-resolution model parameters, such as the weight matrix.

[0283] The present application relates to the entropy coding / decoding process in the aforementioned encoding and decoding process. At present, the industry provides a process for an entropy coding scheme for HEVC intra-frame prediction mode based on a neural network, in which entropy coding is implemented through a convolutional neural network (CNN) module and a multi-level arithmetic coding engine. The input of the CNN module is the reconstructed blocks to the upper left, upper and left of the current coding block and the three MPMs of the coding block, and the output is the probability distribution of the intra-frame prediction mode of the current coding block. Among them, the probability distribution of the intra-frame prediction mode is a 35×1 vector, which contains 35 probability values, respectively representing the probability of the coding block taking the corresponding numerical prediction mode. For example, the convolutional neural network outputs the probability distribution of the intra-frame prediction mode of the current coding block as P, where P(5)=0.3 means that the probability that the current coding block prediction mode is 5 is 0.3.

[0284] The input of the multi-stage arithmetic encoder is the syntax elements of the best intra-frame prediction mode for the current coding block, as well as the probability distribution of the intra-frame prediction mode estimated by the convolutional neural network module. The output of the multi-stage arithmetic encoder is the bitstream of the prediction mode for the current coding block. For example, if the convolutional neural network outputs the probability distribution of the intra-frame prediction mode for the current coding block as P, the best prediction mode for the current coding block is 5, and the probability of the best prediction mode for the current coding block being 5 is 0.3, that is, P(5) = 0.3, then encoding this prediction mode requires approximately -log2P(5) ≈ 1.73 bits.

[0285] Figure 6 The network structure of a convolutional neural network is illustrated. The convolutional neural network consists of 2 convolutional layers, 2 pooling layers, and 2 fully connected layers. The reconstructed blocks to the upper left, upper, and left of the current coding block are combined into a three-channel data block as the neural network input. The corresponding texture topology information is extracted through 2 convolution operations, 2 ReLU calculations, and 2 pooling operations. Then, a 919×1 vector is output based on the texture topology information through 1 fully connected calculation. The 3 MPM values ​​are then converted into 3 35×1 One-Hot vectors. The specific conversion method is that the One-Hot vector only takes the value of 1 at the MPM value and the value of 0 at other positions. For example, if the first MPM value of the current coding block is 0, then the first element of the first One-Hot vector takes the value of 1, and the values ​​of other positions are 0. The role of this One-Hot vector is to assist the neural network in analyzing texture topology information. Then, the convolutional layer output and the three one-hot vectors converted by MPM are spliced ​​into a 1024×1 vector. Finally, the texture topology information is integrated and analyzed through the fully connected layer to output a 35×1 vector as the probability distribution of the prediction mode of the current coding block.

[0286] This scheme uses the neighboring reconstructed pixel blocks and MPM information of the current coding block as the input of the convolutional neural network to obtain the probability distribution of the current block prediction mode. It can obtain a more accurate prediction mode probability model, thereby reducing the codewords required for entropy coding.

[0287] In this entropy coding scheme, when there are multiple syntax elements related to the intra prediction module, the syntax elements related to the intra prediction mode are encoded sequentially, which increases the complexity of hardware implementation.

[0288] Based on this, the present application provides a coding and decoding method, which takes the values ​​of multiple (for example, all) syntax elements related to the intra-frame prediction mode as a set of intra-frame prediction mode syntax element values, and the intra-frame prediction mode syntax element value set corresponds to an index. The index of the intra-frame prediction mode syntax element value set is understood as a syntax element, which is equivalent to converting the values ​​of multiple entropy-coded syntax elements into the value of one entropy-coded syntax element. The syntax element can represent the intra-frame prediction mode state of the current image block (for example, it can indicate the values ​​of multiple syntax elements related to the intra-frame prediction mode actually adopted by the current image block). During entropy coding, only the index of the intra-frame prediction mode syntax element value set needs to be entropy coded once to replace the entropy coding of many intra-frame prediction mode-related syntax element values, thereby reducing the complexity of hardware implementation and improving coding performance.

[0289] The solution provided by the present invention can be applied to Figure 7 In the video transmission system architecture shown in Figure 7 In the illustrated system architecture, the encoder uses a video capture device to capture video, pre-process it, and then compresses and encodes it using an encoder to generate a coded stream. The transmitter then sends the coded stream over the transmission network to the decoder. The decoder's receiver receives the coded stream, decodes it, and renders it for display.

[0290] In another possible implementation, the bitstream after video encoding can also be directly stored. When needed, the decoding end obtains the encoded bitstream and renders and displays it after decoding it by the decoder.

[0291] The working process of the encoder in the application scenario of this application can be as described above Figure 2 Corresponding to the encoding process, the working process of the decoder can be as described above Figure 3 The decoding process is illustrated and will not be described in detail here. It should be noted that the present application solution is mainly used for the intra-frame prediction encoding process, which exists at both the encoding and decoding ends and is present in most video codecs, such as those corresponding to standards such as H.263, H.264, MPEG-2, MPEG-4, VP8, and VP9.

[0292] The present application solution can be applied to devices or products that include video encoder and / or decoder functions, such as video processing software and hardware products, chips, and products or devices that include such chips. The present application embodiments are not limited to specific product forms. For example, the present application solution can be applied to media products such as mobile phones.

[0293] Before describing the solution provided by the embodiment of the present application, Figure 8 、 Figure 9A neural network for predicting the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets in entropy encoding / decoding is described in detail. Figure 8 、 Figure 9 These are several exemplary architectures of neural networks used to predict the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets in entropy encoding / entropy decoding in the embodiments of the present application, and do not constitute specific limitations.

[0294] The main structure of the neural network provided in the embodiments of the present application can be based on the densely connected network (DenseNet) widely used in the field of computer vision. For example, for different coding blocks (e.g., of different sizes), the embodiments of the present application provide corresponding neural network input data formats and corresponding neural network structures for predicting or estimating the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets to maximize coding performance.

[0295] Figure 8 The figure shows a neural network structure that can be applied to a 4×4 coding block to predict the probability distribution of multiple candidate intra prediction mode syntax element value sets. Figure 8 As shown, the neural network includes a fully connected network, a convolutional network, a concat layer, a first fully connected layer and a Softmax layer. The fully connected network includes multiple second fully connected layers and activated PReLU layers connected one-to-one to the second fully connected layers. The fully connected network is used to extract feature information of the input data. The convolutional network includes multiple densely connected blocks and pooling layers between densely connected blocks. The densely connected blocks include convolutional layers, excitation ReLU layers and batch normalization BN layers. The convolutional network is used to extract feature information from the input data. The concat layer is used to splice the feature information extracted by the fully connected network and the feature information extracted by the convolutional network. The first fully connected layer and the Softmax layer perform probability estimation on the feature information after splicing the concat layer, and output the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets.

[0296] exist Figure 8 In the illustrated neural network architecture, the first type of information in the input data can be an M-channel data block composed of information from neighboring image blocks and the current image block, where M is the number of first-type information. Information other than the first type of information in the input data can be a one-dimensional vector concatenated by one-hot vectors converted from information from neighboring image blocks and the current image block.

[0297] The first type of information is image block pixel-level information, which may refer to information related to image block pixel values, such as reconstructed values, predicted values, and residual values.

[0298] based on Figure 8 The illustrated neural network can realize the probability estimation processing of the input data through the neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets. The specific implementation can be: Figure 8 The convolutional network in the illustrated neural network extracts the feature information of the first type of information in the input data; Figure 8 The fully connected network in the illustrated neural network extracts feature information of information other than the first type of information in the input data; Figure 8 The concat layer in the illustrated neural network concatenates the feature information extracted by the fully connected network and the feature information extracted by the convolutional network, and performs probability estimation on the concatenated feature information of the concat layer through the first fully connected layer and the Softmax layer, and outputs the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets.

[0299] In a possible implementation, the neighboring image blocks of the current image block may include an upper left adjacent image block, an upper adjacent image block, and a left adjacent image block.

[0300] In a possible implementation, the neighboring information may include a reconstructed value of the image block.

[0301] In another possible implementation, the information of the neighboring image blocks may include one or more of the following information: a reconstruction value, a prediction value, a residual value, an index of a set of intra-frame prediction mode syntax element values, and partition information.

[0302] For example, based on Figure 8 The illustrated neural network performs probability estimation processing on the input data through the neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets. Its specific implementation can be: the one-dimensional vector converted from the index and partition information of the intra-frame prediction mode syntax element value set of the neighboring image blocks of the current image block is input into the fully connected network in the neural network, and the 3-channel data block spliced ​​with the reconstruction value, prediction value and residual value of the neighboring image block and the filling default value of the current image block is input into the convolution network of the neural network. Then, the output of the fully connected network and the output of the convolution network are spliced. The spliced ​​result is probability estimated through the first fully connected layer and the Softmax layer, and the probability distribution of the current image block using multiple candidate intra-frame prediction mode syntax element value sets is output. In this example, Figure 8In the example neural network architecture, the fully connected network consists of three layers, each consisting of a fully connected layer followed by a Pre-ReLU layer. The dimensions of the first fully connected layer are 300, 300, and 112, respectively. The convolutional network architecture is based on the DenseNet. The overall structure of the DenseNet is divided into multiple densely connected blocks. In each densely connected block, the base layer consists of a convolutional layer, a ReLU layer, and a batch normalization (BN) layer. Between densely connected blocks, pooling layers are used to downsample the feature maps. After the last densely connected block, a fully connected layer is added to convert the feature map into a multi-dimensional vector.

[0303] In one possible implementation, in the neural network used by the 4×4 coding block, DenseNet includes two densely connected blocks, each of which includes 12 base layers. The probability distribution of multiple candidate intra-frame prediction mode syntax element value sets is a multidimensional vector whose dimension is the maximum value of the intra-frame prediction mode syntax element value set plus 1. For example, assuming that the index of the intra-frame prediction mode syntax element value set of the 4×4 coding block has a total of 112 values, the dimension of the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets of the 4×4 coding block can be 112.

[0304] Figure 9 The structure of another neural network is shown, which can be applied to coding blocks of other sizes besides 4×4 to predict the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets. Figure 9 As shown, the neural network may include a convolutional network, a fully connected layer, and a softmax layer. The convolutional network comprises multiple densely connected blocks and pooling layers between them. The densely connected blocks include convolutional layers, stimulated ReLU layers, and batch normalization layers. The convolutional network is used to extract feature information from the input data. The fully connected layer and the softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network and output the probability distribution of multiple candidate intra prediction mode syntax element value sets.

[0305] For example, based on Figure 9 The illustrated neural network performs probability estimation processing on the input data through the neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets. Its specific implementation can be: extracting feature information in the input data through the convolutional network; performing probability estimation on the feature information extracted by the convolutional network through the fully connected layer and the Softmax layer, and outputting the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets.

[0306] exist Figure 9In the illustrated neural network architecture, the input data may be an M-channel data block formed by concatenating information of adjacent image blocks and information of the current image block, where M is the type of information in the image block information.

[0307] For example, based on Figure 9 The neural network shown in the figure performs probability estimation processing on the input data through the neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets. Its specific implementation can be: the 5-channel data block (reconstructed value, predicted value, residual value, index and partition information of the intra-frame prediction mode syntax element value set) spliced ​​with the information of the adjacent image block and the information of the current image block is input into the convolutional network, and the convolutional network adopts the same method as the above Figure 8 The convolutional network in the illustrated neural network has the same DenseNet structure. After the last densely connected block, a fully connected layer and a Softmax layer are added to estimate the probability distribution of the current image block using multiple candidate intra-frame prediction mode syntax element value sets.

[0308] Among them, when constructing the input data of the neural network in the embodiment of the present application, the information of the spliced ​​current image block can be used to fill the default value, and the embodiment of the present application does not limit the content of the filling default value.

[0309] For example, for coding blocks of different sizes, the number of densely connected blocks included in the DenseNet used by the convolutional network included in the neural network, and the dimensions of the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets are different, as shown in Table 1.

[0310] Table 1 Neural network configuration parameters used by encoding blocks of different sizes

[0311]

[0312]

[0313] It should be noted that Table 1 only illustrates the neural network configuration parameters used for coding blocks of different sizes by way of example, and is not a specific limitation thereto.

[0314] Furthermore, in the embodiments of the present application, PyTorch software can be used to train neural networks on NVIDIA GTX1080Ti graphics processing units (GPUs).

[0315] For example, in an embodiment of the present application, a stochastic gradient descent (SGD) algorithm may be used to train a neural network. The uncompressed color image dataset (UCID) and the DIV2K dataset are used to prepare training data. The images in the UCID and DIV2K are compressed using VTM6.0 software. The input data and target data of the neural network are extracted. The input data of the neural network is the information of the neighboring image blocks of the current image block, and the target data is the index of the intra-frame prediction mode syntax element value set of the current image block. The neural network is trained to minimize the following loss function:

[0316]

[0317] Among them, Θ represents the parameters of the neural network, T i Represents the One-Hot vector of the i-th training target, Y i represents the Softmax output of the i-th neural network, · represents the inner product multiplication, and m represents the batch size. It can be seen that the greater the probability of the current image block at the index value of the intra prediction mode syntax element value set, the smaller the loss function value will be.

[0318] In one possible implementation, the neural network described in this application may be a trained neural network with a probability estimation function.

[0319] In another possible implementation, the neural network described in this application is used to receive a training data set; and to train the neural network based on the training data set.

[0320] In one possible implementation, the training data set for training the neural network described in this application may be information of multiple neighboring image blocks of an image block.

[0321] It should be noted that the above content illustrates an implementation of a training neural network provided in an embodiment of the present application, but is not a specific limitation. In practical applications, a specific solution for training the neural network can be selected according to actual needs, and the embodiments of the present application will not be repeated.

[0322] It should be noted that if Figure 8 、 Figure 9 Only several exemplary architectures of the neural network for predicting the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets in the embodiments of the present application are shown, which does not constitute a limitation on the neural network architecture. The number of layers, layer structure, addition, multiplication or merging and other processing included in the neural network, as well as the number and size of input and / or output matrices can be determined according to actual conditions, and the present application does not make specific limitations on this.

[0323] Figure 10 The process diagram of the encoding method provided in the embodiment of the present application is shown. The embodiment can be divided into three steps: determining the index of the intra-frame prediction mode syntax element value set of the current image block, obtaining the information of the neighboring image blocks of the current image block and the index of the intra-frame prediction mode syntax element value set, obtaining the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets of the current image block through a neural network, and performing arithmetic coding to obtain a code stream based on the probability distribution and the index of the intra-frame prediction mode syntax element value set of the current image block.

[0324] Figure 11 This is a flowchart of the encoding method provided in the embodiment of the present application. The encoding method can be performed by the video encoder 20, for example, by the intra-frame prediction unit 254 and the entropy encoding unit 270 in the video encoder 20. The encoding method provided in the embodiment of the present application is described as a series of steps or operations. It should be understood that the encoding method can be performed in various orders and / or simultaneously, not limited to Figure 11 Assume that a video data stream having a plurality of image frames is using a video encoder, and execute an encoding method including the following steps to encode an image or an image block.

[0325] like Figure 11 As shown, the encoding method provided in the embodiment of the present application may include:

[0326] S1101. Determine an index of a syntax element value set of an intra prediction mode of a current image block.

[0327] The intra-frame prediction mode syntax element value set of the current image block includes the values ​​of multiple syntax elements involved in the intra-frame prediction mode of the current image block, and the values ​​of the multiple syntax elements include the index of the intra-frame prediction mode adopted by the current image block.

[0328] Specifically, you can Figure 2 The intra prediction unit 254 in the illustrated encoder 20 obtains a set of intra prediction mode syntax element values ​​for the current image block.

[0329] The index of the intra-frame prediction mode syntax element value set of the current image block is used to indicate the intra-frame coding state of the current image block, and corresponds to the intra-frame prediction mode syntax element value set of the current image block.

[0330] In one possible implementation, in S1101, after obtaining the set of syntax element values ​​of the intra-frame prediction mode of the current image block, the encoder determines the index of the set of syntax element values ​​of the intra-frame prediction mode of the current image block according to the index of the intra-frame prediction mode adopted by the current image block.

[0331] Exemplarily, determining the index of the set of syntax element values ​​of the intra-frame prediction mode for the current image block based on the index of the intra-frame prediction mode used by the current image block can be implemented by determining the sum of the index of the intra-frame prediction mode used by the current image block and the additional value of the first type of intra-frame prediction mode as the index of the set of syntax element values ​​of the intra-frame prediction mode for the current image block. The intra-frame prediction mode used by the current image block belongs to the first type of intra-frame prediction mode, and the intervals formed by the minimum index value plus the additional value and the maximum index value plus the additional value of intra-frame prediction modes of different types do not intersect.

[0332] The additional value of the intra-frame prediction mode is used to determine the index of the intra-frame prediction mode syntax element value set. The additional values ​​of different types of intra-frame prediction modes can be configured according to actual needs, as long as the intervals formed by the sum of the minimum index value and the corresponding additional value and the sum of the maximum index value and the corresponding additional value of different types of intra-frame prediction modes do not intersect. The embodiments of this application do not limit the method for selecting and taking values ​​of the additional value for each type of intra-frame prediction mode.

[0333] For example, the intra prediction mode of type A includes five intra prediction modes, the index range of which is 1 to 5, and the additional value corresponding to the intra prediction mode of type A is X. The intra prediction mode of type B includes three intra prediction modes, the index range of which is 1 to 3, and the additional value corresponding to the intra prediction mode of type B is Y. Therefore, the interval [1+X, 5+X] does not intersect with the interval [1+Y, 3+Y]. Here, "[" and "]" are closed interval symbols.

[0334] In one possible implementation, the additional value of the Nth intra-frame prediction mode includes: the sum of the label values ​​of each intra-frame prediction mode before the Nth intra-frame prediction mode in a predetermined order. The Nth intra-frame prediction mode is any intra-frame prediction mode.

[0335] Among them, the label value of the intra-frame prediction mode can be a parameter configured for the intra-frame prediction mode, which can be pre-configured or obtained according to the characteristics of the intra-frame prediction mode, or obtained in other ways. The embodiment of the present application does not limit the content of the label value of the intra-frame prediction mode.

[0336] Optionally, the label values ​​of two different intra-frame prediction modes may be the same or different.

[0337] Exemplarily, the label value of each type of intra-frame prediction mode may include: the total number of multiple intra-frame prediction modes in each type of intra-frame prediction mode. In other words, the label value of a type of intra-frame prediction mode may be the total number of indexes in that type of intra-frame prediction mode, or the label value of a type of intra-frame prediction mode may be the maximum index of that type of intra-frame prediction mode plus 1.

[0338] For example, assuming that the intra-frame prediction mode adopted by the current image block belongs to the first type of intra-frame prediction mode, the values ​​of multiple syntax elements included in the intra-frame prediction mode syntax element value set also include the flag bit of the first type of intra-frame prediction mode is true, the flag bit of the second type of intra-frame prediction mode is false, and the flag bit of the third type of intra-frame prediction mode is false; the total number of multiple intra-frame prediction modes under the second type of intra-frame prediction mode is B, and the total number of multiple intra-frame prediction modes under the third type of intra-frame prediction mode is C; the predetermined order is the second type of intra-frame prediction mode, the third type of intra-frame prediction mode, and the first type of intra-frame prediction mode, determining the sum of the index of the intra-frame prediction mode adopted by the current image block and the additional value of the first type of intra-frame prediction mode, as the index of the intra-frame prediction mode syntax element value set of the current image block, including: determining the index of the intra-frame prediction mode adopted by the current image block, the sum of B and C, as the index of the intra-frame prediction mode syntax element value set of the current image block.

[0339] Specifically, the above-mentioned predetermined order can be configured according to actual needs, and the embodiments of the present application are not limited to this.

[0340] For example, the predetermined order may be the order from the left branch to the right branch in the tree diagram structure corresponding to the encoding process, or the predetermined order may be the order of branches from high to low priority in the tree diagram corresponding to the encoding process.

[0341] Exemplarily, the index of the set of syntax element values ​​of the intra-frame prediction mode of the current image block may be the sum of the value ranges of all leaf nodes in the tree graph structure corresponding to the encoding process of the current image block.

[0342] The following describes the process of S1101 by taking the intra-frame prediction encoding process of a 4×4 image block as an example. First, the encoding process of the syntax elements related to the intra-frame prediction mode of a 4×4 image block in VTM6.0 is described as follows: Figure 12 In the tree structure shown, the value range of the index of the intra prediction mode syntax element value set of the image block can be the sum of the value ranges of all leaf nodes in the tree. Figure 12 As shown in the figure, we can see that the syntax elements related to the 4×4 image block include the MIP flag, MIP mode, multi-line reference index, MPM index, and intra-frame prediction mode. In VVC, the MIP mode of a 4×4 image block has 35 values, the multi-line reference index has 3 values, the MPM index has 5 values, and the prediction mode has 67 values. Therefore, the index of the intra-frame prediction mode syntax element value set of a 4×4 image block has a total of 35+5+5+67=112 values.

[0343] Figure 12The schematic tree diagram includes multiple chain structure examples from top to bottom. Each chain structure example corresponds to a set of syntax element values ​​for a candidate intra-frame prediction mode, and different chain structure examples correspond to different sets of syntax element values ​​for candidate intra-frame prediction modes. It should be understood that each chain structure example represents or reflects a dependency relationship between multiple syntax elements.

[0344] In one example, the number of candidate intra prediction mode syntax element value sets corresponds to the number of chain structure examples. Figure 12 As shown, the maximum number of candidate intra-frame prediction mode syntax element value sets can be 112 (or the index of the intra-frame prediction mode syntax element value set of the current image block has a total of 112 values, or the index value range is (0, 111)).

[0345] Specifically, based on Figure 12 In the tree structure shown, the index of the intra-frame prediction mode syntax element value set of the 4×4 current image block in S1101 can be obtained by the intra-frame prediction mode syntax element value set of the current image block and Figure 12 The judgment conditions described in the tree diagram structure are derived. The following is the pseudo code corresponding to the process of determining the index of the intra prediction mode syntax element value set of the current 4×4 image block:

[0346]

[0347]

[0348] Among them, MIP Flag represents the MIP flag, MIPMode represents the MIP mode, PredMode represents the index of the intra-frame prediction mode syntax element value set of the current image block, multiRefIdx represents the multi-line reference index, IntraDir represents the intra-frame prediction mode, and MPMIdx represents the MPM index. When the MIP flag is true, the index of the intra-frame prediction mode syntax element value set of the current image block is the value of the MIP mode. Otherwise, when the multi-line reference index takes a value of 0, the index of the intra-frame prediction mode syntax element value set of the current image block is 35+intra-frame prediction mode value. For example, when the multi-line reference index takes a value of 1, the index of the intra-frame prediction mode syntax element value set of the current image block is 35+67+MPM index. When the multi-line reference index takes a value of 3, the index of the intra-frame prediction mode syntax element value set of the current image block is 35+67+5+MPM index. When the best prediction mode of a 4×4 coding block is 1, the MIP flag is false, and the multi-row reference index is 0, the index of the intra prediction mode syntax element value set for the current image block is 35. It is worth noting that for coding blocks in the first row of a coding tree unit (CTU), the multi-reference index defaults to 0, and the range of the index of the intra prediction mode syntax element value set for the current image block is the original range minus 10.

[0349] It should be noted that for image blocks of other sizes, the intra-frame coding tree diagram structure description and the derivation process of the index of the intra-frame prediction mode syntax element value set on the encoding side are basically the same as those for 4×4 image blocks. The specific details are slightly different based on the intra-frame prediction coding configuration of VTM6.0 for coding blocks of different sizes, and will not be repeated in the embodiments of this application.

[0350] S1102: Perform probability estimation processing on the input data through a neural network to obtain probability distributions of multiple candidate intra-frame prediction mode syntax element value sets.

[0351] The input data is used to represent the features of the current image block (eg, texture features of the current image block).

[0352] Exemplarily, the input data may include information of neighboring image blocks of the current image block, or the above-mentioned input data may include information of neighboring image blocks of the preprocessed current image block, where the neighboring image blocks include image blocks adjacent to the upper left, adjacent to the upper side, and adjacent to the left side; the information of the neighboring image blocks includes reconstructed values ​​of the neighboring image blocks.

[0353] In a possible implementation, the information of the neighboring image blocks of the preprocessed current image block may include: an M-channel data block formed by concatenating the information of the neighboring image blocks and the information of the current image block; M is greater than or equal to 1.

[0354] For example, M may be less than or equal to the type of information included in the information of the image block.

[0355] In a possible implementation, the pre-processed information of the neighboring image blocks of the current image block may include a one-dimensional vector formed by connecting the information of the neighboring image blocks and the information conversion vector of the current image block.

[0356] The vector of the information conversion of the image block may be a one-dimensional valid One-Hot vector.

[0357] Among them, when constructing the input data of the neural network in the embodiment of the present application, the information of the spliced ​​current image block can be used to fill the default value, and the embodiment of the present application does not limit the content of the filling default value.

[0358] In another possible implementation, the information of the neighboring image blocks may further include one or more of the following information: a prediction value, a residual value, an index of a set of intra-frame prediction mode syntax element values, and partition information.

[0359] The division information is used to indicate the boundary of the image block. For example, the division information may be a data block with a boundary pixel value of 1 and a middle pixel value of 0.

[0360] Furthermore, the above probability distribution represents the probability values ​​of each of the multiple candidate intra-frame prediction mode syntax element value sets. In other words, the probability distribution represents the intra-frame mode syntax element value sets obtained when performing intra-frame prediction on the current image block, which is the probability of each set in the multiple candidate intra-frame prediction mode syntax element value sets. The multiple candidate intra-frame prediction mode syntax element value sets refer to all available (or selectable) intra-frame prediction mode syntax element value sets.

[0361] Specifically, in S1102, the encoder first obtains information about the neighboring image blocks of the current image block. For example, the encoder can obtain information about the neighboring image blocks of the current image block. Figure 2 In the illustrated encoding process, the mode selection unit 260 obtains information of neighboring image blocks of the current image block to perform the operation S1102.

[0362] For example, for a 4×4 coding block in VVC, the reconstruction values, prediction values, and residual values ​​of the adjacent upper left, upper, and left adjacent blocks of the same size as the current image block can be spliced ​​with the default value filled by the current coding image block to obtain a 3-channel data block as the first type of information in the input data. Figure 8The convolutional network in the neural network is shown. The index and partition information of the intra-frame prediction mode syntax element value set of the adjacent upper left, upper and left adjacent blocks of the same size as the current image block are converted into One-Hot vectors respectively. After obtaining 6 One-Hot vectors, all One-Hot vectors are connected into a one-dimensional vector. For the index of the intra-frame prediction mode syntax element value set, the length of the converted One-Hot vector is 230. For the partition information, the length of the converted One-Hot vector is 17. The 17 coding block sizes supported in VVC are numbered so that each size of image block corresponds to a unique number. Finally, the length of the one-dimensional vector after the 6 One-Hot vectors are connected is 741, which is used as information other than the first type of information in the input data and input. Figure 8 The fully connected network in the illustrated neural network obtains the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets.

[0363] For example, in S1102, the encoder may concatenate the reconstructed values, predicted values, residual values, indexes and partitioning information of the set of intra-frame prediction mode syntax elements of the adjacent upper left, upper and left adjacent blocks of the same size as the current image block with the default value of the current image block to obtain a 5-channel data block as input data. Figure 9 In the illustrated neural network, probability distributions of multiple candidate intra prediction mode syntax element value sets are obtained.

[0364] It should be noted that for Figure 8 、 Figure 9 The functions and training methods of the illustrated neural network have been described in detail in the previous content and will not be repeated here.

[0365] It should be understood that the neural network is a trained neural network with probability estimation function.

[0366] S1103 . Perform entropy coding on probability values ​​related to the index of the intra-frame prediction mode syntax element value set of the current image block according to the probability distribution of the multiple candidate intra-frame prediction mode syntax element value sets to obtain a bitstream.

[0367] The entropy coding may include binary arithmetic coding or multi-level arithmetic coding. Of course, the type of entropy coding performed in S1103 can be selected according to actual needs, and the embodiment of the present application does not limit this.

[0368] In one possible implementation, the probability value associated with the index of the intra-frame prediction mode syntax element value set of the current image block can be any probability value in the probability interval corresponding to the arithmetic coding target intra-frame prediction mode syntax element value set (the intra-frame prediction mode syntax element value set actually adopted by the current image block) in multiple probability intervals constructed based on the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets according to the arithmetic coding principle.

[0369] For example, assuming that there are three sets of multiple candidate intra-frame prediction mode syntax element value sets, which are recorded in order as set 1, set 2, and set 3, and the probability distribution of the multiple candidate intra-frame prediction mode syntax element value sets obtained in S1102 is {0.2, 0.5, 0.3}, it means that the probability of the current image block adopting set 1 is 20%, the probability of adopting set 2 is 50%, and the probability of adopting set 3 is 30%. According to the arithmetic coding principle, multiple probability intervals constructed for the probability distribution of the multiple candidate intra-frame prediction mode syntax element value sets are: the interval corresponding to set 1 is [0, 0.2), the interval corresponding to set 2 is [0.2, 0.7), and the interval corresponding to set 3 is [0.7, 1). Where "[" represents a closed interval boundary, and ")" represents an open interval boundary.

[0370] Assuming that the intra-prediction mode syntax element value set actually used by the current image block is set 2, the probability value associated with the index of the intra-prediction mode syntax element value set for the current image block is any probability value in the range [0.2, 0.7). In S1103, any probability value in the range [0.2, 0.7) can be selected for entropy coding to obtain a bitstream.

[0371] Through the encoding method provided by the embodiments of the present application, the values ​​of multiple (for example, all) syntax elements involved in the intra-frame prediction mode of the current image block are used as a set of intra-frame prediction mode syntax element values. The intra-frame prediction mode syntax element value set corresponds to an index, and the index of the intra-frame prediction mode syntax element value set is understood as a syntax element, which is equivalent to converting the values ​​of multiple entropy-coded syntax elements into the value of one entropy-coded syntax element. The syntax element can represent the intra-frame prediction mode state of the current image block (for example, it can indicate the values ​​of multiple syntax elements involved in the intra-frame prediction mode actually adopted by the current image block). During entropy coding, only the index of the intra-frame prediction mode syntax element value set needs to be entropy coded once to replace the entropy coding of many intra-frame prediction mode-related syntax element values, thereby reducing the complexity of hardware implementation and improving coding performance.

[0372] Furthermore, the neural network for obtaining probability distribution provided in the embodiment of the present application can obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets based on the reconstruction value information, prediction value information, residual value information, index and division information of the intra-frame prediction mode syntax element value set of the neighboring image blocks of the current image block, and make full use of the correlation between the neighboring image blocks of the current image block and the current image block. Since the neighboring image blocks have been reconstructed, the neighboring image blocks can represent the characteristics of the current image block. The above-mentioned probability distribution obtained by inputting the information of the neighboring image blocks into the neural network has higher accuracy.

[0373] Further, such as Figure 13 As shown, the encoding method provided in the embodiment of the present application may further include S1104.

[0374] S1104. Determine a neural network suitable for the current image block based on the network structure differentiation information.

[0375] The network structure distinguishing information includes the size of the current image block, or the network structure distinguishing information depends on the size of the current image block.

[0376] It should be noted that the network structure distinguishing information can also be a feature that can configure different neural network structures in the image block, and this application does not limit the type of indication information.

[0377] Correspondingly, in S1102 , the neural network determined in S1104 is used to perform probability estimation processing on the input data to obtain probability distributions of multiple candidate intra prediction mode syntax element value sets.

[0378] In this way, different neural networks can be designed for different image blocks (for example, different sizes) to maximize the encoding performance.

[0379] Figure 14 This is a flowchart of a decoding method provided in an embodiment of the present application. The decoding method may be performed by the video decoder 30, specifically, for example, the entropy decoding unit 304 and the intra-frame prediction unit 354 in the video decoder 30. The decoding method provided in the embodiment of the present application is described as a series of steps or operations. It should be understood that the decoding method may be performed in various orders and / or simultaneously, not limited to Figure 14 Assume that a video data stream having multiple image frames is using a video codec, and a decoding method including the following steps is executed to decode a code stream of an image or an image block.

[0380] like Figure 14 As shown, the decoding method provided in the embodiment of the present application may include:

[0381] S1401: Perform probability estimation processing on input data through a neural network to obtain probability distributions of multiple candidate intra prediction mode syntax element value sets.

[0382] The probability distribution represents the probability values ​​of each of a plurality of candidate intra prediction mode syntax element value sets.

[0383] It should be understood that the neural network is a trained neural network with probability estimation function.

[0384] It should be noted that the operation of the decoder in S1401 is the same as the operation of the encoder in S1102, and will not be repeated here. The multiple candidate intra prediction mode syntax element value sets have been described in detail above and will not be repeated here.

[0385] S1402: Perform entropy decoding on the received bitstream according to the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets to obtain the index of the intra-frame prediction mode syntax element value set of the current image block.

[0386] Specifically, in S1402, the decoder performs entropy decoding on the received bitstream according to the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets to obtain the index of the intra-frame prediction mode syntax element value set of the current image block as follows:

[0387] First, in S1401, the probability distributions of multiple candidate intra-frame prediction mode syntax element value sets are obtained, thereby obtaining the probability intervals corresponding to the different candidate intra-frame prediction mode syntax element value sets. The process of obtaining the probability intervals has been described in detail in S1103 and will not be repeated here. Different candidate intra-frame prediction mode syntax element value sets have their own index values, which are determined as described in S1101 and will not be repeated here.

[0388] Next, the decoder performs entropy decoding to obtain the probability values ​​encoded into the bitstream by the encoder. As described in S1103, the encoder encodes the probability values ​​associated with the index of the set of syntax element values ​​for the intra-frame prediction mode of the image block into the bitstream, and the decoder obtains these probability values ​​through entropy decoding. It should be noted that the decoding scheme used by the decoder for entropy decoding in S1402 corresponds to the entropy encoding scheme used by the encoder.

[0389] Optionally, the entropy decoding may be binary arithmetic decoding, or multi-order arithmetic decoding. Of course, the type of entropy decoding performed in S1302 may be selected according to actual needs, and the embodiment of the present application does not limit this.

[0390] The decoding end then compares the probability value obtained by entropy decoding with the probability intervals corresponding to the obtained sets of different candidate intra-frame prediction mode syntax element values, and determines the probability interval to which the entropy decoded probability value belongs. This allows the set of candidate intra-frame prediction mode syntax element values ​​corresponding to the probability interval to be obtained. The index of the obtained set of candidate intra-frame prediction mode syntax element values ​​is then the index of the set of intra-frame prediction mode syntax element values ​​for the current image block.

[0391] S1403: Determine a set of syntax element values ​​for the intra-frame prediction mode of the current image block according to an index of the set of syntax element values ​​for the intra-frame prediction mode of the current image block.

[0392] The syntax element value set includes values ​​of multiple syntax elements, and the multiple syntax elements include an index of an intra-frame prediction mode adopted by the current image block.

[0393] Specifically, the operation of S1403 is the reverse operation of the operation of S1101.

[0394] In one possible implementation, the decoder determines the intra-frame prediction mode syntax element value set of the current image block based on the index of the intra-frame prediction mode syntax element value set of the current image block. Specifically, it can be implemented as follows: determine the syntax element value set corresponding to the first numerical interval as the intra-frame prediction mode syntax element value set of the current image block; the index of the intra-frame prediction mode syntax element value set of the current image block belongs to the first numerical interval.

[0395] The starting value of the first numerical interval is the sum of the minimum index value of the first type of intra-frame prediction mode and the additional value of the first type of intra-frame prediction mode, and the ending value of the first numerical interval is the sum of the maximum index value of the first type of intra-frame prediction mode and the additional value of the first type of intra-frame prediction mode. The intra-frame prediction mode used by the current image block belongs to the first type of intra-frame prediction mode; the intervals formed by the sum of the minimum index value and the corresponding additional value and the sum of the maximum index value and the corresponding additional value of different types of intra-frame prediction modes do not intersect.

[0396] It should be noted that the added value has been described in detail in S1101 and will not be repeated here.

[0397] Exemplarily, if the predetermined order of the multiple intra-frame prediction modes is the second type of intra-frame prediction mode, the third type of intra-frame prediction mode, and the first type of intra-frame prediction mode, the total number of the multiple intra-frame prediction modes in the first type of intra-frame prediction mode is A, the total number of the multiple intra-frame prediction modes in the second type of intra-frame prediction mode is B, and the total number of the multiple intra-frame prediction modes in the third type of intra-frame prediction mode is C, determining the syntax element value set corresponding to the first numerical interval as the intra-frame prediction mode syntax element value set of the current image block can be specifically implemented as follows:

[0398] If the first numerical interval is [0, B), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the second type of intra-frame prediction mode is true, the index of the intra-frame prediction mode adopted by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block, the flag bit of the first type of intra-frame prediction mode is false, and the flag bit of the third type of intra-frame prediction mode is false.

[0399] If the first numerical interval is [B, B+C), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the third type of intra-frame prediction mode is true, the index of the intra-frame prediction mode adopted by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block minus B, the flag bit of the first type of intra-frame prediction mode is false, and the flag bit of the second type of intra-frame prediction mode is false.

[0400] If the first numerical interval is [B+C, A+B+C), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the first type of intra-frame prediction mode is true, the index of the intra-frame prediction mode adopted by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block minus (B+C), the flag bit of the second type of intra-frame prediction mode is false, and the flag bit of the third type of intra-frame prediction mode is false.

[0401] The following describes the process of S1403 by taking the intra prediction decoding process of a 4×4 image block as an example. In S1403, the intra prediction mode syntax element value set of the current image block can be obtained by the index of the intra prediction mode syntax element value set of the current image block decoded in S1402, and Figure 12 The judgment conditions in the schematic tree diagram are derived. The following is the pseudo code corresponding to the process of determining the value set of the intra prediction mode syntax element of the current 4×4 image block:

[0402]

[0403] from Figure 12As can be seen from the tree structure shown, if the index of the intra-prediction mode syntax element value set is between [0, 35), the MIP flag is true and the MIP mode value is equal to the index of the intra-prediction mode syntax element value set. If the index of the intra-prediction mode syntax element value set is between [35, 102), the MIP flag is false, the multi-row reference index is 0, and the intra-prediction mode value is -35 of the intra-prediction mode syntax element value set. If the index of the intra-prediction mode syntax element value set is between [102, 107), the MIP flag is false, the multi-row reference index is 1, and the MPM index value is -35-67 of the intra-prediction mode syntax element value set. If the index of the intra-prediction mode syntax element value set is between [107, 112), the MIP flag is false, the multi-row reference index is 3, and the MPM index value is -35-67-5 of the intra-prediction mode syntax element value set. For example, when the index of the intra-frame prediction mode syntax element value set of a 4×4 image block is 35, the decoding end can parse out that the intra-frame prediction mode syntax element value set of the current image block includes: the MIP flag is false, the multi-row reference index value is 0, and the intra-frame prediction mode of the current image block is 0.

[0404] It should be noted that for image blocks of other sizes, the intra-frame coding tree diagram description and the derivation process of the index of the intra-frame prediction mode syntax element value set on the encoding side are basically the same as those for 4×4 image blocks. The specific details are slightly different based on the intra-frame prediction coding configuration of VTM6.0 for coding blocks of different sizes, and will not be repeated in the embodiments of this application.

[0405] S1404: Perform intra-frame prediction processing based on the values ​​of multiple syntax elements in the intra-frame prediction mode syntax element value set of the current image block.

[0406] It should be noted that the specific process of performing intra-frame prediction processing in S1404 can refer to the aforementioned Figure 2 or Figure 3 The intra-frame prediction process described above will not be repeated here.

[0407] Through the decoding method provided by the embodiments of the present application, the values ​​of multiple (for example, all) syntax elements involved in the intra-frame prediction mode of the current image block are used as a set of intra-frame prediction mode syntax element values. The intra-frame prediction mode syntax element value set corresponds to an index, and the index of the intra-frame prediction mode syntax element value set is understood as a syntax element, which is equivalent to converting the values ​​of multiple entropy-decoded syntax elements into the value of one entropy-decoded syntax element. The syntax element can represent the intra-frame prediction mode coding state of the current image block (for example, it can indicate the values ​​of multiple syntax elements involved in the intra-frame prediction mode actually adopted by the current image block). During entropy decoding, only one entropy decoding of the index of the intra-frame prediction mode syntax element value set is required to replace the entropy decoding of many intra-frame prediction mode-related syntax element values, thereby reducing the complexity of hardware implementation and improving decoding performance.

[0408] Furthermore, the neural network for obtaining probability distribution provided in the embodiment of the present application can obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets based on the reconstruction value information, prediction value information, residual value information, index and division information of the intra-frame prediction mode syntax element value set of the neighboring image blocks of the current image block, and make full use of the correlation between the neighboring image blocks of the current image block and the current image block. Since the neighboring image blocks have been reconstructed, the neighboring image blocks can represent the characteristics of the current image block. The above-mentioned probability distribution obtained by inputting the information of the neighboring image blocks into the neural network has higher accuracy.

[0409] Further, such as Figure 15 As shown, the decoding method provided in the embodiment of the present application may further include S1405 and S1406.

[0410] S1405. Determine a neural network suitable for the current image block based on the network structure differentiation information.

[0411] The network structure distinguishing information includes the size of the current image block, or the network structure distinguishing information depends on the size of the current image block.

[0412] It should be noted that the network structure distinguishing information can also be a feature that can configure different neural network structures in the image block, and this application does not limit the type of indication information.

[0413] Correspondingly, in S1401, probability estimation processing is performed on the input data through a neural network to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, including: probability estimation processing is performed on the input data through the neural network determined in S405 to obtain the probability distribution of multiple candidate intra-frame prediction mode syntax element value sets.

[0414] In this way, different neural networks can be designed for different image blocks (for example, different sizes) to maximize decoding performance.

[0415] Example 1, the encoding and decoding method provided by the embodiment of the present application is implemented on the VTM6.0 platform, and the BD-Rate performance of encoding 1 frame of all test sequences in the All Intra mode is tested under the VVC standard test conditions. The experimental results are shown in Table 2. The test platform for comparing the VTM6.0 baseline and the proposed solution is the same CPU cluster. The three columns Y, U, and V represent the encoding performance improvement achieved by the encoding and decoding method provided by the embodiment of the present application on the luminance component and chrominance component relative to the VTM6.0 baseline, and the encoding time and decoding time represent the changes in the encoding and decoding time of VTM6.0 integrated with the embodiment of the present application relative to the VTM6.0 baseline. After the neural network-based VVC intra-frame prediction mode encoding and decoding method proposed in this application is integrated into the VTM6.0 platform, a 1.01% encoding performance improvement is achieved compared to the VTM6.0 baseline under the VVC standard test.

[0416] Table 2

[0417]

[0418] Example 2: Based on Example 1, the number of densely connected blocks contained in the DenseNet neural network is set to 2 for all size encoding blocks, and the number of layers contained in each densely connected block is set to 12. The same experiment as Example 1 is performed, and the experimental results are shown in Table 3.

[0419] Table 3

[0420]

[0421] Example 3: Based on Example 1, the number of densely connected blocks contained in the DenseNet neural network is set to 2 for all size encoding blocks, and the number of layers contained in each densely connected block is set to 6. The same experiment as Example 1 is performed, and the experimental results are shown in Table 4.

[0422] Table 4

[0423]

[0424] The above mainly introduces the solution provided by the embodiment of the present invention from the perspective of the working principle of the encoder and decoder. It can be understood that in order to realize the above functions, the encoder, decoder, etc. include hardware structures and / or software modules corresponding to the execution of each function. It should be easy for those skilled in the art to realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0425] In the embodiments of the present invention, encoders, decoders, and the like can be divided into functional modules according to the above-described method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into a single processing module. The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the module division in the embodiments of the present invention is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used.

[0426] Figure 16 The embodiment of the present application provides an encoder 160, which is used to implement the functions of the encoder in the above embodiment. Figure 16 As shown, the encoder 160 may include: an intra-frame prediction module 1601 and an entropy coding module 1602. The intra-frame prediction module 1601 is used to perform Figure 11 Process S1101 in; entropy coding module 1602 is used to perform Figure 11 Process S1102 and S1103 in the entropy coding module 1602 is also used to perform Figure 13 All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here. In one example, the intra-frame prediction module 1601 can correspond to Figure 2 In an example, the entropy coding module 1602 may correspond to the intra-frame prediction unit 254; Figure 2 The entropy coding unit 270 in . Figure 16 For further details on the functionality of the intra prediction module 1601 in the encoder, see Figure 2 The description in , will not be repeated here. Figure 16 For further details on the functionality of the entropy coding module 1602 in the encoder, see Figure 2 The description in , will not be repeated here.

[0427] As mentioned above, the encoder 160 provided in the embodiment of the present application can be used to implement the functions of the encoder in the method implemented by the above-mentioned embodiments of the present application. For the convenience of explanation, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the embodiments of the present application.

[0428] Figure 17 The decoder 170 provided in the embodiment of the present application is illustrated, which is used to implement the functions of the processing device in the above embodiment. Figure 17 As shown, the decoder 170 may include: an entropy decoding module 1701 and an intra-frame prediction module 1702. The entropy decoding module 1701 is used to perform Figure 14 Process S1401, S1402 in; Entropy decoding module 1701 is used to perform Figure 15 In the process S1405, S1406; the intra-frame prediction module 1702 is used to perform Figure 14 In the above method embodiment, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here. In one example, the intra-frame prediction module 1702 can correspond to Figure 3 In an example, the entropy decoding module 1701 may correspond to the intra-frame prediction unit 354; Figure 3 The entropy decoding unit 304 in . Figure 17 For further details on the functionality of the entropy decoding module 1701 in the decoder, see Figure 3 The description in , will not be repeated here. Figure 17 For further details on the functionality of the intra prediction module 1702 in the decoder, see Figure 3 The description in , will not be repeated here.

[0429] As mentioned above, the decoder 170 provided in the embodiment of the present application can be used to implement the functions of the decoder in the above-mentioned embodiment of the present application. For the convenience of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the embodiments of the present application.

[0430] The embodiment of the present application further provides a coding and decoding system, including an encoder 160 and a decoder 170 .

[0431] As another form of this embodiment, a non-transitory computer-readable storage medium is provided, including program code, which performs the method in the above method embodiment when executed.

[0432] As another form of this embodiment, a computer program product containing instructions is provided. When the computer program product is run on a computer, the computer is caused to perform the method in the above method embodiment.

[0433] The embodiment of the present application further provides a chip system, which includes a processor for implementing the technical method of the embodiment of the present application. In one possible design, the chip system also includes a memory for storing the necessary program instructions and / or data of the embodiment of the present invention. In one possible design, the chip system also includes a memory for the processor to call the application code stored in the memory. The chip system can be composed of one or more chips, or can include chips and other discrete devices, which is not specifically limited in the embodiment of the present application.

[0434] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0435] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0436] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0437] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0438] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0439] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A coding method, characterized in that: The method comprises: Determining an index of a set of syntax element values ​​for an intra-frame prediction mode of a current image block, the syntax element value set including a plurality of syntax element values, the plurality of syntax element values ​​being used to indicate values ​​of a plurality of syntax elements involved in an intra-frame prediction mode actually adopted by the current image block; the plurality of syntax element values ​​including the index of the intra-frame prediction mode adopted by the current image block; Probability estimation processing is performed on input data through a neural network to obtain probability distributions of multiple candidate intra-prediction mode syntax element value sets, where the probability distributions represent probability values ​​of each of the multiple candidate intra-prediction mode syntax element value sets, and the input data is used to characterize features of a current image block; the values ​​of multiple syntax elements included in each candidate intra-prediction mode syntax element value set in the multiple candidate intra-prediction mode syntax element value sets refer to values ​​of multiple syntax elements involved in an intra-prediction mode that can be used when performing intra-prediction on the current image block; the values ​​of the multiple syntax elements included in one candidate intra-prediction mode syntax element value set are mutually dependent; and the intra-prediction mode syntax element value set for the current image block is one of the multiple candidate intra-prediction mode syntax element value sets; According to the probability distribution of the multiple candidate intra-frame prediction mode syntax element value sets, entropy coding is performed on the probability value related to the index of the intra-frame prediction mode syntax element value set of the current image block to obtain a code stream.

2. The method according to claim 1, characterized in that The determining of the index of the intra prediction mode syntax element value set of the current image block includes: According to the index of the intra-frame prediction mode adopted by the current image block, the index of the intra-frame prediction mode syntax element value set of the current image block is determined.

3. The method according to claim 2, characterized in that The determining, according to the index of the intra-frame prediction mode adopted by the current image block, the index of the intra-frame prediction mode syntax element value set of the current image block includes: Determine the sum of the index of the intra-frame prediction mode used by the current image block and the additional value of the first type of intra-frame prediction mode as the index of the intra-frame prediction mode syntax element value set of the current image block; The intra-frame prediction mode adopted by the current image block belongs to the first type of intra-frame prediction mode; the intervals formed by the sum of the minimum index value and the corresponding additional value and the sum of the maximum index value and the corresponding additional value of different types of intra-frame prediction modes do not intersect.

4. The method according to claim 3, characterized in that The additional values ​​of the Nth type of intra prediction mode include: The sum of the label values ​​of each type of intra-frame prediction mode before the Nth type of intra-frame prediction mode in a predetermined order.

5. The method according to claim 4, characterized in that The label value of each type of intra-frame prediction mode includes: the total number of multiple intra-frame prediction modes in each type of intra-frame prediction mode.

6. The method according to claim 5, characterized in that The intra-frame prediction mode adopted by the current image block belongs to a first type of intra-frame prediction mode, and the values ​​of the multiple syntax elements further include: a flag bit of the first type of intra-frame prediction mode is true, a flag bit of the second type of intra-frame prediction mode is false, and a flag bit of the third type of intra-frame prediction mode is false; a total number of the multiple intra-frame prediction modes under the second type of intra-frame prediction mode is B, and a total number of the multiple intra-frame prediction modes under the third type of intra-frame prediction mode is C; and the predetermined order is the second type of intra-frame prediction mode, the third type of intra-frame prediction mode, and the first type of intra-frame prediction mode; The sum of the index of the intra-frame prediction mode adopted by the current image block and the additional value of the first type of intra-frame prediction mode is determined as the index of the intra-frame prediction mode syntax element value set of the current image block, including: Determine the index of the intra-frame prediction mode adopted by the current image block, the sum of B and C, and use it as the index of the intra-frame prediction mode syntax element value set of the current image block.

7. The method according to any one of claims 1 to 6, characterized in that Before performing probability estimation processing on the input data by the neural network to obtain probability distributions of multiple candidate intra prediction mode syntax element value sets, the method further includes: determining a neural network suitable for the current image block according to network structure distinguishing information, wherein the network structure distinguishing information includes a size of the current image block or the network structure distinguishing information depends on the size of the current image block; Accordingly, the probability estimation process of the input data is performed by the neural network to obtain the probability distribution of multiple candidate intra prediction mode syntax element value sets, including: Probability estimation processing is performed on the input data through a determined neural network to obtain probability distributions of multiple candidate intra-frame prediction mode syntax element value sets.

8. The method according to any one of claims 1 to 7, characterized in that The input data includes information of neighboring image blocks of the current image block, or the input data includes preprocessed information of neighboring image blocks of the current image block, where the neighboring image blocks include image blocks adjacent to the upper left, adjacent to the upper side, and adjacent to the left side; and the information of the neighboring image blocks includes reconstructed values ​​of the neighboring image blocks.

9. The method according to claim 8, characterized in that The preprocessed information of the neighboring image blocks of the current image block includes: A data block of M channels formed by concatenating information of the adjacent image blocks and information of the current image block; wherein M is greater than or equal to 1; or, A one-dimensional vector formed by connecting vectors converted from the information of the adjacent image blocks and the information of the current image block.

10. The method according to claim 8 or 9, characterized in that The information of the neighboring image blocks further includes one or more of the following information: a prediction value, a residual value, an index of a set of syntax element values ​​of an intra-frame prediction mode, and partition information.

11. The method according to any one of claims 8 to 10, characterized in that: The neural network includes a convolutional network, a fully connected layer, and a normalized exponential function Softmax layer; the convolutional network includes multiple densely connected blocks and pooling layers between densely connected blocks, and the densely connected blocks include a convolutional layer, an excitation ReLU layer, and a batch normalization BN layer, wherein: The performing probability estimation processing on the input data by the neural network to obtain probability distribution of multiple candidate intra prediction mode syntax element value sets includes: The convolutional network is used to extract feature information from the input data; the fully connected layer and the Softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network, and a probability distribution of multiple candidate intra-frame prediction mode syntax element value sets is output.

12. The method according to any one of claims 8 to 10, characterized in that The neural network includes a fully connected network, a convolutional network, a concat layer, a first fully connected layer and a Softmax layer; the fully connected network includes multiple second fully connected layers and activated PReLU layers connected one-to-one to the second fully connected layers, the convolutional network includes multiple densely connected blocks and pooling layers between densely connected blocks, and the densely connected blocks include convolution layers, activated ReLU layers and batch normalization BN layers, wherein: The performing probability estimation processing on the input data by the neural network to obtain probability distribution of multiple candidate intra prediction mode syntax element value sets includes: Extracting feature information of a first category of information from the input data through the convolutional network; extracting feature information of information other than the first category of information from the input data through the fully connected network; concatenating the feature information extracted by the fully connected network and the feature information extracted by the convolutional network through the concat layer, and performing probability estimation on the concatenated feature information through the first fully connected layer and the Softmax layer, and outputting probability distributions of multiple candidate intra-frame prediction mode syntax element value sets; The first type of information is image block pixel-level information.

13. A decoding method, characterized in that: The method comprises: Probability estimation processing is performed on input data through a neural network to obtain probability distributions of multiple candidate intra-frame prediction mode syntax element value sets, where the probability distributions represent probability values ​​of each of the multiple candidate intra-frame prediction mode syntax element value sets, and the input data is used to characterize features of a current image block; the values ​​of multiple syntax elements included in each candidate intra-frame prediction mode syntax element value set in the multiple candidate intra-frame prediction mode syntax element value sets refer to values ​​of multiple syntax elements involved in an intra-frame prediction mode that can be used when performing intra-frame prediction on the current image block; and the values ​​of the multiple syntax elements included in one candidate intra-frame prediction mode syntax element value set are dependent on each other; performing entropy decoding on the received bitstream according to the probability distribution to obtain an index of a set of syntax element values ​​of an intra-frame prediction mode of the current image block; the set of syntax element values ​​of the intra-frame prediction mode of the current image block is one of the multiple candidate intra-frame prediction mode syntax element value sets; determining, according to an index of a syntax element value set of the intra prediction mode of the current image block, a syntax element value set for the intra prediction mode of the current image block, where the syntax element value set includes values ​​of a plurality of syntax elements, and the plurality of syntax elements include an index of the intra prediction mode used by the current image block; Based on the values ​​of the plurality of syntax elements, an intra prediction process is performed.

14. The method according to claim 13, characterized in that Determining, according to the index of the intra-prediction mode syntax element value set of the current image block, the intra-prediction mode syntax element value set of the current image block, comprising: Determine a set of intra-frame prediction mode syntax element values ​​corresponding to a first numerical interval, which is the set of intra-frame prediction mode syntax element values ​​of the current image block; the index of the set of intra-frame prediction mode syntax element values ​​of the current image block belongs to the first numerical interval; Among them, the starting value of the first numerical interval is the sum of the minimum index value of the first type of intra-frame prediction mode and the additional value of the first type of intra-frame prediction mode, and the ending value of the first numerical interval is the sum of the maximum index value of the first type of intra-frame prediction mode and the additional value of the first type of intra-frame prediction mode; the intra-frame prediction mode adopted by the current image block belongs to the first type of intra-frame prediction mode; the intervals formed by the sum of the minimum index value and the corresponding additional value and the sum of the maximum index value and the corresponding additional value of different types of intra-frame prediction modes do not intersect.

15. The method according to claim 14, characterized in that The additional values ​​of the Nth type of intra prediction mode include: The sum of the label values ​​of each type of intra-frame prediction mode before the Nth type of intra-frame prediction mode in a predetermined order.

16. The method according to claim 15, characterized in that The label value of each type of intra-frame prediction mode includes: the total number of multiple intra-frame prediction modes in each type of intra-frame prediction mode.

17. The method according to claim 16, characterized in that If the predetermined order is the second type of intra-frame prediction mode, the third type of intra-frame prediction mode, and the first type of intra-frame prediction mode; the total number of the multiple intra-frame prediction modes in the first type of intra-frame prediction mode is A, the total number of the multiple intra-frame prediction modes in the second type of intra-frame prediction mode is B, and the total number of the multiple intra-frame prediction modes in the third type of intra-frame prediction mode is C; Determining a set of intra-frame prediction mode syntax element values ​​corresponding to the first numerical interval as the set of intra-frame prediction mode syntax element values ​​of the current image block includes: If the first numerical interval is [0, B), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the second type of intra-frame prediction mode is true, the index of the intra-frame prediction mode used by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block, the flag bit of the first type of intra-frame prediction mode is false, and the flag bit of the third type of intra-frame prediction mode is false; If the first numerical interval is [B, B+C), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the third type of intra-frame prediction mode is true, the index of the intra-frame prediction mode used by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block minus B, the flag bit of the first type of intra-frame prediction mode is false, and the flag bit of the second type of intra-frame prediction mode is false; If the first numerical interval is [B+C, A+B+C), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the first type of intra-frame prediction mode is true, the index of the intra-frame prediction mode adopted by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block minus (B+C), the flag bit of the second type of intra-frame prediction mode is false, and the flag bit of the third type of intra-frame prediction mode is false.

18. The method according to any one of claims 13 to 17, characterized in that: Before performing probability estimation processing on the input data by the neural network to obtain probability distributions of multiple candidate intra prediction mode syntax element value sets, the method further includes: determining a neural network suitable for the current image block according to network structure distinguishing information, wherein the network structure distinguishing information includes a size of the current image block or the network structure distinguishing information depends on the size of the current image block; Accordingly, the probability estimation process of the input data is performed by the neural network to obtain the probability distribution of multiple candidate intra prediction mode syntax element value sets, including: Probability estimation processing is performed on the input data through a determined neural network to obtain probability distributions of multiple candidate intra-frame prediction mode syntax element value sets.

19. The method according to any one of claims 13 to 18, characterized in that: The input data includes information of neighboring image blocks of the current image block, or the input data includes preprocessed information of neighboring image blocks of the current image block, where the neighboring image blocks include image blocks adjacent to the upper left, adjacent to the upper side, and adjacent to the left side; and the information of the neighboring image blocks includes reconstructed values ​​of the neighboring image blocks.

20. The method according to claim 19, characterized in that The preprocessed information of the neighboring image blocks of the current image block includes: A data block of M channels formed by concatenating information of the adjacent image blocks and information of the current image block; wherein M is greater than or equal to 1; or, A one-dimensional vector formed by connecting vectors converted from the information of the adjacent image blocks and the information of the current image block.

21. The method according to claim 19 or 20, characterized in that The information of the neighboring image blocks further includes one or more of the following information: a prediction value, a residual value, an index of a set of syntax element values ​​of an intra-frame prediction mode, and partition information.

22. The method according to any one of claims 19 to 21, characterized in that The neural network includes a convolutional network, a fully connected layer, and a normalized exponential function Softmax layer; the convolutional network includes multiple densely connected blocks and pooling layers between densely connected blocks, and the densely connected blocks include a convolutional layer, an excitation ReLU layer, and a batch normalization BN layer, wherein: The performing probability estimation processing on the input data by the neural network to obtain probability distribution of multiple candidate intra prediction mode syntax element value sets includes: The convolutional network is used to extract feature information from the input data; the fully connected layer and the Softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network, and a probability distribution of multiple candidate intra-frame prediction mode syntax element value sets is output.

23. The method according to any one of claims 19 to 21, characterized in that The neural network includes a fully connected network, a convolutional network, a concat layer, a first fully connected layer and a Softmax layer; the fully connected network includes multiple second fully connected layers and multiple activated PReLU layers connected one-to-one with the second fully connected layers; the convolutional network includes multiple densely connected blocks and pooling layers between densely connected blocks, and the densely connected blocks include convolution layers, activated ReLU layers and batch normalization BN layers, wherein: The performing probability estimation processing on the input data by the neural network to obtain probability distribution of multiple candidate intra prediction mode syntax element value sets includes: Extracting feature information of a first category of information from the input data through the convolutional network; extracting feature information of information other than the first category of information from the input data through the fully connected network; concatenating the feature information extracted by the fully connected network and the feature information extracted by the convolutional network through the concat layer, and performing probability estimation on the concatenated feature information through the first fully connected layer and the Softmax layer, and outputting probability distributions of multiple candidate intra-frame prediction mode syntax element value sets; The first type of information is image block pixel-level information.

24. An encoder, characterized in that The encoder comprises: an intra-frame prediction module, configured to determine an index of a set of syntax element values ​​for an intra-frame prediction mode of a current image block, the syntax element value set comprising a plurality of syntax element values, the plurality of syntax element values ​​being used to indicate values ​​of a plurality of syntax elements involved in an intra-frame prediction mode actually adopted by the current image block; the plurality of syntax element values ​​comprising the index of the intra-frame prediction mode adopted by the current image block; an entropy coding module, configured to perform probability estimation processing on input data through a neural network to obtain probability distributions of multiple candidate intra-frame prediction mode syntax element value sets, wherein the probability distributions represent probability values ​​of each of the multiple candidate intra-frame prediction mode syntax element value sets, wherein the input data is used to characterize features of a current image block; the values ​​of the multiple syntax elements included in each candidate intra-frame prediction mode syntax element value set refer to values ​​of multiple syntax elements involved in an intra-frame prediction mode that can be used when performing intra-frame prediction on the current image block; the values ​​of the multiple syntax elements included in one candidate intra-frame prediction mode syntax element value set are mutually dependent; and the intra-frame prediction mode syntax element value set for the current image block is one of the multiple candidate intra-frame prediction mode syntax element value sets; The entropy coding module is further configured to perform entropy coding on probability values ​​associated with the index of the intra-frame prediction mode syntax element value set of the current image block according to the probability distribution of the multiple candidate intra-frame prediction mode syntax element value sets to obtain a code stream.

25. The encoder according to claim 24, characterized in that The intra-frame prediction module is specifically used for: According to the index of the intra-frame prediction mode adopted by the current image block, the index of the intra-frame prediction mode syntax element value set of the current image block is determined.

26. The encoder according to claim 25, characterized in that The intra-frame prediction module is specifically used for: Determine the sum of the index of the intra-frame prediction mode used by the current image block and the additional value of the first type of intra-frame prediction mode as the index of the intra-frame prediction mode syntax element value set of the current image block; The intra-frame prediction mode adopted by the current image block belongs to the first type of intra-frame prediction mode; the intervals formed by the sum of the minimum index value and the corresponding additional value and the sum of the maximum index value and the corresponding additional value of different types of intra-frame prediction modes do not intersect.

27. The encoder according to claim 26, characterized in that The additional values ​​of the Nth type of intra prediction mode include: The sum of the label values ​​of each type of intra-frame prediction mode before the Nth type of intra-frame prediction mode in a predetermined order.

28. The encoder according to claim 27, wherein The label value of each type of intra-frame prediction mode includes: the total number of multiple intra-frame prediction modes in each type of intra-frame prediction mode.

29. The encoder according to claim 28, wherein The intra-frame prediction mode adopted by the current image block belongs to a first type of intra-frame prediction mode, and the values ​​of the multiple syntax elements further include: a flag bit of the first type of intra-frame prediction mode is true, a flag bit of the second type of intra-frame prediction mode is false, and a flag bit of the third type of intra-frame prediction mode is false; the total number of the second type of intra-frame prediction modes is B, and the total number of the third type of intra-frame prediction modes is C; and the predetermined order is the second type of intra-frame prediction mode, the third type of intra-frame prediction mode, and the first type of intra-frame prediction mode; The sum of the index of the intra-frame prediction mode adopted by the current image block and the additional value of the first type of intra-frame prediction mode is determined as the index of the intra-frame prediction mode syntax element value set of the current image block, including: Determine the index of the intra-frame prediction mode adopted by the current image block, the sum of B and C, and use it as the index of the intra-frame prediction mode syntax element value set of the current image block.

30. The encoder according to any one of claims 24 to 29, characterized in that The entropy coding module is further configured to: Determine a neural network suitable for the current image block based on network structure distinguishing information, where the network structure distinguishing information includes the size of the current image block, or the network structure distinguishing information depends on the size of the current image block; perform probability estimation processing on the input data through the determined neural network to obtain probability distributions of multiple candidate intra-frame prediction mode syntax element value sets.

31. The encoder according to any one of claims 24 to 30, characterized in that The input data includes information of neighboring image blocks of the current image block, or the input data includes preprocessed information of neighboring image blocks of the current image block, where the neighboring image blocks include image blocks adjacent to the upper left, adjacent to the upper side, and adjacent to the left side; and the information of the neighboring image blocks includes reconstructed values ​​of the neighboring image blocks.

32. The encoder according to claim 31, wherein The preprocessed information of the neighboring image blocks of the current image block includes: A data block of M channels formed by concatenating information of the adjacent image blocks and information of the current image block; wherein M is greater than or equal to 1; or, A one-dimensional vector formed by connecting vectors converted from the information of the adjacent image blocks and the information of the current image block.

33. The encoder according to claim 31 or 32, characterized in that The information of the neighboring image blocks further includes one or more of the following information: a prediction value, a residual value, an index of a set of syntax element values ​​of an intra-frame prediction mode, and partition information.

34. The encoder according to any one of claims 31 to 33, characterized in that The neural network includes a convolutional network, a fully connected layer and a normalized exponential function Softmax layer; the convolutional network includes multiple dense connection blocks and pooling layers between dense connection blocks, and the dense connection blocks include a convolutional layer, an excitation ReLU layer and a batch normalization BN layer; The entropy coding module is specifically used for: The convolutional network is used to extract feature information from the input data; the fully connected layer and the Softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network, and a probability distribution of multiple candidate intra-frame prediction mode syntax element value sets is output.

35. The encoder according to any one of claims 31 to 33, characterized in that The neural network includes a fully connected network, a convolutional network, a concat layer, a first fully connected layer and a Softmax layer; the fully connected network includes multiple second fully connected layers and activated PReLU layers connected one-to-one to the second fully connected layers, the convolutional network includes multiple densely connected blocks and pooling layers between densely connected blocks, and the densely connected blocks include convolution layers, excitation ReLU layers and batch normalization BN layers; The entropy coding module is specifically used for: Extracting feature information of a first category of information in the input data through the convolutional network; extracting feature information of information other than the first category of information in the input data through the fully connected network; splicing the feature information extracted by the fully connected network and the feature information extracted by the convolutional network through the concat layer, and performing probability estimation on the feature information spliced ​​by the concat layer through the first fully connected layer and the Softmax layer, and outputting probability distributions of multiple candidate intra prediction mode syntax element value sets; The first type of information is image block pixel-level information.

36. A decoder, characterized in that The decoder comprises: an entropy decoding module configured to perform probability estimation processing on input data through a neural network to obtain a probability distribution of multiple candidate intra-frame prediction mode syntax element value sets, wherein the probability distribution represents the probability value of each of the multiple candidate intra-frame prediction mode syntax element value sets; the input data is used to characterize the characteristics of the current image block; based on the probability distribution, the received bitstream is entropy decoded to obtain an index of the intra-frame prediction mode syntax element value set for the current image block; the multiple syntax element values ​​included in each candidate intra-frame prediction mode syntax element value set in the multiple candidate intra-frame prediction mode syntax element value sets refer to the values ​​of multiple syntax elements involved in the intra-frame prediction mode that can be used when performing intra-frame prediction on the current image block; the multiple syntax element values ​​included in one candidate intra-frame prediction mode syntax element value set are dependent on each other; and the intra-frame prediction mode syntax element value set for the current image block is one of the multiple candidate intra-frame prediction mode syntax element value sets; an intra-frame prediction module, configured to determine a set of syntax element values ​​for the intra-frame prediction mode of the current image block according to an index of the set of syntax element values ​​for the intra-frame prediction mode of the current image block, wherein the syntax element value set includes values ​​of multiple syntax elements, and the multiple syntax elements include an index of the intra-frame prediction mode adopted by the current image block; and perform intra-frame prediction processing based on the values ​​of the multiple syntax elements.

37. The decoder according to claim 36, characterized in that The intra-frame prediction module is specifically used for: Determine a set of intra-frame prediction mode syntax element values ​​corresponding to a first numerical interval, which is the set of intra-frame prediction mode syntax element values ​​of the current image block; the index of the set of intra-frame prediction mode syntax element values ​​of the current image block belongs to the first numerical interval; Among them, the starting value of the first numerical interval is the sum of the minimum index value of the first type of intra-frame prediction mode and the additional value of the first type of intra-frame prediction mode, and the ending value of the first numerical interval is the sum of the maximum index value of the first type of intra-frame prediction mode and the additional value of the first type of intra-frame prediction mode; the intra-frame prediction mode adopted by the current image block belongs to the first type of intra-frame prediction mode; the intervals formed by the sum of the minimum index value and the corresponding additional value and the sum of the maximum index value and the corresponding additional value of different types of intra-frame prediction modes do not intersect.

38. The decoder according to claim 37, characterized in that The additional values ​​of the Nth type of intra prediction mode include: The sum of the label values ​​of each type of intra-frame prediction mode before the Nth type of intra-frame prediction mode in a predetermined order.

39. The decoder according to claim 38, characterized in that The label value of each type of intra-frame prediction mode includes: the total number of multiple intra-frame prediction modes in each type of intra-frame prediction mode.

40. The decoder according to claim 39, wherein If the predetermined order is the second type of intra-frame prediction mode, the third type of intra-frame prediction mode, and the first type of intra-frame prediction mode; the total number of the multiple intra-frame prediction modes in the first type of intra-frame prediction mode is A, the total number of the multiple intra-frame prediction modes in the second type of intra-frame prediction mode is B, and the total number of the multiple intra-frame prediction modes in the third type of intra-frame prediction mode is C; The intra-frame prediction module is specifically used for: If the first numerical interval is [0, B), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the second type of intra-frame prediction mode is true, the index of the intra-frame prediction mode used by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block, the flag bit of the first type of intra-frame prediction mode is false, and the flag bit of the third type of intra-frame prediction mode is false; If the first numerical interval is [B, B+C), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the third type of intra-frame prediction mode is true, the index of the intra-frame prediction mode used by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block minus B, the flag bit of the first type of intra-frame prediction mode is false, and the flag bit of the second type of intra-frame prediction mode is false; If the first numerical interval is [B+C, A+B+C), the intra-frame prediction mode syntax element value set of the current image block includes: the flag bit of the first type of intra-frame prediction mode is true, the index of the intra-frame prediction mode adopted by the current image block is the index of the intra-frame prediction mode syntax element value set of the current image block minus (B+C), the flag bit of the second type of intra-frame prediction mode is false, and the flag bit of the third type of intra-frame prediction mode is false.

41. The decoder according to any one of claims 36 to 40, characterized in that The entropy decoding module is further used for: Determine a neural network suitable for the current image block based on network structure distinguishing information, where the network structure distinguishing information includes the size of the current image block, or the network structure distinguishing information depends on the size of the current image block; perform probability estimation processing on the input data through the determined neural network to obtain probability distributions of multiple candidate intra-frame prediction mode syntax element value sets.

42. The decoder according to any one of claims 36 to 41, characterized in that The input data includes information of neighboring image blocks of the current image block, or the input data includes preprocessed information of neighboring image blocks of the current image block, where the neighboring image blocks include image blocks adjacent to the upper left, adjacent to the upper side, and adjacent to the left side; and the information of the neighboring image blocks includes reconstructed values ​​of the neighboring image blocks.

43. The decoder according to claim 42, characterized in that The preprocessed information of the neighboring image blocks of the current image block includes: A data block of M channels formed by concatenating information of the adjacent image blocks and information of the current image block; wherein M is greater than or equal to 1; or, A one-dimensional vector formed by connecting vectors converted from the information of the adjacent image blocks and the information of the current image block.

44. The decoder according to claim 42 or 43, characterized in that The information of the neighboring image blocks further includes one or more of the following information: a prediction value, a residual value, an index of a set of syntax element values ​​of an intra-frame prediction mode, and partition information.

45. The decoder according to any one of claims 42 to 44, characterized in that The neural network includes a convolutional network, a fully connected layer and a normalized exponential function Softmax layer; the convolutional network includes multiple dense connection blocks and pooling layers between dense connection blocks, and the dense connection blocks include a convolutional layer, an excitation ReLU layer and a batch normalization BN layer; The entropy decoding module is specifically used for: The convolutional network is used to extract feature information from the input data; the fully connected layer and the Softmax layer are used to perform probability estimation on the feature information extracted by the convolutional network, and a probability distribution of multiple candidate intra-frame prediction mode syntax element value sets is output.

46. ​​The decoder according to any one of claims 42 to 44, characterized in that The neural network includes a fully connected network, a convolutional network, a concat layer, a first fully connected layer and a Softmax layer; the fully connected network includes multiple second fully connected layers and multiple activated PReLU layers connected one-to-one with the second fully connected layers; the convolutional network includes multiple densely connected blocks and pooling layers between densely connected blocks, and the densely connected blocks include convolution layers, excitation ReLU layers and batch normalization BN layers; The entropy decoding module is specifically used for: Extracting feature information of a first category of information in the input data through the convolutional network; extracting feature information of information other than the first category of information in the input data through the fully connected network; splicing the feature information extracted by the fully connected network and the feature information extracted by the convolutional network through the concat layer, and performing probability estimation on the feature information spliced ​​by the concat layer through the first fully connected layer and the Softmax layer, and outputting probability distributions of multiple candidate intra prediction mode syntax element value sets; The first type of information is image block pixel-level information.

47. An encoder, characterized in that comprising a processing circuit for executing the method of any one of claims 1 to 12.

48. A decoder, characterized in that comprising processing circuitry for performing the method of any one of claims 13 to 23.

49. A computer program product, characterized in that The method comprises a program code for executing the method according to any one of claims 1 to 23 when the program code is executed on a computer or a processor.

50. An encoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the encoder to perform the method of any one of claims 1 to 12.

51. A decoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program to be executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method of any one of claims 13 to 23.

52. A non-transitory computer-readable storage medium, characterized in that The method comprises program code for executing the method according to any one of claims 1 to 23 when the program code is executed by a computer device.

Images