Inter-frame prediction method and device

Abstract

The present application provides an inter-frame prediction method and device. It relates to the field of video or image compression technology based on artificial intelligence (AI), and specifically to the field of video compression technology based on neural networks. The method includes: obtaining the motion vectors of each of the P reconstructed image blocks in the surrounding area of ​​the current block; obtaining Q prior candidate motion vectors and Q probability values ​​of the current block according to the motion vectors of each of the P reconstructed image blocks; obtaining M weight factors corresponding to the M prior candidate motion vectors according to the M probability values ​​corresponding to the M prior candidate motion vectors; M, P and Q are positive integers; performing motion compensation according to the M prior candidate motion vectors to obtain M prediction values; and obtaining the prediction value of the current block by weighted summation of the M prediction values ​​and the corresponding M weight factors. The present application can improve the accuracy of inter-frame prediction, reduce the error of inter-frame prediction, and improve the RDO efficiency of inter-frame prediction.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of video or image compression based on artificial intelligence (AI), and in particular to an inter-frame prediction method and device. Background Art

[0002] Video coding (video encoding and decoding) is widely used in digital video applications such as broadcast digital television, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, Digital Versatile Disc (DVD) and Blu-ray Disc, video content acquisition and editing systems, and security applications in camcorders.

[0003] Even in the case of short films, large amounts of video data are required to describe them, which can create difficulties when the data is to be sent or otherwise transmitted across networks with limited bandwidth capacity. Therefore, video data is often compressed before being transmitted across modern telecommunications networks. The size of the video can also be an issue when storing the video on a storage device, as memory resources may be limited. Video compression devices typically use software and / or hardware on the source side to encode the video data prior to transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received by a video decompression device on the destination side. With limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase compression rates with little to no impact on image quality.

[0004] Prediction in video coding can be divided into intra-frame prediction and inter-frame prediction. Inter-frame prediction is to find a matching reference block for the current block in the current image in the reconstructed image, and use the value of the pixel in the reference block as the predicted value of the pixel in the current block. The encoder tries multiple reference blocks for the current block in the reference image, then decides on a reference block suitable for the current block and transmits motion information to the decoder. The decoder can find the reference block of the corresponding image block through the motion information in the bitstream, and then obtain the prediction of the image block. The motion information includes one or two motion vectors (MV) pointing to the reference block, and the indication information of the image where the reference block is located (usually recorded as the reference frame index (RI)). In the high efficiency video coding (HEVC) standard, two inter-frame prediction modes are defined, namely advanced motion vector prediction (AMVP) mode and merge mode. Both modes first construct a candidate motion information list through the motion information of the reconstructed image blocks adjacent to the current block in the spatial domain or temporal domain, then determine the optimal motion information from the candidate motion information list as the motion information of the current block, and then obtain the prediction of the current block based on the motion information of the current block.

[0005] Therefore, how to obtain the prediction of the current block based on multiple candidate motion information is the key to achieving inter-frame prediction. Summary of the invention

[0006] The present application provides an inter-frame prediction method and device to improve the accuracy of inter-frame prediction, reduce the error of inter-frame prediction, and improve the RDO efficiency of inter-frame prediction.

[0007] In a first aspect, the present application provides an inter-frame prediction method, comprising: obtaining motion vectors of P reconstructed image blocks in a peripheral area of ​​a current block, wherein the peripheral area includes a spatial neighborhood and / or a temporal neighborhood of the current block; obtaining Q prior candidate motion vectors of the current block and Q probability values ​​corresponding to the Q prior candidate motion vectors according to the motion vectors of the P reconstructed image blocks; obtaining M weight factors corresponding to the M prior candidate motion vectors according to the M probability values ​​corresponding to the M prior candidate motion vectors; M, P and Q are positive integers; performing motion compensation respectively according to the M prior candidate motion vectors to obtain M prediction values; and obtaining a prediction value of the current block according to a weighted sum of the M prediction values ​​and the corresponding M weight factors.

[0008] The surrounding area of ​​the current block includes the spatial neighborhood and / or the temporal neighborhood of the current block, wherein the image blocks of the spatial neighborhood may include a left candidate image block located on the left side of the current block and an upper candidate image block located above the current block.

[0009] The reconstructed image block may refer to an encoded image block that has been encoded and reconstructed by the encoding end or a decoded image block that has been decoded and reconstructed by the decoding end. The reconstructed image block may also refer to a basic unit image block of a preset size obtained by dividing an encoded image block or a decoded image block into equal sizes.

[0010] The motion vector of the reconstructed image block may include: (1) multiple a posteriori motion vectors of the reconstructed image block, which are determined based on the reconstruction value of the reconstructed image block and the prediction values ​​corresponding to multiple a posteriori candidate motion vectors; or (2) the optimal motion vector of the reconstructed image block, which is the a posteriori motion vector with the largest probability value or the smallest prediction error value among the above multiple a posteriori motion vectors.

[0011] The multiple posterior candidate motion vectors of the reconstructed image block are obtained based on the multiple a priori candidate motion vectors of the reconstructed image block. For any one of the multiple a priori candidate motion vectors of the reconstructed image block, it can be offset within a preset search window to generate multiple offset candidate motion vectors. It can be seen that one a priori candidate motion vector of the reconstructed image block can obtain multiple offset candidate motion vectors. The multiple a priori candidate motion vectors of the reconstructed image block are all operated as above, and all the offset candidate motion vectors obtained are the multiple posterior candidate motion vectors of the reconstructed image block. The above-mentioned P reconstructed image blocks can obtain their respective multiple posterior candidate motion vectors according to this method, which will not be described one by one here.

[0012] The multiple a posteriori motion vectors of the reconstructed image block may refer to the multiple a posteriori candidate motion vectors mentioned above; or may refer to some motion vectors among the multiple a posteriori candidate motion vectors mentioned above, such as multiple specified motion vectors selected from the multiple a posteriori candidate motion vectors mentioned above. The P reconstructed image blocks mentioned above can obtain their respective multiple a posteriori motion vectors according to this method, which will not be described one by one here.

[0013] The motion vectors of the P reconstructed image blocks can be input into the trained neural network to obtain Q prior candidate motion vectors of the current block and Q probability values ​​corresponding to the Q prior candidate motion vectors. The neural network can refer to the description of the training engine 25 below, which will not be repeated here.

[0014] The Q prior candidate motion vectors of the current block may refer to all motion vectors remaining after deduplication of multiple a posteriori motion vectors of the P reconstructed image blocks, or may refer to some motion vectors among all motion vectors remaining after deduplication of multiple a posteriori motion vectors of the P reconstructed image blocks.

[0015] Optionally, M=Q, in which case the M probability values ​​refer to the Q probability values ​​mentioned above, and the M priori candidate motion vectors refer to the Q priori candidate motion vectors mentioned above.

[0016] Optionally, M<Q, in which case the M probability values ​​are all greater than the other probability values ​​except the M probability values ​​among the Q probability values, and the M prior candidate motion vectors corresponding to the M probability values ​​are selected from the Q prior candidate motion vectors of the current block. That is, the first M probability values ​​with the largest probability values ​​are selected from the Q probability values ​​corresponding to the Q prior candidate motion vectors of the current block, and the M prior candidate motion vectors corresponding to the M probability values ​​are selected from the Q prior candidate motion vectors of the current block, and the weight factors and prediction values ​​are calculated based on the M probability values ​​and the M prior candidate motion vectors, thereby obtaining the prediction value of the current block. The remaining probability values ​​except the aforementioned M probability values ​​among the multiple probability values ​​corresponding to the multiple prior candidate motion vectors can be ignored due to their small values, which can reduce the amount of calculation and improve the efficiency of inter-frame prediction.

[0017] It should be noted that the “corresponding” in the M probability values ​​corresponding to the M prior candidate motion vectors does not mean a one-to-one correspondence. For example, the current block has 5 prior candidate motion vectors, and the multiple probability values ​​corresponding to them may be 5 probability values ​​or less than 5 probability values.

[0018] When the sum of the M probability values ​​is 1, the probability value corresponding to the first prior candidate motion vector is used as the weight factor corresponding to the first prior candidate motion vector. That is, the weight factor of each of the M prior candidate motion vectors is the probability value of each of the M prior candidate motion vectors; or, when the sum of the M probability values ​​is not 1, the M probability values ​​are normalized; the normalized value of the probability value corresponding to the first prior candidate motion vector is used as the weight factor corresponding to the first prior candidate motion vector. That is, the weight factor of each of the M prior candidate motion vectors is the normalized value of the probability value of each of the M prior candidate motion vectors. The above-mentioned first prior candidate motion vector is a term used only for the convenience of description. It does not refer to a specific prior candidate motion vector, but represents any one of the Q prior candidate motion vectors. It can be seen that the sum of the M weight factors corresponding to the M prior candidate motion vectors is 1.

[0019] According to the principle of inter-frame prediction, a candidate motion vector can find a reference block in the reference frame of the current block, and the current block is inter-frame predicted based on the reference block to obtain the prediction value corresponding to the candidate motion vector. It can be seen that the prediction value of the current block corresponds to the candidate motion vector. Therefore, according to the M prior candidate motion vectors, motion compensation is performed respectively, and M prediction values ​​of the current block can be obtained.

[0020] The prediction value of the current block is obtained by weighted summing the M prediction values ​​and the corresponding M weight factors. As described above, the M prediction values ​​correspond to the M prior candidate motion vectors, and the M weight factors also correspond to the M prior candidate motion vectors. Therefore, for the same prior candidate motion vector, a corresponding relationship is established between the corresponding prediction value and the weight factor. The weight factor corresponding to the same prior candidate motion vector is multiplied with the prediction value, and then multiple products corresponding to multiple prior candidate motion vectors are added to obtain the prediction value of the current block.

[0021] The present application obtains multiple weight factors and multiple prediction values ​​of the current block based on the motion vectors of multiple reconstructed image blocks in the surrounding area of ​​the current block, multiplies the weight factors and prediction values ​​corresponding to the same prior candidate motion vector, and then adds multiple products corresponding to multiple prior candidate motion vectors to obtain the prediction value of the current block. The prediction value of the current block obtained in this way combines multiple prior candidate motion vectors, so that it can better fit the rich and varied textures in the real world, improve the accuracy of inter-frame prediction, reduce the error of inter-frame prediction, and improve the overall rate-distortion optimization (RDO) efficiency of inter-frame prediction.

[0022] In a possible implementation, in addition to obtaining the motion vectors of the P reconstructed image blocks, relevant information of the P reconstructed image blocks may be obtained. The relevant information of the reconstructed image block may be a plurality of prediction error values ​​corresponding to a plurality of a posteriori motion vectors of the reconstructed image block, and the plurality of prediction error values ​​are also determined based on the reconstruction value of the reconstructed image block and the prediction values ​​corresponding to the plurality of a posteriori candidate motion vectors.

[0023] By performing motion compensation on the reconstructed image blocks according to the multiple a posteriori candidate motion vectors of the reconstructed image blocks, multiple prediction values ​​can be obtained, and the multiple prediction values ​​correspond to the aforementioned multiple a posteriori candidate motion vectors.

[0024] The multiple prediction values ​​are compared with the reconstruction values ​​of the reconstructed image blocks to obtain multiple prediction error values, which correspond to multiple a posteriori candidate motion vectors. The present application can use the sum of absolute difference (SAD) or the sum of squared difference (SSE) to obtain the prediction error value corresponding to a certain a posteriori candidate motion vector.

[0025] If the multiple a posteriori motion vectors of the reconstructed image block refer to the above-mentioned multiple a posteriori candidate motion vectors, then the multiple prediction error values ​​corresponding to the above-mentioned multiple a posteriori motion vectors of the reconstructed image block refer to the multiple prediction error values ​​corresponding to the above-mentioned multiple a posteriori candidate motion vectors; if the multiple a posteriori motion vectors of the reconstructed image block refer to some motion vectors among the above-mentioned multiple a posteriori candidate motion vectors, then the multiple prediction error values ​​corresponding to the above-mentioned multiple a posteriori motion vectors of the reconstructed image block refer to the prediction error values ​​corresponding to the partial motion vectors selected from the multiple prediction error values ​​corresponding to the above-mentioned multiple a posteriori candidate motion vectors.

[0026] Correspondingly, the input to the neural network includes a plurality of posterior motion vectors of each of the P reconstructed image blocks and a plurality of prediction error values ​​corresponding to the plurality of posterior motion vectors.

[0027] In a possible implementation, in addition to obtaining the motion vectors of the P reconstructed image blocks, relevant information of the P reconstructed image blocks may be obtained. The relevant information of the reconstructed image block may be a plurality of probability values ​​corresponding to a plurality of a posteriori motion vectors of the reconstructed image block, and the plurality of probability values ​​are also determined according to the reconstructed value of the reconstructed image block and the predicted values ​​corresponding to the plurality of a posteriori candidate motion vectors.

[0028] There are two methods for obtaining multiple probability values ​​corresponding to multiple posterior motion vectors of the reconstructed image block:

[0029] One method is to obtain multiple probability values ​​of the reconstructed image block based on the multiple prediction error values ​​of the reconstructed image block obtained in the above method. For example, the multiple prediction error values ​​of the reconstructed image block can be normalized using a normalized exponential function, a linear normalization method, or the like to obtain normalized values ​​of the multiple prediction error values, which are the multiple probability values ​​of the reconstructed image block. Based on the correspondence between the multiple prediction error values ​​of the reconstructed image block and the multiple posterior motion vectors, the multiple probability values ​​of the reconstructed image block also correspond to the multiple posterior motion vectors of the reconstructed image block, and the probability value can represent the probability that the corresponding posterior motion vector becomes the optimal motion vector of the reconstructed image block.

[0030] Another method is to input the reconstruction value of the reconstructed image block and the multiple predicted values ​​of the reconstructed image block obtained in the first method into a trained neural network to obtain multiple probability values ​​of the reconstructed image block corresponding to multiple posterior motion vectors. The neural network can refer to the description of the training engine 25 above, which will not be repeated here.

[0031] Correspondingly, the input to the neural network includes a plurality of posterior motion vectors of each of the P reconstructed image blocks and a plurality of probability values ​​corresponding to the plurality of posterior motion vectors.

[0032] Therefore, after obtaining multiple prediction error values ​​or probability values ​​corresponding to multiple posterior motion vectors through the above two methods, the optimal motion vector of the reconstructed image block can be obtained by the following two methods:

[0033] One is to use the a posteriori motion vector corresponding to the minimum prediction error value among multiple prediction error values ​​corresponding to multiple a posteriori motion vectors as the optimal motion vector of the reconstructed image block.

[0034] The other is to use the posterior motion vector corresponding to the maximum probability value among multiple probability values ​​corresponding to multiple posterior motion vectors as the optimal motion vector of the reconstructed image block.

[0035] It should be noted that the optimal motion vector in this application only refers to the motion vector obtained by one of the above two methods, which is one of the multiple posterior motion vectors of the reconstructed image block, but the optimal motion vector is not the only motion vector used for inter-frame prediction of the reconstructed image block.

[0036] In a possible implementation, after obtaining the reconstruction value of the current block, the posterior motion vector of the current block and related information thereof may be immediately obtained, and the obtaining method includes:

[0037] 1. Obtain multiple a posteriori motion vectors of the current block and multiple prediction error values ​​corresponding to the multiple a posteriori motion vectors according to the reconstructed value of the current block and the prediction values ​​corresponding to the multiple a posteriori candidate motion vectors of the current block. The multiple a posteriori motion vectors of the current block are obtained according to the multiple a priori candidate motion vectors of the current block.

[0038] 2. Input the reconstructed value of the current block and the predicted value corresponding to the multiple posterior candidate motion vectors of the current block into the neural network to obtain multiple posterior motion vectors of the current block and multiple probability values ​​corresponding to the multiple posterior motion vectors. The multiple posterior motion vectors of the current block are obtained based on the multiple a priori candidate motion vectors of the current block, or, the multiple probability values ​​corresponding to the multiple posterior motion vectors of the current block are obtained based on the multiple prediction error values ​​of the current block.

[0039] 3. Determine the posterior motion vector with the largest probability value or the smallest prediction error value among the multiple posterior motion vectors of the current block as the optimal motion vector of the current block.

[0040] In one possible implementation, the training data set used by the training engine when training the neural network includes information of multiple groups of image blocks, wherein the information of each group of image blocks includes multiple posterior motion vectors of multiple reconstructed image blocks, multiple probability values ​​corresponding to the multiple posterior motion vectors, and multiple posterior motion vectors of a current block, and multiple probability values ​​corresponding to the multiple posterior motion vectors, wherein the multiple reconstructed image blocks are image blocks in the spatial neighborhood and / or temporal neighborhood of the current block; and the neural network is trained based on the training data set.

[0041] In one possible implementation, the training data set used by the training engine when training the neural network includes information of multiple groups of image blocks, wherein the information of each group of image blocks includes multiple posterior motion vectors of multiple reconstructed image blocks, multiple prediction error values ​​corresponding to the multiple posterior motion vectors, and multiple posterior motion vectors of a current block, and multiple probability values ​​corresponding to the multiple posterior motion vectors, wherein the multiple reconstructed image blocks are image blocks in the spatial neighborhood and / or temporal neighborhood of the current block; and the neural network is trained based on the training data set.

[0042] In one possible implementation, the training data set used by the training engine when training the neural network includes information of multiple groups of image blocks, wherein the information of each group of image blocks includes optimal motion vectors of multiple reconstructed image blocks, multiple posterior motion vectors of a current block, and multiple probability values ​​corresponding to the multiple posterior motion vectors, and the multiple reconstructed image blocks are image blocks in the spatial neighborhood and / or temporal neighborhood of the current block; the neural network is trained based on the training data set.

[0043] Optionally, the neural network includes at least a convolution layer and an activation layer. The depth of the convolution kernel of the convolution layer is 2, 3, 4, 5, 6, 16, 24, 32, 48, 64 or 128; the size of the convolution kernel in the convolution layer is 1×1, 3×3, 5×5 or 7×7. For example, the size of a convolution layer is 3×3×2×10, wherein 3×3 represents the size of the convolution kernel in the convolution layer; 2 represents the depth of the convolution kernel contained in the convolution layer, and the number of data channels input to the convolution layer is consistent with the depth of the convolution kernel contained in the convolution layer, that is, the number of data channels input to the convolution layer is also 2; 10 represents the number of convolution kernels contained in the convolution layer, and the number of data channels output from the convolution layer is consistent with the number of convolution kernels contained in the convolution layer, that is, the number of data channels output from the convolution layer is also 10.

[0044] Optionally, the neural network includes a convolutional neural network (CNN), a deep neural network (DNN), or a recurrent neural network (RNN).

[0045] In a second aspect, the present application provides an encoder, comprising a processing circuit, for executing the method according to any one of the above-mentioned first aspects.

[0046] In a third aspect, the present application provides a decoder, comprising a processing circuit, for executing the method described in any one of the first aspects above.

[0047] In a fourth aspect, the present application provides a computer program product, comprising a program code, which, when executed on a computer or a processor, is used to execute any of the methods described in the first aspect above.

[0048] In a fifth aspect, the present application provides an encoder comprising: one or more processors; a non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the decoder executes the method described in any one of the first aspects above.

[0049] In a sixth aspect, the present application provides a decoder comprising: one or more processors; a non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the encoder executes the method described in any one of the first aspects above.

[0050] In a seventh aspect, the present application provides a non-transitory computer-readable storage medium, comprising a program code, which, when executed by a computer device, is used to execute the method described in any one of the first aspects above.

[0051] In an eighth aspect, the present invention relates to an inter-frame prediction device, and the beneficial effects can be found in the description of the first aspect and will not be described in detail here. The inter-frame prediction device has the function of implementing the behavior in the method embodiment of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the inter-frame prediction device includes: a motion estimation unit and an inter-frame prediction processing unit, wherein the motion estimation unit is used to obtain the motion vectors of P reconstructed image blocks in the surrounding area of ​​the current block, and the surrounding area includes the spatial neighborhood and / or temporal neighborhood of the current block; the inter-frame prediction processing unit is used to implement the method described in any one of the first aspects above. These modules can perform the corresponding functions in the method example of the first aspect above, and refer to the detailed description in the method example for details, which will not be described here.

[0052] One or more embodiments will be described in detail in the drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1a is an exemplary block diagram of a decoding system 10 according to an embodiment of the present application;

[0054] Figure 1b is an exemplary block diagram of a video decoding system 40 according to an embodiment of the present application;

[0055] Figure 2 is an exemplary block diagram of a video encoder 20 according to an embodiment of the present application;

[0056] Figure 3 is an exemplary block diagram of a video decoder 30 according to an embodiment of the present application;

[0057] Figure 4 is an exemplary block diagram of a video decoding device 400 according to an embodiment of the present application;

[0058] Figure 5 is an exemplary block diagram of a device 500 according to an embodiment of the present application;

[0059] Figure 6a-6e Several exemplary architectures of neural networks for inter-frame prediction according to embodiments of the present application;

[0060] Figure 7 is an exemplary schematic diagram of a candidate image block according to an embodiment of the present application;

[0061] Figure 8 A flowchart of a process 800 of an inter-frame prediction method according to an embodiment of the present application;

[0062] Fig. 9 A flowchart of a process 900 of an inter-frame prediction method according to an embodiment of the present application;

[0063] Fig.10 This is an exemplary schematic diagram of a search window according to an embodiment of the present application;

[0064] Fig.11 A flowchart of a process 1100 of an inter-frame prediction method according to an embodiment of the present application;

[0065] Fig.12 A flowchart of a process 1200 of an inter-frame prediction method according to an embodiment of the present application;

[0066] Fig.13 Schematic diagram of the structure of the inter-frame prediction device 1300 according to an embodiment of the present application. DETAILED DESCRIPTION

[0067] The embodiments of the present application provide an AI-based video compression technology, and more particularly, provide a neural network-based video compression technology, and specifically provide an inter-frame prediction technology based on a neural network (NN) to improve a traditional hybrid video encoding and decoding system.

[0068] Video coding generally refers to processing a sequence of images that form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms. Video coding (or commonly referred to as coding) includes two parts: video encoding and video decoding. Video coding is performed on the source side and generally includes processing (e.g., compressing) the original video image to reduce the amount of data required to represent the video image (thereby more efficiently storing and / or transmitting). Video decoding is performed on the destination side and generally includes inverse processing relative to the encoder to reconstruct the video image. The "encoding" of the video image (or commonly referred to as the image) involved in the embodiment should be understood as the "encoding" or "decoding" of the video image or video sequence. The encoding part and the decoding part are also collectively referred to as codec (encoding and decoding, CODEC).

[0069] In the case of lossless video coding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed by quantization, etc. to reduce the amount of data required to represent the video image, but the decoder side cannot fully reconstruct the video image, that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0070] Several video coding standards belong to the category of "lossy hybrid video codecs" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is usually divided into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder usually processes, i.e., encodes the video at the block (video block) level, for example, by generating a prediction block through spatial (intra-frame) prediction and temporal (inter-frame) prediction; subtracting the prediction block from the current block (currently processed / to be processed block) to obtain a residual block; transforming the residual block in the transform domain and quantizing the residual block to reduce the amount of data to be transmitted (compressed), while the decoder side applies the inverse processing part relative to the encoder to the encoded or compressed block to reconstruct the current block for representation. In addition, the encoder needs to repeat the processing steps of the decoder so that the encoder and the decoder generate the same predictions (e.g., intra-frame predictions and inter-frame predictions) and / or reconstructed pixels for processing, i.e., encoding subsequent blocks.

[0071] In the following embodiment of the decoding system 10, the encoder 20 and the decoder 30 are based on Figures 1a to 3 Give a description.

[0072] Figure 1a 1 is an exemplary block diagram of a decoding system 10 of an embodiment of the present application, for example, a video decoding system 10 (or simply a decoding system 10) that can utilize the techniques of the present application. The video encoder 20 (or simply an encoder 20) and the video decoder 30 (or simply a decoder 30) in the video decoding system 10 represent devices that can be used to perform various techniques according to various examples described in the present application.

[0073] like Figure 1a As shown, the decoding system 10 includes a source device 12, which is used to provide encoded image data 21 such as encoded images to a destination device 14 for decoding the encoded image data 21.

[0074] The source device 12 includes an encoder 20 , and may additionally or optionally include an image source 16 , a preprocessor (or a preprocessing unit) 18 such as an image preprocessor, and a communication interface (or a communication unit) 22 .

[0075] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0076] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data) 17 may also be referred to as a raw image (or raw image data) 17 .

[0077] The preprocessor 18 is used to receive the original image data 17 and preprocess the original image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment, or denoising. It is understood that the preprocessing unit 18 may be an optional component.

[0078] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide the encoded image data 21 (hereinafter referred to as Figure 2 etc. for further description).

[0079] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0080] The destination device 14 includes a decoder 30 and, in addition or alternatively, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0081] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0082] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any type of combination thereof.

[0083] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0084] The communication interface 28 corresponds to the communication interface 22 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0085] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1a A unidirectional communication interface, or a bidirectional communication interface, indicated by an arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0086] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as Figure 3 etc. for further description).

[0087] The post-processor 32 is used to post-process the decoded image data 31 (also called reconstructed image data) such as the decoded image to obtain the post-processed image data 33 such as the post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, cropping or resampling, or any other processing for generating the decoded image data 31 for display by the display device 34 or the like.

[0088] The display device 34 is used to receive the post-processed image data 33 to display the image to a user or viewer, etc. The display device 34 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or display. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0089] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 (especially the inter-frame prediction unit in the encoder 20) or the decoder 30 (especially the inter-frame prediction unit in the decoder 30) to process the input image or image region or image block to generate a prediction value of the input image or image region or image block.

[0090] Optionally, the training data set in the embodiment of the present application includes: information of multiple groups of image blocks, wherein the information of each group of image blocks includes multiple posterior motion vectors of multiple reconstructed image blocks, multiple probability values ​​corresponding to the multiple posterior motion vectors, and multiple posterior candidate motion vectors of the current block, and multiple probability values ​​corresponding to the multiple posterior candidate motion vectors, and the multiple reconstructed image blocks are image blocks in the spatial neighborhood and / or temporal neighborhood of the current block. A neural network is obtained by training with the training data set, the input of which is the multiple posterior motion vectors of multiple reconstructed image blocks in the surrounding area of ​​the current block, and the multiple probability values ​​corresponding to the multiple posterior motion vectors, and the output is the multiple a priori candidate motion vectors of the current block, and the multiple probability values ​​corresponding to the multiple a priori candidate motion vectors.

[0091] Optionally, the training data set in the embodiment of the present application includes: information of multiple groups of image blocks, wherein the information of each group of image blocks includes multiple posterior motion vectors of multiple reconstructed image blocks, multiple prediction error values ​​corresponding to the multiple posterior motion vectors, and multiple posterior candidate motion vectors of the current block, and multiple probability values ​​corresponding to the multiple posterior candidate motion vectors, and the multiple reconstructed image blocks are image blocks in the spatial neighborhood and / or temporal neighborhood of the current block. A neural network is obtained by training with the training data set, the input of which is the multiple posterior motion vectors of multiple reconstructed image blocks in the surrounding area of ​​the current block, and the multiple prediction error values ​​corresponding to the multiple posterior motion vectors, and the output is the multiple a priori candidate motion vectors of the current block, and the multiple probability values ​​corresponding to the multiple a priori candidate motion vectors.

[0092] Optionally, the training data set in the embodiment of the present application includes: information of multiple groups of image blocks, wherein the information of each group of image blocks includes the optimal motion vectors of multiple reconstructed image blocks, and multiple a posteriori candidate motion vectors of the current block, and multiple probability values ​​corresponding to the multiple a posteriori candidate motion vectors, and the multiple reconstructed image blocks are image blocks in the spatial neighborhood and / or temporal neighborhood of the current block. A neural network is obtained by training the training data set, the input of the neural network is the optimal motion vectors of multiple reconstructed image blocks in the surrounding area of ​​the current block, and the output is multiple a priori candidate motion vectors of the current block, and multiple probability values ​​corresponding to the multiple a priori candidate motion vectors.

[0093] Optionally, the training data set in the embodiment of the present application includes: information of multiple groups of image blocks, wherein the information of each group of image blocks includes the reconstruction value of the image block and the predicted value corresponding to multiple posterior candidate motion vectors, as well as multiple posterior motion vectors of the image block and multiple probability values ​​corresponding to the multiple posterior motion vectors. A neural network is obtained by training the training data set, the input of which is the reconstruction value of the current block and the predicted value corresponding to the multiple posterior candidate motion vectors, and the output is the multiple posterior motion vectors of the current block and the multiple probability values ​​corresponding to the multiple posterior motion vectors.

[0094] The process of training the neural network by the training engine 25 makes the output multiple prior candidate motion vectors of the current block approximate multiple a posteriori motion vectors of the current block, and the multiple probability values ​​corresponding to the multiple prior candidate motion vectors approximate the multiple probability values ​​corresponding to the multiple a posteriori motion vectors. Each training process can use a small batch size of 64 images and an initial learning rate of 1e-4, following a step size of 10. The information of multiple groups of image blocks can be data generated when an encoder performs inter-frame encoding on multiple current blocks. The neural network can be used to implement the inter-frame prediction method provided in an embodiment of the present application, that is, the motion vectors of multiple reconstructed image blocks in the surrounding area of ​​the current block and their related information are input into the neural network, and multiple prior candidate motion vectors of the current block and multiple probability values ​​corresponding to the multiple prior candidate motion vectors can be obtained. The following will be combined with Figure 6a-6e Describe the neural network in detail.

[0095] The training data in the embodiment of the present application can be stored in a database (not shown), and the training engine 25 obtains a target model (for example, a neural network for image frame prediction) based on the training data. It should be noted that the embodiment of the present application does not limit the source of the training data, for example, the training data can be obtained from the cloud or other places for model training.

[0096] The target model trained by the training engine 25 can be applied to the decoding system 10, for example, Figure 1a The source device 12 (e.g., encoder 20) or the destination device 14 (e.g., decoder 30) shown. The training engine 25 can be trained in the cloud to obtain a target model, and then the decoding system 10 downloads and uses the target model from the cloud; or, the training engine 25 can be trained in the cloud to obtain a target model and use the target model, and the decoding system 10 directly obtains the processing result from the cloud. For example, the training engine 25 trains to obtain a target model with an inter-frame prediction function, and the decoding system 10 downloads the target model from the cloud, and then the inter-frame prediction unit 244 in the encoder 20 or the inter-frame prediction unit 344 in the decoder 30 can perform inter-frame prediction on the input image or image block according to the target model to obtain a prediction of the image or image block. For another example, the training engine 25 trains to obtain a target model with an inter-frame prediction function, and the decoding system 10 does not need to download the target model from the cloud. The encoder 20 or the decoder 30 transmits the image or image block to the cloud, and the cloud performs inter-frame prediction on the image or image block through the target model to obtain a prediction of the image or image block and transmit it to the encoder 20 or the decoder 30.

[0097] although Figure 1aThe source device 12 and the destination device 14 are shown as independent devices, but the device embodiment may also include the source device 12 and the destination device 14 or the functions of the source device 12 and the destination device 14 at the same time, that is, the source device 12 or the corresponding function and the destination device 14 or the corresponding function at the same time. In these embodiments, the source device 12 or the corresponding function and the destination device 14 or the corresponding function can be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0098] According to the description, Figure 1a The presence and (exact) division of different units or functions in the source device 12 and / or the destination device 14 shown may vary depending on the actual device and application, which will be obvious to the skilled person.

[0099] The encoder 20 (eg, video encoder 20) or the decoder 30 (eg, video decoder 30) or both may be configured as follows: Figure 1b The processing circuitry shown may be implemented, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding dedicated processors, or any combination thereof. The encoder 20 may be implemented by processing circuitry 46 to include reference Figure 2 The various modules discussed in encoder 20 and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented by processing circuit 46 to include reference Figure 3 The processing circuit 46 may be used to perform the various operations discussed below. Figure 5 As shown, if part of the technology is implemented in software, the device can store the instructions of the software in a suitable non-transitory computer-readable storage medium, and use one or more processors to execute the instructions in hardware, thereby performing the technology of the present application. One of the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as Figure 1b shown.

[0100] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or a content distribution server), a broadcast receiving device, a broadcast transmitting device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0101] In some cases, Figure 1a The video decoding system 10 shown is merely exemplary, and the techniques provided herein may be applicable to video encoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from a local memory, sent over a network, and so on. A video encoding device may encode data and store the data in a memory, and / or a video decoding device may retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to a memory and / or retrieve and decode data from a memory.

[0102] Figure 1b FIG. 4 is an exemplary block diagram of a video decoding system 40 according to an embodiment of the present application. Figure 1b As shown, the video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by a processing circuit 46), an antenna 42, one or more processors 43, one or more memory storages 44 and / or a display device 45.

[0103] like Figure 1b As shown, imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory storage 44 and / or display device 45 can communicate with each other. In different examples, video decoding system 40 can include only video encoder 20 or only video decoder 30.

[0104] In some instances, antenna 42 can be used to transmit or receive a coded bit stream of video data. In addition, in some instances, display device 45 can be used to present video data. Processing circuit 46 can include application-specific integrated circuit (ASIC) logic, graphics processor, general processor, etc. Video decoding system 40 can also include optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, graphics processor, general processor, etc. In addition, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory storage 44 can be implemented by cache memory. In other instances, processing circuit 46 can include memory (e.g., cache, etc.) for implementing image buffer, etc.

[0105] In some examples, video encoder 20 implemented by logic circuits may include an image buffer (e.g., implemented by processing circuit 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuit 46 to implement reference Figure 2 The various modules discussed in connection with the encoder and / or any other encoder system or subsystem described herein. Logic circuits may be used to perform the various operations discussed herein.

[0106] In some examples, video decoder 30 may be implemented in a similar manner by processing circuitry 46 to implement reference Figure 3 The various modules discussed in connection with the video decoder 30 of the present invention and / or any other decoder system or subsystem described herein. In some examples, the video decoder 30 implemented by logic circuitry may include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (implemented, for example, by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented by processing circuitry 46 to implement the video decoder 30 of the present invention. Figure 3 and / or the various modules discussed above for any other decoder system or subsystem described herein.

[0107] In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frames discussed herein, indicators, index values, mode selection data, etc., such as data related to the encoded partitions (e.g., transform coefficients or quantized transform coefficients, (as discussed) optional indicators, and / or data defining the encoded partitions). Video decoding system 40 may also include video decoder 30 coupled to antenna 42 and used to decode the encoded bitstream. Display device 45 is used to present the video frames.

[0108] It should be understood that for the examples described with reference to the video encoder 20 in the embodiments of the present application, the video decoder 30 can be used to perform the reverse process. With respect to the signaling syntax elements, the video decoder 30 can be used to receive and parse such syntax elements and decode the related video data accordingly. In some examples, the video encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, the video decoder 30 can parse such syntax elements and decode the related video data accordingly.

[0109] For ease of description, the embodiments of the present application are described with reference to the universal video coding (VVC) reference software or the high-efficiency video coding (HEVC) developed by the video coding joint working group (joint collaboration team on video coding, JCT-VC) of the ITU-T video coding experts group (VCEG) and the ISO / IEC motion picture experts group (MPEG). Those of ordinary skill in the art understand that the embodiments of the present application are not limited to HEVC or VVC.

[0110] Encoders and encoding methods

[0111] Figure 2 FIG. 2 is an exemplary block diagram of a video encoder 20 according to an embodiment of the present application. Figure 2As shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270 and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254 and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The illustrated video encoder 20 may also be referred to as a hybrid video encoder or a hybrid video codec-based video encoder.

[0112] See also Figure 2 The inter-frame prediction unit is a trained target model (also called a neural network) that processes an input image or image region or image block to generate a prediction value of the input image block. For example, the neural network for inter-frame prediction is used to receive an input image or image region or image block and generate a prediction value of the input image or image region or image block. Figure 6a-6e Describe the neural network used for inter-frame prediction in detail.

[0113] The residual calculation unit 204, the transform processing unit 206, the quantization unit 208 and the mode selection unit 260 constitute the forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 constitute the backward signal path of the encoder, wherein the backward signal path of the encoder 20 corresponds to the signal path of the decoder (see Figure 3 The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded image buffer 230, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 also constitute a “built-in decoder” of the video encoder 20.

[0114] Images and Image Segmentation (Images and Blocks)

[0115] The encoder 20 may be configured to receive, via an input 201 or the like, an image (or image data) 17, e.g., an image in a sequence of images forming a video or a video sequence. The received image or image data may also be a pre-processed image (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 may also be referred to as a current image or an image to be encoded (particularly when distinguishing the current image from other images in video encoding, such as previously encoded and / or decoded images in the same video sequence, i.e., a video sequence also including the current image).

[0116] A (digital) image is or can be considered as a two-dimensional array or matrix of pixels with intensity values. The pixels in the array can also be called pixels (pixel or pel) (short for picture elements). The number of pixels in the array or image in the horizontal and vertical directions (or axes) determines the size and / or resolution of the image. In order to represent color, three color components are usually used, that is, the image can be represented as or include three pixel arrays. In the RBG format or color space, the image includes corresponding red, green and blue pixel arrays. However, in video coding, each pixel is usually represented in a brightness / chrominance format or color space, such as YCbCr, including a brightness component indicated by Y (sometimes also represented by L) and two chrominance components represented by Cb and Cr. The brightness (luma) component Y represents the brightness or grayscale level intensity (for example, the two are the same in grayscale images), while the two chrominance (abbreviated as chroma) components Cb and Cr represent the chrominance or color information components. Accordingly, an image in YCbCr format includes a luminance pixel array of luminance pixel values ​​(Y) and two chrominance pixel arrays of chrominance values ​​(Cb and Cr). An image in RGB format can be converted or transformed into YCbCr format and vice versa, a process also referred to as color conversion or transformation. If the image is black and white, the image may include only a luminance pixel array. Accordingly, an image may be, for example, a luminance pixel array in a monochrome format or a luminance pixel array and two corresponding chrominance pixel arrays in 4:2:0, 4:2:2 and 4:4:4 color formats.

[0117] In one embodiment, an embodiment of the video encoder 20 may include an image segmentation unit ( Figure 2), for dividing the image 17 into a plurality of (usually non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC) or coding tree blocks (CTB), or coding tree units (CTU) in the H.265 / HEVC and VVC standards. The partitioning unit may be used to use the same block size for all images in a video sequence and a corresponding grid of defined block sizes, or to change the block size between images or subsets of images or groups of images, and to partition each image into corresponding blocks.

[0118] In other embodiments, the video encoder may be configured to directly receive a block 203 of the image 17, for example, one, several or all blocks constituting the image 17. The image block 203 may also be referred to as a current image block or an image block to be encoded.

[0119] Like image 17, image block 203 is also or can be considered as a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 203 is smaller than image 17. In other words, block 203 may include one pixel array (e.g., a brightness array in the case of monochrome image 17 or a brightness array or chrominance array in the case of a color image) or three pixel arrays (e.g., one brightness array and two chrominance arrays in the case of a color image 17) or any other number and / or type of arrays according to the color format adopted. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, the block can be an M×N (M columns×N rows) pixel array, or an M×N transform coefficient array, etc.

[0120] In one embodiment, Figure 2 The video encoder 20 shown is used to encode the image 17 block by block, for example, encoding and prediction are performed on each block 203.

[0121] In one embodiment, Figure 2 The illustrated video encoder 20 may also be used to partition and / or encode an image using slices (also referred to as video slices), wherein an image may be partitioned or encoded using one or more slices (usually non-overlapping). Each slice may include one or more blocks (e.g., coding tree units CTUs) or one or more groups of blocks (e.g., coding tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0122] In one embodiment, Figure 2The video encoder 20 shown can also be used to partition and / or encode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), wherein an image can be partitioned or encoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTU) or one or more coding blocks, etc., wherein each coding block may be in a rectangular shape, etc., and may include one or more complete or partial blocks (e.g., CTU).

[0123] Residual calculation

[0124] The residual calculation unit 204 is used to calculate the residual block 205 (the prediction block 265 is described in detail later) based on the image block (or original block) 203 and the prediction block 265 in the following manner: for example, the pixel value of the prediction block 265 is subtracted from the pixel value of the image block 203 pixel by pixel (pixel by pixel) to obtain the residual block 205 in the pixel domain.

[0125] Transform

[0126] The transform processing unit 206 is used to perform discrete cosine transform (DCT) or discrete sine transform (DST) on the pixel values ​​of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be called transform residual coefficients, representing the residual block 205 in the transform domain.

[0127] The transform processing unit 206 may be used to apply an integerized approximation of the DCT / DST, such as the transform specified for H.265 / HEVC. This integerized approximation is typically scaled by a factor compared to an orthogonal DCT transform. In order to maintain the norm of the residual block processed by the forward transform and the inverse transform, other scaling factors are used as part of the transform process. The scaling factor is typically selected based on certain constraints, such as whether the scaling factor is a power of 2 for the shift operation, the bit depth of the transform coefficients, a trade-off between accuracy and implementation cost, etc. For example, a specific scaling factor is specified for the inverse transform on the encoder 20 side by the inverse transform processing unit 212 (and for the corresponding inverse transform on the decoder 30 side by, for example, the inverse transform processing unit 312), and correspondingly, a corresponding scaling factor may be specified for the forward transform on the encoder 20 side by the transform processing unit 206.

[0128] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206) may be used to output transform parameters such as one or more transform types, for example, directly output or output after encoding or compression by the entropy coding unit 270, for example, so that the video decoder 30 may receive and use the transform parameters for decoding.

[0129] Quantification

[0130] The quantization unit 208 is used to quantize the transform coefficient 207 by, for example, scalar quantization or vector quantization to obtain a quantized transform coefficient 209 . The quantized transform coefficient 209 may also be referred to as a quantized residual coefficient 209 .

[0131] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, an n-bit transform coefficient may be rounded down to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different degrees of scaling may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to a finer quantization, while a larger quantization step size corresponds to a coarser quantization. A suitable quantization step size may be indicated by a quantization parameter (QP). For example, a quantization parameter may be an index of a predefined set of suitable quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (a smaller quantization step size), a larger quantization parameter may correspond to coarse quantization (a larger quantization step size), and vice versa. Quantization may include dividing by a quantization step size, while a corresponding or inverse dequantization performed by an inverse quantization unit 210 or the like may include multiplying by a quantization step size. Embodiments according to some standards, such as HEVC, may be used to determine a quantization step size using a quantization parameter. In general, the quantization step size may be calculated using a fixed-point approximation of an equation containing division according to the quantization parameter. Other scaling factors may be introduced for quantization and dequantization to recover the norm of the residual block that may be modified due to the scaling used in the fixed-point approximation of the equations for the quantization step size and the quantization parameter. In one exemplary implementation, the scaling of the inverse transform and dequantization may be combined. Alternatively, a custom quantization table may be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the larger the quantization step size, the greater the loss.

[0132] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) may be configured to output a quantization parameter (QP), for example, directly output or output after being encoded or compressed by the entropy coding unit 270, for example, so that the video decoder 30 may receive and use the quantization parameter for decoding.

[0133] Dequantization

[0134] The inverse quantization unit 210 is used to perform inverse quantization of the quantization unit 208 on the quantized coefficients to obtain dequantized coefficients 211, for example, performing an inverse quantization scheme of the quantization scheme performed by the quantization unit 208 according to or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 may also be referred to as dequantized residual coefficients 211, corresponding to the transform coefficients 207, but due to the loss caused by quantization, the dequantized coefficients 211 are usually not completely the same as the transform coefficients.

[0135] Inverse Transform

[0136] The inverse transform processing unit 212 is used to perform an inverse transform of the transform performed by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to obtain a reconstructed residual block 213 (or a corresponding dequantized coefficient 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 213.

[0137] reconstruction

[0138] The reconstruction unit 214 (e.g., the summer 214) is used to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 213 and the pixel point values ​​of the prediction block 265.

[0139] Filtering

[0140] The loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstruction block 215 to obtain the filter block 221, or is generally used to filter the reconstructed pixel points to obtain the filtered pixel point value. For example, the loop filter unit is used to smoothly perform pixel conversion or improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 2 2. Although shown as a loop filter in FIG. 2, in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filter block 221 may also be referred to as a filter reconstruction block 221.

[0141] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) may be used to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly or after being entropy encoded by the entropy encoding unit 270, such that the decoder 30 may receive and use the same or different loop filter parameters for decoding.

[0142] Decoded Image Buffer

[0143] The decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use by the video encoder 20 when encoding video data. The DPB 230 may be formed by any of a variety of memory devices, such as a dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. The decoded picture buffer 230 may be used to store one or more filter blocks 221. The decoded picture buffer 230 may also be used to store other previous filter blocks of the same current picture or a different picture such as a previous reconstructed picture, such as a previously reconstructed and filtered block 221, and may provide a complete previously reconstructed, i.e., decoded picture (and corresponding reference blocks and pixels) and / or a partially reconstructed current picture (and corresponding reference blocks and pixels), such as for inter-frame prediction. The decoded image buffer 230 may also be used to store one or more unfiltered reconstructed blocks 215, or generally to store unfiltered reconstructed pixels, for example, reconstructed blocks 215 that have not been filtered by the loop filter unit 220, or reconstructed blocks or reconstructed pixels that have not undergone any other processing.

[0144] Mode selection (segmentation and prediction)

[0145] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, which are used to receive or obtain original image data such as an original block 203 (current block 203 of the current image 17) and reconstructed image data from the decoded image buffer 230 or other buffers (e.g., a column buffer, not shown in the figure), such as filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images. The reconstructed image data is used as reference image data required for prediction such as inter-frame prediction or intra-frame prediction to obtain a prediction block 265 or a prediction value 265.

[0146] The mode selection unit 260 may be used to determine or select a partition for the current block (including no partition) and prediction mode (eg, intra-frame or inter-frame prediction mode), generate a corresponding prediction block 265 , calculate the residual block 205 , and reconstruct the reconstruction block 215 .

[0147] In one embodiment, the mode selection unit 260 may be used to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260) that provides the best match or minimum residual (minimum residual means better compression in transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression in transmission or storage), or considers or balances both of the above. The mode selection unit 260 may be used to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., select the prediction mode that provides minimum rate distortion optimization. The terms "best", "lowest", "optimal", etc. herein do not necessarily mean "best", "lowest", "optimal" in general, but may also refer to situations where termination or selection criteria are met, for example, values ​​exceeding or falling below a threshold or other restrictions may result in a "suboptimal selection" but reduce complexity and processing time.

[0148] In other words, the partitioning unit 262 may be used to partition an image in a video sequence into a sequence of coding tree units (CTUs), and the CTU 203 may be further partitioned into smaller block portions or sub-blocks (again forming blocks), for example, by iteratively using quad-tree partitioning (QT), binary-tree partitioning (BT) or triple-tree partitioning (TT) or any combination thereof, and for performing prediction on each of the block portions or sub-blocks, for example, wherein the mode selection comprises selecting a tree structure of the partition block 203 and selecting a prediction mode to be applied to each of the block portions or sub-blocks.

[0149] The segmentation (eg, performed by segmentation unit 262) and prediction processes (eg, performed by inter-prediction unit 244 and intra-prediction unit 254) performed by video encoder 20 are described in detail below.

[0150] segmentation

[0151] The segmentation unit 262 may segment (or divide) an image block (or CTU) 203 into smaller parts, such as small blocks in a square or rectangular shape. For an image with three pixel arrays, a CTU consists of N×N luminance pixel blocks and two corresponding chrominance pixel blocks. The maximum allowed size of a luminance block in a CTU is specified as 128×128 in the developing universal video coding (VVC) standard, but may be specified as a value different from 128×128 in the future, such as 256×256. The CTUs of an image may be concentrated / grouped into slices / coding block groups, coding blocks, or bricks. A coding block covers a rectangular area of ​​an image, and a coding block may be divided into one or more bricks. A brick consists of multiple CTU rows within a coding block. A coding block that is not segmented into multiple bricks may be called a brick. However, a brick is a true subset of a coding block and is therefore not called a coding block. VVC supports two coding block group modes, namely raster scan slice / coding block group mode and rectangular slice mode. In raster scan coded block group mode, a slice / coded block group contains a sequence of coded blocks in a raster scan of the coded blocks of an image. In rectangular slice mode, a slice contains multiple bricks of an image, which together form a rectangular area of ​​the image. The bricks within a rectangular slice are arranged in the order of the brick raster scan of the slice. These smaller blocks (also called sub-blocks) can be further split into smaller parts. This is also called tree partitioning or hierarchical tree partitioning, where a root block at root tree level 0 (hierarchy level 0, depth 0), etc., can be recursively split into two or more blocks at the next lower tree level, such as nodes at tree level 1 (hierarchy level 1, depth 1). These blocks can be split into two or more blocks at the next lower level, such as tree level 2 (hierarchy level 2, depth 2), etc., until the partitioning ends (because the end criteria are met, such as reaching the maximum tree depth or minimum block size). Blocks that are not further split are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary tree (BT), a tree divided into three parts is called a ternary tree (TT), and a tree divided into four parts is called a quadtree (QT).

[0152] For example, a coding tree unit (CTU) may be or include a CTB of luminance pixels, two corresponding CTBs of chrominance pixels of an image with three pixel arrays, or a CTB of pixels of a monochrome image, or a CTB of pixels of an image encoded using three independent color planes and syntax structures (for encoding pixels). Accordingly, a coding tree block (CTB) may be an N×N pixel block, where N may be set to a certain value so that the component is divided into CTBs, which is segmentation. A coding unit (coding unit, CU) may be or include a coding block of luminance pixels, two corresponding coding blocks of chrominance pixels of an image with three pixel arrays, or a coding block of pixels of a monochrome image, or a coding block of pixels of an image encoded using three independent color planes and syntax structures (for encoding pixels). Accordingly, a coding block (CB) may be an M×N pixel block, where M and N may be set to a certain value so that the CTB is divided into coding blocks, which is segmentation.

[0153] For example, in an embodiment, according to HEVC, a coding tree unit (CTU) may be divided into multiple CUs using a quadtree structure represented as a coding tree. A decision is made at the leaf-CU level whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode an image area. Each leaf-CU may be further divided into one, two, or four PUs according to the PU partition type. The same prediction process is used within a PU, and relevant information is transmitted to the decoder in units of PUs. After the prediction process is applied according to the PU partition type to obtain the residual block, the leaf-CU may be partitioned into transform units (TUs) according to other quadtree structures similar to the coding tree for the CU.

[0154] For example, in an embodiment, according to the latest video coding standard currently under development (referred to as Versatile Video Coding (VVC), a combined quadtree of nested multi-type trees (e.g., binary trees and ternary trees) is used to partition a segment structure for partitioning a coding tree unit. In the coding tree structure within the coding tree unit, the CU may be a square or a rectangle. For example, a coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a multi-type tree structure. The multi-type tree structure has four partition types: vertical binary tree partition (SPLIT_BT_VER), horizontal binary tree partition (SPLIT_BT_HOR), vertical ternary tree partition The multi-type tree leaf nodes are called coding units (CUs), unless the CU is too large for the maximum transform length, such segmentation is used for prediction and transform processing without any other segmentation. In most cases, this means that the block sizes of CU, PU and TU in the coding block structure of the quadtree nested multi-type tree are the same. This exception occurs when the maximum supported transform length is less than the width or height of the color components of the CU. VVC has established a unique signaling mechanism for the segmentation information in the coding structure with quadtree nested multi-type trees. In the signaling mechanism, the coding The tree unit (CTU) as the root of the quadtree is first split by the quadtree structure. Then each quadtree leaf node (when large enough) is further split into a multi-type tree structure. In the multi-type tree structure, the first flag (mtt_split_cu_flag) indicates whether the node is further split. When the node is further split, the second flag (mtt_split_cu_vertical_flag) is used to indicate the division direction, and then the third flag (mtt_split_cu_binary_flag) is used to indicate whether the division is a binary tree division or a ternary tree division. According to mtt_split_c The values ​​of u_vertical_flag and mtt_split_cu_binary_flag allow the decoder to derive the multi-type tree splitting mode (MttSplitMode) of the CU based on predefined rules or tables. It should be noted that for certain designs, such as the 64×64 luma block and 32×32 chroma pipeline design in the VVC hardware decoder, TT splitting is not allowed when the width or height of the luma coding block is greater than 64. TT splitting is also not allowed when the width or height of the chroma coding block is greater than 32. The pipeline design divides the image into multiple virtual pipeline data units (VPDUs), each VPDU is defined as a non-overlapping unit in the image. In the hardware decoder, consecutive VPDUs are processed simultaneously in multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it is necessary to keep the VPDU small.In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, the ternary tree (TT) and binary tree (BT) partitioning may increase the VPDU size.

[0155] In addition, it should be noted that when a part of the tree node block exceeds the bottom or the right boundary of the image, the tree node block is forcibly divided until all the pixels of each coding CU are located within the image boundary.

[0156] For example, the intra sub-partitions (ISP) tool may divide the luma intra prediction block vertically or horizontally into two or four sub-partitions according to the block size.

[0157] In one example, mode select unit 260 of video encoder 20 may be used to perform any combination of the segmentation techniques described above.

[0158] As described above, the video encoder 20 is operable to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, an intra prediction mode and / or an inter prediction mode.

[0159] Intra-frame prediction

[0160] The intra prediction mode set may include 35 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and plane mode, or directional modes as defined in HEVC, or may include 67 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and plane mode, or directional modes as defined in VVC. For example, several traditional angle intra prediction modes are adaptively replaced with wide-angle intra prediction modes for non-square blocks defined in VVC. For another example, in order to avoid the division operation of DC prediction, only the longer side is used to calculate the average value of non-square blocks. In addition, the intra prediction result of the plane mode can also be modified using the position-dependent intra prediction combination (PDPC) method.

[0161] The intra prediction unit 254 is configured to generate an intra prediction block 265 using reconstructed pixels of adjacent blocks of the same current image according to an intra prediction mode in the intra prediction mode set.

[0162] The intra-frame prediction unit 254 (or generally the mode selection unit 260) is also used to output intra-frame prediction parameters (or generally information indicating the selected intra-frame prediction mode of the block) in the form of syntax elements 266 to the entropy coding unit 270 for inclusion in the encoded image data 21, so that the video decoder 30 can perform operations, such as receiving and using the prediction parameters for decoding.

[0163] The intra prediction modes in HEVC include a DC prediction mode, a planar prediction mode and 33 angular prediction modes, with a total of 35 candidate prediction modes. Figure 3 It is a schematic diagram of HEVC intra-frame prediction direction, such as Figure 3 As shown, the current block can use the pixels of the reconstructed image blocks on the left and above as references for intra-frame prediction. The image blocks in the surrounding area of ​​the current block used for intra-frame prediction of the current block are called reference blocks, and the pixels in the reference blocks are called reference pixels. Among the 35 candidate prediction modes, the DC prediction mode is applicable to the area with flat texture in the current block. All pixels in this area use the average value of the reference pixels in the reference block as prediction; the plane prediction mode is applicable to image blocks with smoothly changing textures. The current block that meets this condition uses the reference pixels in the reference block for bilinear interpolation as the prediction of all pixels in the current block; the angle prediction mode uses the characteristic that the texture of the current block is highly correlated with the texture of the adjacent reconstructed image block, and copies the value of the reference pixel in the corresponding reference block along a certain angle as the prediction of all pixels in the current block.

[0164] The HEVC encoder gives the current block from Figure 3 An optimal intra-frame prediction mode is selected from the 35 candidate prediction modes shown, and the optimal intra-frame prediction mode is written into the video bitstream. To improve the coding efficiency of intra-frame prediction, the encoder / decoder will derive three most likely modes from the optimal intra-frame prediction modes of the reconstructed image blocks in the surrounding area using intra-frame prediction. If the optimal intra-frame prediction mode selected for the current block is one of the three most likely modes, a first index is encoded to indicate that the selected optimal intra-frame prediction mode is one of the three most likely modes; if the selected optimal intra-frame prediction mode is not one of the three most likely modes, a second index is encoded to indicate that the selected optimal intra-frame prediction mode is one of the other 32 modes (other modes among the 35 candidate prediction modes except the aforementioned three most likely modes). The HEVC standard uses a 5-bit fixed-length code as the aforementioned second index.

[0165] The method for the HEVC encoder to derive the three most likely modes includes: selecting the optimal intra-frame prediction mode of the left adjacent image block and the upper adjacent image block of the current block and putting them into a set. If the two optimal intra-frame prediction modes are the same, only one is retained in the set. If the two optimal intra-frame prediction modes are the same and both are angle prediction modes, two angle prediction modes adjacent to the angle direction are selected to join the set; otherwise, the plane prediction mode, the DC mode and the vertical prediction mode are selected in turn to join the set until the number of modes in the set reaches 3.

[0166] After the HEVC decoder performs entropy decoding on the bitstream, it obtains the mode information of the current block, which includes an indication flag indicating whether the optimal intra-frame prediction mode of the current block is among the three most likely modes, and the index of the optimal intra-frame prediction mode of the current block among the three most likely modes or the index of the optimal intra-frame prediction mode of the current block among the other 32 modes.

[0167] Inter prediction

[0168] In a possible implementation, the set of inter-frame prediction modes depends on the available reference image (i.e., at least part of the previously decoded image stored in DBP230 as mentioned above) and other inter-frame prediction parameters, for example, on whether to use the entire reference image or only a part of the reference image, such as a search window area near the area of ​​the current block, to search for the best matching reference block, and / or on whether to perform pixel interpolation such as half-pixel, quarter-pixel and / or 1 / 16 interpolation, for example.

[0169] In addition to the above prediction modes, skip mode and / or direct mode may also be employed.

[0170] For example, extended merge prediction, the merge candidate list of this mode consists of the following five candidate types in order: spatial MVP from spatially adjacent CUs, temporal MVP from collocated CUs, history-based MVP from FIFO tables, pairwise average MVP, and zero MV. Decoder side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of the MV of the merge mode. Merge mode with MVD (MMVD) is derived from the merge mode with motion vector difference. The MMVD flag is sent immediately after the skip flag and merge flag are sent to specify whether the CU uses the MMVD mode. The CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding the MVD of the CU with different precisions. The MVD of the current CU is adaptively selected according to the prediction mode of the current CU. When the CU is encoded in merge mode, the combined inter / intra prediction (CIIP) mode can be applied to the current CU. The CIIP prediction is obtained by weighted averaging the inter and intra prediction signals. For affine motion compensated prediction, the affine motion field of the block is described by the motion information of 2 control points (4 parameters) or 3 control points (6 parameters) motion vectors. Subblock-based temporal motion vector prediction (SbTMVP) is similar to the temporal motion vector prediction (TMVP) in HEVC, but the motion vector of the sub-CU within the current CU is predicted. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces calculations, especially in terms of the number of multiplications and the size of the multipliers. In triangle partitioning mode, the CU is evenly divided into two triangular parts in two ways: diagonal partitioning and anti-diagonal partitioning. In addition, the bidirectional prediction mode is extended on the basis of simple averaging to support weighted averaging of two prediction signals.

[0171] The inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in Figure 2). The motion estimation unit may be configured to receive or acquire the image block 203 (the current image block 203 of the current image 17) and the decoded image 231, or at least one or more previously reconstructed blocks, for example, reconstructed blocks of one or more other / different previously decoded images 231, for motion estimation. For example, a video sequence may include the current image and the previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or form a sequence of images forming the video sequence.

[0172] For example, the encoder 20 may be configured to select a reference block from a plurality of reference blocks of the same or different images in a plurality of other images, and provide the reference image (or reference image index) and / or the offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. The offset is also referred to as a motion vector (MV).

[0173] The motion compensation unit is used to obtain, for example, receive, inter-frame prediction parameters, and perform inter-frame prediction based on or using the inter-frame prediction parameters to obtain an inter-frame prediction block 246. The motion compensation performed by the motion compensation unit may include extracting or generating a prediction block based on a motion / block vector determined by motion estimation, and may also include performing interpolation with sub-pixel accuracy. Interpolation filtering can generate pixel points of other pixels from pixel points of known pixels, thereby potentially increasing the number of candidate prediction blocks that can be used to encode the image block. Once a motion vector corresponding to a PU of a current image block is received, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0174] The motion compensation unit may also generate syntax elements associated with blocks and video slices for use by the video decoder 30 when decoding image blocks of the video slice. In addition, or as an alternative to slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be generated or used.

[0175] In the process of obtaining a candidate motion vector list in an advanced motion vector prediction (AMVP) mode, motion vectors (MVs) that can be added to the candidate motion vector list as alternatives include MVs of image blocks that are spatially adjacent and temporally adjacent to the current block, wherein the MVs of the image blocks that are spatially adjacent may include the MV of a left candidate image block located on the left side of the current block and the MV of an upper candidate image block located above the current block. Exemplarily, Figure 7 is an exemplary schematic diagram of a candidate image block in an embodiment of the present application, such as Figure 7As shown, the set of left candidate image blocks includes {A0, A1}, the set of upper candidate image blocks includes {B0, B1, B2}, and the set of temporally adjacent candidate image blocks includes {C, T}. All three sets can be added to the candidate motion vector list as alternatives. However, according to the existing coding standard, the maximum length of the candidate motion vector list of AMVP is 2, so it is necessary to determine the MV of adding up to two image blocks to the candidate motion vector list from the three sets according to the prescribed order. The order can be to give priority to the set of left candidate image blocks {A0, A1} of the current block (consider A0 first, and then consider A1 if A0 is not available), then consider the set of upper candidate image blocks {B0, B1, B2} of the current block (consider B0 first, and then consider B1 if B0 is not available, and then consider B2 if B1 is not available), and finally consider the set of temporally adjacent candidate image blocks {C, T} of the current block (consider T first, and then consider C if T is not available).

[0176] After obtaining the above candidate motion vector list, the optimal MV is determined from the candidate motion vector list by the rate distortion cost (RDcost), and the candidate motion vector with the smallest RD cost is used as the motion vector predictor (MVP) of the current block. The rate distortion cost is calculated by the following formula:

[0177] J=SAD+λR

[0178] Wherein, J represents RD cost, SAD is the sum of absolute differences (SAD) between the pixel value of the predicted block obtained after motion estimation using the candidate motion vector and the pixel value of the current block, R represents the bit rate, and λ represents the Lagrange multiplier.

[0179] The encoder passes the index of the determined MVP in the candidate motion vector list to the decoder. Furthermore, a motion search can be performed in the neighborhood centered on the MVP to obtain the actual motion vector of the current block. The encoder calculates the motion vector difference (MVD) between the MVP and the actual motion vector, and also passes the MVD to the decoder. The decoder parses the index, finds the corresponding MVP in the candidate motion vector list based on the index, parses the MVD, and adds the MVD to the MVP to obtain the actual motion vector of the current block.

[0180] In the process of obtaining the candidate motion information list in the merging mode, the motion information that can be added to the candidate motion information list as an alternative includes the motion information of the image blocks that are spatially adjacent or temporally adjacent to the current block, wherein the image blocks that are spatially adjacent and the image blocks that are temporally adjacent can refer to Figure 7 , the candidate motion information corresponding to the spatial domain in the candidate motion information list comes from the five spatially adjacent blocks (A0, A1, B0, B1 and B2). If the spatially adjacent block is not available or is intra-predicted, its motion information will not be added to the candidate motion information list. The candidate motion information of the current block in the time domain is obtained by scaling the MV of the block at the corresponding position in the reference frame according to the picture order count (POC) of the reference frame and the current frame. First, determine whether the block at position T in the reference frame is available. If not, select the block at position C. After obtaining the above candidate motion information list, the optimal motion information is determined from the candidate motion information list through the RD cost as the motion information of the current block. The encoder passes the index value (denoted as mergeindex) of the position of the optimal motion information in the candidate motion information list to the decoder.

[0181] Entropy Coding

[0182] The entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC scheme (CALVC), an arithmetic coding scheme, a binarization algorithm, a context adaptive binary arithmetic coding (CABAC), a syntax-based context-adaptive binary arithmetic coding (SBAC), a probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements to obtain coded image data 21 that can be output in the form of a coded bit stream 21 through an output terminal 272, so that the video decoder 30, etc. can receive and use the parameters for decoding. The coded bit stream 21 can be transmitted to the video decoder 30, or it can be stored in a memory for later transmission or retrieval by the video decoder 30.

[0183] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform based encoder 20 may directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another implementation, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0184] Decoder and decoding method

[0185] Figure 3 3 is an exemplary block diagram of a video decoder 30 according to an embodiment of the present application. The video decoder 30 is used to receive, for example, coded image data 21 (for example, coded bitstream 21) encoded by an encoder 20, and obtain a decoded image 331. The coded image data or bitstream includes information for decoding the coded image data, such as data representing image blocks of a coded video slice (and / or a coding block group or coding block) and related syntax elements.

[0186] exist Figure 3 In the example of FIG. 3 , decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. Inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same as reference Figure 2 The decoding process of the video encoder 100 is the reverse of the encoding process described above.

[0187] See also Figure 3 The inter-frame prediction unit includes a trained target model (also called a neural network) for processing an input image or an image region or an image block to generate a prediction value of the input image block. For example, the neural network for inter-frame prediction is used to receive an input image or an image region or an image block and generate a prediction value of the input image or an image region or an image block. Figure 6a-6e Describe the neural network used for inter-frame prediction in detail.

[0188] As described for encoder 20, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer DPB 230, inter prediction unit 344, and intra prediction unit 354 also constitute a “built-in decoder” of video encoder 20. Accordingly, inverse quantization unit 310 may be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 may be functionally identical to inverse transform processing unit 122, reconstruction unit 314 may be functionally identical to reconstruction unit 214, loop filter 320 may be functionally identical to loop filter 220, and decoded picture buffer 330 may be functionally identical to decoded picture buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 applies accordingly to the corresponding units and functions of video decoder 30.

[0189] Entropy decoding

[0190] The entropy decoding unit 304 is used to parse the bit stream 21 (or generally the encoded image data 21) and perform entropy decoding on the encoded image data 21 to obtain the quantization coefficients 309 and / or the decoded encoding parameters ( Figure 3 The entropy decoding unit 304 may be used to provide the inter-frame prediction parameters, intra-frame prediction parameters and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice and / or video block level. In addition, or as an alternative to slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be received or used.

[0191] Dequantization

[0192] The inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or generally information related to inverse quantization) and a quantization coefficient from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and inverse quantize the decoded quantization coefficient 309 based on the quantization parameter to obtain an inverse quantization coefficient 311, which may also be referred to as a transform coefficient 311. The inverse quantization process may include using the quantization parameter calculated by the video encoder 20 for each video block in the video slice to determine a degree of quantization, and also determine a degree of inverse quantization that needs to be performed.

[0193] Inverse Transform

[0194] The inverse transform processing unit 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain a reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 313. The transform may be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may also be configured to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform applied to the dequantized coefficients 311.

[0195] reconstruction

[0196] The reconstruction unit 314 (eg, the summer 314 ) is used to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 313 and the pixel point values ​​of the prediction block 365 .

[0197] Filtering

[0198] The loop filter unit 320 (in or after the encoding loop) is used to filter the reconstruction block 315 to obtain a filter block 321, so as to smoothly perform pixel conversion or improve video quality, etc. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 3 3. Although shown as a loop filter in FIG. 3, in other configurations, the loop filter unit 320 may be implemented as a post-loop filter.

[0199] Decoded Image Buffer

[0200] The decoded video blocks 321 in one picture are then stored in a decoded picture buffer 330 which stores the decoded pictures 331 as reference pictures for subsequent motion compensation of other pictures and / or for respective output display.

[0201] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., to display it to the user or for the user to view it.

[0202] predict

[0203] The inter-frame prediction unit 344 may be functionally the same as the inter-frame prediction unit 244 (particularly the motion compensation unit), and the intra-frame prediction unit 354 may be functionally the same as the inter-frame prediction unit 254, and may determine the division or segmentation and perform prediction based on the segmentation and / or prediction parameters or corresponding information received from the coded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304). The mode application unit 360 may be used to perform prediction (intra-frame or inter-frame prediction) of each block according to the reconstructed image, block or corresponding pixel point (filtered or unfiltered), and obtain a prediction block 365.

[0204] When the video slice is encoded as an intra coded (I) slice, the intra prediction unit 354 in the mode application unit 360 is used to generate a prediction block 365 for the image block of the current video slice based on the indicated intra prediction mode and data from the previously decoded block of the current image. When the video image is encoded as an inter coded (i.e., B or P) slice, the inter prediction unit 344 (e.g., motion compensation unit) in the mode application unit 360 is used to generate a prediction block 365 for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 304. For inter prediction, these prediction blocks can be generated from one of the reference images in one of the reference image lists. The video decoder 30 can construct reference frame list 0 and list 1 using a default construction technique based on the reference images stored in the DPB 330. The same or similar processes may be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks) in addition to or as an alternative to slices (e.g., video slices), for example, a video may be encoded using I, P or B coding block groups and / or coding blocks.

[0205] The mode application unit 360 is used to determine prediction information for a video block of a current video slice by parsing motion vectors and other syntax elements, and use the prediction information to generate a prediction block for the current video block being decoded. For example, the mode application unit 360 uses some of the received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction) for encoding the video block of the video slice, the inter-frame prediction slice type (e.g., B slice, P slice, or GPB slice), the construction information of one or more reference picture lists for the slice, the motion vector for each inter-frame coded video block of the slice, the inter-frame prediction state for each inter-frame coded video block of the slice, and other information to decode the video block in the current video slice. In addition to slices (e.g., video slices) or as an alternative to slices, the same or similar process can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), for example, the video can be encoded using I, P, or B coding block groups and / or coding blocks.

[0206] In one embodiment, Figure 3 The video encoder 30 may also be configured to partition and / or decode an image using slices (also referred to as video slices), wherein an image may be partitioned or decoded using one or more slices (usually non-overlapping). Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coding blocks in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0207] In one embodiment, Figure 3 The video decoder 30 shown can also be used to segment and / or decode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), wherein an image can be segmented or decoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTU) or one or more coding blocks, etc., wherein each coding block may be in a rectangular shape, etc., and may include one or more complete or partial blocks (e.g., CTU).

[0208] Other variations of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, a non-transform based decoder 30 may directly dequantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, the video decoder 30 may have the dequantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0209] It should be understood that the processing result of the current step can be further processed in the encoder 20 and the decoder 30 and then output to the next step. For example, after interpolation filtering, motion vector derivation or loop filtering, the processing result of interpolation filtering, motion vector derivation or loop filtering can be further operated, such as clipping or shifting operation.

[0210] It should be noted that the derived motion vector of the current block (including but not limited to the control point motion vector of the affine mode, the sub-block motion vector of the affine, plane, ATMVP mode, the temporal motion vector, etc.) can be further operated. For example, the value of the motion vector is limited to a predefined range according to the representation bit of the motion vector. If the representation bit of the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents the power. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of 4 4×4 sub-blocks in an 8×8 block) is limited so that the maximum difference between the integer parts of the 4 4×4 sub-block MVs does not exceed N pixels, for example, not more than 1 pixel. Two methods of limiting motion vectors according to bitDepth are provided here.

[0211] Although the above embodiments mainly describe video coding and decoding, it should be noted that the embodiments of the decoding system 10, the encoder 20 and the decoder 30 and other embodiments described herein can also be used for still image processing or coding and decoding, that is, the processing or coding and decoding of a single image in video coding and decoding that is independent of any previous or consecutive images. In general, if the image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and the inter-frame prediction unit 344 (decoder) may not be available. All other functions (also called tools or techniques) of the video encoder 20 and the video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transformation 206, quantization 208, inverse quantization 210 / 310, (inverse) transformation 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270 and entropy decoding 304.

[0212] Figure 4 4 is an exemplary block diagram of a video decoding device 400 according to an embodiment of the present application. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder, such as Figure 1a The video decoder 30 in the embodiment may also be an encoder, for example Figure 1aThe video encoder 20 in.

[0213] The video decoding device 400 includes: an input port 410 (or input port 410) and a receiving unit (Rx) 420 for receiving data; a processor, a logic unit or a central processing unit (CPU) 430 for processing data; for example, the processor 430 here can be a neural network processor 430; a transmitter unit (Tx) 440 and an output port 450 (or output port 450) for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the input port 410, the receiving unit 420, the transmitting unit 440 and the output port 450 for the output or output of optical or electrical signals.

[0214] The processor 430 is implemented by hardware and software. The processor 430 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the input port 410, the receiving unit 420, the sending unit 440, the output port 450, and the memory 460. The processor 430 includes a decoding module 470 (e.g., a decoding module 470 based on a neural network). The decoding module 470 implements the embodiments disclosed above. For example, the decoding module 470 performs, processes, prepares, or provides various encoding operations. Therefore, the decoding module 470 provides substantial improvements to the functions of the video decoding device 400 and affects the switching of the video decoding device 400 to different states. Alternatively, the decoding module 470 is implemented with instructions stored in the memory 460 and executed by the processor 430.

[0215] The memory 460 includes one or more disks, tape drives, and solid-state hard disks, and can be used as an overflow data storage device for storing such programs when such programs are selected for execution, and for storing instructions and data read during program execution. The memory 460 can be volatile and / or non-volatile, and can be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0216] Figure 5 is an exemplary block diagram of a device 500 according to an embodiment of the present application. The device 500 can be used as Figure 1a Either or both of the source device 12 and the destination device 14 in .

[0217] The processor 502 in the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices that are currently available or will be developed in the future and are capable of manipulating or processing information. Although a single processor such as the processor 502 shown in the figure may be used to implement the disclosed implementation, using more than one processor is faster and more efficient.

[0218] In one implementation, the memory 504 in the apparatus 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 accessed by the processor 502 via a bus 512. The memory 504 may also include an operating system 508 and an application 510, which includes at least one program that allows the processor 502 to perform the methods described herein. For example, the application 510 may include applications 1 to N, and also include a video decoding application that performs the methods described herein.

[0219] The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element that may be used to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0220] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. In addition, auxiliary storage may be directly coupled to other components of device 500 or accessed through a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, device 500 may have a variety of configurations.

[0221] Since the embodiments of the present application involve the application of neural networks, in order to facilitate understanding, some nouns or terms used in the embodiments of the present application are explained below, and these nouns or terms are also considered as part of the content of the invention.

[0222] (1) Neural Network

[0223] A neural network (NN) is a machine learning model. A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0224]

[0225] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0226] (2) Deep Neural Networks

[0227] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0228] (3) Convolutional Neural Network

[0229] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it. A convolutional neural network contains a feature extractor consisting of a convolutional layer and a pooling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve an input image or convolution feature plane (feature map).

[0230] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution on the input signal. A convolutional layer can include many convolution operators, also known as kernels. In image processing, the convolution operator is essentially a weight matrix, which is usually predefined. During the convolution operation on the image, the weight matrix is ​​usually processed horizontally on the input image one pixel after another (or two pixels after two pixels... depending on the value of the stride) to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, instead of using a single weight matrix, multiple weight matrices of the same size (rows × columns) are applied, that is, multiple isotype matrices. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, where the dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise points in the image. The multiple weight matrices have the same size (rows × columns), and the size of the feature maps extracted by the multiple weight matrices of the same size is also the same. The extracted multiple feature maps of the same size are then merged to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network can make correct predictions. When the convolutional neural network has multiple convolutional layers, the initial convolutional layer often extracts more general features, which can also be called low-level features; as the depth of the convolutional neural network deepens, the features extracted by the later convolutional layers become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0231] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. It can be a convolution layer followed by a pooling layer, or multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator can calculate the pixel values ​​in the image within a specific range to produce an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value in the range as the result of maximum pooling within a specific range. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0232] After being processed by the convolution layer / pooling layer, the convolution neural network is not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0233] Optionally, after the multiple hidden layers in the neural network layer, an output layer of the entire convolutional neural network is also included. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the back propagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0234] (4) Recurrent Neural Network

[0235] Recurrent neural networks (RNN) are used to process sequence data. In the traditional neural network model, from the input layer to the hidden layer and then to the output layer, the layers are fully connected, and the nodes within each layer are disconnected. Although this ordinary neural network has solved many difficult problems, it is still powerless for many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. The same error back propagation algorithm is used, but there is a difference: that is, if the RNN is expanded, the parameters, such as W, are shared; this is not the case with the traditional neural network mentioned above. And when using the gradient descent algorithm, the output of each step depends not only on the network at the current step, but also on the state of the network at the previous steps. This learning algorithm is called the Back Propagation Through Time (BPTT).

[0236] Since we already have convolutional neural networks, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, there is a premise that the elements are independent of each other, and the input and output are also independent, such as cats and dogs. But in the real world, many elements are interconnected, such as the change of stocks over time, or someone said: I like traveling, and my favorite place is Yunnan. I must go there if I have the chance in the future. Fill in the blank here, humans should all know to fill in "Yunnan". Because humans will make inferences based on the content of the context, but how can machines do this? RNN came into being. RNN is designed to give machines the ability to remember like humans. Therefore, the output of RNN needs to rely on the current input information and historical memory information.

[0237] (5) Loss Function

[0238] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0239] (6) Back propagation algorithm

[0240] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0241] (7) Generative Adversarial Networks

[0242] Generative adversarial networks (GAN) are a type of deep learning model. The model includes at least two modules: one is the generative model and the other is the discriminative model. These two modules learn from each other through competition to produce better output. Both the generative model and the discriminative model can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: Taking the GAN that generates images as an example, suppose there are two networks, G (Generator) and D (Discriminator), where G is a network that generates images. It receives a random noise z and generates images through this noise, denoted as G(z); D is a discriminative network that is used to determine whether a picture is "real". Its input parameter is x, which represents a picture. The output D(x) represents the probability that x is a real picture. If it is 1, it means that it is 100% a real picture, and if it is 0, it means that it cannot be a real picture. In the process of training the generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from the real images as much as possible. In this way, G and D constitute a dynamic "game" process, which is also the "adversary" in the "generative adversarial network". The final result of the game is that, under ideal conditions, G can generate images G(z) that are "real" enough, while D has difficulty determining whether the images generated by G are real, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate images.

[0243] The following will be combined Figure 6a-6e The target model (also called neural network) for inter-frame prediction is described in detail. Figure 6a-6e Several exemplary architectures of neural networks for inter-frame prediction according to embodiments of the present application.

[0244] like Figure 6a As shown in the figure, the neural network includes, in the order of processing, a 3×3 convolutional layer (3×3Conv), an activation layer (Relu), a block processing layer (Res-Block), ..., a block processing layer, a 3×3 convolutional layer, an activation layer, and a 3×3 convolutional layer. The original matrix of the input neural network is processed by the above layers and then added to the original matrix to obtain the final output matrix.

[0245] like Figure 6bAs shown, the neural network includes, in the order of processing, two 3×3 convolutional layers and activation layers, one block processing layer, ..., block processing layer, 3×3 convolutional layer, activation layer and 3×3 convolutional layer. The first matrix passes through one 3×3 convolutional layer and activation layer, and the second matrix passes through another 3×3 convolutional layer and activation layer. The two processed matrices are combined (contact) and then processed through the block processing layer, ..., block processing layer, 3×3 convolutional layer, activation layer and 3×3 convolutional layer to obtain a matrix, which is then added to the first matrix to obtain the final output matrix.

[0246] like Figure 6c As shown, the neural network includes, in the order of processing, two 3×3 convolutional layers and activation layers, one block processing layer, ..., block processing layer, 3×3 convolutional layer, activation layer and 3×3 convolutional layer. Before inputting into the neural network, the first matrix and the second matrix are multiplied, and then the first matrix passes through one 3×3 convolutional layer and activation layer, and the multiplied matrix passes through another 3×3 convolutional layer and activation layer. The two processed matrices are added and then passed through the block processing layer, ..., block processing layer, 3×3 convolutional layer, activation layer and 3×3 convolutional layer to obtain a matrix, which is then added to the first matrix to obtain the final output matrix.

[0247] like Figure 6d As shown in the figure, the block processing layers include, in order of processing, a 3×3 convolution layer, an activation layer, and a 3×3 convolution layer. After the input matrix is ​​processed by these three layers, the processed matrix is ​​added to the initial input matrix to obtain the output matrix. Figure 6c As shown in FIG. 1 , the block processing layers include, in order of processing, a 3×3 convolution layer, an activation layer, a 3×3 convolution layer, and an activation layer. After the input matrix is ​​processed by the 3×3 convolution layer, the activation layer, and the 3×3 convolution layer, the processed matrix is ​​added to the initial input matrix and then passed through an activation layer to obtain the output matrix.

[0248] It should be noted that if Figure 6a-6e Only several exemplary architectures of the neural network for inter-frame prediction in the embodiments of the present application are shown, which does not constitute a limitation on the neural network architecture. The number of layers, layer structure, addition, multiplication or merging and other processing included in the neural network, as well as the number and size of input and / or output matrices can be determined according to actual conditions, and the present application does not make specific limitations on this.

[0249] Figure 8800 is a flowchart of the process 800 of the inter-frame prediction method according to an embodiment of the present application. The process 800 may be performed by the video encoder 20 or the video decoder 30, and specifically, may be performed by the inter-frame prediction unit 244, 344 of the video encoder 20 or the video decoder 30. The process 800 is described as a series of steps or operations, and it should be understood that the process 800 may be performed in various orders and / or may occur simultaneously, not limited to Figure 8 Assuming that a video data stream having multiple image frames is using a video encoder or a video decoder, a process 800 including the following steps is performed to perform inter-frame prediction on an image or an image block. The process 800 may include:

[0250] Step 801: Obtain motion vectors of multiple reconstructed image blocks in the surrounding area of ​​the current block.

[0251] The surrounding area of ​​the current block includes the spatial neighborhood and / or the temporal neighborhood of the current block, wherein the image blocks of the spatial neighborhood may include the left candidate image block located on the left side of the current block and the upper candidate image block located above the current block. Figure 7 As shown, the set of candidate image blocks on the left includes {A0, A1}, the set of candidate image blocks above includes {B0, B1, B2}, and the set of candidate image blocks adjacent in the time domain includes {C, T}. The reconstructed image block may refer to a coded image block that has been encoded and reconstructed by the encoding end or a decoded image block that has been decoded and reconstructed by the decoding end. The reconstructed image block may also refer to a basic unit image block of a preset size obtained by dividing a coded image block or a decoded image block of equal size. For example, the size of the coded image block or the decoded image block may be 16×16, 64×64 or 32×16, and the size of the basic unit image block may be 4×4 or 8×8.

[0252] The following description is made by taking a certain reconstructed image block as an example. The reconstructed image block may be any one of a plurality of reconstructed image blocks in the surrounding area. The method may be referred to for other reconstructed image blocks.

[0253] The motion vector of the reconstructed image block may include: (1) multiple a posteriori motion vectors of the reconstructed image block, which are determined based on the reconstruction value of the reconstructed image block and the prediction values ​​corresponding to multiple a posteriori candidate motion vectors; or (2) the optimal motion vector of the reconstructed image block, which is the a posteriori motion vector with the largest probability value or the smallest prediction error value among the above multiple a posteriori motion vectors.

[0254] Multiple a priori candidate motion vectors are obtained based on multiple a priori candidate motion vectors of the reconstructed image block. For any one of the multiple a priori candidate motion vectors of the reconstructed image block, it can be offset within a preset search window to generate multiple offset candidate motion vectors. It can be seen that one a priori candidate motion vector of the reconstructed image block can obtain multiple offset candidate motion vectors. The multiple a priori candidate motion vectors of the reconstructed image block are all operated as above, and all the offset candidate motion vectors obtained are the multiple a priori candidate motion vectors of the reconstructed image block. For example, Fig.10 An exemplary schematic diagram of a search window in an embodiment of the present application is shown in FIG. Fig.10 As shown in the figure, assuming that a priori candidate motion vector of the reconstructed image block is (0,0), the priori candidate motion vector is offset within a 3×3 search window, and 9 offset candidate motion vectors can be obtained: (-1,-1), (-1,0), (-1,1), (0,-1), (0,0), (0,1), (1,-1), (1,0), (1,1). The 9 offset candidate motion vectors are multiple a posteriori candidate motion vectors of the reconstructed image block.

[0255] The multiple a posteriori motion vectors of the reconstructed image block may refer to the multiple a posteriori candidate motion vectors mentioned above; or may refer to some motion vectors among the multiple a posteriori candidate motion vectors, for example, multiple designated motion vectors selected from the multiple a posteriori candidate motion vectors mentioned above.

[0256] The probability values ​​or prediction error values ​​of multiple posterior motion vectors may be described below.

[0257] In a possible implementation, in addition to obtaining the motion vector of the reconstructed image block, related information of the reconstructed image block may also be obtained. The related information and the method for obtaining the related information are as follows:

[0258] 1. Multiple prediction error values ​​corresponding to multiple a posteriori motion vectors of the reconstructed image block, wherein the multiple prediction error values ​​are also determined based on the reconstruction value of the reconstructed image block and the prediction values ​​corresponding to the multiple a posteriori candidate motion vectors.

[0259] By performing motion compensation on the reconstructed image blocks according to the multiple a posteriori candidate motion vectors, multiple prediction values ​​can be obtained, and the multiple prediction values ​​correspond to the multiple a posteriori candidate motion vectors.

[0260] The multiple prediction values ​​are compared with the reconstruction values ​​of the reconstructed image blocks to obtain multiple prediction error values, which correspond to multiple a posteriori candidate motion vectors. The present application can use the sum of absolute difference (SAD) or the sum of squared difference (SSE) to obtain the prediction error value corresponding to a certain a posteriori candidate motion vector.

[0261] If the multiple a posteriori motion vectors of the reconstructed image block refer to the above-mentioned multiple a posteriori candidate motion vectors, then the multiple prediction error values ​​corresponding to the above-mentioned multiple a posteriori motion vectors of the reconstructed image block refer to the multiple prediction error values ​​corresponding to the above-mentioned multiple a posteriori candidate motion vectors; if the multiple a posteriori motion vectors of the reconstructed image block refer to some motion vectors among the above-mentioned multiple a posteriori candidate motion vectors, then the multiple prediction error values ​​corresponding to the above-mentioned multiple a posteriori motion vectors of the reconstructed image block refer to the prediction error values ​​corresponding to the partial motion vectors selected from the multiple prediction error values ​​corresponding to the above-mentioned multiple a posteriori candidate motion vectors.

[0262] 2. Multiple probability values ​​corresponding to multiple a posteriori motion vectors of the reconstructed image block, wherein the multiple probability values ​​are also determined based on the reconstruction value of the reconstructed image block and the prediction values ​​corresponding to the multiple a posteriori candidate motion vectors.

[0263] There are two methods for obtaining multiple probability values ​​corresponding to multiple posterior motion vectors of the reconstructed image block:

[0264] One method is to obtain multiple probability values ​​of the reconstructed image block based on the multiple prediction error values ​​of the reconstructed image block obtained in the first method. For example, the multiple prediction error values ​​of the reconstructed image block can be normalized using a normalized exponential function, a linear normalization method, or the like to obtain normalized values ​​of the multiple prediction error values, which are the multiple probability values ​​of the reconstructed image block. Based on the correspondence between the multiple prediction error values ​​of the reconstructed image block and the multiple posterior motion vectors, the multiple probability values ​​of the reconstructed image block also correspond to the multiple posterior motion vectors of the reconstructed image block, and the probability value can represent the probability that the corresponding posterior motion vector becomes the optimal motion vector of the reconstructed image block.

[0265] Another method is to input the reconstruction value of the reconstructed image block and the multiple predicted values ​​of the reconstructed image block obtained in the first method into a trained neural network to obtain multiple probability values ​​of the reconstructed image block corresponding to multiple posterior motion vectors. The neural network can refer to the description of the training engine 25 above, which will not be repeated here.

[0266] Therefore, after obtaining multiple prediction error values ​​or probability values ​​corresponding to multiple posterior motion vectors through the above two methods, the optimal motion vector of the reconstructed image block can be obtained by the following two methods:

[0267] One is to use the a posteriori motion vector corresponding to the minimum prediction error value among multiple prediction error values ​​corresponding to multiple a posteriori motion vectors as the optimal motion vector of the reconstructed image block.

[0268] The other is to use the posterior motion vector corresponding to the maximum probability value among multiple probability values ​​corresponding to multiple posterior motion vectors as the optimal motion vector of the reconstructed image block.

[0269] In the present application, the above-mentioned motion vector and related information of the reconstructed image block can be directly read from the memory. After the reconstructed image block is encoded or decoded, the above-mentioned method can be used to immediately obtain the motion vector or motion vector and related information of the reconstructed image block, and then store it. When performing inter-frame prediction on the subsequent image block (current block), it can be directly read from the corresponding position of the memory. In this way, the inter-frame prediction efficiency of the current block can be improved.

[0270] In the present application, the motion vector or the motion vector and related information of the reconstructed image block can also be calculated when the current block is inter-frame predicted, that is, when the current block is inter-frame predicted, the motion vector or the motion vector and related information of the reconstructed image block can be obtained by the above method. In this way, the calculation is performed only after determining which reconstructed image block is needed, which can save storage space.

[0271] If the above-mentioned multiple reconstructed image blocks all adopt inter-frame prediction during the encoding or decoding process, the above-mentioned method can be used to obtain the motion vectors or motion vectors and related information of the multiple reconstructed image blocks. If some of the multiple reconstructed image blocks do not adopt inter-frame prediction during the encoding or decoding process, the motion vectors or motion vectors and related information of the part of the image blocks can also be obtained according to any one of the methods described in the above three situations.

[0272] If the reconstructed image block includes multiple basic unit image blocks, the motion vector or the motion vector and related information of the reconstructed image block can be used as the motion vector or the motion vector and related information of all the basic unit image blocks included in it. Further, the motion vector or the motion vector and related information of the reconstructed image block can be used as the motion vector or the motion vector and related information of all the pixels included in it.

[0273] Step 802: Obtain multiple a priori candidate motion vectors of the current block and multiple probability values ​​corresponding to the multiple a priori candidate motion vectors according to the motion vectors of the multiple reconstructed image blocks.

[0274] The multiple prior candidate motion vectors of the current block may refer to all motion vectors remaining after deduplication of multiple a posteriori motion vectors of multiple reconstructed image blocks, or may refer to some motion vectors among all motion vectors remaining after deduplication of multiple a posteriori motion vectors of multiple reconstructed image blocks.

[0275] The motion vectors of the reconstructed image blocks can be input into the trained neural network to obtain multiple a priori candidate motion vectors of the current block and multiple probability values ​​corresponding to the multiple a priori candidate motion vectors. The neural network can refer to the description of the training engine 25 above, which will not be repeated here.

[0276] Optionally, multiple posterior motion vectors of multiple reconstructed image blocks and multiple prediction error values ​​corresponding to the multiple posterior motion vectors can be input into a trained neural network to obtain multiple prior candidate motion vectors of the current block and multiple probability values ​​corresponding to the multiple prior candidate motion vectors.

[0277] Optionally, multiple posterior motion vectors of multiple reconstructed image blocks and multiple probability values ​​corresponding to the multiple posterior motion vectors can be input into a trained neural network to obtain multiple prior candidate motion vectors of the current block and multiple probability values ​​corresponding to the multiple prior candidate motion vectors.

[0278] Optionally, the optimal motion vectors of multiple reconstructed image blocks may be input into a trained neural network to obtain multiple a priori candidate motion vectors of the current block and multiple probability values ​​corresponding to the multiple a priori candidate motion vectors.

[0279] Step 803: Obtain a plurality of weight factors corresponding to the plurality of a priori candidate motion vectors according to a plurality of probability values ​​corresponding to the plurality of a priori candidate motion vectors.

[0280] When the sum of multiple probability values ​​is 1, the probability value corresponding to the first prior candidate motion vector is used as the weight factor corresponding to the first prior candidate motion vector. That is, the weight factors of the multiple prior candidate motion vectors are the probability values ​​of the multiple prior candidate motion vectors; or, when the sum of multiple probability values ​​is not 1, the multiple probability values ​​are normalized; the normalized value of the probability value corresponding to the first prior candidate motion vector is used as the weight factor corresponding to the first prior candidate motion vector. That is, the weight factors of the multiple prior candidate motion vectors are the normalized values ​​of the probability values ​​of the multiple prior candidate motion vectors. The above-mentioned first prior candidate motion vector is any one of the multiple prior candidate motion vectors. It can be seen that the sum of the multiple weight factors corresponding to the multiple prior candidate motion vectors is 1.

[0281] Step 804: Perform motion compensation according to the multiple prior candidate motion vectors to obtain multiple prediction values.

[0282] According to the principle of inter-frame prediction, a candidate motion vector can find a reference block in the reference frame of the current block, and the current block is inter-frame predicted based on the reference block to obtain the prediction value corresponding to the candidate motion vector. It can be seen that the prediction value of the current block corresponds to the candidate motion vector. Therefore, by performing motion compensation based on multiple prior candidate motion vectors, multiple prediction values ​​of the current block can be obtained.

[0283] Step 805: Obtain a prediction value of the current block by weighted summing multiple weight factors and multiple prediction values.

[0284] The weight factor corresponding to the same a priori candidate motion vector and the prediction value are multiplied, and then multiple products corresponding to multiple a priori candidate motion vectors are added to obtain the prediction value of the current block.

[0285] The present application obtains multiple weight factors and multiple prediction values ​​of the current block based on the motion vectors of multiple reconstructed image blocks in the surrounding area of ​​the current block, multiplies the weight factors and prediction values ​​corresponding to the same prior candidate motion vector, and then adds multiple products corresponding to multiple prior candidate motion vectors to obtain the prediction value of the current block. The prediction value of the current block obtained in this way combines multiple prior candidate motion vectors, so that it can better fit the rich and varied textures in the real world, improve the accuracy of inter-frame prediction, reduce the error of inter-frame prediction, and improve the overall rate-distortion optimization (RDO) efficiency of inter-frame prediction.

[0286] In a possible implementation, after the reconstruction value of the current block is obtained, the motion vector of the current block and related information thereof may be obtained immediately. The motion vector and related information thereof may refer to step 801, and the obtaining method thereof includes:

[0287] 1. Obtain multiple a posteriori motion vectors of the current block and multiple prediction error values ​​corresponding to the multiple a posteriori motion vectors according to the reconstructed value of the current block and the prediction values ​​corresponding to the multiple a posteriori candidate motion vectors of the current block. The multiple a posteriori motion vectors of the current block are obtained according to the multiple a priori candidate motion vectors of the current block.

[0288] 2. Input the reconstructed value of the current block and the predicted value corresponding to the multiple posterior candidate motion vectors of the current block into the neural network to obtain multiple posterior motion vectors of the current block and multiple probability values ​​corresponding to the multiple posterior motion vectors. The multiple posterior motion vectors of the current block are obtained based on the multiple a priori candidate motion vectors of the current block, or, the multiple probability values ​​corresponding to the multiple posterior motion vectors of the current block are obtained based on the multiple prediction error values ​​of the current block.

[0289] 3. Determine the posterior motion vector with the largest probability value or the smallest prediction error value among the multiple posterior motion vectors of the current block as the optimal motion vector of the current block.

[0290] In a possible implementation, the multiple probability values ​​of the current block include M probability values, and the M probability values ​​are all greater than other probability values ​​except the M probability values ​​in the multiple probability values ​​of the current block. Therefore, M prior candidate motion vectors corresponding to the M probability values ​​can be selected from the multiple prior candidate motion vectors of the current block, and then M weight factors are obtained according to the M probability values, and motion compensation is performed respectively according to the M prior candidate motion vectors to obtain M prediction values ​​of the current block, and finally the prediction value of the current block is obtained by weighted summation according to the M weight factors and the M prediction values. That is, the first M probability values ​​with the largest probability values ​​are selected from the multiple probability values ​​corresponding to the multiple prior candidate motion vectors of the current block, and the M prior candidate motion vectors corresponding to the M probability values ​​are selected from the multiple prior candidate motion vectors of the current block, and the weight factors and prediction values ​​are calculated based on the M probability values ​​and the M prior candidate motion vectors, so as to obtain the prediction value of the current block. Among the multiple probability values ​​corresponding to the multiple prior candidate motion vectors, the remaining probability values ​​except the aforementioned M probability values ​​can be ignored due to their small values, which can reduce the amount of calculation and improve the efficiency of inter-frame prediction.

[0291] Several specific embodiments are used below. Figure 8 The technical solution of the method embodiment shown is described in detail.

[0292] Embodiment 1

[0293] In this embodiment, multiple a priori candidate motion vectors of the current block and multiple probability values ​​corresponding to the multiple a priori candidate motion vectors are determined based on multiple a posteriori motion vectors of each of the multiple reconstructed image blocks in the surrounding area and multiple prediction error values ​​corresponding to the multiple a priori motion vectors.

[0294] Fig. 9 Flow chart of process 900 of the inter-frame prediction method of the embodiment of the present application. Process 900 may be performed by the video encoder 20 or the video decoder 30, and specifically, may be performed by the inter-frame prediction unit 244, 344 of the video encoder 20 or the video decoder 30. Process 900 is described as a series of steps or operations, and it should be understood that process 900 may be performed in various orders and / or occur simultaneously, not limited to Fig. 9 Assuming that a video data stream having multiple image frames is using a video encoder or a video decoder, a process 900 including the following steps is performed to perform inter-frame prediction on an image or an image block. The process 900 may include:

[0295] Step 901: Obtain a plurality of posterior motion vectors of a plurality of reconstructed image blocks in a peripheral region and a plurality of prediction error values ​​corresponding to the plurality of posterior motion vectors.

[0296] The following is an explanation using a certain reconstructed image block as an example. The reconstructed image block may be any one of a plurality of reconstructed image blocks in the surrounding area. Other reconstructed image blocks may refer to this method to obtain a plurality of posterior motion vectors and a plurality of prediction error values ​​corresponding to the plurality of posterior motion vectors.

[0297] There are N4 a posteriori candidate motion vectors for the reconstructed image block, and the N4 a posteriori candidate motion vectors are obtained based on multiple a priori candidate motion vectors of the reconstructed image block. The acquisition method can refer to the description of step 801 above. Motion compensation is performed according to the N4 a posteriori candidate motion vectors respectively, and N4 prediction values ​​of the reconstructed image block can be obtained. The N4 prediction values ​​correspond to the N4 a posteriori candidate motion vectors, that is, inter-frame prediction is performed on the reconstructed image block according to a reference block corresponding to a posterior candidate motion vector, and a prediction value of the reconstructed image block can be obtained. The N4 prediction values ​​are compared with the reconstruction value of the reconstructed image block respectively to obtain N4 prediction error values ​​of the reconstructed image block, and the N4 prediction error values ​​correspond to the N4 a posteriori candidate motion vectors. The present application can use methods such as SAD or SSE to obtain the prediction error value of the reconstructed image block corresponding to a certain a posteriori candidate motion vector.

[0298] The N2 a posteriori motion vectors of the reconstructed image block may refer to the N4 a posteriori candidate motion vectors; or may refer to some motion vectors among the N4 a posteriori candidate motion vectors, for example, multiple specified motion vectors selected from the N4 a posteriori candidate motion vectors.

[0299] Correspondingly, the number of prediction error values ​​corresponding to N2 posterior motion vectors of the reconstructed image block is also N2.

[0300] All the posterior motion vectors of multiple reconstructed image blocks can be expressed as a two-dimensional matrix of N2×Q, where N2 is the number of posterior motion vectors and Q is the number of reconstructed image blocks, and the elements are expressed as k=0, 1, ..., Q-1, represents the index of the reconstructed image block, n=0, 1, ..., N2-1, represents the index of the posterior motion vector, which means the posterior motion vector indicated by n of the reconstructed image block indicated by k.

[0301] The total prediction error values ​​of multiple reconstructed image blocks can also be expressed as a two-dimensional matrix of N2×Q, where the elements are expressed as k=0, 1, ..., Q-1, represents the index of the reconstructed image block, n=0, 1, ..., N2-1, represents the index of the posterior motion vector, which means the prediction error value corresponding to the posterior motion vector indicated by n of the reconstructed image block indicated by k.

[0302] Step 902: Obtain multiple a priori candidate motion vectors of the current block and multiple prediction error values ​​corresponding to the multiple a priori candidate motion vectors according to the multiple a priori motion vectors of the multiple reconstructed image blocks and the multiple prediction error values ​​corresponding to the multiple a priori candidate motion vectors.

[0303] In the present application, all prediction error values ​​and all a posteriori motion vectors of multiple reconstructed image blocks, i.e., the above two N2×Q two-dimensional matrices, can be input into a trained neural network, and the neural network can output multiple a priori candidate motion vectors of the current block and multiple prediction error values ​​corresponding to the multiple a priori candidate motion vectors. The neural network can refer to the description of the above training engine 25, which will not be repeated here.

[0304] The multiple prior candidate motion vectors of the current block can be represented as a two-dimensional matrix of N1×S, where N1 is the number of prior candidate motion vectors of the current block, and S is the number of basic unit image blocks or pixels contained in the current block. If the current block is not further divided, S=1. The elements in the matrix are represented as l=0, 1, ..., S-1, represents the index of the basic unit image block or pixel, n=0, 1, ..., N1-1, represents the index of the prior candidate motion vector, which means the prior candidate motion vector indicated by n of the basic unit image block or pixel indicated by l.

[0305] The multiple prediction error values ​​corresponding to the multiple prior candidate motion vectors of the current block can also be expressed as a two-dimensional matrix of N1×S. The elements in the matrix are expressed as l=0, 1, ..., S-1, represents the index of the basic unit image block or pixel, n=0, 1, ..., N1-1, represents the index of the prior candidate motion vector, which means the probability that the prior candidate motion vector indicated by n of the basic unit image block or pixel indicated by l becomes the optimal motion vector of the basic unit image block or pixel.

[0306] Optionally, if l remains unchanged, That is, the sum of the N1 probability values ​​corresponding to the N1 prior candidate motion vectors of the basic unit image block or pixel indicated by l is 1. Alternatively, Using integer expression, we can get 256 and The binary bits of the integer value are related to the number of bits, which represents The integer value of is represented by 8 bits, so It can also be equal to 128 or 512, etc.

[0307] Step 903: Obtain multiple weight factors corresponding to the multiple priori candidate motion vectors according to the multiple prediction error values ​​corresponding to the multiple priori candidate motion vectors of the current block.

[0308] The multiple weight factors corresponding to the multiple prior candidate motion vectors of the current block can also be expressed as a two-dimensional matrix of N1×S. The elements in the matrix are expressed as l=0, 1, ..., S-1, represents the index of the basic unit image block or pixel, n=0, 1, ..., N1-1, represents the index of the prior candidate motion vector, which means the weight factor of the prior candidate motion vector indicated by n of the basic unit image block or pixel indicated by l.

[0309] If the N1 probability values ​​corresponding to the N1 prior candidate motion vectors of the basic unit image block or pixel indicated by l in the current block are normalized, that is, The N1 probability values ​​can be used as N1 weight factors corresponding to the N1 prior candidate motion vectors, for example If the N1 probability values ​​corresponding to the N1 prior candidate motion vectors of the basic unit image block or pixel indicated by l in the current block have not been normalized, the N1 probability values ​​may be normalized first, and then the normalized values ​​of the N1 probability values ​​are used as the N1 weight factors corresponding to the N1 prior candidate motion vectors. Therefore, when l remains unchanged,

[0310] Step 904: Perform motion compensation according to the multiple prior candidate motion vectors to obtain multiple prediction values.

[0311] A certain a priori candidate motion vector is taken as an example for explanation. The a priori candidate motion vector is any one of multiple a priori candidate motion vectors. Other a priori candidate motion vectors can refer to this method.

[0312] Motion compensation is performed according to the prior candidate motion vector to obtain a prediction value of the current block, so N1 prior candidate motion vectors can obtain N1 prediction values.

[0313] The multiple prediction values ​​of the current block can be expressed as a three-dimensional matrix of BH×WH×S, where BH×WH represents the size of the basic unit image block contained in the current block, and S represents the number of basic unit image blocks or pixels contained in the current block. If the current block is not further divided, S=1. The elements in the matrix are expressed as l=0, 1, ..., S-1, represents the index of the basic unit image block or pixel, n=0, 1, ..., N1-1, represents the index of the prior candidate motion vector, which means the pixel in the i-th row and j-th column in the basic unit image block indicated by l, and the predicted value corresponding to the prior candidate motion vector indicated by n.

[0314] Step 905: Obtain a prediction value of the current block by weighted summing multiple weight factors and multiple prediction values.

[0315] The weight factor corresponding to the same prior candidate motion vector and the predicted value are multiplied, and then the multiple products corresponding to multiple prior candidate motion vectors are added to obtain the predicted value of the current block. In the current block, the predicted value of the pixel in the i-th row and j-th column in the basic unit image block indicated by l can be expressed as:

[0316]

[0317] Embodiment 2

[0318] In this embodiment, multiple a priori candidate motion vectors of the current block and multiple probability values ​​corresponding to the multiple a priori candidate motion vectors are determined based on multiple a posteriori motion vectors of each of the multiple reconstructed image blocks in the surrounding area and multiple probability values ​​corresponding to the multiple a priori motion vectors.

[0319] Fig.11 1 is a flowchart of a process 1100 of an inter-frame prediction method according to an embodiment of the present application. The process 1100 may be performed by the video encoder 20 or the video decoder 30, and specifically, may be performed by the inter-frame prediction unit 244, 344 of the video encoder 20 or the video decoder 30. The process 1100 is described as a series of steps or operations, and it should be understood that the process 1100 may be performed in various orders and / or may occur simultaneously, not limited to Fig.11 Assuming that a video data stream having multiple image frames is using a video encoder or a video decoder, a process 1100 including the following steps is performed to perform inter-frame prediction on an image or an image block. The process 1100 may include:

[0320] Step 1101: Obtain a plurality of posterior motion vectors of a plurality of reconstructed image blocks in a peripheral region and a plurality of probability values ​​corresponding to the plurality of posterior motion vectors.

[0321] Step 1101 of this embodiment is different from step 901 of the first embodiment described above in that a plurality of prediction error values ​​corresponding to a plurality of posterior motion vectors are changed into a plurality of probability values ​​corresponding to a plurality of posterior motion vectors.

[0322] The following is an explanation using a certain reconstructed image block as an example. The reconstructed image block may be any one of a plurality of reconstructed image blocks in the surrounding area. Other reconstructed image blocks may refer to the method to obtain a plurality of posterior motion vectors and a plurality of probability values ​​corresponding to the plurality of posterior motion vectors.

[0323] The N2 posterior motion vectors of the reconstructed image block may be obtained by referring to the method in step 901 above, which will not be described in detail here.

[0324] The N2 probability values ​​corresponding to the N2 posterior motion vectors of the reconstructed image block can be obtained by the following two methods:

[0325] One is to obtain N2 probability values ​​of the reconstructed image block according to the N2 prediction error values ​​of the reconstructed image block obtained in the first embodiment.

[0326] The N2 prediction error values ​​of the reconstructed image blocks correspond to an N2-dimensional vector of all prediction error values ​​of the reconstructed image blocks, the elements of which are represented as k1 is the index of the reconstructed image block, n = 0, 1, ..., N2-1, which represents the index of the posterior motion vector. The N2 probability values ​​of the reconstructed image block can be calculated based on the N2 prediction error values ​​of the reconstructed image block. The N2 probability values ​​of the reconstructed image block can also be expressed as an N2-dimensional vector, the elements of which are expressed as k1 is the index of the reconstructed image block, n=0, 1, ..., N2-1, represents the index of the posterior motion vector, which means the probability that the posterior motion vector indicated by n of the reconstructed image block indicated by k1 becomes the optimal motion vector of the reconstructed image block.

[0327] Alternatively, the following normalized exponential function can be used to convert Convert to

[0328]

[0329] For example, we can use the linear normalization method to Convert to

[0330] Therefore, when k remains unchanged,

[0331] Another method is to input the reconstruction value of the reconstructed image block and the N2 prediction values ​​corresponding to the N2 posterior motion vectors into the trained neural network to obtain the N2 probability values ​​corresponding to the N2 posterior motion vectors of the reconstructed image block. The neural network can refer to the description of the training engine 25 above, which will not be repeated here.

[0332] The reconstruction value of the reconstructed image block can be obtained after encoding the reconstructed image block, and the N2 prediction values ​​of the reconstructed image block corresponding to the N2 posterior motion vectors can be obtained by referring to the method in the above step 901, which will not be repeated here.

[0333] All the posterior motion vectors of multiple reconstructed image blocks can be expressed as a two-dimensional matrix of N2×Q, where N2 is the number of posterior motion vectors and Q is the number of reconstructed image blocks, and the elements are expressed as k=0, 1, ..., Q-1, represents the index of the reconstructed image block, n=0, 1, ..., N2-1, represents the index of the posterior motion vector, which means the posterior motion vector indicated by n of the reconstructed image block indicated by k.

[0334] All probability values ​​of multiple reconstructed image blocks can also be expressed as a two-dimensional matrix of N2×Q, where the elements are expressed as k=0, 1, ..., Q-1, represents the index of the reconstructed image block, n=0, 1, ..., N2-1, represents the index of the posterior motion vector, which means the probability that the posterior motion vector indicated by n of the reconstructed image block indicated by k becomes the optimal motion vector of the reconstructed image block.

[0335] Step 1102: Obtain multiple a priori candidate motion vectors of the current block and multiple probability values ​​corresponding to the multiple a priori candidate motion vectors according to the multiple a priori motion vectors of the multiple reconstructed image blocks and the multiple probability values ​​corresponding to the multiple a priori candidate motion vectors.

[0336] Step 1102 of this embodiment is different from step 902 of the first embodiment described above, in that a plurality of prediction error values ​​corresponding to a plurality of posterior motion vectors input into the neural network are changed into a plurality of probability values ​​corresponding to a plurality of posterior motion vectors.

[0337] Step 1103: Obtain multiple weight factors corresponding to the multiple priori candidate motion vectors according to the multiple probability values ​​corresponding to the multiple priori candidate motion vectors of the current block.

[0338] Step 1104: Perform motion compensation according to the multiple prior candidate motion vectors to obtain multiple prediction values.

[0339] Step 1105: Obtain a prediction value of the current block by weighted summing multiple weight factors and multiple prediction values.

[0340] Steps 1103-1105 of this embodiment may refer to steps 903-905 of the first embodiment, and will not be described in detail here.

[0341] Embodiment 3

[0342] In this embodiment, a plurality of a priori candidate motion vectors of the current block and a plurality of probability values ​​corresponding to the plurality of a priori candidate motion vectors are determined according to respective optimal motion vectors of a plurality of reconstructed image blocks in the surrounding area.

[0343] Fig.12 1 is a flowchart of a process 1200 of an inter-frame prediction method according to an embodiment of the present application. The process 1200 may be performed by the video encoder 20 or the video decoder 30, and specifically, may be performed by the inter-frame prediction unit 244, 344 of the video encoder 20 or the video decoder 30. The process 1200 is described as a series of steps or operations, and it should be understood that the process 1200 may be performed in various orders and / or may occur simultaneously, not limited to Fig.12 Assuming that a video data stream having multiple image frames is using a video encoder or a video decoder, a process 1200 including the following steps is performed to perform inter-frame prediction on an image or an image block. The process 1200 may include:

[0344] Step 1201: Obtain the optimal motion vector of each of a plurality of reconstructed image blocks in a peripheral area.

[0345] Step 1201 of this embodiment is different from step 901 of the first embodiment in that: a plurality of a posteriori motion vectors and a plurality of prediction error values ​​corresponding to the plurality of a posteriori motion vectors become the optimal motion vector.

[0346] The following description is made by taking a certain reconstructed image block as an example. The reconstructed image block may be any one of a plurality of reconstructed image blocks in the surrounding area. Other reconstructed image blocks may refer to the method to obtain the optimal motion vector.

[0347] The optimal motion vector of the reconstructed image block can be obtained by the following two methods:

[0348] One is to obtain the optimal motion vector of the reconstructed image block according to the N2 prediction error values ​​of the reconstructed image block obtained in Example 1, that is, to use the posterior motion vector corresponding to the minimum prediction error value among the N2 prediction error values ​​of the reconstructed image block as the optimal motion vector of the reconstructed image block.

[0349] Another method is to obtain the optimal motion vector of the reconstructed image block according to the N2 probability values ​​of the reconstructed image block obtained in Example 2, that is, to use the posterior motion vector corresponding to the maximum probability value among the N2 probability values ​​of the reconstructed image block as the optimal motion vector of the reconstructed image block.

[0350] Step 1202: Obtain multiple a priori candidate motion vectors of the current block and multiple probability values ​​corresponding to the multiple a priori candidate motion vectors according to the respective optimal motion vectors of the multiple reconstructed image blocks.

[0351] Step 1202 of this embodiment is different from step 902 of the above-mentioned embodiment 1, and the difference is that: multiple posterior motion vectors input into the neural network and multiple prediction error values ​​corresponding to the multiple posterior motion vectors become optimal motion vectors of multiple reconstructed image blocks.

[0352] Step 1203: Obtain multiple weight factors corresponding to the multiple priori candidate motion vectors according to the multiple probability values ​​corresponding to the multiple priori candidate motion vectors of the current block.

[0353] Step 1204: Perform motion compensation according to the multiple prior candidate motion vectors to obtain multiple prediction values.

[0354] Step 1205: Obtain a prediction value of the current block by weighted summing multiple weight factors and multiple prediction values.

[0355] Steps 1203-1205 of this embodiment may refer to steps 903-905 of the first embodiment, and will not be described in detail here.

[0356] Fig.13 The structure diagram of the inter-frame prediction device 1300 of the embodiment of the present application is shown in FIG. The inter-frame prediction device 1300 includes: a motion estimation unit 1301 and an inter-frame prediction processing unit 1302, wherein the motion estimation unit 1301 is used to obtain the motion vectors of P reconstructed image blocks in the surrounding area of ​​the current block, and the surrounding area includes the spatial neighborhood and / or the temporal neighborhood of the current block; the inter-frame prediction processing unit 1302 is used to obtain Q prior candidate motion vectors of the current block and Q probability values ​​corresponding to the Q prior candidate motion vectors according to the motion vectors of the P reconstructed image blocks; obtain M weight factors corresponding to the M prior candidate motion vectors according to the M probability values ​​corresponding to the M prior candidate motion vectors; M, P and Q are positive integers, and M is less than or equal to Q; perform motion compensation according to the M prior candidate motion vectors to obtain M prediction values; and obtain the prediction value of the current block by weighted summation of the M prediction values ​​and the corresponding M weight factors. In an example, the inter-frame prediction device 1300 including the motion estimation unit 1301 and the inter-frame prediction processing unit 1302 may correspond to Figure 2 The inter-frame prediction unit 244 in, or corresponding to Figure 3 The inter-frame prediction unit 344 in.

[0357] In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The processor can be a general-purpose processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware coding processor to be executed, or the hardware and software modules in the coding processor are combined to be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0358] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0359] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0360] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0361] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0362] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0363] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0364] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0365] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An inter-frame prediction method, characterized in that: include: Obtaining motion vectors of P reconstructed image blocks in a surrounding area of ​​a current block, wherein the surrounding area includes a spatial neighborhood and / or a temporal neighborhood of the current block; Obtaining Q a priori candidate motion vectors of the current block and Q probability values ​​corresponding to the Q a priori candidate motion vectors according to the motion vectors of the P reconstructed image blocks; Obtaining, according to the M probability values ​​corresponding to the M priori candidate motion vectors, M weight factors corresponding to the M priori candidate motion vectors; M, P and Q are positive integers, M is less than or equal to Q; Perform motion compensation according to the M prior candidate motion vectors to obtain M prediction values; The prediction value of the current block is obtained by weighted summing the M prediction values ​​and the corresponding M weight factors.

2. The method according to claim 1, characterized in that The step of obtaining Q a priori candidate motion vectors of the current block and Q probability values ​​corresponding to the Q a priori candidate motion vectors according to the motion vectors of the P reconstructed image blocks includes: The motion vectors of the P reconstructed image blocks are input into a trained neural network to obtain the Q priori candidate motion vectors and Q probability values ​​corresponding to the Q priori candidate motion vectors.

3. The method according to claim 1, characterized in that The obtaining, according to the M probability values ​​corresponding to the M a priori candidate motion vectors, M weight factors corresponding to the M a priori candidate motion vectors comprises: When the sum of the M probability values ​​is 1, the probability value corresponding to the first a priori candidate motion vector is used as the weight factor corresponding to the first a priori candidate motion vector; or, When the sum of the M probability values ​​is not 1, normalizing the M probability values; using the normalized value of the probability value corresponding to the first a priori candidate motion vector as the weight factor corresponding to the first a priori candidate motion vector; The first a priori candidate motion vector is any one of the M a priori candidate motion vectors.

4. The method according to any one of claims 1 to 3, characterized in that M is equal to Q, and the M probability values ​​are the Q probability values.

5. The method according to any one of claims 1 to 3, characterized in that: M is smaller than Q, and the M probability values ​​are all greater than other probability values ​​among the Q probability values ​​except the M probability values.

6. The method according to any one of claims 1 to 3, characterized in that The step of obtaining Q a priori candidate motion vectors of the current block and Q probability values ​​corresponding to the Q a priori candidate motion vectors according to the motion vectors of the P reconstructed image blocks includes: Inputting a plurality of a posteriori motion vectors of each of the P reconstructed image blocks and a plurality of probability values ​​corresponding to the plurality of a posteriori motion vectors into a trained neural network, Q a priori candidate motion vectors of the current block and Q probability values ​​corresponding to the Q a priori candidate motion vectors are obtained; the plurality of a posteriori motion vectors of the reconstructed image block and the plurality of probability values ​​corresponding to the plurality of a posteriori motion vectors are determined based on a reconstruction value of the reconstructed image block and prediction values ​​corresponding to the plurality of a posteriori candidate motion vectors, and the reconstructed image block is any one of the P reconstructed image blocks.

7. The method according to any one of claims 1 to 3, characterized in that The step of obtaining Q a priori candidate motion vectors of the current block and Q probability values ​​corresponding to the Q a priori candidate motion vectors according to the motion vectors of the P reconstructed image blocks includes: Inputting a plurality of a posteriori motion vectors of each of the P reconstructed image blocks and a plurality of prediction error values ​​corresponding to the plurality of a posteriori motion vectors into a trained neural network, obtaining Q a priori candidate motion vectors of the current block and Q probability values ​​corresponding to the Q a priori candidate motion vectors; the plurality of a posteriori motion vectors of the reconstructed image block and the plurality of prediction error values ​​corresponding to the plurality of a posteriori motion vectors are determined based on the reconstruction value of the reconstructed image block and the prediction values ​​corresponding to the plurality of a posteriori candidate motion vectors, and the reconstructed image block is any one of the P reconstructed image blocks.

8. The method according to any one of claims 1 to 3, characterized in that The step of obtaining Q a priori candidate motion vectors of the current block and Q probability values ​​corresponding to the Q a priori candidate motion vectors according to the motion vectors of the P reconstructed image blocks includes: The optimal motion vectors of the P reconstructed image blocks are input into a trained neural network to obtain Q a priori candidate motion vectors of the current block and Q probability values ​​corresponding to the Q a priori candidate motion vectors; the optimal motion vector of the reconstructed image block is a posterior motion vector with the largest probability value or the smallest prediction error value among multiple a priori motion vectors of the reconstructed image block, and the reconstructed image block is any one of the multiple reconstructed image blocks; wherein, The multiple a posteriori motion vectors of the reconstructed image block correspond to multiple probability values, and the multiple a posteriori motion vectors and the multiple probability values ​​corresponding to the multiple a posteriori motion vectors are determined according to the reconstruction value of the reconstructed image block and the prediction values ​​corresponding to the multiple a posteriori candidate motion vectors; or, The multiple a posteriori motion vectors of the reconstructed image block correspond to multiple prediction error values, and the multiple a posteriori motion vectors and the multiple prediction error values ​​corresponding to the multiple a posteriori motion vectors are determined based on the reconstruction value of the reconstructed image block and the prediction values ​​corresponding to the multiple a posteriori candidate motion vectors.

9. The method according to claim 6, characterized in that Also includes: Acquire a training data set, wherein the training data set includes information of multiple groups of image blocks, wherein the information of each group of image blocks includes multiple a posteriori motion vectors of multiple reconstructed image blocks, multiple probability values ​​corresponding to the multiple a posteriori motion vectors, and multiple a posteriori motion vectors of a current block, and multiple probability values ​​corresponding to the multiple a posteriori motion vectors, wherein the multiple reconstructed image blocks are image blocks in a spatial neighborhood and / or a temporal neighborhood of the current block; The neural network is obtained by training according to the training data set.

10. The method according to claim 7, characterized in that Also includes: Acquire a training data set, wherein the training data set includes information of multiple groups of image blocks, wherein the information of each group of image blocks includes multiple a posteriori motion vectors of multiple reconstructed image blocks, multiple prediction error values ​​corresponding to the multiple a posteriori motion vectors, and multiple a posteriori motion vectors of a current block, and multiple probability values ​​corresponding to the multiple a posteriori motion vectors, wherein the multiple reconstructed image blocks are image blocks in a spatial neighborhood and / or a temporal neighborhood of the current block; The neural network is obtained by training according to the training data set.

11. The method according to claim 8, characterized in that Also includes: Acquire a training data set, wherein the training data set includes information of multiple groups of image blocks, wherein the information of each group of image blocks includes optimal motion vectors of multiple reconstructed image blocks, multiple a posteriori motion vectors of a current block, and multiple probability values ​​corresponding to the multiple a posteriori motion vectors, and the multiple reconstructed image blocks are image blocks in a spatial neighborhood and / or a temporal neighborhood of the current block; The neural network is obtained by training according to the training data set.

12. The method according to any one of claims 9 to 11, characterized in that: The neural network includes at least a convolutional layer and an activation layer.

13. The method according to claim 12, characterized in that The depth of the convolution kernel of the convolution layer is 2, 3, 4, 5, 6, 16, 24, 32, 48, 64 or 128; the size of the convolution kernel in the convolution layer is 1×1, 3×3, 5×5 or 7×7.

14. The method according to any one of claims 9 to 11 and 13, characterized in that: The neural network includes a convolutional neural network (CNN), a deep neural network (DNN), or a recurrent neural network (RNN).

15. An encoder, characterized in that: Comprising a processing circuit for executing the method of any one of claims 1 to 14.

16. A decoder, characterized in that: Comprising a processing circuit for executing the method of any one of claims 1 to 14.

17. A computer program product, characterized in that The method comprises a program code for executing the method according to any one of claims 1 to 14 when the program code is executed on a computer or a processor.

18. An encoder, characterized in that: include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the encoder to perform the method of any one of claims 1 to 14.

19. A decoder, characterized in that: include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method of any one of claims 1 to 14.

20. A non-transitory computer-readable storage medium, characterized in that: The method comprises a program code which, when executed by a computer device, is used to perform the method according to any one of claims 1 to 14.

Images