Encoding method, decoding method, encoder, decoder, code stream, and storage medium

Abstract

Embodiments of the present application provide an encoding method, a decoding method, an encoder, a decoder, a code stream, and a storage medium. The decoding method comprises: parsing a code stream, and determining a reconstructed block of a current block; and, on the basis of the reconstructed block, determining first motion information of the current block.

Description

Coding and decoding method, codec, code stream and storage medium Technical Field

[0001] The present application relates to the field of video coding and decoding technology, and in particular to a coding and decoding method, a codec, a bit stream, and a storage medium. Background Art

[0002] During inter-frame prediction, the current block determines a reference block in the reference image based on motion information to serve as the prediction block for the current block. After the prediction is complete, this motion information is used by the current block or other blocks. However, the reference block indicated by this motion information in the reconstructed reference image may not necessarily match the reconstructed block of the current block, which may affect codec performance. Therefore, how to improve motion information is a problem that needs to be solved.

[0003] Summary of the Invention

[0004] The present application provides a coding and decoding method, a codec, a bit stream, and a storage medium. The following introduces various aspects of the present application.

[0005] In a first aspect, a decoding method is provided. The decoding method is applied to a decoder, and the decoding method includes: parsing a code stream to determine a reconstructed block of a current block; and determining first motion information of the current block according to the reconstructed block.

[0006] In a second aspect, a coding method is provided. The coding method is applied to an encoder, and the coding method includes: determining a reconstructed block of a current block; and determining first motion information of the current block according to the reconstructed block.

[0007] According to a third aspect, a decoder is provided, comprising: a first determination unit configured to parse a code stream and determine a reconstructed block of a current block; and a second determination unit configured to determine first motion information of the current block according to the reconstructed block.

[0008] In a fourth aspect, a decoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method of the first aspect when running the computer program.

[0009] In a fifth aspect, an encoder is provided, comprising: a first determination unit configured to determine a reconstructed block of a current block; and a second determination unit configured to determine first motion information of the current block based on the reconstructed block.

[0010] In a sixth aspect, an encoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method of the second aspect when running the computer program.

[0011] In a seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the method of the first aspect or the second aspect is implemented.

[0012] In an eighth aspect, a computer program product is provided, comprising a computer program, which implements the method of the first aspect or the second aspect when the computer program is executed.

[0013] In a ninth aspect, a non-volatile computer-readable storage medium for storing a bit stream is provided, wherein the bit stream is generated by an encoding method using an encoder, or the bit stream is decoded by a decoding method using a decoder, wherein the decoding method is the method of the first aspect and the encoding method is the method of the second aspect.

[0014] In a tenth aspect, a code stream is provided, comprising a code stream generated according to the method of the second aspect.

[0015] The embodiment of the present application determines motion information in a reconstructed reference image based on a reconstructed block of the current block. Since the reconstructed block can determine the most matching reference block in the reconstructed reference image, the accuracy of the motion information can be improved, thereby helping to improve the performance of the codec. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG1 is a structural diagram illustrating an example of a video encoder to which an embodiment of the present application may be applied.

[0017] FIG2 is a diagram showing an example structure of a video decoder to which an embodiment of the present application can be applied.

[0018] FIG3 is a schematic diagram of a reference image of a GOP structure.

[0019] FIG4 is an example diagram of a block division method.

[0020] FIG5 is a schematic diagram of adjacent blocks of a current block.

[0021] FIG6 is a schematic diagram of a temporal motion information prediction method.

[0022] FIG7 is a schematic diagram of determining a motion vector for a sub-block.

[0023] FIG8 is a schematic diagram of an MVD distribution.

[0024] FIG9 is a schematic diagram of calculating a motion vector based on affine.

[0025] FIG10 is a schematic diagram of a method for deriving a motion vector based on sub-blocks.

[0026] FIG11 is a schematic diagram of the weights of the 64 modes based on GPM.

[0027] FIG12 is a schematic diagram of searching for a motion vector based on DMVR.

[0028] FIG13 is a schematic diagram of determining a motion vector based on template matching.

[0029] FIG14 is a schematic diagram of reference pixels for intra-frame prediction.

[0030] FIG15 is a schematic diagram of intra prediction using four reference rows / columns.

[0031] FIG16 is a schematic diagram of nine prediction modes for intra-frame prediction.

[0032] FIG17 is a schematic diagram of 35 prediction modes for intra-frame prediction.

[0033] FIG18 is a schematic diagram of 67 prediction modes for intra-frame prediction.

[0034] FIG19 is a schematic diagram of a wide-angle mode of intra-frame prediction.

[0035] FIG. 20 is an example diagram of screen content.

[0036] Figure 21 is a flowchart of the decoding method provided in an embodiment of the present application.

[0037] Figure 22 is a flow chart of the encoding method provided in an embodiment of the present application.

[0038] FIG23A is a schematic diagram of a decoding process according to an embodiment of the present application.

[0039] FIG23B is a schematic diagram of a decoding process provided in another embodiment of the present application.

[0040] FIG24A is a schematic diagram of the current block decoding process.

[0041] FIG24B is a schematic diagram of a process of decoding adjacent blocks of a current block.

[0042] FIG25 is a schematic diagram of the structure of a decoder provided in an embodiment of the present application.

[0043] FIG26 is a schematic diagram of the structure of a decoder provided in another embodiment of the present application.

[0044] FIG27 is a schematic diagram of the structure of an encoder provided in one embodiment of the present application.

[0045] FIG28 is a schematic diagram of the structure of an encoder provided in another embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0048] In the following description, reference is made to “some implementations” which describe a subset of all possible embodiments, but it will be understood that “some implementations” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0050] FIG1 is a schematic block diagram of a video encoder according to an embodiment of the present application.

[0051] It should be understood that the video encoder 100 can be used to perform lossy compression or lossless compression on an image. The lossless compression can be visually lossless compression or mathematically lossless compression.

[0052] The video encoder 100 can be applied to image data in a luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in terms of color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0053] For example, the video encoder 100 reads video data and, for each image in the video data, divides the image into a number of coding tree units (CTUs). In some examples, a CTU may be referred to as a "tree block," "largest coding unit" (LCU) or "coding tree block" (CTB). Each CTU may be associated with a pixel block of equal size within the image. Each pixel may correspond to one luminance (luminance or luma) sample and two chrominance (chroma) samples. Therefore, each CTU may be associated with one luminance sample block and two chrominance sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU may be further divided into a number of coding units (CUs) for encoding. A CU may be a rectangular block or a square block. A CU may correspond to a prediction unit (PU) and a transform unit (TU).

[0054] In some embodiments, as shown in FIG1 , the video encoder 100 may include a prediction module 110, a residual module 120, a transform / quantization module 130, an inverse transform / quantization module 140, a reconstruction module 150, a loop filter module 160, a decoded image buffer 170, and an entropy coding module 180. It should be noted that the video encoder 100 may include more, fewer, or different functional components.

[0055] Optionally, in this application, the current block may be referred to as the current coding unit (CU). The prediction block may also be referred to as a predicted image block or an image prediction block, and the reconstructed image block may also be referred to as a reconstructed block or an image reconstruction block. Due to the need for parallel processing, an image may be divided into slices. Slices in the same image may be processed in parallel, meaning that there is no data dependency between them. The term "frame" is commonly used, and it can generally be understood that a frame is an image. The term "frame" herein may also be replaced by "image" or "slice," etc.

[0056] In some embodiments, the prediction module 110 includes an inter-frame prediction module 111 and an intra-frame prediction module 112. Because there is a strong correlation between adjacent pixels in a video image, intra-frame prediction is used in video coding and decoding technologies to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent images in a video, inter-frame prediction is used in video coding and decoding technologies to eliminate temporal redundancy between adjacent images, thereby improving coding efficiency.

[0057] The inter-frame prediction module 111 can be used for inter-frame prediction. Inter-frame prediction can include motion estimation and motion compensation. It can refer to image information from different images. Inter-frame prediction uses motion information to find a reference block from a reference image and generate a prediction block based on the reference block to eliminate temporal redundancy. Inter-frame prediction uses motion information to find a reference block from a reference image and generate a prediction block based on the reference block. Motion information includes the reference image list in which the reference image is located, the reference image index, and the motion vector. The motion vector can be integer pixel or fractional pixel. If the motion vector is fractional pixel, interpolation filtering is required to generate the required fractional pixel block in the reference image. Here, the integer pixel or fractional pixel block in the reference image found based on the motion vector is called a reference block. Some technologies directly use the reference block as the prediction block, while others further process the reference block to generate a prediction block. Reprocessing the reference block to generate a prediction block can also be understood as using the reference block as the prediction block and then processing the prediction block to generate a new prediction block.

[0058] The intra-frame prediction module 112 only refers to information of the same image to predict pixel information within the current code image block to eliminate spatial redundancy.

[0059] Intra-frame prediction has multiple prediction modes. For example, the H-series international digital video coding standard H.264 / AVC has eight angular prediction modes and one non-angular prediction mode. H.265 / HEVC expands this to 33 angular prediction modes and two non-angular prediction modes. High-efficiency video coding (HEVC) uses planar, direct current (DC), and 33 angular modes for a total of 35 intra-frame prediction modes. Versatile video coding (VVC) uses planar, DC, and 65 angular modes for a total of 67 intra-frame prediction modes.

[0060] It should be noted that with the increase of angle modes, intra-frame prediction will be more accurate and more in line with the needs of high-definition and ultra-high-definition digital video development.

[0061] Residual module 120 may generate a residual block for a CU based on the pixel block of the CU and the prediction block of the CU. For example, residual module 120 may generate a residual block for the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel block of the CU and the corresponding sample in the prediction block of the CU.

[0062] The transform / quantization module 130 may quantize the transform coefficients. The transform / quantization module 130 may quantize the transform coefficients associated with the CU based on a quantization parameter (QP) value associated with the CU. The video encoder 100 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0063] The inverse transform / quantization module 140 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients to reconstruct a residual block from the quantized transform coefficients.

[0064] Reconstruction module 150 can add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by prediction module 110 to generate a reconstructed image block associated with the CU. By reconstructing each sample block of the CU in this manner, video encoder 100 can reconstruct the pixel blocks of the CU.

[0065] The loop filter module 160 is used to process the inverse transformed and inverse quantized pixels to compensate for distortion information and provide a better reference for subsequent pixel encoding. For example, a deblocking filtering operation may be performed to reduce the blocking effect of pixel blocks associated with the CU.

[0066] In some embodiments, the loop filtering module 160 includes a deblocking filtering module and a sample adaptive offset / adaptive loop filtering (SAO / ALF) module, wherein the deblocking filtering module is used to remove blocking effects, and the SAO / ALF module is used to remove ringing effects and reduce other distortions.

[0067] The decoded image buffer 170 may store the reconstructed pixel blocks. The inter prediction module 111 may use a reference image containing the reconstructed pixel blocks to perform inter prediction on PUs of other images. In addition, the intra prediction module 112 may use the reconstructed pixel blocks in the decoded image buffer 170 to perform intra prediction on other PUs in the same image as the CU.

[0068] The entropy encoding module 180 may receive the quantized transform coefficients from the transform / quantization module 130. The entropy encoding module 180 may perform one or more entropy encoding operations on the quantized transform coefficients to generate entropy-encoded data.

[0069] FIG2 is a schematic block diagram of a video decoder according to an embodiment of the present application.

[0070] 2 , video decoder 200 includes an entropy decoding module 210, a prediction module 220, an inverse quantization / transformation module 230, a reconstruction module 240, a loop filter module 250, and a decoded image buffer 260. It should be noted that video decoder 200 may include more, fewer, or different functional components.

[0071] The video decoder 200 may receive a bitstream. The entropy decoding module 210 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, the entropy decoding module 210 may parse the entropy-encoded syntax elements in the bitstream. The prediction module 220, the inverse quantization / transformation module 230, the reconstruction module 240, and the loop filter module 250 may decode the video data based on the syntax elements extracted from the bitstream, thereby generating decoded video data.

[0072] In some embodiments, the prediction module 220 includes an intra-frame prediction module 222 and an inter-frame prediction module 221 .

[0073] The intra prediction module 222 may perform intra prediction to generate a prediction block for a PU. The intra prediction module 222 may use an intra prediction mode to generate a prediction block for the PU based on pixel blocks of spatially neighboring PUs. The intra prediction module 222 may also determine the intra prediction mode for the PU based on one or more syntax elements parsed from the codestream.

[0074] The inter-frame prediction module 221 may construct a first reference picture list (List 0) and a second reference picture list (List 1) based on syntax elements parsed from the codestream. Furthermore, if a PU is encoded using inter-frame prediction, the entropy decoding module 210 may parse the motion information of the PU. The inter-frame prediction module 221 may determine one or more reference blocks for the PU based on the motion information of the PU. The inter-frame prediction module 221 may generate a prediction block for the PU based on the one or more reference blocks of the PU.

[0075] The inverse quantization / transform module 230 may inversely quantize (ie, dequantize) the transform coefficients associated with the TU. The inverse quantization / transform module 230 may use the QP value associated with the CU of the TU to determine the degree of quantization.

[0076] After inverse quantizing the transform coefficients, inverse quantization / transform module 230 may apply one or more inverse transforms to the inverse quantized transform coefficients in order to generate a residual block associated with the TU.

[0077] Reconstruction module 240 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, reconstruction module 240 can add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.

[0078] The loop filtering module 250 may perform a deblocking filtering operation to reduce blocking artifacts of pixel blocks associated with a CU.

[0079] The video decoder 200 may store the reconstructed image of the CU in the decoded image buffer 260. The video decoder 200 may use the reconstructed image in the decoded image buffer 260 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0080] The basic process of video encoding and decoding is as follows: At the encoder end, an image is divided into blocks. For the current block, the prediction module 110 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual module 120 calculates a residual block based on the predicted block and the original block of the current block. This residual block is the difference between the predicted block and the original block of the current block. This residual block can also be referred to as residual information. This residual block undergoes transformation and quantization by the transform / quantization module 130, removing information that is insensitive to the human eye and eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform / quantization module 130 can be referred to as a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform / quantization module 130 can be referred to as a frequency residual block or a frequency-domain residual block. The entropy coding module 180 receives the quantized change coefficients output by the transform and quantization module 130 and performs entropy coding on these quantized change coefficients to output a bitstream. For example, the entropy coding module 180 can eliminate character redundancy based on the target context model and the probability information of the binary bitstream.

[0081] At the decoding end, the entropy decoding module 210 can parse the code stream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction module 220 uses intra-frame prediction or inter-frame prediction on the current block based on the prediction information to generate a prediction block for the current block. The inverse quantization / transformation module 230 uses the quantization coefficient matrix obtained from the code stream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction module 240 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering module 250 performs loop filtering on the reconstructed image based on the image or block to obtain a decoded image. The encoding end also requires similar operations as the decoding end to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be used as a reference image for inter-frame prediction of subsequent images.

[0082] It should be noted that the block division information determined by the encoder, as well as mode information or parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, etc., are carried in the bitstream when necessary. The decoder parses the bitstream and analyzes the existing information to determine the same block division information, prediction, transform, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoder, thereby ensuring that the decoded image obtained by the encoder and the decoder are identical.

[0083] The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. This application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

[0084] The above describes in detail the coding and decoding framework applicable to the embodiments of the present application. Next, some prediction techniques applicable to the above coding and decoding framework are introduced.

[0085] Inter-frame prediction uses temporal correlation to eliminate redundancy. To minimize visual jitter, video frame rates are typically set to 30, 50, 60, or even 120 frames per second. In such videos, adjacent frames within the same scene are highly correlated. Inter-frame prediction leverages this correlation to predict the current encoding content by referencing previously encoded and decoded frames. Inter-frame prediction can significantly improve encoding and decoding performance.

[0086] The most basic inter-frame prediction method is translational prediction, which assumes that the content being predicted is moving in translation between the current image and the reference image. For example, if the current block is moving in translation between the current image and the reference image, a motion vector (MV) can be used to find the reference block of the current block in the reference image and use this reference block as the prediction block for the current block. Translational motion is a common feature in video. For example, static backgrounds, objects that move in translation, and camera pans can all be processed using translational prediction.

[0087] Some content in natural videos is not simply a translation; for example, there are subtle changes during translation, including changes in shape and color. Bidirectional prediction finds two reference blocks from the reference image and performs a weighted average of these two reference blocks to obtain a predicted block that is as similar as possible to the current block. For example, in some scenes, a weighted average of a reference block from both the forward and backward directions of the current frame may be closer to the current block than a single reference block. Compared to unidirectional prediction, bidirectional prediction improves the compression performance of video codecs. The following is a detailed introduction to the reference image for inter-frame prediction.

[0088] In a video sequence, each image has a unique picture order count (POC). POC can be used as an identifier for an image. The identification order of POC is the same as the playback order of the corresponding images. For example, a P-image (P Frame) is an image that can only be predicted using a reference image whose POC is before the current image. In the relevant standards, the current P-image has only one reference picture list (RPL), which can be represented as RPL0. The reference images indicated in RPL0 are all reference images whose POC is before the current image. B-images (B Frames) can use reference images whose POC is before the current image, as well as reference images whose POC is after the current image. In addition, B-images have two reference picture lists, which can be represented as RPL0 and RPL1. For the above two reference picture lists of B images, one configuration method is that RPL0 contains reference images whose POC is before the current image, and RPL1 contains reference images whose POC is after the current image. For the current block, it can refer only to the reference blocks of a reference image in RPL0, which is called forward prediction; or it can refer only to the reference blocks of a reference image in RPL1, which is called backward prediction; or it can refer to both the reference blocks of a reference image in RPL0 and the reference blocks of a reference image in RPL1, which is called bidirectional prediction. One method of simultaneously referring to two reference blocks is to average the pixels at each corresponding position in the two reference blocks to obtain the prediction block for the current block. In some related technologies, B-pictures are no longer restricted to reference images with a Point of Occurrence (POC) before the current image in RPL0, or to reference images with a Point of Occurrence (POC) after the current image in RPL1. Therefore, RPL0 can contain reference images with a Point of Occurrence (POC) after the current image, and RPL1 can contain reference images with a Point of Occurrence (POC) before the current image. Furthermore, the current block can also reference reference images with a Point of Occurrence (POC) before or after the current image. This type of B-picture is also called a generalized B-picture.

[0089] Figure 3 shows a schematic diagram of reference images in a GOP structure. As shown in Figure 3, the arrows in the figure indicate reference relationships. An I-picture does not require a reference image. After an I-picture with a POC of 0 is decoded, a P-picture with a POC of 4 is decoded. Subsequently, when decoding the P-picture with a POC of 4, the I-picture with a POC of 0 can be referenced. Then, when decoding a B-picture with a POC of 2, the I-picture with a POC of 0 and the P-picture with a POC of 4 can be referenced.

[0090] In other related technologies, images can also be encoded and decoded based on a random access (RA) or low delay (LD) encoding and decoding order. The encoding and decoding order in the RA configuration differs from the POC order described above. That is, B-pictures can reference information before and after the current picture, thereby improving encoding and decoding performance. The encoding and decoding order in the LD configuration is the same as the POC order. Therefore, the current picture can only reference information before the current picture. The LD configuration can be further divided into two types: Low Delay P and Low Delay B. Low Delay P is the traditional Low Delay configuration, with a typical structure of IPPP..., meaning an I picture is encoded and decoded first, and subsequent encoded and decoded pictures are all P pictures. Low Delay B has a typical structure of IBBB..., which differs from Low Delay P in that each inter-frame picture is a B picture, using two reference picture lists. This means that the current block can simultaneously reference a reference block from a reference picture in RPL0 and a reference block from a reference picture in RPL1.

[0091] Generally, the RA configuration has higher compression efficiency than the LD configuration, while the LDB configuration has higher compression efficiency than the LDP configuration. This is partly because bidirectional prediction can refer to backward information and partly because bidirectional prediction can reduce prediction errors through some techniques (such as weighted averaging).

[0092] A reference picture list for a current image can have multiple reference pictures, such as 2, 3, or N. When encoding a current image, the number of reference pictures in RPL0 and RPL1 is determined by a certain configuration or algorithm. However, the same reference picture may appear in both RPL0 and RPL1. This means that the codec allows the current block to reference two reference blocks of the same reference picture at the same time.

[0093] Codecs typically use the index value in the reference picture list to correspond to the reference picture. If a reference picture list is 4 in length, then index has four values: 0, 1, 2, and 3. For example, RPL0 of the current frame has four reference pictures with POC values ​​of 5, 4, 3, and 0. Then RPL0 index 0 is the reference picture with POC 5, RPL0 index 1 is the reference picture with POC 4, RPL0 index 2 is the reference picture with POC 3, and RPL0 index 3 is the reference picture with POC 0.

[0094] The above section introduces the reference image for inter-frame prediction. Determining the reference block within the reference image requires motion information. Inter-frame prediction uses motion information to represent "motion." Basic motion information can include reference picture information and motion vector (MV) information. For a block to use bidirectional prediction, two reference blocks must be found, which means two sets of reference picture information and motion vector information are required. Each set can be considered a unidirectional motion information, and combining the two sets together forms bidirectional motion information. Specifically, unidirectional motion information and bidirectional motion information can use the same data structure, except that both sets of reference frame information and motion vector information for bidirectional motion information are valid, while one set of reference frame information and motion vector information for unidirectional motion information is invalid. "Valid" here can also be understood as "used," and "invalid" can also be understood as "not used."

[0095] In the relevant standards, inter-frame prediction supports two reference picture lists, which can be represented as RPL0 and RPL1. For the above-mentioned bidirectional motion information, the relevant standards can use the reference picture index refIdxL0 and motion vector mvL0 corresponding to RPL0, and the reference picture index refIdxL1 and motion vector mvL0 corresponding to RPL1. Here, the reference picture index corresponding to RPL0 and the reference picture index corresponding to RPL1 can be understood as the above-mentioned reference picture information. The relevant standards use two flags to indicate whether the motion information corresponding to RPL0 and the motion information corresponding to RPL1 are used, respectively, represented as predFlagL0 and predFlagL1. Alternatively, it can be understood that predFlagL0 and predFlagL1 indicate whether the above-mentioned unidirectional motion information is "valid". Although the relevant standards do not explicitly mention the data structure of motion information, the reference picture index, motion vector and "valid" flag corresponding to each reference picture list can be used to represent motion information. Relevant standards do not mention motion information, but instead use motion vectors. The reference image index and the flag indicating whether to use the corresponding motion information can also be considered as appendages of the motion vector. It should be understood that "motion information" in this application can also be described as "motion vector." "Motion information" can also be called "motion parameters."

[0096] For a two-dimensional image, a motion vector can be represented by (x, y), with a horizontal component and a vertical component. Since videos are represented in pixels, there are distances between pixels. The motion of an object in adjacent images may not always correspond to a whole-pixel distance. For example, in a distant video, the distance between two pixels is 1 meter for the distant object, while the object moves 0.5 meters between two frames. This scenario cannot be well represented using a whole-pixel motion vector. Therefore, motion vectors can be expressed at the sub-pixel level, with precision of 1 / 2, 1 / 4, 1 / 8, or 1 / 16 pixels, to represent motion more precisely. Interpolation is then used to obtain the pixel value at the sub-pixel position in the reference image.

[0097] Both the unidirectional and bidirectional predictions in the aforementioned translation prediction are block-based, such as coding units (CUs) or prediction units (PUs). This means that prediction is performed using a pixel matrix as the unit. The most basic block is a rectangular block, such as a square or rectangle. Video codec standards such as HEVC and VVC allow the encoder to determine the size and division of CUs and PUs based on the video content. Regions with simple textures or motion tend to use larger blocks, while regions with complex textures or motion tend to use smaller blocks. The deeper the block division level, the more complex the blocks that more closely resemble the actual texture or motion, but the corresponding overhead for representing these divisions increases. Motion information may also need to be transmitted in the bitstream. Furthermore, generally, the finer the block division, the greater the motion information overhead.

[0098] The original method of representing motion information directly writes the complete motion information. Later, motion vectors were represented using motion vector prediction (MVP) plus motion vector difference (MVD): MV = MVP + MVD. The more accurate the MVP, the smaller the MVD, which reduces the bitstream overhead.

[0099] It's understandable that a block can use not only two reference blocks but also more reference blocks, using these blocks together to generate a predicted block. This approach is called multi-hypothesis prediction. Correspondingly, if multi-hypothesis prediction is used, motion information can be not limited to just two reference images and motion vectors, but can also include multiple reference images and motion vectors. The data structure of the motion information can be expanded accordingly. Next, we'll introduce another prediction technique, the merge mode.

[0100] Each inter-frame coded block requires motion information. To simplify the problem, we assume that the division of CU is equal to the division of PU, which is equal to the division of TU. That is, a coding unit has a prediction unit of the same size and position and a transform unit of the same size and position. Differences in any of the links in prediction, transformation, quantization, and entropy coding may lead to the division of CU. For example, if the motion information of two regions is different, the encoder may divide these two regions into different CUs. For another example, if the motion information of two regions is the same or similar, but the residual characteristics are very different, the encoder may also divide these two regions into different CUs. How to divide is determined by the overall compression efficiency, not entirely by a single factor. Therefore, the same object or regions with the same or similar motion will be divided into different CUs.

[0101] Figure 4 shows an example diagram of a block division method, as shown in Figure 4, where a is the original image, in which an iron rod moves in the direction indicated by the arrow, and the background area moves less. Figure b shows the block division in the relevant standard 1, and Figure c removes the boundaries of the blocks with the same motion information in Figure b. It can be seen from Figure c that many adjacent blocks use the same motion information. In this case, if the motion information is encoded separately for each block, it will result in obvious waste. As mentioned above, the complete motion information includes the reference image index, MV and the flag of whether it is used for RPL0, and the reference image index, MV and the flag of whether it is used for RPL1. The basic principle of the merge mode is that the current block can inherit the motion information of the adjacent blocks, including the reference image information and the motion vector information.

[0102] Merge mode builds a merge candidate list. If the current block uses merge mode, an index can be used to indicate which block's motion information the current block should merge with, eliminating the need to encode the complete motion information. When building the merge candidate list, motion information from spatially adjacent blocks of the current block, temporal motion information, motion information from non-adjacent blocks in the spatial domain, motion information from non-adjacent blocks in the temporal domain, historical motion information, and synthesized motion information can be included.

[0103] The above-mentioned adjacent blocks in the spatial domain refer to blocks adjacent to the current block in the same image, and the above-mentioned non-adjacent blocks in the spatial domain refer to blocks non-adjacent to the current block in the same image. The above-mentioned motion information in the temporal domain and the motion information of non-adjacent blocks in the temporal domain refer to motion information at a specified position in a collocated reference image. Figure 5 shows a schematic diagram of adjacent blocks of a current block. As shown in Figure 5, the gray block is the current block, where positions 1, 2, 3, 4, and 5 are the positions of the spatially adjacent blocks used for merge, and the other white positions are the positions of the spatially non-adjacent blocks used for merge. Position 6 is the position used for motion information in the temporal domain. If the corresponding position in the lower right corner of the current block is not available, the position corresponding to the center of the current block is used. The other gray positions are the positions used for motion information of non-adjacent blocks in the temporal domain. The method for deriving temporal motion information based on motion information at corresponding positions in a collocated reference image will be described below.

[0104] Historical motion information is independent of position. The codec maintains a first-in, first-out list of motion information. Each time a block is encoded or decoded, the codec updates the list with the motion information of that block, ensuring that there is no duplication with existing motion information in the list. Historical motion information is obtained from this list.

[0105] Temporal motion information prediction is used as a complement to spatial motion information prediction. Generally speaking, the correlation between adjacent regions within the same image is stronger than that between different images, but there are cases where temporal motion information has a stronger correlation. For example, the current block and the surrounding blocks in the current image belong to different objects and have distinct motions. However, the motion of blocks in a reference image that belong to the same object as the current block can provide a better motion prediction for the current block. Temporal motion information prediction is described in detail below with reference to Figure 6.

[0106] Figure 6 shows a schematic diagram of temporal motion information prediction. As shown in Figure 6, the motion vector of the co-located block (i.e., the block from which temporal motion information is derived) in the co-located reference image is the vector from the co-located reference image col_pic to the co-located block's reference image col_ref. For the current block, the required motion vector is the vector from the current image curr_pic to the current block's reference image curr_ref. Let the Point of Order (POC) distance between col_pic and col_ref be td, and the POC distance between curr_pic and curr_ref be tb. Assuming that the motion from the co-located block to the current block is constant, the scaling factor can be determined based on td and tb. Let the motion vector of the co-located block be (col_mv_x, col_mv_y). The temporal motion vector prediction (tmvp_x, tmvp_y) can be derived as follows: tmvp_x = col_mv_x × tb / td, and tmvp_y = col_mv_y × tb / td.

[0107] In relevant standards, the minimum unit for storing motion information in a co-located reference image is 4×4. In other words, each 4×4 sub-block stores a set of motion information. It's understandable that, if hardware implementation costs are not considered, a co-located reference image can also store a set of motion information per pixel. The following describes subblock-based temporal motion vector prediction (SbTMVP).

[0108] The relevant standards introduce sub-block-based temporal motion vector prediction, or SbTMVP. While MVP and TMVP typically share the same MVP for the entire block, SbTMVP performs prediction based on sub-blocks, generating a unique MVP for each sub-block.

[0109] On the other hand, TMVP uses the position of the lower right corner of the current block or the position of the center of the current block to locate the co-located block, while SbTMVP finds a motion offset based on the motion of the surrounding blocks to determine the position. Figure 7 shows a schematic diagram of a sub-block determining the motion vector. As shown in Figure 7, if the block at position A1 refers to the co-located reference image, then the motion offset is set to the motion vector of A1 using the co-located reference image; otherwise, the motion offset is set to (0, 0). In Figure 7, the position is found according to the motion offset, and then the MV corresponding to the position of each sub-block in the "co-located block" is scaled to obtain the MVP of each sub-block.

[0110] The merge mode directly uses the motion information selected in the merge candidate list as the motion information of the current block. In actual videos, there are sometimes some differences between the actual motion vector of the current block and the motion vector in the merge candidate list. The merge mode with motion vector difference (MMVD) is a special merge mode that uses an efficient method to encode the MVD in this case. Figure 8 shows a schematic diagram of the MVD distribution. As shown in Figure 8, MVD is more distributed in a single horizontal direction or a single vertical direction, and there are more MVDs with small values ​​and fewer MVDs with larger values. MMVD uses this characteristic of MVD to encode and decode it.

[0111] MMVD cannot represent any MVD, but can only represent MVDs of specific values ​​in some specific directions. MMVD uses mmvd_direction_idx to represent the direction of the MVD. Of course, it can also be understood as whether the x and y of the MVD are non-zero and the positive and negative signs. mmvd_distance_idx is used to represent the absolute value of the non-zero x and y of the MVD (MmvdDistance). Table 1 below shows the relationship between mmvd_distance_idx[x0][y0] and MmvdDistance[x0][y0], where ph_mmvd_fullpel_only_flag is a picture header identifier (flag) that can set 2 different combinations of MMVD. Table 2 shows the relationship between mmvd_direction_idx[x0][y0] and MmvdSign[x0][y0].

[0112] Table 1

[0113] Table 2

[0114] The MVD of MMVD can be calculated according to formula (1) and formula (2).

[0115] MmvdOffset[x0][y0][0]=(MmvdDistance[x0][y0]<<2)×MmvdSign[x0][y0][0] (1)

[0116] MmvdOffset[x0][y0][1]=(MmvdDistance[x0][y0]<<2)×MmvdSign[x0][y0][1] (2)

[0117] The previous article introduced the simplest and most commonly used translational motion. However, in the real world, there are more than just translations. There are many other forms of motion, such as reduction, magnification, rotation, perspective motion (for example, objects close to the lens appear larger, and objects far from the lens appear smaller), and many irregular forms of motion. Affine can be used to represent motions that are more complex than translation. Figure 9 is a schematic diagram of a motion vector calculation based on affine. As shown in Figure 9, affine uses a linear model to calculate the motion vector of each sub-block or each pixel in the current block based on the motion vectors of two control points (four parameters, a motion vector includes one parameter x and one parameter y) or three control points (six parameters).

[0118] For the 4-parameter affine model, the motion vector at the (x, y) position in the current block is derived as follows:

[0119] For the 6-parameter affine model, the motion vector at the (x, y) position in the current block is derived as follows:

[0120] Among them, (mv0x, mv0y) is the motion vector of the control point in the upper left corner of the current block, (mv1x, mv1y) is the motion vector of the control point in the upper right corner of the current block, and (mv2x, mv2y) is the motion vector of the control point in the lower left corner of the current block.

[0121] In order to simplify the complexity of hardware implementation, affine can divide the current block into 4x4 sub-blocks, calculate an MV for each sub-block and perform motion compensation. Figure 10 shows a schematic diagram of deducing motion vectors based on sub-blocks. As shown in Figure 10, affine only needs a few control points to derive the respective motion vectors for each sub-block. Compared with motion compensation based on the entire block, affine can achieve more precise predictions. And compared with dividing smaller CUs, the overhead of affine is much smaller. It is understandable that with the improvement of hardware processing capabilities, affine can also perform pixel-based processing. That is, a motion vector is derived for each pixel, and motion compensation is performed on a pixel based on the motion vector.

[0122] HEVC supports a maximum CTU of 64x64 and can recursively perform quadtree division. VVC supports a maximum CTU of 128x128, including quadtree, ternary tree and binary tree division. Although these division methods are becoming more and more flexible in block division, whether it is CU, PU, ​​or TU, they can only be divided into rectangular blocks. The boundaries of texture or motion in natural videos are diverse. For example, if you encounter an oblique object boundary, if you simply use rectangular blocks to approach the boundary, many small blocks will be divided, which will significantly increase the overhead. The geometric partitioning mode (GPM) can better handle the texture and boundaries in natural videos.

[0123] GPM uses two prediction blocks of the same size as the current block. Some pixel locations within the GPM prediction block use 100% of the pixel values ​​corresponding to the first prediction block, while others use 100% of the pixel values ​​corresponding to the second prediction block. In the boundary region, or transition region, pixel values ​​from both prediction blocks are used in a certain proportion. The weights in the boundary region also gradually transition. Of course, for scenarios such as encoding screen content, the transition region can be omitted. The weighting of the pixel values ​​corresponding to the two prediction blocks is determined by the GPM's "partitioning" mode, which determines the weight of each pixel location. However, in certain situations, such as with very small block sizes, some GPM modes may not guarantee that some pixel locations will use 100% of the pixel values ​​corresponding to the first prediction block, while others will use 100% of the pixel values ​​corresponding to the second prediction block. GPM can also be considered to use two prediction blocks of different sizes from the current block, each taking a required portion.

[0124] Figure 11 shows a schematic diagram of the weights for the 64 GPM modes. As shown in Figure 11, black indicates a weight of 0% for the position corresponding to the first prediction block, white indicates a weight of 100%, and gray areas, depending on the depth of the color, indicate a weight greater than 0% or less than 100% for the position corresponding to the first prediction block. The weight for the position corresponding to the second reference block is 100% minus the weight for the position corresponding to the first reference block.

[0125] GPM can be considered a prediction mode or method, or rather, a "partitioning" mode that simulates partitioning of the prediction block. This approach is similar to implementing PU partitioning, but without actually partitioning. The first and second prediction blocks used in the GPM can be prediction blocks generated by intra-frame prediction, inter-frame unidirectional prediction, or inter-frame bidirectional prediction.

[0126] The two parts of GPM can use inter-frame prediction or intra-frame prediction, respectively. That is, GPM can combine two inter-frame prediction blocks, or one inter-frame prediction block and one intra-frame prediction block. The following describes decoder-side motion vector refinement (DMVR) and bidirectional optical flow (BDOF) techniques, respectively.

[0127] The bitrate of general consumer video is limited, so video compression typically seeks a compromise between bitrate overhead and distortion. For example, for the same content, within a certain range, finer the block division, the greater the overhead and less distortion; coarser the block division, the less overhead and greater distortion. For example, for the encoding of motion information, within a certain range, for the same content, more precise the motion information, the greater the overhead and less distortion; coarser the motion information, the less overhead and greater distortion. Some decoding-side methods utilize decoding-side information for processing and calculation without incurring overhead, improving motion information and prediction, thereby reducing distortion. Two typical decoding-side methods are DMVR and BDOF.

[0128] One condition for DMVR to start is that the two reference images of the current block come from the forward and backward directions of the current image respectively, and the two reference images are equidistant from the current image. Another starting condition is that the current CU uses the whole-block merge mode (including skip). The whole-block merge mode does not include sub-block-based merges such as SbTMVP and affine merge, because the motion vector in the merge mode is prone to inaccuracy. DMVR uses bilateral matching (BM), which is to calculate the matching cost of the reference blocks on both sides, such as the sum of absolute difference (SAD).

[0129] Figure 12 illustrates a schematic diagram of a DMVR-based motion vector search. As shown in Figure 12, DMVR searches for the matching costs of MVs surrounding the original MV. During the motion, the MVs of the two reference images are mirrored, meaning that one side moves by MVdiff and the other side moves by -MVdiff based on their respective original MVs. The search also supports sub-pixel search, allowing DMVR to potentially find MVs with higher precision than the original MVs. DMVR can perform searches based on specific rules. For example, it typically first searches for integer-pixel MVs within a certain range to find the integer-pixel MV with the lowest matching cost. Then, based on this integer-pixel MV, it searches for sub-pixel MVs. If an MV with a lower matching cost than the original MV is found, the MV with the lower matching cost is used for motion-compensated prediction. The MVs refined by DMVR can be used to store the MVs and surrounding blocks. For example, when constructing a merge candidate list for the current block, if the MVs of surrounding blocks have been refined using DMVR, using these refined MVs to construct the merge candidate list can achieve better compression.

[0130] DMVR can be processed on a sub-block basis. If the horizontal or vertical size of a block is larger than 16 pixels, it will be split into sub-blocks of 16 pixels. This is partly due to hardware implementation complexity, as DMVR requires searching on the decoding end, and limiting the size of sub-blocks can reduce cache costs. On the other hand, dividing the processing into sub-blocks provides greater flexibility, as each sub-block can independently improve the MV, which, to a certain extent, improves the division accuracy and thus improves compression efficiency.

[0131] BDOF is also a decoder-side method. It improves MV and prediction based on the principle of optical flow. Optical flow is the instantaneous velocity of pixels moving across the observation imaging plane for a moving object. Optical flow has several basic assumptions, such as constant brightness, meaning that the brightness of the same object does not change when it moves between images. It also assumes temporal continuity or small motion, meaning that changes in time do not cause drastic changes in the object's position.

[0132] A condition for BDOF to be enabled is that the two reference images for the current block come from the previous and next images, and that the two reference images are equidistant from the current image. For each 4x4 sub-block, BDOF derives a motion vector deviation (v_x, v_y) by minimizing the difference between the predicted values ​​in the two directions. This motion vector deviation is also used to adjust the predicted value in the corresponding sub-block. The derivation process is as follows:

[0133] First, calculate the horizontal and vertical gradients of the two prediction blocks and

[0134] Among them I (k) (i, j) is the predicted value of the coordinate (i, j) in the reference image list k, k = 0, 1, and shift1 is calculated according to the bit depth bitDepth of the brightness, shift1 = max(6, bitDepth-6).

[0135] S1, S2, S3, S5 and S6 are calculated according to the following formula: S1 = ∑ (i,j)∈Ω Abs(ψ x (i,j)) (7) S2=∑ (i,j)∈Ω ψ x (i,j)·Sign(ψ y (i,j)) (8) S3=∑ (i,j)∈Ω θ(i,j)·Sign(ψ x (i,j) (9) S5=∑ (i,j)∈Ω Abs(ψ y (i,j)) (10) S6=∑ (i,j)∈Ω θ(i,j)·Sign(ψ y (i,j)) (11)

[0136] In the above formula, ψ x (i,j) and ψ y (i,j) is calculated according to the following formula: θ(i,j)=(I (1) (i,j)>>n b )-(I (0) (i,j)>>n b ) (14)

[0137] Where Ω is a 6x6 window around the current 4x4 sub-block, n a is min(1,bitDepth-11), n b is min(4,bitDepth-8). Motion vector deviation (v x ,v y ) is calculated as follows:

[0138] in, th′ BIO =2max(5,BD-7), symbol is rounded down, Based on the motion vector deviation and gradient, each prediction value within the 4x4 sub-block is adjusted as follows:

[0139] The final BDOF prediction value is calculated as follows: pred BDOF (x,y)=(I (0) (x,y)+I (1) (x,y)+b(x,y)+o offset )>>shift (18)

[0140] Among them, offset and shift are calculated based on the bit depth of brightness. a , n b Both the shift and the shift are performed to reduce the bit width during the calculation process. The BDOF motion vector deviation can achieve high precision, making the prediction more accurate, and the sub-block-based processing also improves flexibility.

[0141] Both DMVR and BDOF improve motion vectors. DMVR is based on block matching, while BDOF is based on optical flow. They can be used in combination. An example is shown below, which can be called multi-round decoder-side motion vector improvement (MDMVR).

[0142] The first step is to improve the motion vector based on bidirectional matching of the entire block. The second step is to improve the motion vector based on bidirectional matching of sub-blocks. The block size for this step can be 16x16. The third step is to improve the motion vector based on bidirectional optical flow of sub-blocks. The sub-block size for this step can be 8x8. Further steps can be added to this foundation, such as a fourth step to improve the motion vector based on bidirectional optical flow of 4x4 sub-blocks. Or, further, a step to improve the motion vector based on bidirectional optical flow of points. Next, we will introduce template matching technology.

[0143] Template matching exploits the correlation between adjacent pixels, using areas surrounding the current block as templates. Template matching was first used in inter-frame prediction. When generating predictions for inter-coded blocks, the surrounding reconstructed pixels are not required, allowing the prediction process for inter-frame blocks to proceed in parallel. When applying template matching to intra-coded blocks, the reconstructed pixels to the left and above are used as reference pixels. Therefore, when encoding and decoding the current block, the adjacent blocks to its left and above must have completed encoding and decoding. Of course, existing hardware decoder implementations do not guarantee that the adjacent blocks to its left and above have completed encoding and decoding by the time decoding of the current block begins.

[0144] Figure 13 illustrates a schematic diagram of a motion vector determination method based on template matching. As shown in Figure 13, the rectangular areas to the left and above the current block are used as templates. The height of the left template portion is generally the same as the height of the current block, and the width of the upper template portion is generally the same as the width of the current block, but can also be different. The best matching position of the template is found in the reference image to determine the motion information, or motion vector, of the current block. This process can be roughly described as: starting from a starting position in a reference image, a search is conducted within a certain surrounding range. Search rules, such as the search range and search step size, can be predefined. At each position, the degree of match between the template corresponding to that position and the templates surrounding the current block is calculated. The degree of match can be measured using distortion costs, such as SAD (Sum of Absolute Transformed Difference), or SATD (Sum of Absolute Transformed Difference). SATD typically uses transforms such as the Hadamard transform and mean-squared error (MSE). Lower values ​​of SAD, SATD, or MSE indicate a higher degree of match. The cost is calculated using the predicted block of the template corresponding to that position and the reconstructed blocks of the templates surrounding the current block. In addition to searching at integer pixel positions, sub-pixel positions can also be searched. The motion information for the current block is determined based on the position with the highest degree of match. By leveraging the correlation between adjacent pixels, the motion information appropriate for the template may also be appropriate for the current block. Of course, template matching may not be applicable to all blocks. Therefore, methods can be used to determine whether to use the template matching method for the current block, such as using a control switch to indicate whether template matching is used for the current block. This template matching method is called decoder-side motion vector derivation (DMVD). Both the encoder and decoder can use the template to search to derive motion information or find better motion information based on the existing motion information. Both the encoder and decoder perform the same search rules to ensure consistent encoding and decoding, eliminating the need for the encoder to transmit specific motion vectors or motion vector differences. Template matching can improve compression performance, but it requires a "search" on the decoder side, increasing decoding complexity.

[0145] In relevant standards, motion information can be used and stored at the block or sub-block level. This means that motion compensation can be performed down to the sub-block level, with the smallest storage unit being 4x4. It's understandable that, if hardware and software conditions permit, motion information can also be used and stored down to the pixel level. This means that each pixel can have its own motion information. Finer granularity in the use and storage of motion information leads to higher compression performance, but also incurs greater hardware and software costs. Next, we'll introduce intra-frame prediction.

[0146] There is a strong spatial correlation between adjacent parts or pixels within an image. Intra-frame prediction is a prediction method that exploits the spatial correlation between the coded and decoded pixels surrounding the current block and the pixels within the current block. For example, Figure 14 shows a schematic diagram of reference pixels for intra-frame prediction. As shown in Figure 14, the white 4x4 block is the current block, and the gray pixels in the column to the left and the row above the current block are the reference pixels for the current block. Intra-frame prediction uses these reference pixels to predict the current block. These reference pixels may all be available, meaning they have all been coded and decoded. Alternatively, some may not be available. For example, if the current block is at the leftmost edge of the entire frame, the reference pixels to the left of the current block may not be available. Alternatively, when encoding and decoding the current block, the lower left portion of the current block has not yet been coded and decoded, so the reference pixels to the lower left are also not available. In cases where reference pixels are unavailable, available reference pixels, certain values, or certain methods can be used for padding, or no padding can be performed.

[0147] Multiple reference line intra prediction (MRL) can use more reference pixels, thereby improving coding efficiency. Figure 15 shows a schematic diagram of intra prediction using four reference rows / columns. As shown in Figure 15, in MRL, the available reference rows are expanded to three rows. Not only is the most adjacent reference row 0 used as a reference sample, but reference row 1 in the second adjacent row, reference row 2 in the third adjacent row, and reference row 3 in the fourth adjacent row can also be used as reference pixels.

[0148] There are multiple prediction modes for intra-frame prediction. Figure 16 shows a schematic diagram of the nine prediction modes for intra-frame prediction. Mode 0 copies the pixels above the current block vertically to the current block as the prediction value, and mode 1 copies the reference pixels on the left horizontally to the current block as the prediction value. Mode 2 (DC) uses the average value of the eight points A~D and I~L as the prediction value for all points. Modes 3-8 copy the reference pixels to the corresponding positions of the current block at a certain angle. Because some positions of the current block cannot correspond exactly to the reference pixels, it may be necessary to use the weighted average of the reference pixels, or the interpolated sub-pixels of the reference pixels. In addition, there are plane, planar and other modes. With the development of technology and the expansion of blocks, there are more and more angle prediction modes.

[0149] Figure 17 shows a schematic diagram of 35 prediction modes for intra-frame prediction. As shown in Figure 17, the intra-frame prediction modes include planar, DC, and 33 angle modes, totaling 35 prediction modes. Figure 18 shows a schematic diagram of 67 prediction modes for intra-frame prediction. As shown in Figure 18, the intra-frame modes include planar, DC, and 65 angle modes, totaling 67 prediction modes. Of course, in addition to the above 67 modes, wide-angle modes are also provided for some rectangular blocks with a large difference in length and width. As shown in Figure 19, the modes indicated by the dotted lines are the two intervals of -14 to -1 and 67 to 80, which will replace some conventional prediction modes. Next, we will introduce intra block copy (IBC).

[0150] IBC can significantly improve the compression efficiency of screen content encoding, so IBC is usually used for screen content encoding. Unlike the content captured by the camera, the screen content is generated by a computer. The screen content contains noise-free text and computer graphics, etc., so the boundaries of the screen content are clearer. Figure 20 shows an example diagram of screen content. As shown in Figure 20, there is a large amount of repeated content in the screen content. As mentioned above, inter-frame prediction uses the reference block on the reference image as the prediction block of the current block, and the reference image is not the current image. IBC finds a block from the encoded and decoded part or the reconstructed part of the current image as the prediction block of the current block. IBC can also be called current picture referencing (CPR).

[0151] IBC uses a block vector (BV) to represent the position difference between the current block and the reference block, which is similar to the MV of the inter-frame prediction mentioned above. The encoder determines the best matching block of the current block through block matching within the search range and encodes the BV. There are many ways to encode BV, for example, the merge mode can be used. IBC can be considered as an intra-frame prediction, or as another prediction technology independent of intra-frame prediction and inter-frame prediction. IBC is very efficient in encoding screen content and can also improve compression efficiency in natural sequences captured by the camera.

[0152] In related technologies, there is also a prediction method that includes inter-frame prediction and intra-frame prediction, namely combined inter-frame and intra-frame prediction (CIIP). CIIP is a prediction technology that performs a weighted average of inter-frame prediction values ​​and intra-frame prediction values. When predicting a block, both inter-frame prediction technology (such as merge mode) and inter-frame prediction value P are generated. inter , and use intra-frame prediction technology (such as using planar mode) to generate intra-frame prediction value P intra, and the predicted value of CIIP P CIIP By P inter and P intra The weighted generation is shown in formula (19). CIIP =((4-wt)×P inter +wt×P intra +2)>>2 (19)

[0153] Among them, wt is set according to the number of adjacent blocks that are intra-frame coded blocks or inter-frame coded blocks.

[0154] As mentioned above, various prediction techniques are based on motion information. Taking inter-frame prediction as an example, when performing inter-frame prediction on the current block, a reference block is determined in the reference image based on motion information to serve as the prediction block for the current block. After obtaining the reconstructed block for the current block, this motion information is continued to be used by the current block or other blocks. However, the reference block indicated by this motion information in the reconstructed reference image does not necessarily match the reconstructed block for the current block, which may affect codec performance. Therefore, how to improve motion information is a problem that needs to be solved.

[0155] In response to the above problem, an embodiment of the present application provides an encoding method, comprising: determining a reconstructed block of a current block; and determining first motion information of the current block based on the reconstructed block.

[0156] In addition, an embodiment of the present application further provides a decoding method, including: parsing a code stream to determine a reconstructed block of a current block; and determining first motion information of the current block based on the reconstructed block.

[0157] The embodiment of the present application determines motion information in a reconstructed reference image based on a reconstructed block of the current block. Since the reconstructed block can determine the most matching reference block in the reconstructed reference image, the accuracy of the motion information can be improved, thereby helping to improve the performance of the codec.

[0158] The decoding method of the embodiment of the present application is described in detail below with reference to FIG21 .

[0159] Figure 21 is a flowchart of a decoding method provided by an embodiment of the present application. The method of Figure 21 can be applied to a decoder.

[0160] Referring to FIG. 21 , in step S2110 , the code stream is parsed to determine a reconstructed block of the current block. The current block may refer to the current block to be decoded. In some implementations, the current block is a luma block. In other implementations, the current block may also be a chroma block.

[0161] The method of determining the reconstructed block of the current block may include: parsing the code stream to determine the prediction block of the current block; and then determining the reconstructed block according to the prediction block and the residual block.

[0162] The method of determining the reconstructed block according to the prediction block and the residual block may include, for example, taking the sum of the pixel value of the prediction block and the pixel value of the residual block (obtained by parsing the code stream) as the pixel value of the reconstructed block.

[0163] In step S2120 , first motion information of the current block is determined according to the reconstructed block.

[0164] Before determining the first motion information, in some implementations, second motion information may be determined first, and then prediction may be performed based on the second motion information to determine a prediction block for the current block.

[0165] The second motion information may refer to the motion information of the current block. Alternatively, the motion information of other blocks (such as adjacent blocks of the current block) may be used as the "second motion information" of the current block. In other words, the current block may use the motion information of other blocks to determine its own motion information. Alternatively, the MVD may be added to the motion information of other blocks (such as adjacent blocks of the current block) to determine the "second motion information" of the current block. The second motion information may be the motion information determined by the current block using a prediction mode such as merge mode, SbTMVP mode, MMVD mode, or affine mode.

[0166] The method of determining the prediction block of the current block according to the second motion information may include, for example: determining a reference block in a reference image of the current block according to the second motion information; and then using pixel values ​​of the reference block as pixel values ​​of the prediction block of the current block.

[0167] The embodiment of the present application determines motion information in a reconstructed reference image based on a reconstructed block of the current block. Since the reconstructed block can determine the most matching reference block in the reconstructed reference image, the accuracy of the motion information can be improved, thereby helping to improve the performance of the codec.

[0168] The first motion information may include one or more of the following information: a motion vector, index information of a reference image, and identification information of whether to use a reference image.

[0169] The first motion information may refer to block-level motion information. Alternatively, the first motion information may refer to motion information of a portion of the current block. For example, the first motion information refers to motion information of a sub-block or pixel of the current block.

[0170] After determining the first motion information, the first motion information may be used in a decoding process of the current block. For example, the first motion information may be used in a processing process of a reconstructed block of the current block, such as loop filtering of the reconstructed block.

[0171] As shown in Figures 23A and 23B, the process of improving motion information (i.e., determining the first motion information) can occur before or after loop filtering. If it occurs after loop filtering, the first motion information cannot be used in the loop filtering process of the current block. Because loop filtering includes multiple processes, such as deblocking filtering, SAO, ALF, etc., the process of improving motion information can occur before some processes in the loop filtering or after others. For example, improving motion information after the deblocking filtering process, or performing SAO, ALF, etc. after improving motion information.

[0172] Alternatively, in some implementations, the first motion information may be used in the decoding process of other blocks (e.g., the first coding block) that are later in decoding order than the current block, or in other words, the first motion information is used to determine the motion information of the first coding block. For example, the first motion information may be directly used as the motion information of the first coding block, or the first motion information may be used as candidate motion information in the prediction process of the first coding block.

[0173] For example, the first motion information can be used in a specific prediction mode to determine the prediction block of the first coding block. As an example, the first motion information can be introduced into the prediction modes such as the merge mode, SbTMVP mode, MMVD mode, affine mode, etc. mentioned above to predict the first coding block, or directly used as the motion information for inter-frame prediction to predict the first coding block. Taking the merge mode as an example, the first motion information can be used as candidate motion information in the merge candidate list, and then an index can be associated with the first motion information. If the first coding block selects the index, it indicates that the first coding block is predicted based on the first motion information.

[0174] Of course, the first motion information may also be used in a process of processing a reconstructed block of the first coding block, such as loop filtering of the reconstructed block.

[0175] In relevant standards, the deblocking filter determines the boundary filtering strength based on motion information. As the name suggests, the deblocking filter can eliminate the block effect to a certain extent. Since the image content is processed based on blocks, different blocks are processed differently, and the block boundaries may show obvious differences. Of course, the difference may be caused by codec distortion, or it may be the texture of the image itself. Therefore, the deblocking filter can determine the boundary filtering strength based on whether the motion information of adjacent blocks is the same. If the number of reference images of two blocks is different or the reference images are different, the boundary filtering strength of the luminance component is set to 1. If the motion vectors of adjacent blocks are different (the difference in the horizontal or vertical direction is greater than or equal to half a pixel), the boundary filtering strength of the luminance component is set to 1. If the reference images of adjacent blocks are the same and the motion vectors are the same, the boundary filtering strength of the luminance component is set to 0.

[0176] Based on the above introduction to deblocking filtering, when performing deblocking filtering between a first coding block and its adjacent blocks, or between sub-blocks within the first coding block, the boundary filter strength can be determined based on the first motion information. Different motion vectors will affect the boundary filter strength setting. Using the first motion information for deblocking filtering is more accurate than the second motion information, which helps the deblocking filter make more reasonable boundary filter strength settings, thereby helping to improve compression efficiency.

[0177] As described above, the first coding block can be decoded using the first motion information. The first coding block may not consider other conditions when using the first motion information. Alternatively, a certain condition (first condition) may be set, and the first motion information can be used for decoding only when the first coding block meets the first condition. In some implementations, the first condition is related to the positional relationship of the current block, that is, if the positional relationship between the first coding block and the current block does not meet the first condition, the first motion information is not allowed to be used for decoding.

[0178] The embodiments of the present application do not specifically limit the first condition. For example, the first condition may be that the first coding block and the current block are not adjacent. For another example, the first condition may be that the first coding block and the current block belong to different CTUs. For another example, the first condition may be that the first coding block and the current block belong to different pictures.

[0179] As an example, if a first coding block is adjacent to the current block, the first coding block is not allowed to be decoded using the first motion information, or in other words, the first coding block is only allowed to be decoded using the second motion information; if a coding block is not adjacent to the current block, the first coding block is allowed to be decoded using the first motion information.

[0180] As another example, if the first coding block and the current block belong to the same CTU, the first coding block is not allowed to be decoded using the first motion information, or in other words, the first coding block is only allowed to be decoded using the second motion information; if the first coding block and the current block belong to different CTUs, the first coding block is allowed to be decoded using the first motion information.

[0181] As another example, if the first coding block and the current block belong to the same image, the first coding block is not allowed to be decoded using the first motion information, or in other words, the first coding block is only allowed to be decoded using the second motion information; if the first coding block and the current block belong to different images, the first coding block is allowed to be decoded using the first motion information.

[0182] According to the introduction above, the first motion information is determined after the reconstruction of the current block is completed. If the first coding block is decoded based on the first motion information, the first coding block needs to wait for a certain time to obtain the first motion information, thereby generating a delay time. The following is a detailed description using Figures 24A and 24B as examples. Figure 24A shows a flow chart of decoding the current block. As shown in Figure 24A, in the decoding process of the current block, the motion information needs to be determined first, and then motion compensation is performed to generate a prediction block. Figure 24B shows a flow chart of decoding the adjacent blocks of the current block. As shown in Figure 24B, the adjacent blocks of the current block can enter the process of determining the motion information only after the motion information that needs to be referenced is determined. If the adjacent blocks use the first motion information, the adjacent blocks can enter the process of determining the motion information only after the current block determines the first motion information. Therefore, the decoding process of the adjacent blocks may be delayed.

[0183] The embodiment of the present application prohibits some coding blocks from using the first motion information for decoding by setting the above-mentioned first condition, thereby avoiding delays in the decoding process of these coding blocks, and avoiding affecting the decoding efficiency while improving the decoding performance.

[0184] As previously described, the first motion information is determined based on the reconstructed block. In some implementations, a certain condition (second condition) can be set to determine whether the first motion information needs to be determined during the decoding process of the current block. For example, if the current block meets the second condition, the first motion information can be determined based on the reconstructed block; if the current block does not meet the second condition, the first motion information is not determined based on the reconstructed block.

[0185] The second condition may be associated with one or more of the residual information and prediction mode of the current block. For example, the second condition may be that the current block has residual information. Another example may be that the current block uses an intra prediction mode.

[0186] If the current block does not have residual information, it may indicate that the reference block determined based on the second motion information closely matches the reconstructed block of the current block, meaning that the second motion information is relatively accurate. Therefore, not determining the first motion information at this point can save computational overhead during decoding, thereby improving decoding efficiency.

[0187] If the current block uses intra prediction, it means that the current block does not carry its own motion information in the code stream. In this case, other coding blocks (such as adjacent blocks of the current block) cannot use the motion information of the current block for prediction. In this case, the motion information of the current block can be restored by performing the operation of determining the first motion information, so that other coding blocks can use this motion information, thereby improving the prediction effect of other coding blocks and further helping to improve the performance of encoding and decoding.

[0188] It should be understood that the first condition and the second condition mentioned above can be used in combination or separately in the decoding process, and the embodiments of the present application do not specifically limit this.

[0189] The embodiment of the present application does not specifically limit the manner in which step S2120 is determined. In some implementations, a predefined rule may be used at the codec end to determine by default whether motion information is determined based on reconstruction information. For example, as long as a coding block (block level) meets certain conditions (such as the first condition and / or the second condition described above), motion information may be determined based on reconstruction information. In other implementations, whether motion information is determined based on reconstruction information may be determined by parsing information in syntax elements.

[0190] In some implementations, the first identification information is obtained by parsing the bitstream. The first identification information is used to indicate whether the motion information is determined based on the reconstruction information.

[0191] The first identification information may indicate whether the motion information of the current block or a portion of the current block is determined based on the reconstruction information. Alternatively, in some implementations, the first identification information may indicate whether the motion information of a higher-level image is determined based on the reconstruction information.

[0192] For example, the first identification information may be sequence-level or slice-level information to indicate whether a slice or a sequence determines motion information based on reconstruction information. For example, the first identification information may be carried in a sequence parameter set (SPS) or a slice header.

[0193] The first identification information may be composed of any combination of letters and / or numbers. For example, the first identification information may be represented as sps_mirar_enabled_flag.

[0194] The first identification information may include a first value (such as 1 or true) and a second value (such as 0 or false). Accordingly, the first value is used to indicate that the current block determines motion information based on reconstruction information, and the second value is used to indicate that the current block does not determine motion information based on reconstruction information.

[0195] As an example, if the sps_mirar_enabled_flag obtained by parsing the bitstream is 1, it indicates that the current block determines motion information based on reconstruction information; if the sps_mirar_enabled_flag obtained by parsing the bitstream is 0, it indicates that the current block does not determine motion information based on reconstruction information.

[0196] As previously described, the first motion information of the current block may be determined based on the reconstructed block. Further, in some implementations, the first motion information may be determined based on the reconstructed block and the second motion information. The second motion information is used to determine the starting position of one or more reference blocks.

[0197] For example, an initial reference block can be determined in the reference image based on the position information of the reconstructed block and the indication of the second motion information; then, a search is performed around the initial reference block to determine multiple reference blocks; finally, a target reference block is selected from the initial reference block and the multiple reference blocks, and the motion information indicating the target reference block is used as the first motion information.

[0198] In some implementations, the first motion information may be determined based on a degree of matching between the current block and one or more reference blocks in a reference frame. The degree of matching may refer to a cost between the reconstructed block and the one or more reference blocks. The cost may be, for example, SAD, SATD, or the sum of squares due to error (SSE).

[0199] For example, after determining multiple reference blocks in the reference image, the SAD between the reconstructed block and the multiple reference blocks is calculated, thereby obtaining multiple SAD values. A smaller SAD value indicates a higher degree of match between the reference block and the reconstructed block, while a larger SAD value indicates a lower degree of match between the reference block and the reconstructed block. The reference block corresponding to the minimum SAD value is used as the target reference block, and motion information indicating the target reference block is used as the first motion information.

[0200] In some implementations, the aforementioned multiple reference blocks may be determined in a reference image based on a preset search range and / or search order. For example, a sparse search may be performed first in the reference image, followed by a detailed search around the reference block corresponding to a smaller SAD value. For another example, an integer pixel search may be performed first in the reference image, followed by a sub-pixel search. For another example, when searching a reference image, an entire block may be searched first, followed by searching the entire block into sub-blocks. For another example, when searching a reference image, the search may be performed in multiple layers, such as first searching the entire block, and then dividing the entire block into 16x16 sub-blocks after determining the optimal MV, and then searching the entire block again based on the 16x16 sub-blocks, and then dividing the entire block into 8x8 sub-blocks after determining the optimal MV, and then dividing the entire block into 4x4 sub-blocks after determining the optimal MV.

[0201] A more specific example is given below to introduce in more detail the method of determining the first motion information from the reconstructed block and the second motion information.

[0202] As an example, based on the indication of the second motion information, the starting position for searching for a reference block in the reference image is determined, for example, using the position indicated by the second motion information as the starting position. If the current block or a sub-block does not have second motion information, for example, if the current block or a sub-block uses only intra-frame prediction, a default motion information can be set for the current block or sub-block as the starting position for the search, such as using the motion information of a neighboring position. A search range and search order can then be preset, and the search performed according to the preset search range and search order. For example, the search range can be set to a rectangular area of ​​8 pixels above, below, and to the left and right of the search starting point, and the search order can be performed in a grid scan from left to right and top to bottom. For example, the search can begin with a sparse search followed by a detailed search around the less costly motion vectors, or alternatively, a whole-pixel search can be performed first followed by a sub-pixel search. Ultimately, one or more reference blocks are determined within the reference frame. The degree of match between the reconstructed block and the multiple reference blocks can be represented by one of the SAD, SATD, or SSE values ​​between the reconstructed block and the reference blocks. Taking SAD as an example, the smaller the SAD value between the reconstructed block and the reference block, the higher the matching degree, and the reference block corresponding to the minimum SAD value is used as the most matching reference block to determine the first motion information.

[0203] In addition, when determining mv, you can first search the entire block, and then divide the entire block into sub-blocks for search. The search can be performed in multiple layers. For example, first search the entire block, and after determining the optimal motion information, divide the entire block into 16x16 sub-blocks, and then search the 16x16 sub-blocks again. After determining the optimal motion information, divide the entire block into 8x8 sub-blocks, and after determining the optimal motion information, divide the entire block into 4x4 sub-blocks, and finally determine the improved motion information of each sub-block. Of course, if the current block is motion compensated by sub-block, then it can also be improved directly by sub-block. When searching, you can first search the entire block, and then divide the entire block into sub-blocks for search. The search can be performed in multiple layers, such as searching by the entire block first, and then dividing the entire block into 16x16 sub-blocks after determining the optimal mv, and then searching by 16x16 sub-blocks again, and then dividing the entire block into 8x8 sub-blocks after determining the optimal mv, and then dividing the entire block into 4x4 sub-blocks after determining the optimal mv, and finally determining the improved motion information of each sub-block. Of course, if the current block is originally motion compensated by sub-block, then it can also be improved directly by sub-block. The above is a detailed description of the decoding method provided by the embodiment of the present application in conjunction with Figure 21. The following is a detailed description of the encoding method provided by the embodiment of the present application in conjunction with Figure 22.

[0204] Figure 22 is a flow chart of an encoding method provided by an embodiment of the present application. The method of Figure 22 can be applied to an encoder.

[0205] Referring to FIG. 22 , in step S2210 , a reconstructed block of a current block is determined. The current block may refer to a current block to be encoded. In some implementations, the current block is a luminance block. In other implementations, the current block may also be a chrominance block.

[0206] The method of determining the reconstructed block of the current block may include: determining a prediction block of the current block; and then determining the reconstructed block according to the prediction block and the residual block.

[0207] The method of determining the reconstructed block according to the prediction block and the residual block may include, for example, taking the sum of the pixel value of the prediction block and the pixel value of the residual block as the pixel value of the reconstructed block.

[0208] In step S2220, first motion information of the current block is determined according to the reconstructed block.

[0209] Before determining the first motion information, in some implementations, second motion information may be determined first, and then prediction may be performed based on the second motion information to determine a prediction block for the current block.

[0210] The second motion information may refer to the motion information of the current block. Alternatively, the motion information of other blocks (such as adjacent blocks of the current block) may be used as the "second motion information" of the current block. In other words, the current block may use the motion information of other blocks to determine its own motion information. Alternatively, the MVD may be added to the motion information of other blocks (such as adjacent blocks of the current block) to determine the "second motion information" of the current block. The second motion information may be the motion information determined by the current block using a prediction mode such as merge mode, SbTMVP mode, MMVD mode, or affine mode.

[0211] The method of determining the prediction block of the current block according to the second motion information may include, for example: determining a reference block in a reference image of the current block according to the second motion information; and then using pixel values ​​of the reference block as pixel values ​​of the prediction block of the current block.

[0212] The embodiment of the present application determines motion information in a reconstructed reference image based on a reconstructed block of the current block. Since the reconstructed block can determine the most matching reference block in the reconstructed reference image, the accuracy of the motion information can be improved, thereby helping to improve encoding performance.

[0213] The first motion information may include, for example, one or more of the following information: a motion vector, index information of a reference frame, and identification information of whether to use a reference image.

[0214] The first motion information may refer to block-level motion information. Alternatively, the first motion information may refer to motion information of a portion of the current block. For example, the first motion information refers to motion information of a sub-block or pixel of the current block.

[0215] After the first motion information is determined, the first motion information may be used in an encoding process of the current block. For example, the first motion information may be used in a processing process of a reconstructed block of the current block, such as loop filtering of the reconstructed block.

[0216] As shown in Figures 23A and 23B, the process of improving motion information (i.e., determining the first motion information) can occur before or after loop filtering. If it occurs after loop filtering, the first motion information cannot be used in the loop filtering process of the current block. Because loop filtering includes multiple processes, such as deblocking filtering, SAO, ALF, etc., the process of improving motion information can occur before some processes in the loop filtering or after others. For example, improving motion information after the deblocking filtering process, or performing SAO, ALF, etc. after improving motion information.

[0217] Alternatively, in some implementations, the first motion information may be used in the encoding process of other blocks (e.g., the first coding block) that are later in the coding order than the current block, or in other words, the first motion information is used to determine the motion information of the first coding block. For example, the first motion information may be directly used as the motion information of the first coding block, or the first motion information may be used as candidate motion information in the prediction process of the first coding block.

[0218] For example, the first motion information can be used in a specific prediction mode to determine the prediction block of the first coding block. As an example, the first motion information can be introduced into the prediction modes such as the merge mode, SbTMVP mode, MMVD mode, Affine mode mentioned above to predict the first coding block, or directly used as the motion information for inter-frame prediction to predict the first coding block. Taking the merge mode as an example, the first motion information can be used as candidate motion information in the merge candidate list, and then an index can be associated with the first motion information. If the first coding block selects the index, it indicates that the first coding block is predicted based on the first motion information.

[0219] Of course, the first motion information may also be used in a process of processing a reconstructed block of the first coding block, such as loop filtering of the reconstructed block.

[0220] In relevant standards, the deblocking filter determines the boundary filtering strength based on motion information. As the name suggests, the deblocking filter can eliminate the block effect to a certain extent. Since the image content is processed based on blocks, different blocks are processed differently, and the block boundaries may show obvious differences. Of course, the difference may be caused by codec distortion, or it may be the texture of the image itself. Therefore, the deblocking filter can determine the boundary filtering strength based on whether the motion information of adjacent blocks is the same. If the number of reference images of two blocks is different or the reference images are different, the boundary filtering strength of the luminance component is set to 1. If the motion vectors of adjacent blocks are different (the difference in the horizontal or vertical direction is greater than or equal to half a pixel), the boundary filtering strength of the luminance component is set to 1. If the reference images of adjacent blocks are the same and the motion vectors are the same, the boundary filtering strength of the luminance component is set to 0.

[0221] Based on the above introduction to deblocking filtering, when performing deblocking filtering between a first coding block and its adjacent blocks, or between sub-blocks within the first coding block, the boundary filter strength can be determined based on the first motion information. Different motion vectors will affect the boundary filter strength setting. Using the first motion information for deblocking filtering is more accurate than the second motion information, which helps the deblocking filter make more reasonable boundary filter strength settings, thereby helping to improve compression efficiency.

[0222] As described above, the first coding block can be encoded using the first motion information. The first coding block may not consider other conditions when using the first motion information. Alternatively, a certain condition (first condition) may be set, and the first motion information can be used for encoding only when the first coding block meets the first condition. In some implementations, the first condition is related to the positional relationship of the current block, that is, if the positional relationship between the first coding block and the current block does not meet the first condition, the first motion information is not allowed to be used for encoding.

[0223] The embodiments of the present application do not specifically limit the first condition. For example, the first condition may be that the first coding block and the current block are not adjacent. For another example, the first condition may be that the first coding block and the current block belong to different CTUs. For another example, the first condition may be that the first coding block and the current block belong to different pictures.

[0224] As an example, if a first coding block is adjacent to the current block, the first coding block is not allowed to be encoded using the first motion information, or in other words, the first coding block is only allowed to be encoded using the second motion information; if a coding block is not adjacent to the current block, the first coding block is allowed to be encoded using the first motion information.

[0225] As another example, if the first coding block and the current block belong to the same CTU, the first coding block is not allowed to be encoded using the first motion information, or in other words, the first coding block is only allowed to be encoded using the second motion information; if the first coding block and the current block belong to different CTUs, the first coding block is allowed to be encoded using the first motion information.

[0226] As another example, if the first coding block and the current block belong to the same image, the first coding block is not allowed to be encoded using the first motion information, or in other words, the first coding block is only allowed to be encoded using the second motion information; if the first coding block and the current block belong to different images, the first coding block is allowed to be encoded using the first motion information.

[0227] According to the introduction above, the first motion information is determined after the reconstruction of the current block is completed. If the first coding block is encoded based on the first motion information, then the first coding block needs to wait for a certain time to obtain the first motion information, thereby generating a delay time. The following is a detailed description using Figures 24A and 24B as examples. Figure 24A shows a flow chart of decoding the current block. As shown in Figure 24A, in the decoding process of the current block, it is necessary to first determine the motion information before performing motion compensation to generate a prediction block. Figure 24B shows a flow chart of decoding the adjacent blocks of the current block. As shown in Figure 24B, the adjacent blocks of the current block can enter the process of determining the motion information only after the motion information that needs to be referenced is determined. If the adjacent blocks use the first motion information, the adjacent blocks can enter the process of determining the motion information only after the current block determines the first motion information. Therefore, the decoding process of the adjacent blocks may be delayed.

[0228] The embodiment of the present application prohibits some coding blocks from using the first motion information for encoding by setting the above-mentioned first condition, thereby avoiding delays in the encoding process of these coding blocks, and avoiding affecting the encoding efficiency while improving the encoding performance.

[0229] As previously described, the first motion information is determined based on the reconstructed block. In some implementations, a certain condition (second condition) can be set to determine whether the first motion information needs to be determined during the encoding process of the current block. For example, if the current block meets the second condition, the first motion information can be determined based on the reconstructed block; if the current block does not meet the second condition, the first motion information is not determined based on the reconstructed block.

[0230] The second condition may be associated with one or more of the residual information and prediction mode of the current block. For example, the second condition may be that the current block has residual information. Another example may be that the current block uses an intra prediction mode.

[0231] If the current block does not have residual information, it may indicate that the reference block determined based on the second motion information closely matches the reconstructed block of the current block, meaning that the second motion information is relatively accurate. Therefore, not determining the first motion information at this point can save computational overhead and improve encoding efficiency.

[0232] If the current block uses intra prediction, it means that the current block does not carry its own motion information in the code stream. In this case, other coding blocks (such as adjacent blocks of the current block) cannot use the motion information of the current block for prediction. In this case, by performing the operation of determining the first motion information, the motion information of the current block can be restored, so that other coding blocks can use this motion information, thereby improving the prediction effect of other coding blocks and further helping to improve encoding performance.

[0233] It should be understood that the first condition and the second condition mentioned above can be used in combination during the encoding process or can be used separately, and the embodiments of the present application do not specifically limit this.

[0234] The embodiment of the present application does not specifically limit the manner in which step S2220 is determined. In some implementations, a predefined rule may be used at the codec end to default to whether motion information is determined based on reconstruction information. For example, as long as a coding block (block level) meets certain conditions (such as the first condition and / or the second condition described above), motion information can be determined based on reconstruction information. In other implementations, it may be determined at the encoding end whether motion information is determined based on reconstruction information, and the result may be transmitted to the decoding end in the form of a syntax element.

[0235] In some implementations, the first identification information may be written into the bitstream. The first identification information is used to indicate whether the motion information is determined based on the reconstruction information.

[0236] The first identification information may indicate whether the motion information of the current block or a portion of the current block is determined based on the reconstruction information. Alternatively, in some implementations, the first identification information may indicate whether the motion information of a higher-level image is determined based on the reconstruction information.

[0237] For example, the first identification information may be sequence-level or slice-level information to indicate whether a slice or a sequence determines motion information based on reconstruction information. For example, the first identification information may be carried in a sequence parameter set (SPS) or a slice header.

[0238] The first identification information may be composed of any combination of letters and / or numbers. For example, the first identification information may be represented as sps_mirar_enabled_flag.

[0239] The first identification information may include a first value (such as 1 or true) and a second value (such as 0 or false). Accordingly, the first value is used to indicate that the current block determines motion information based on reconstruction information, and the second value is used to indicate that the current block does not determine motion information based on reconstruction information.

[0240] As an example, if the value of sps_mirar_enabled_flag is 1, it indicates that the current block determines motion information based on reconstruction information; if the value of sps_mirar_enabled_flag is 0, it indicates that the current block does not determine motion information based on reconstruction information.

[0241] As previously described, the first motion information of the current block may be determined based on the reconstructed block. Further, in some implementations, the first motion information may be determined based on the reconstructed block and the second motion information. The second motion information is used to determine the starting position of one or more reference blocks.

[0242] For example, an initial reference block can be determined in the reference image based on the position information of the reconstructed block and the indication of the second motion information; then, a search is performed around the initial reference block to determine multiple reference blocks; finally, a target reference block is selected from the initial reference block and the multiple reference blocks, and the motion information indicating the target reference block is used as the first motion information.

[0243] In some implementations, the first motion information may be determined based on a degree of matching between the current block and one or more reference blocks in a reference frame. The degree of matching may refer to a cost between the reconstructed block and the one or more reference blocks. The cost may be, for example, SAD, SATD, or SSE.

[0244] For example, after determining multiple reference blocks in the reference image, the SAD between the reconstructed block and the multiple reference blocks is calculated, thereby obtaining multiple SAD values. A smaller SAD value indicates a higher degree of match between the reference block and the reconstructed block, while a larger SAD value indicates a lower degree of match between the reference block and the reconstructed block. The reference block corresponding to the minimum SAD value is used as the target reference block, and motion information indicating the target reference block is used as the first motion information.

[0245] In some implementations, the aforementioned multiple reference blocks may be determined in a reference image based on a preset search range and / or search order. For example, a sparse search may be performed first in the reference image, followed by a detailed search around the reference block corresponding to a smaller SAD value. For another example, an integer pixel search may be performed first in the reference image, followed by a sub-pixel search. For another example, when searching a reference image, an entire block may be searched first, followed by searching the entire block into sub-blocks. For another example, when searching a reference image, the search may be performed in multiple layers, such as first searching the entire block, and then dividing the entire block into 16x16 sub-blocks after determining the optimal MV, and then searching the entire block again based on the 16x16 sub-blocks, and then dividing the entire block into 8x8 sub-blocks after determining the optimal MV, and then dividing the entire block into 4x4 sub-blocks after determining the optimal MV.

[0246] A more specific example is given below to introduce in more detail the method of determining the first motion information from the reconstructed block and the second motion information.

[0247] As an example, based on the indication of the second motion information, the starting position for searching for a reference block is determined in the reference image. If the current block or a sub-block does not have second motion information, for example, if the current block or a sub-block uses only intra-frame prediction, a default motion information can be set for the current block or sub-block as the starting position for the search, such as the motion information of a neighboring position. A search range and search order can then be preset, and the search performed according to the preset search range and search order. For example, the search range can be set to a rectangular area of ​​8 pixels above, below, and to the left and right of the search starting point, and the search order can be performed in a grid scan from left to right and top to bottom. For example, the search can begin with a sparse search followed by a detailed search around the less costly motion vectors, or a whole-pixel search can be performed first followed by a sub-pixel search. Ultimately, one or more reference blocks are determined within the reference frame. The degree of match between the reconstructed block and the multiple reference blocks can be represented by one of the SAD, SATD, or SSE values ​​between the reconstructed block and the reference blocks. Taking SAD as an example, the smaller the SAD value between the reconstructed block and the reference block, the higher the matching degree, and the reference block corresponding to the minimum SAD value is used as the most matching reference block to determine the first motion information.

[0248] In addition, when determining the MV, you can first search the entire block, then divide the entire block into sub-blocks and search again. The search can be performed in multiple layers. For example, first search the entire block, and after determining the optimal motion information, divide the entire block into 16x16 sub-blocks, and then search the 16x16 sub-blocks again. After determining the optimal motion information, divide the entire block into 8x8 sub-blocks, and after determining the optimal motion information, divide the entire block into 4x4 sub-blocks, and finally determine the improved motion information for each sub-block. Of course, if the current block is motion compensated by sub-block, then it can also be improved directly by sub-block. When searching, you can first search the entire block, and then divide the entire block into sub-blocks and search again. The search can be performed in multiple layers. For example, first search the entire block, determine the optimal MV, then divide the entire block into 16x16 sub-blocks, search the 16x16 sub-blocks again, determine the optimal MV, then divide the entire block into 8x8 sub-blocks, and finally determine the improved motion information for each sub-block. Of course, if the current block is already motion compensated by sub-block, then it can also be improved directly by sub-block.

[0249] The following describes the embodiments of the present application in more detail with reference to specific examples. It should be noted that the examples below are only intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. It is obvious that those skilled in the art can make various equivalent modifications or changes based on the examples given below, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0250] When performing inter-frame prediction on the current block, motion information is used to identify a reference block in the reference image to serve as the prediction block for the current block. After the current block's reconstructed block is obtained, this motion information is used by the current block and other blocks. However, the reference block indicated by this motion information in the reconstructed reference image may not necessarily match the reconstructed block of the current block, which may affect codec performance. Therefore, improving motion information is a problem that needs to be solved.

[0251] This example re-determines motion information in a reconstructed reference image based on the reconstructed block of the current block. Because the reconstructed block can determine the most closely matching reference block in the reconstructed reference image, the accuracy of the motion information is improved, thereby enhancing codec performance. This example solution is described in detail below.

[0252] In this example, a possible solution to improve motion information is to use the reconstructed block to search on the reference image to determine the improved motion information. Specifically, the existing motion vector of the current block can be used as the starting position for the search. The degree of matching between the reconstructed block and the reference block can be determined. The degree of matching can be represented by the SAD (sum of absolute difference), SATD or SSE of the reconstructed block and the reference block. Taking SAD as an example, the smaller the SAD, the higher the degree of matching, and vice versa. The search range and search order can be preset, and the search can be performed according to the preset search range and search order to determine the improved motion information. When searching, a sparse search can be performed first and then a detailed search can be performed around the mv with a smaller SAD. A whole pixel search can be performed first and then a sub-pixel search can be performed.

[0253] When searching, you can first search the entire block, and then divide the entire block into sub-blocks for searching. The search can be performed in multiple layers, such as searching the entire block first, and then dividing the entire block into 16x16 sub-blocks after determining the optimal mv, and then searching the 16x16 sub-blocks again, and then dividing the entire block into 8x8 sub-blocks after determining the optimal mv, and then dividing the entire block into 4x4 sub-blocks after determining the optimal mv, and finally determining the improved motion information of each sub-block. Of course, if the current block is originally motion compensated by sub-blocks, then it can also be improved directly by sub-blocks. If the current block or a sub-block does not have valid motion information, such as the current block or a sub-block only uses intra-frame prediction, then a possible solution is to set default motion information for the current block or sub-block as the starting position of the search, such as taking the motion information of a certain adjacent position.

[0254] The solution of this example may be referred to as motion information refinement after reconstruction (MIRAR), or motion vector refinement after reconstruction (MVRAR). Hereinafter, MIRAR is used to represent the solution of this example.

[0255] The improved motion information based on this example can be used for subsequent applications. Usage scenarios include but are not limited to the following:

[0256] 1. As motion information for the current block, when encoding or decoding subsequent blocks, the motion information of the current block can be obtained from adjacent or non-adjacent locations in the spatial domain, or from adjacent or non-adjacent locations in the temporal domain. This information can also be used to update the HMVP's history-based motion vector prediction list. In short, the improved motion information can be used by any method that can obtain it. This provides more accurate motion information for subsequent encoded and decoded blocks, such as blocks in the same image or blocks in other images.

[0257] 2. Used as the motion information of the current block in subsequent processing involving the current block. For example, in the process of loop filtering, if the motion information of the current block is needed, the improved motion information can be used.

[0258] In Usage Scenario 1, it was mentioned that blocks in the same image and blocks in other images can be used. An example of using blocks in other images is when the current image is used as a co-located reference image for deriving temporal motion information, resulting in blocks in other images using the current image's motion information. When used as a co-located reference image, the current image's motion information is preserved at a certain granularity.

[0259] MIRAR may introduce latency when used with adjacent blocks within the same image. Specifically, hardware considerations generally require a certain degree of parallelism, allowing multiple processes to be pipelined between blocks. Figure 24A illustrates a schematic flow for decoding the current block. Figure 24B illustrates a schematic flow for decoding adjacent blocks of the current block. As shown in Figure 24A, the decoding process for the current block requires first determining motion information before performing motion compensation to generate a prediction block. Assuming there is no dependency between the prediction block and the residual block generation process, the residual block can be determined as shown in Figure 24A, and the reconstructed block can then be determined based on the motion-compensated prediction block and residual block. As shown in Figure 24B, adjacent blocks of the current block can enter the motion information determination process after all reference motion information has been determined. However, when using MIRAR, adjacent blocks that may require motion information from the current block can only enter the motion information determination process after the current block has processed and improved its motion information. This may result in latency in the decoding of adjacent blocks. To avoid latency in the decoding of blocks, certain conditions should be set to prohibit the use of MIRAR for blocks that may be prone to latency.

[0260] To reduce latency, the use of MIRAR's improved motion information can be restricted, as shown below.

[0261] 1. When a block adjacent to the current block obtains motion information of the current block, it can only use MIRAR to improve the previous motion information. However, when a block not adjacent to the current block obtains motion information of the current block, it can use the motion information improved by MIRAR.

[0262] 2. When obtaining motion information of the current block from a block in the same CTU as the current block, only the previous motion information can be improved using MIRAR. However, when obtaining motion information of the current block from a block in a different CTU, the improved motion information can be used.

[0263] 3. The motion information improved by MIRAR is only used for the derivation of temporal motion information, i.e., it is used by different images.

[0264] It should be noted that the above restrictions can also be used to reduce the delay caused by motion information improved by solutions such as DMVR, BDOF, and template matching.

[0265] This example can trigger MIRAR in any case, as long as the current image or slice is an inter-coded image or slice. Of course, you can also set certain conditions, as shown below.

[0266] 1. A possible triggering condition for MIRAR is that the current block has coded residuals, or the current block has coded coefficients, or the coefficients of the current block are not all zero.

[0267] 2. A possible condition for triggering MIRAR is that the current block uses intra prediction.

[0268] It is understood that MIRAR can be used under only one of the conditions, or can be used as long as one of the two conditions is met.

[0269] This example does not require identification information at the block level to indicate whether the current block should perform MIRAR. It can be automatically triggered as long as the conditions are met.

[0270] However, in high-level syntax, a flag can be set to indicate whether MIRAR can be used at a certain level. For example, in the sequence parameter set (SPS) or slice header. The first flag can be represented by sps_mirar_enabled_flag. If the value of sps_mirar_enabled_flag is 1, it indicates that MIRAR can be used in the current sequence. If the value of sps_mirar_enabled_flag is 0, it indicates that MIRAR cannot be used in the current sequence.

[0271] The granularity of MIRAR's improved motion information can be block-level, sub-block-level, or pixel-level. The granularity of the improved motion information refers to the range of pixels that share a set of motion information.

[0272] In the case of irregular motion, MIRAR utilizes the information of the reconstructed block to provide more accurate motion information for subsequent encoded and decoded blocks without increasing the bitrate overhead, thereby improving compression efficiency.

[0273] The method embodiment of the present application is described in detail above in conjunction with Figures 1 to 24. The device embodiment of the present application is described in detail below in conjunction with Figures 25 to 28. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment. Therefore, for parts not described in detail, reference can be made to the above method embodiment.

[0274] FIG25 is a schematic diagram of the structure of a decoder provided by an embodiment of the present application. As shown in FIG25 , the decoder 2500 includes: a first determining unit 2510 and a second determining unit 2520 .

[0275] The first determining unit 2510 is configured to parse the code stream and determine a reconstructed block of the current block.

[0276] The second determining unit 2520 is configured to determine first motion information of the current block according to the reconstructed block.

[0277] In some implementations, the first motion information is used to determine motion information of a first coded block that is later than the current block in decoding order.

[0278] In some implementations, a positional relationship between the first coding block and the current block satisfies a first condition.

[0279] In some implementations, the first condition includes: the first coding block is not adjacent to the current block; or, the first coding block and the current block belong to different CTUs; or, the first coding block and the current block belong to different images.

[0280] In some implementations, the decoder 2500 further includes a third determining unit 2530 configured to determine a prediction block of a second coding block based on second motion information of the current block, where a positional relationship between the second coding block and the current block does not satisfy the first condition.

[0281] In some implementations, the first motion information is used to process the reconstructed block.

[0282] In some implementations, the first motion information is used to perform loop filtering on the reconstructed block.

[0283] In some implementations, the second determining unit 2520 is further configured to determine the first motion information based on the reconstructed block if a second condition is satisfied.

[0284] In some implementations, the second condition is related to one or more of the following: residual information of the current block; or a prediction mode of the current block.

[0285] In some implementations, the second condition includes one or more of the following: residual information exists in the current block; and the current block uses an intra-frame prediction mode.

[0286] In some implementations, the decoder 2500 further includes a fourth determining unit 2540 configured to parse the code stream and determine first identification information, where the first identification information is used to indicate whether the motion information is determined based on the reconstruction information.

[0287] In some implementations, the first identification information is sequence-level or stripe-level information.

[0288] In some implementations, the first motion information is determined based on the reconstructed block and the second motion information.

[0289] In some implementations, the first motion information is determined based on a degree of matching between the current block and one or more reference blocks in a reference image, and the second motion information is used to determine a starting position of the one or more reference blocks.

[0290] In some implementations, the first motion information is determined based on a degree of matching between the current block and one or more reference blocks in a reference image, and the degree of matching is determined based on SAD, SATD and / or SSE between the reconstructed block and the one or more reference blocks.

[0291] In some implementations, the first motion information includes one or more of the following: block-level motion information, sub-block-level motion information, or pixel-level motion information.

[0292] In some implementations, the first motion information includes one or more of the following: a motion vector and / or index information of a reference image.

[0293] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0294] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0295] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the decoder 2500. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the decoding method in the first embodiment.

[0296] Based on the composition of the above-mentioned decoder 2500 and the computer-readable storage medium, refer to Figure 26, which shows a specific hardware structure diagram of the decoder 2600 provided in an embodiment of the present application. As shown in Figure 26, the decoder 2600 may include: a communication interface 2610, a memory 2620 and a processor 2630; each component is coupled together through a bus system 2640. It can be understood that the bus system 2640 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 2640 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 2640 in Figure 26. Among them,

[0297] The communication interface 2610 is used to receive and send signals when sending and receiving information with other external network elements.

[0298] The memory 2620 is used to store computer programs.

[0299] The processor 2630 is configured to, when running the computer program, perform the following operations: parsing a bitstream to determine a reconstructed block of a current block; and determining first motion information of the current block according to the reconstructed block.

[0300] It is understood that the memory 2620 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 2620 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0301] The processor 2630 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 2630 or by software instructions. The above-mentioned processor 2630 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 2620, and the processor 2630 reads the information in the memory 2620 and completes the steps of the above method in combination with its hardware.

[0302] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0303] Optionally, as another embodiment, the processor 2630 is further configured to execute the decoding method described in the above embodiment when running the computer program.

[0304] FIG27 is a schematic diagram of the structure of an encoder provided by an embodiment of the present application. As shown in FIG27 , the encoder 2700 includes: a first determining unit 2710 and a second determining unit 2720 .

[0305] The first determining unit 2710 is configured to determine a reconstructed block of the current block.

[0306] The second determining unit 2720 is configured to determine first motion information of the current block according to the reconstructed block.

[0307] In some implementations, the first motion information is used to determine motion information of a first coded block that is later than the current block in a coding order.

[0308] In some implementations, a positional relationship between the first coding block and the current block satisfies a first condition.

[0309] In some implementations, the first condition includes: the first coding block is not adjacent to the current block; or, the first coding block and the current block belong to different CTUs; or, the first coding block and the current block belong to different frames.

[0310] In some implementations, the encoder 2700 further includes a third determining unit 2730 configured to determine a prediction block of a second coding block based on second motion information of the current block, where a positional relationship between the second coding block and the current block does not satisfy the first condition.

[0311] In some implementations, the first motion information is used to process the reconstructed block.

[0312] In some implementations, the first motion information is used to perform loop filtering on the reconstructed block.

[0313] In some implementations, the second determining unit 2720 is further configured to determine the first motion information according to the reconstructed block if a second condition is satisfied.

[0314] In some implementations, the second condition is related to one or more of the following: residual information of the current block; or a prediction mode of the current block.

[0315] In some implementations, the second condition includes one or more of the following: residual information exists in the current block; and the current block uses an intra-frame prediction mode.

[0316] In some implementations, the encoder 2700 further includes a fourth determining unit 2740 configured to write first identification information into the bitstream, where the first identification information is used to indicate whether to determine the motion information based on the reconstruction information.

[0317] In some implementations, the first identification information is sequence-level or stripe-level information.

[0318] In some implementations, the first motion information is determined based on the reconstructed block and the second motion information.

[0319] In some implementations, the first motion information is determined based on a degree of matching between the current block and one or more reference blocks in a reference image, and the second motion information is used to determine a starting position of the one or more reference blocks.

[0320] In some implementations, the first motion information is determined based on a degree of matching between the current block and one or more reference blocks in a reference image, and the degree of matching is determined based on SAD, SATD and / or SSE between the reconstructed block and the one or more reference blocks.

[0321] In some implementations, the first motion information includes one or more of the following: block-level motion information, sub-block-level motion information, or pixel-level motion information.

[0322] In some implementations, the first motion information includes one or more of the following: a motion vector and / or index information of a reference image.

[0323] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0324] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0325] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the encoder 2700. The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the decoding method in the aforementioned first embodiment.

[0326] Based on the composition of the above-mentioned encoder 2700 and the computer-readable storage medium, refer to Figure 28, which shows a specific hardware structure diagram of the encoder 2800 provided in an embodiment of the present application. As shown in Figure 28, the encoder 2800 may include: a communication interface 2810, a memory 2820 and a processor 2830; each component is coupled together through a bus system 2840. It can be understood that the bus system 2840 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 2840 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus systems 2840 in Figure 28. Among them,

[0327] The communication interface 2810 is used to receive and send signals when sending and receiving information with other external network elements.

[0328] The memory 2820 is used to store computer programs.

[0329] The processor 2830 is configured to, when running the computer program, execute: determining a reconstructed block of the current block; and determining first motion information of the current block according to the reconstructed block.

[0330] It is understood that the memory 2820 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 2820 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0331] The processor 2830 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the processor 2830. The above-mentioned processor 2830 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 2820, and the processor 2830 reads the information in the memory 2820 and completes the steps of the above method in combination with its hardware.

[0332] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0333] Optionally, as another embodiment, the processor 2830 is further configured to execute the encoding method described in the above embodiment when running the computer program.

[0334] It should be noted that, in this application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0335] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0336] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0337] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0338] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0339] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A decoding method, applied to a decoder, comprising: Parse the code stream and determine the reconstructed block of the current block; First motion information of the current block is determined according to the reconstructed block.

2. The method according to claim 1, wherein The first motion information is used to determine motion information of a first coding block, which is later than the current block in a decoding order.

3. The method according to claim 2, wherein: A positional relationship between the first coding block and the current block satisfies a first condition.

4. The method according to claim 3, wherein: The first coding block is not adjacent to the current block; or, The first coding block and the current block belong to different coding tree units CTU; or, The first coding block and the current block belong to different pictures.

5. The method according to claim 3 or 4, wherein: The method further comprises: A prediction block of a second coding block is determined according to the second motion information of the current block, and a positional relationship between the second coding block and the current block does not satisfy the first condition.

6. The method according to any one of claims 1 to 5, wherein The first motion information is used to process the reconstructed block.

7. The method according to claim 6, wherein: The first motion information is used to perform loop filtering on the reconstructed block.

8. The method according to any one of claims 1 to 7, wherein The determining the first motion information of the current block according to the reconstructed block includes: If the second condition is met, the first motion information is determined according to the reconstructed block.

9. The method according to claim 8, wherein The second condition is related to one or more of the following: The residual information of the current block; or The prediction mode of the current block.

10. The method according to claim 9, wherein: The second condition includes one or more of the following: There is residual information in the current block; The current block uses an intra prediction mode.

11. The method according to any one of claims 1 to 10, wherein The method further comprises: Parse the code stream to determine first identification information, where the first identification information is used to indicate whether motion information is determined based on reconstruction information.

12. The method according to claim 11, wherein The first identification information is sequence-level or stripe-level information.

13. The method according to any one of claims 1 to 12, wherein The first motion information is determined based on the reconstructed block and second motion information.

14. The method according to claim 13, wherein The first motion information is determined based on a matching degree between the current block and one or more reference blocks in a reference image, and the second motion information is used to determine a starting position of the one or more reference blocks.

15. The method according to any one of claims 1 to 14, wherein The first motion information is determined based on a degree of matching between the current block and one or more reference blocks in a reference image, and the degree of matching is determined based on a sum of absolute errors (SAD), a sum of absolute transform errors (SATD) and / or a sum of squared residual errors (SSE) between the reconstructed block and the one or more reference blocks.

16. The method according to any one of claims 1 to 15, wherein The first motion information includes one or more of the following: block-level motion information, sub-block-level motion information, or pixel-level motion information.

17. The method according to any one of claims 1 to 16, wherein The first motion information includes one or more of the following: a motion vector and / or index information of a reference image.

18. A coding method, applied to an encoder, comprising: Determine a reconstruction block of the current block; First motion information of the current block is determined according to the reconstructed block.

19. The method according to claim 18, wherein The first motion information is used to determine motion information of a first coding block, where the first coding block is later than the current block in a coding order.

20. The method according to claim 19, wherein A positional relationship between the first coding block and the current block satisfies a first condition.

21. The method according to claim 20, wherein: The first coding block is not adjacent to the current block; or, The first coding block and the current block belong to different coding tree units CTU; or, The first coding block and the current block belong to different pictures.

22. The method according to claim 20 or 21, wherein The method further comprises: A prediction block of a second coding block is determined according to the second motion information of the current block, and a positional relationship between the second coding block and the current block does not satisfy the first condition.

23. The method according to any one of claims 18 to 22, wherein The first motion information is used to process the reconstructed block.

24. The method according to claim 23, wherein The first motion information is used to perform loop filtering on the reconstructed block.

25. The method according to any one of claims 18 to 24, wherein The determining the first motion information of the current block according to the reconstructed block includes: If the second condition is met, the first motion information is determined according to the reconstructed block.

26. The method according to claim 25, wherein The second condition is related to one or more of the following: The residual information of the current block; or The prediction mode of the current block.

27. The method according to claim 26, wherein The second condition includes one or more of the following: There is residual information in the current block; The current block uses an intra prediction mode.

28. The method according to any one of claims 18 to 27, wherein The method further comprises: First identification information is written into the bitstream, where the first identification information is used to indicate whether motion information is determined based on the reconstruction information.

29. The method according to claim 28, wherein The first identification information is sequence-level or stripe-level information.

30. The method according to any one of claims 18 to 29, wherein The first motion information is determined based on the reconstructed block and second motion information.

31. The method according to claim 30, wherein The first motion information is determined based on a matching degree between the current block and one or more reference blocks in a reference image, and the second motion information is used to determine a starting position of the one or more reference blocks.

32. The method according to any one of claims 18 to 31, wherein The first motion information is determined based on a degree of matching between the current block and one or more reference blocks in a reference image, and the degree of matching is determined based on a sum of absolute errors (SAD), a sum of absolute transform errors (SATD) and / or a sum of squared residual errors (SSE) between the reconstructed block and the one or more reference blocks.

33. The method according to any one of claims 18 to 32, wherein The first motion information includes one or more of the following: block-level motion information, sub-block-level motion information, or pixel-level motion information.

34. A method according to any one of claims 18 to 33, wherein The first motion information includes one or more of the following: a motion vector and / or index information of a reference image.

35. A decoder comprising: A first determining unit is configured to parse the bitstream and determine a reconstructed block of the current block; The second determining unit is configured to determine first motion information of the current block according to the reconstructed block.

36. A decoder comprising: memory for storing computer programs; A processor, configured to execute the method according to any one of claims 1 to 17 when running the computer program.

37. An encoder comprising: a first determining unit configured to determine a reconstructed block of a current block; The second determining unit is configured to determine first motion information of the current block according to the reconstructed block.

38. An encoder comprising: memory for storing computer programs; A processor, configured to perform the method according to any one of claims 18 to 34 when running the computer program.

39. A non-volatile computer-readable storage medium storing a bit stream, wherein the bit stream is generated by an encoding method using an encoder, or the bit stream is decoded by a decoding method using a decoder, wherein: The decoding method is the method according to any one of claims 1 to 17, and the encoding method is the method according to any one of claims 18 to 34.

40. A code stream, comprising a code stream generated by the method according to any one of claims 18 to 34.

41. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 17 or 18 to 34 is implemented.

Images