A method and apparatus for inter prediction

Abstract

This application discloses a method and apparatus for inter-frame prediction. The method includes: determining a plurality of first image blocks in the image block to be processed based on a preset image segmentation width, a preset image segmentation height, and the width and height of the image block to be processed; performing bidirectional prediction based on optical flow on the plurality of first image blocks to obtain a prediction value for each first image block; and combining the prediction values ​​of the plurality of first image blocks to obtain a prediction value for the image block to be processed. The apparatus includes a determination module, a prediction module, and a combination module. This application can reduce the implementation complexity of inter-frame prediction and improve processing efficiency.

Description

[0001] This application is a divisional application. The original application has the application number 201910325612.0 and the original application date is April 22, 2019. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of video encoding and decoding, and more particularly to a method and apparatus for inter-frame prediction. Background Technology

[0003] Inter-frame prediction utilizes the correlation, or temporal correlation, between video image frames to achieve image compression. It is widely used in compression encoding and decoding for scenarios such as regular television, video conferencing, video telephony, and high-definition television. Inter-frame prediction operations are used to process images on both the encoding and decoding sides.

[0004] When performing inter-frame prediction on an image, the image is first divided into multiple image blocks based on the height and width of the corresponding image patch. Then, inter-frame prediction is performed on each of these blocks. When the width and height of the corresponding image patch are large, the area of ​​each resulting image block is large, which increases the complexity of inter-frame prediction. Summary of the Invention

[0005] This application provides an inter-frame prediction method and apparatus that can reduce the implementation complexity of inter-frame prediction and improve processing efficiency.

[0006] Firstly, this application provides a method for inter-frame prediction. In this method, multiple first image blocks are determined within the image block to be processed based on a preset image segmentation width, a preset image segmentation height, and the width and height of the image block to be processed. Bidirectional optical flow-based prediction is then performed on each of the multiple first image blocks to obtain a prediction value for each first image block. Finally, the prediction values ​​of the multiple first image blocks are combined to obtain a prediction value for the image block to be processed. Because multiple first image blocks are determined within the image block to be processed based on the preset image segmentation width, preset image segmentation height, and the width and height of the image block to be processed, the size of each first image block is constrained by the preset image segmentation width and height, preventing each determined first image block from having a large area. This reduces the consumption of hardware resources such as memory, lowers the complexity of inter-frame prediction, and improves processing efficiency.

[0007] In one possible implementation, the width and height of the image block to be processed are the same as the width and height of the first image block, respectively; that is, the image block to be processed contains only one first image block. Correspondingly, when the image block to be processed is determined to be the first image block based on the preset image segmentation width, the preset image segmentation height, and the width and height of the image block to be processed, the image block to be processed is used as the processing unit to perform bidirectional prediction based on optical flow to obtain the predicted value of the image block to be processed.

[0008] In one possible implementation, the width of the first image block is determined by comparing the preset image segmentation width with the width of the image block to be processed; the height of the first image block is determined by comparing the preset image segmentation height with the height of the image block to be processed; and the plurality of first image blocks are determined from the image block to be processed based on the width and height of the first image block. This constrains the width and height of the first image blocks by the preset image segmentation width, preventing each determined first image block from having a large area. This reduces the consumption of hardware resources such as memory, lowers the complexity of inter-frame prediction, and improves processing efficiency.

[0009] In one possible implementation, the width of the first image block is the minimum value between the preset image segmentation width and the width of the image block to be processed, and the height of the first image block is the minimum value between the preset image segmentation height and the height of the image block to be processed. This reduces the area required to determine the first image, minimizing the complexity of inter-frame prediction and improving processing efficiency.

[0010] In one possible implementation, a first prediction block of the first image block is obtained based on the motion information of the image block to be processed; gradient calculation is performed on the first prediction block to obtain a first gradient value matrix of the first image block; motion information correction amount of each basic processing unit in the first image block is calculated based on the first prediction block and the first gradient value matrix; and a predicted value of the first image block is obtained based on the motion information correction amount of each basic processing unit. Since the predicted value of the first image block is obtained based on the motion information correction amount of each basic processing unit, the accuracy of the predicted value of the first image block can be improved.

[0011] In one possible implementation, based on the pixel values ​​at the block edge positions of the first prediction block, the width and height of the first prediction block are first expanded so that the width and height of the first prediction block after the first expansion are each 2 pixels larger than the width and height of the first image block; and / or, based on the gradient values ​​at the matrix edge positions of the first gradient value matrix, the width and height of the first gradient value matrix are first expanded so that the width and height of the first gradient value matrix after the first expansion are each 2 pixels larger than the width and height of the first image block; correspondingly, the motion information correction amount of each basic processing unit in the first image block is calculated according to the first expanded first prediction block and / or the first expanded first gradient value matrix. By first expanding the width and height of the first prediction block so that the width and height of the first expanded first prediction block are each 2 pixels larger than the width and height of the first image block, the size of the first prediction block can be reduced when bidirectional prediction is performed on the image block in the reference image to obtain the first prediction block. Correspondingly, the size of the image block is also reduced, thereby reducing the amount of data for bidirectional prediction and reducing the occupation of hardware resources.

[0012] In one possible implementation, interpolation filtering is performed on the pixel values ​​of the block edge region of the first prediction block, or the pixel values ​​at the block edge positions of the first prediction block are copied to perform a second expansion of the width and height of the first prediction block; correspondingly, gradient calculation is performed on the second-expanded first prediction block. Since the second expansion of the width and height of the first prediction block can be performed by copying the pixel values ​​at the block edge positions of the first prediction block, the implementation method is simple and the computational complexity is low.

[0013] In one possible implementation, the first prediction block includes a forward prediction block and a backward prediction block, and the first gradient value matrix includes a forward horizontal gradient value matrix, a forward vertical gradient value matrix, a backward horizontal gradient value matrix, and a backward vertical gradient value matrix.

[0014] In one possible implementation, the preset image segmentation width is 64, 32, or 16, and the preset image segmentation height is 64, 32, or 16. This allows for a reduction in the size of the determined first image block based on constraints on the predicted image segmentation width and height.

[0015] In one possible implementation, the basic processing unit is a 4x4 pixel matrix.

[0016] Secondly, this application provides an inter-frame prediction apparatus, comprising: a determining module, a predicting module, and a combining module. The determining module determines multiple first image blocks within the image block to be processed based on a preset image segmentation width, a preset image segmentation height, and the width and height of the image block to be processed. The predicting module performs bidirectional optical flow-based prediction on each of the multiple first image blocks to obtain a predicted value for each first image block. The combining module combines the predicted values ​​of the multiple first image blocks to obtain a predicted value for the image block to be processed. Since the determining module determines multiple first image blocks within the image block to be processed based on the preset image segmentation width, preset image segmentation height, and the width and height of the image block to be processed, the size of the first image blocks is constrained by the preset image segmentation width and preset image segmentation height, preventing each determined first image block from having a large area. This reduces the consumption of hardware resources such as memory, lowers the complexity of inter-frame prediction, and improves processing efficiency.

[0017] In one possible implementation, the determining module, the predicting module, and the combining module can also be used to perform the operations of the method in any possible implementation of the first aspect, which will not be described in detail here.

[0018] Thirdly, embodiments of this application provide an apparatus for inter-frame prediction, the apparatus comprising: a processor and a memory connected together; the memory storing one or more programs configured to be executed by the processor, the one or more programs containing instructions for performing a method for performing the first aspect or any possible implementation of the first aspect.

[0019] Fourthly, this application provides a non-volatile computer-readable storage medium for storing a computer program, which is loaded by a processor to execute instructions of the method described in the first aspect or any possible implementation thereof.

[0020] Fifthly, this application provides a chip including programmable logic circuitry and / or program instructions, and a method for implementing the first aspect or any possible implementation thereof when the chip is running.

[0021] In a sixth aspect, embodiments of this application provide a method for inter-frame prediction, comprising: acquiring motion information of an image block to be processed, the image block to be processed comprising a plurality of virtual pipeline data units, the virtual pipeline data unit comprising at least one basic processing unit; obtaining a prediction value matrix for each virtual pipeline data unit based on the motion information; calculating a horizontal prediction gradient matrix and a vertical prediction gradient matrix for each virtual pipeline data unit based on each prediction value matrix; and calculating a motion information correction amount for each basic processing unit in each virtual pipeline data unit based on the prediction value matrix, the horizontal prediction gradient matrix, and the vertical prediction gradient matrix.

[0022] In one feasible implementation of the sixth aspect, obtaining the prediction value matrix of each virtual pipeline data unit based on the motion information includes: obtaining an initial prediction matrix of each virtual pipeline data unit based on the motion information, wherein the initial prediction matrix is ​​equal in size to the virtual pipeline data unit; and using the initial prediction matrix as the prediction value matrix.

[0023] In one feasible implementation of the sixth aspect, after obtaining the initial prediction matrix for each of the virtual pipeline data units, the method further includes: extending the edges of the initial prediction matrix by pixels to obtain an extended prediction matrix, wherein the size of the extended prediction matrix is ​​larger than the size of the initial prediction matrix; correspondingly, using the initial prediction matrix as the prediction value matrix includes: using the extended prediction matrix as the prediction value matrix.

[0024] In one feasible implementation of the sixth aspect, the step of extending the edge of the initial prediction matrix by pixels includes: obtaining the pixel values ​​of pixels outside the initial prediction matrix by interpolation based on the pixel values ​​of pixels within the initial prediction matrix, or using the pixel values ​​of pixels at the edge of the initial prediction matrix as the pixel values ​​of pixels outside the initial prediction matrix adjacent to the edge.

[0025] In one feasible implementation of the sixth aspect, the virtual pipeline data unit includes multiple motion compensation units, and obtaining the prediction value matrix of each virtual pipeline data unit based on the motion information includes: obtaining the compensation value matrix of each motion compensation unit based on the motion information; and combining the compensation value matrices of the multiple motion compensation units to obtain the prediction value matrix.

[0026] In one feasible implementation of the sixth aspect, the step of calculating the horizontal prediction gradient matrix and the vertical prediction gradient matrix of each virtual pipeline data unit based on each of the prediction value matrices includes: performing horizontal gradient calculation and vertical gradient calculation on the prediction value matrices to obtain the horizontal prediction gradient matrix and the vertical prediction gradient matrix, respectively.

[0027] In one feasible implementation of the sixth aspect, before calculating the motion information correction amount of each basic processing unit in each of the virtual pipeline data units based on the predicted value matrix, the horizontal predicted gradient matrix, and the vertical predicted gradient matrix, the method further includes: extending the edges of the predicted value matrix by pixels to obtain a filled predicted matrix, the filled predicted matrix having a preset size; extending the edges of the horizontal predicted gradient matrix and the vertical predicted gradient matrix by gradient values ​​respectively to obtain a filled horizontal gradient matrix and a filled vertical gradient matrix, the filled horizontal gradient matrix and the filled vertical gradient matrix each having the preset size; correspondingly, calculating the motion information correction amount of each basic processing unit in each of the virtual pipeline data units based on the predicted value matrix, the horizontal predicted gradient matrix, and the vertical predicted gradient matrix includes: calculating the motion information correction amount of each basic processing unit in each of the virtual pipeline data units based on the filled predicted matrix, the filled horizontal gradient matrix, and the filled vertical gradient matrix.

[0028] In one feasible implementation of the sixth aspect, before performing pixel-level expansion on the edges of the predicted value matrix, the method further includes: determining that the size of the predicted value matrix is ​​smaller than the preset size.

[0029] In one feasible implementation of the sixth aspect, before performing gradient value expansion on the edges of the horizontal prediction gradient matrix and the edges of the vertical prediction gradient matrix, the method further includes: determining that the size of the horizontal prediction gradient matrix and / or the size of the vertical prediction gradient matrix is ​​smaller than the preset size.

[0030] In one feasible implementation of the sixth aspect, after calculating the motion information correction amount of each of the basic processing units in each of the virtual pipeline data units, the method further includes: obtaining a prediction value for each of the basic processing units based on the prediction value matrix of the virtual pipeline data units and the motion correction amount of each of the basic processing units in the virtual pipeline data units.

[0031] In one feasible implementation of the sixth aspect, the method is used for bidirectional prediction; correspondingly, the motion information includes motion information of a first reference frame list and motion information of a second reference frame list; the prediction value matrix includes a first prediction value matrix and a second prediction value matrix, wherein the first prediction value matrix is ​​obtained based on the motion information of the first reference frame list, and the second prediction value matrix is ​​obtained based on the motion information of the second reference frame list; the horizontal prediction gradient matrix includes a first horizontal prediction gradient matrix and a second horizontal prediction gradient matrix, wherein the first horizontal prediction gradient matrix is ​​calculated based on the first prediction value matrix, and the second horizontal prediction gradient matrix is ​​calculated based on the second prediction value matrix; the vertical prediction gradient matrix includes a first vertical prediction gradient matrix and a second vertical prediction gradient matrix, wherein the first vertical prediction gradient matrix is ​​calculated based on the first prediction value matrix, and the second vertical prediction gradient matrix is ​​calculated based on the second prediction value matrix; the motion information correction amount includes a first reference frame list motion information correction amount and a second reference frame list motion information correction amount, wherein the first reference frame list motion information correction amount is calculated based on the first prediction value matrix, the first horizontal prediction gradient matrix, and the second vertical prediction gradient matrix.

[0032] In one feasible implementation of the sixth aspect, before performing pixel-level expansion on the edges of the initial prediction matrix, the method further includes: determining that the temporal location of the image frame containing the image block to be processed is between the first reference frame indicated by the motion information of the first reference frame list and the second reference frame indicated by the motion information of the second reference frame list.

[0033] In one feasible implementation of the sixth aspect, after obtaining the prediction value matrix for each of the virtual pipeline data units, the method further includes: determining that the difference between the first prediction value matrix and the second prediction value matrix is ​​less than a first threshold.

[0034] In one feasible implementation of the sixth aspect, the motion information correction amount of the basic processing unit corresponds to a basic prediction value matrix in the prediction value matrix. Before calculating the motion information correction amount of each basic processing unit in each virtual pipeline data unit based on the prediction value matrix, the horizontal prediction gradient matrix, and the vertical prediction gradient matrix, the method further includes: determining that the difference between the first basic prediction value matrix and the second basic prediction value matrix is ​​less than a second threshold.

[0035] In one feasible implementation of the sixth aspect, the basic processing unit has a size of 4x4.

[0036] In one feasible implementation of the sixth aspect, the width of the virtual pipeline data unit is W, the height is H, the size of the extended prediction matrix is ​​(W+n+2)x(H+n+2), and correspondingly, the size of the horizontal prediction gradient matrix is ​​(W+n)x(H+n), and the size of the vertical prediction gradient matrix is ​​(W+n)x(H+n), where W and H are positive integers, and n is an even number.

[0037] In one possible implementation of the sixth aspect, n is 0, 2, or -2.

[0038] In one feasible implementation of the sixth aspect, before acquiring the motion information of the image block to be processed, the method further includes: determining that the image block to be processed includes the plurality of virtual pipeline data units.

[0039] In a seventh aspect, embodiments of this application provide an inter-frame prediction apparatus, comprising: an acquisition module for acquiring motion information of an image block to be processed, the image block to be processed comprising a plurality of virtual pipeline data units, the virtual pipeline data unit comprising at least one basic processing unit; a compensation module for obtaining a prediction value matrix for each of the virtual pipeline data units based on the motion information; a calculation module for calculating a horizontal prediction gradient matrix and a vertical prediction gradient matrix for each of the virtual pipeline data units based on each prediction value matrix; and a correction module for calculating a motion information correction amount for each of the basic processing units in each of the virtual pipeline data units based on the prediction value matrix, the horizontal prediction gradient matrix, and the vertical prediction gradient matrix.

[0040] In one feasible implementation of the seventh aspect, the compensation module is specifically used to: obtain an initial prediction matrix for each of the virtual pipeline data units based on the motion information, wherein the initial prediction matrix is ​​equal in size to the virtual pipeline data unit; and use the initial prediction matrix as the prediction value matrix.

[0041] In one feasible implementation of the seventh aspect, the compensation module is specifically used to: extend the edges of the initial prediction matrix by pixels to obtain an extended prediction matrix, the size of which is larger than the size of the initial prediction matrix; and use the extended prediction matrix as the prediction value matrix.

[0042] In one feasible implementation of the seventh aspect, the compensation module is specifically used to: obtain the pixel value of the pixel outside the initial prediction matrix by interpolation based on the pixel value of the pixel within the initial prediction matrix, or to use the pixel value of the pixel at the edge of the initial prediction matrix as the pixel value of the pixel outside the initial prediction matrix adjacent to the edge.

[0043] In one feasible implementation of the seventh aspect, the virtual pipeline data unit includes multiple motion compensation units, and the compensation module is specifically used to: obtain a compensation value matrix for each motion compensation unit based on the motion information; and combine the compensation value matrices of the multiple motion compensation units to obtain the prediction value matrix.

[0044] In one feasible implementation of the seventh aspect, the calculation module is specifically used to: perform horizontal gradient calculation and vertical gradient calculation on the predicted value matrix to obtain the horizontal prediction gradient matrix and the vertical prediction gradient matrix, respectively.

[0045] In one feasible implementation of the seventh aspect, a filling module is further included, configured to: extend the edges of the prediction value matrix by pixels to obtain a filled prediction matrix, the filled prediction matrix having a preset size; extend the edges of the horizontal prediction gradient matrix and the vertical prediction gradient matrix by gradient values ​​respectively to obtain a filled horizontal gradient matrix and a filled vertical gradient matrix, the filled horizontal gradient matrix and the filled vertical gradient matrix having the preset size respectively; and calculate the motion information correction amount of each basic processing unit in each of the virtual pipeline data units based on the filled prediction matrix, the filled horizontal gradient matrix and the filled vertical gradient matrix.

[0046] In one feasible implementation of the seventh aspect, a determination module is further included, used to: determine that the size of the predicted value matrix is ​​less than the preset size.

[0047] In one feasible implementation of the seventh aspect, the determination module is further configured to: determine that the size of the horizontal prediction gradient matrix and / or the size of the vertical prediction gradient matrix is ​​less than the preset size.

[0048] In one feasible implementation of the seventh aspect, the correction module is further configured to: obtain the predicted value of each of the basic processing units based on the predicted value matrix of the virtual pipeline data unit and the motion correction amount of each of the basic processing units in the virtual pipeline data unit.

[0049] In one feasible implementation of the seventh aspect, the device is used for bidirectional prediction; correspondingly, the motion information includes motion information of a first reference frame list and motion information of a second reference frame list; the prediction value matrix includes a first prediction value matrix and a second prediction value matrix, wherein the first prediction value matrix is ​​obtained based on the motion information of the first reference frame list, and the second prediction value matrix is ​​obtained based on the motion information of the second reference frame list; the horizontal prediction gradient matrix includes a first horizontal prediction gradient matrix and a second horizontal prediction gradient matrix, wherein the first horizontal prediction gradient matrix is ​​calculated based on the first prediction value matrix, and the second horizontal prediction gradient matrix is ​​calculated based on the second prediction value matrix; the vertical prediction gradient matrix includes a first vertical prediction gradient matrix and a second vertical prediction gradient matrix, wherein the first vertical prediction gradient matrix is ​​calculated based on the first prediction value matrix, and the second vertical prediction gradient matrix is ​​calculated based on the second prediction value matrix; the motion information correction amount includes a first reference frame list motion information correction amount and a second reference frame list motion information correction amount, wherein the first reference frame list motion information correction amount is calculated based on the first prediction value matrix, the first horizontal prediction gradient matrix, and the second vertical prediction gradient matrix.

[0050] In one feasible implementation of the seventh aspect, the determining module is further configured to: determine that the temporal position of the image frame in which the image block to be processed is located is between the first reference frame indicated by the motion information of the first reference frame list and the second reference frame indicated by the motion information of the second reference frame list.

[0051] In one feasible implementation of the seventh aspect, the judgment module is further configured to: determine that the difference between the first predicted value matrix and the second predicted value matrix is ​​less than a first threshold.

[0052] In one feasible implementation of the seventh aspect, the judgment module is further configured to: determine that the difference between the first basic prediction value matrix and the second basic prediction value matrix is ​​less than a second threshold.

[0053] In one feasible implementation of the seventh aspect, the size of the basic processing unit is 4x4.

[0054] In one feasible implementation of the seventh aspect, the width of the virtual pipeline data unit is W, the height is H, the size of the extended prediction matrix is ​​(W+n+2)x(H+n+2), and correspondingly, the size of the horizontal prediction gradient matrix is ​​(W+n)x(H+n), and the size of the vertical prediction gradient matrix is ​​(W+n)x(H+n), where W and H are positive integers, and n is an even number.

[0055] In one possible implementation of the seventh aspect, n is 0, 2, or -2.

[0056] In one feasible implementation of the seventh aspect, the determination module is further configured to: determine that the image block to be processed includes the plurality of virtual pipeline data units.

[0057] Eighthly, embodiments of this application provide an encoding device, comprising: a non-volatile memory and a processor coupled to each other, wherein the processor invokes program code stored in the memory to perform part or all of the steps of any method of the first aspect, or to perform part or all of the steps of any method of the sixth aspect.

[0058] In a ninth aspect, embodiments of this application provide a decoding device, comprising: a non-volatile memory and a processor coupled to each other, wherein the processor invokes program code stored in the memory to execute part or all of the steps of any method of the first aspect, or to execute part or all of the steps of any method of the sixth aspect.

[0059] In a tenth aspect, embodiments of this application provide a computer-readable storage medium storing program code, wherein the program code includes instructions for performing some or all of the steps of any method of the first aspect, or for performing some or all of the steps of any method of the sixth aspect.

[0060] In the eleventh aspect, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform some or all of the steps of any method of the first aspect, or to perform some or all of the steps of any method of the sixth aspect. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the background art, the accompanying drawings used in the embodiments of the present invention or the background art will be described below.

[0062] Figure 1A This is a block diagram for implementing an example of the video encoding and decoding system 10 of the embodiments of the present invention;

[0063] Figure 1B This is a block diagram for implementing an example of the video decoding system 40 of the embodiments of the present invention;

[0064] Figure 2 This is a block diagram for implementing the encoder 20 instance structure of the present invention;

[0065] Figure 3This is a block diagram for implementing the decoder 30 instance structure of the embodiments of the present invention;

[0066] Figure 4 This is a block diagram for implementing an example of the video decoding device 400 according to an embodiment of the present invention;

[0067] Figure 5 This is a block diagram of another example of an encoding or decoding device for implementing embodiments of the present invention;

[0068] Figure 6 This is a schematic diagram for implementing the motion information candidate positions in embodiments of the present invention;

[0069] Figure 7 This is a schematic diagram illustrating the use of motion information for inter-frame prediction in embodiments of the present invention.

[0070] Figure 8 This is a schematic diagram for implementing bidirectional weighted prediction in embodiments of the present invention;

[0071] Figure 9 This is a schematic diagram for implementing CU boundary padding in embodiments of the present invention;

[0072] Figure 10 This is a schematic diagram of VPDU partitioning used to implement embodiments of the present invention;

[0073] Figure 11 This is a schematic diagram illustrating an illegal VPDU partitioning implementation in an embodiment of the present invention;

[0074] Figure 12 This is a flowchart of a method for implementing inter-frame prediction in an embodiment of the present invention;

[0075] Figure 13 This is another schematic diagram illustrating the use of motion information for inter-frame prediction in embodiments of the present invention;

[0076] Figure 14 This is a flowchart of another method for implementing inter-frame prediction in an embodiment of the present invention;

[0077] Figure 15 This is another schematic diagram illustrating the use of motion information for inter-frame prediction in embodiments of the present invention;

[0078] Figure 16 This is a flowchart of another method for implementing inter-frame prediction in an embodiment of the present invention;

[0079] Figure 17 This is a flowchart of a method for implementing an embodiment of the present invention;

[0080] Figure 18 This is a structural block diagram of a device for implementing inter-frame prediction in an embodiment of the present invention;

[0081] Figure 19 This is a block diagram of a device structure for implementing another inter-frame prediction in an embodiment of the present invention. Detailed Implementation

[0082] The embodiments of the present invention will now be described with reference to the accompanying drawings. In the following description, reference is made to the accompanying drawings, which form part of this disclosure and illustrate specific aspects of the embodiments of the invention or to illustrate specific aspects of the embodiments of the invention. It should be understood that the embodiments of the invention can be used in other aspects and may include structural or logical variations not depicted in the drawings. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of the invention is defined by the appended claims. For example, it should be understood that the disclosure of the described methods is equally applicable to corresponding devices or systems for performing the methods, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units, to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units, each performing one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific apparatus is described based on one or more units, such as functional units, the corresponding method may include a step to perform the functionality of one or more units (e.g., one step performs the functionality of one or more units, or multiple steps, each performing the functionality of one or more of the multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Furthermore, it should be understood that, unless otherwise expressly stated, the features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0083] The technical solutions involved in the embodiments of this invention may be applied not only to existing video coding standards (such as H.264, High Efficiency Video Coding (HEVC), etc.) but also to future video coding standards (such as the H.266 standard). The terminology used in the embodiments of this invention is only for explaining specific embodiments of the invention and is not intended to limit the invention. A brief introduction to some concepts that may be involved in the embodiments of this invention is provided below.

[0084] Video coding generally refers to the processing of a sequence of images that form a video or video sequence. In the field of video coding, the terms "picture," "frame," or "image" can be used synonymously. As used herein, video coding refers to either video encoding or video decoding. Video coding is performed on the source side and typically involves processing (e.g., by compression) the raw video images to reduce the amount of data required to represent them, thereby enabling more efficient storage and / or transmission. Video decoding is performed on the destination side and typically involves inverse processing relative to the encoder to reconstruct the video images. The "encoding" of video images in the embodiments should be understood as involving the "encoding" or "decoding" of a video sequence. The combination of encoding and decoding portions is also referred to as encoding and decoding (encoding and decoding).

[0085] A video sequence consists of a series of pictures, which are further divided into slices, and slices into blocks. Video coding is performed on a block-by-block basis. In some newer video coding standards, the concept of a block has been further expanded. For example, the H.264 standard uses macroblocks (MBs), which can be further divided into multiple prediction blocks (partitions) for predictive coding. The HEVC standard uses basic concepts such as coding units (CUs), prediction units (PUs), and transform units (TUs) to functionally divide various block units, and employs a novel tree-based structure for description. For instance, a CU can be divided into smaller CUs using a quadtree, and these smaller CUs can be further divided, forming a quadtree structure. CUs are the basic units for partitioning and encoding images. PUs and TUs also have similar tree structures. A PU corresponds to a prediction block and is the basic unit for predictive coding. CUs are further divided into multiple PUs according to partitioning patterns. A TU corresponds to a transform block and is the basic unit for transforming the prediction residual. However, regardless of whether it is CU, PU or TU, they all essentially belong to the concept of blocks (or image blocks).

[0086] For example, in HEVC, coding tree units (CTUs) are split into multiple CUs using a quadtree structure represented as a coding tree. At the CU level, a decision is made on whether to use inter-picture (temporal) or intra-picture (spatial) prediction to encode picture regions. Each CU can be further split into one, two, or four PUs based on the PU splitting type. The same prediction process is applied within a PU, and relevant information is transmitted to the decoder based on the PU. After obtaining residual blocks by applying the prediction process based on the PU splitting type, the CUs can be segmented into transform units (TUs) according to other quadtree structures similar to those used for CUs. In the latest developments in video compression technology, quadtree and binary tree (QTBT) segmentation frames are used to segment coding blocks. In the QTBT block structure, CUs can be square or rectangular in shape.

[0087] In this paper, for ease of description and understanding, the image block to be encoded in the current encoded image is referred to as the current block. For example, in encoding, it refers to the block currently being encoded; in decoding, it refers to the block currently being decoded. The decoded image block in the reference image used to predict the current block is called the reference block. That is, the reference block is the block that provides a reference signal for the current block, where the reference signal represents the pixel value within the image block. The block in the reference image that provides the prediction signal for the current block can be called the prediction block, where the prediction signal represents the pixel value, sampled value, or sampled signal within the prediction block. For example, after traversing multiple reference blocks, an optimal reference block is found. This optimal reference block will provide prediction for the current block; this block is called the prediction block.

[0088] In lossless video coding, the original video images can be reconstructed, meaning the reconstructed video images have the same quality as the original (assuming no transmission loss or other data loss during storage or transmission). In lossy video coding, further compression is performed, for example, through quantization, to reduce the amount of data required to represent the video images. However, the decoder cannot fully reconstruct the video images, meaning the quality of the reconstructed video images is lower or worse than the original.

[0089] Several H.261 video coding standards fall under the category of "lossy hybrid video coding and decoding" (i.e., combining spatial and temporal prediction in the sample domain with 2D transform coding in the transform domain for applying quantization). Each frame of a video sequence is typically segmented into a non-overlapping set of blocks, usually encoded at the block level. In other words, the encoder typically processes, i.e., encodes the video at the block (video block) level, for example, by generating prediction blocks through spatial (intra-frame) and temporal (inter-frame) predictions, subtracting the prediction blocks from the current block (the currently processed or pending block) to obtain residual blocks, transforming and quantizing the residual blocks in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse processing relative to the encoder to the encoded or compressed blocks to reconstruct the current block for representation. Additionally, the encoder replicates the decoder processing loop, causing the encoder and decoder to generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstructions for processing, i.e., encoding subsequent blocks.

[0090] The system architecture used in the embodiments of the present invention is described below. See also Figure 1A , Figure 1A A schematic block diagram of a video encoding and decoding system 10 used in an embodiment of the present invention is provided as an example. Figure 1A As shown, the video encoding and decoding system 10 may include a source device 12 and a destination device 14. The source device 12 generates encoded video data; therefore, the source device 12 may be referred to as a video encoding device. The destination device 14 can decode the encoded video data generated by the source device 12; therefore, the destination device 14 may be referred to as a video decoding device. Various embodiments of the source device 12, the destination device 14, or both may include one or more processors and memory coupled to said one or more processors. The memory may include, but is not limited to, RAM, ROM, EEPROM, flash memory, or any other media that can be used to store desired program code in the form of computer-accessible instructions or data structures, as described herein. The source device 12 and the destination device 14 may include a variety of devices, including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, handsets such as so-called "smartphones," televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, wireless communication devices, or the like.

[0091] Although Figure 1ASource device 12 and destination device 14 are illustrated as separate devices, but device embodiments may also include the functionality of both source device 12 and destination device 14, or both; that is, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality. In such embodiments, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0092] A communication connection can be established between source device 12 and destination device 14 via link 13, through which destination device 14 can receive encoded video data from source device 12. Link 13 may include one or more media or devices capable of moving encoded video data from source device 12 to destination device 14. In one example, link 13 may include one or more communication media enabling source device 12 to transmit encoded video data directly to destination device 14 in real time. In this example, source device 12 may modulate the encoded video data according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated video data to destination device 14. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, wide area network, or global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices facilitating communication from source device 12 to destination device 14.

[0093] The source device 12 includes an encoder 20. Optionally, the source device 12 may also include an image source 16, an image preprocessor 18, and a communication interface 22. In specific implementations, the encoder 20, image source 16, image preprocessor 18, and communication interface 22 may be hardware components or software programs within the source device 12. These are described below:

[0094] Image source 16 may include or be any type of image capture device for, for example, capturing real-world images, and / or any type of image or commentary (for screen content encoding, some text on the screen is also considered part of the image or picture to be encoded) generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). Image source 16 may be a camera for capturing images or a memory for storing images. Image source 16 may also include any type of (internal or external) interface for storing previously captured or generated images and / or acquiring or receiving images. When image source 16 is a camera, image source 16 may be, for example, a local or integrated camera integrated into a source device; when image source 16 is a memory, image source 16 may be a local or, for example, integrated memory integrated into a source device. When the image source 16 includes an interface, the interface may be, for example, an external interface for receiving images from an external video source. The external video source may be, for example, an external image capture device, such as a camera, external storage, or an external image generation device. The external image generation device may be, for example, an external computer graphics processor, a computer, or a server. The interface may be any type of interface according to any proprietary or standardized interface protocol, such as a wired or wireless interface, or an optical interface.

[0095] An image can be viewed as a two-dimensional array or matrix of pixels. Pixels in the array are also called sampling points. The number of sampling points in the array or image along the horizontal and vertical directions (or axes) defines the image's size and / or resolution. To represent color, three color components are typically used; that is, an image can be represented as or contain three sampling arrays. For example, in RGB format or color space, an image includes corresponding red, green, and blue sampling arrays. However, in video coding, each pixel is typically represented in a luma / chroma format or color space. For example, for a YUV format image, this includes a luma component indicated by Y (sometimes also indicated by L) and two chroma components indicated by U and V. The luma component Y represents the brightness or grayscale level intensity (e.g., both are the same in grayscale images), while the two chroma components U and V represent chroma or color information components. Accordingly, a YUV format image includes a luma sampling array of luma sample values ​​(Y) and two chroma sampling arrays of chroma values ​​(U and V). Images in RGB format can be converted or transformed to YUV format, and vice versa; this process is also called color transformation or conversion. If the image is black and white, it may only include a luminance sampling array. In this embodiment of the invention, the image transmitted from image source 16 to image processor can also be referred to as raw image data 17.

[0096] Image preprocessor 18 is configured to receive raw image data 17 and perform preprocessing on the raw image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. For example, the preprocessing performed by image preprocessor 18 may include retouching, color format conversion (e.g., from RGB format to YUV format), color correction, or noise reduction.

[0097] Encoder 20 (or video encoder 20) is used to receive preprocessed image data 19 and process the preprocessed image data 19 using a relevant prediction mode (such as the prediction mode in the various embodiments of this document), thereby providing encoded image data 21 (which will be further based on...). Figure 2 or Figure 4 or Figure 5 (Describe the structural details of encoder 20). In some embodiments, encoder 20 can be used to perform the various embodiments described below to implement the chroma block prediction method described in this invention on the encoding side.

[0098] Communication interface 22 can be used to receive encoded image data 21 and transmit the encoded image data 21 to destination device 14 or any other device (such as a memory) via link 13 for storage or direct reconstruction. The other device can be any device used for decoding or storage. Communication interface 22 can, for example, be used to encapsulate the encoded image data 21 into a suitable format, such as data packets, for transmission over link 13.

[0099] Destination device 14 includes decoder 30. Optionally, destination device 14 may also include communication interface 28, image post-processor 32, and display device 34. These are described below:

[0100] Communication interface 28 can be used to receive encoded image data 21 from source device 12 or any other source, such as a storage device, for example, an encoded image data storage device. Communication interface 28 can be used to transmit or receive encoded image data 21 via link 13 between source device 12 and destination device 14 or via any type of network, such as a direct wired or wireless connection, any type of network such as wired or wireless networks or any combination thereof, or any type of private and public network, or any combination thereof. Communication interface 28 can be used, for example, to decapsulate data packets transmitted by communication interface 22 to obtain encoded image data 21.

[0101] Both communication interface 28 and communication interface 22 can be configured as unidirectional or bidirectional communication interfaces, and can be used, for example, to send and receive messages to establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission.

[0102] Decoder 30 (or decoder 30) is used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (hereinafter further based on...) Figure 3 or Figure 4 or Figure 5 (Description of structural details of decoder 30). In some embodiments, decoder 30 can be used to perform the various embodiments described below to implement the chroma block prediction method described in this invention on the decoding side.

[0103] Image post-processor 32 is used to perform post-processing on decoded image data 31 (also known as reconstructed image data) to obtain post-processed image data 33. The post-processing performed by image post-processor 32 may include: color format conversion (e.g., from YUV format to RGB format), color correction, retouching or resampling, or any other processing, and may also be used to transmit the post-processed image data 33 to display device 34.

[0104] Display device 34 is used to receive post-processed image data 33 to display the image to, for example, a user or viewer. Display device 34 can be or may include any class of displays for presenting reconstructed images, such as integrated or external displays or monitors. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other class of display.

[0105] Although, Figure 1A Source device 12 and destination device 14 are illustrated as separate devices, but device embodiments may also include the functionality of both source device 12 and destination device 14, or both; that is, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality. In such embodiments, the functionality of source device 12 or its corresponding functionality and the functionality of destination device 14 or its corresponding functionality may be implemented using the same hardware and / or software, or using separate hardware and / or software, or any combination thereof.

[0106] Based on the description, it is obvious to those skilled in the art that the functionality of different units is different. Figure 1A The presence and (precise) division of the functionality of the source device 12 and / or destination device 14 shown may vary depending on the actual device and application. The source device 12 and destination device 14 may include any of a variety of devices, encompassing any category of handheld or stationary devices, such as laptops or notebooks, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, cameras, in-vehicle devices, display devices, digital media players, video game consoles, video streaming devices (e.g., content service servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc., and may or may not use any type of operating system.

[0107] Both encoder 20 and decoder 30 can be implemented as any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology is implemented in part in software, the device can store software instructions in a suitable non-transitory computer-readable storage medium, and one or more processors can be used to execute the instructions in hardware to perform the technology of this disclosure. Any of the foregoing (including hardware, software, combinations of hardware and software, etc.) can be considered as one or more processors.

[0108] In some cases, Figure 1A The video encoding and decoding system 10 shown is merely an example, and the technology of this application can be applied to video encoding setups (e.g., video encoding or video decoding) that do not necessarily involve any data communication between the encoding and decoding devices. In other instances, data may be retrieved from local storage, streamed over a network, etc. A video encoding device may encode data and store it in storage, and / or a video decoding device may retrieve data from storage and decode it. In some instances, encoding and decoding are performed by devices that do not communicate with each other but only encode data to storage and / or retrieve data from storage and decode the data.

[0109] See Figure 1B , Figure 1B It is a inclusion according to an exemplary embodiment Figure 2 encoder 20 and / or Figure 3 The illustrated diagram shows an example of a video decoding system 40 with decoder 30. The video decoding system 40 can implement various combinations of technologies according to embodiments of the present invention. In the illustrated embodiment, the video decoding system 40 may include an imaging device 41, an encoder 20, a decoder 30 (and / or a video codec implemented by logic circuitry 47 of processing unit 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0110] like Figure 1B As shown, the imaging device 41, antenna 42, processing unit 46, logic circuit 47, encoder 20, decoder 30, processor 43, memory 44, and / or display device 45 are capable of communicating with each other. As discussed, although the video decoding system 40 is illustrated with encoder 20 and decoder 30, in different instances, the video decoding system 40 may contain only encoder 20 or only decoder 30.

[0111] In some instances, antenna 42 can be used to transmit or receive encoded bitstreams of video data. Additionally, in some instances, display device 45 can be used to present video data. In some instances, logic circuitry 47 can be implemented using processing unit 46. Processing unit 46 can include application-specific integrated circuit (ASIC) logic, graphics processor, general-purpose processor, etc. Video decoding system 40 can also include an optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, graphics processor, general-purpose processor, etc. In some instances, logic circuitry 47 can be implemented in hardware, such as dedicated video encoding hardware, while processor 43 can be implemented in general-purpose software, operating system, etc. Furthermore, memory 44 can be any type of memory, such as volatile memory (e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory 44 can be implemented using cache memory. In some instances, logic circuitry 47 may access memory 44 (e.g., for implementing an image buffer). In other instances, logic circuitry 47 and / or processing unit 46 may include memory (e.g., cache, etc.) for implementing image buffers, etc.

[0112] In some instances, the encoder 20 implemented via logic circuitry may include (e.g., implemented via processing unit 46 or memory 44) an image buffer and (e.g., implemented via processing unit 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the encoder 20 implemented via logic circuitry 47 to implement a reference. Figure 2 And / or any other encoder system or subsystem described herein, and the various modules discussed herein. Logic circuits may be used to perform the various operations discussed herein.

[0113] In some instances, decoder 30 can be implemented in a similar manner via logic circuitry 47 to implement reference. Figure 3The decoder 30 and / or any other decoder system or subsystem described herein are various modules discussed. In some instances, the logic circuit-implemented decoder 30 may include an image buffer (implemented via processing unit 2820 or memory 44) and a graphics processing unit (e.g., implemented via processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the decoder 30 implemented via logic circuit 47 to implement reference Figure 3 And / or the various modules discussed in any other decoder system or subsystem described herein.

[0114] In some instances, antenna 42 can be used to receive encoded bitstreams of video data. As discussed herein, the encoded bitstream may contain data related to encoded video frames, indicators, index values, mode selection data, etc., such as data related to encoded segmentation (e.g., transform coefficients or quantized transform coefficients, optional indicators, and / or data defining encoded segmentation). Video decoding system 40 may also include decoder 30 coupled to antenna 42 for decoding the encoded bitstream. Display device 45 is used to display the video frames.

[0115] It should be understood that, in embodiments of the present invention, for the example described with reference to encoder 20, decoder 30 can be used to perform the reverse process. Regarding signaling syntax elements, decoder 30 can be used to receive and parse such syntax elements, and accordingly decode the associated video data. In some examples, encoder 20 can entropy-encode syntax elements into an encoded video bitstream. In such instances, decoder 30 can parse such syntax elements and accordingly decode the associated video data.

[0116] It should be noted that the method described in the embodiments of the present invention is mainly used for the inter-frame prediction process, which exists in both the encoder 20 and the decoder 30. The encoder 20 and decoder 30 in the embodiments of the present invention can be, for example, the encoder / decoder corresponding to video standard protocols such as H.263, H.264, HEVV, MPEG-2, MPEG-4, VP8, VP9, ​​or next-generation video standard protocols (such as H.266).

[0117] See Figure 2 , Figure 2 A schematic / conceptual block diagram illustrating an example of an encoder 20 for implementing embodiments of the present invention is shown. Figure 2In this example, encoder 20 includes a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a buffer 216, a loop filter unit 220, a decoded picture buffer (DPB) 230, a prediction processing unit 260, and an entropy coding unit 270. Prediction processing unit 260 may include inter-frame prediction unit 244, intra-frame prediction unit 254, and mode selection unit 262. Inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0118] For example, the residual calculation unit 204, transform processing unit 206, quantization unit 208, prediction processing unit 260, and entropy coding unit 270 form the forward signal path of the encoder 20, while, for example, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, and prediction processing unit 260 form the backward signal path of the encoder, wherein the backward signal path of the encoder corresponds to the signal path of the decoder (see...). Figure 3 Decoder 30 in the middle.

[0119] Encoder 20 receives, for example, image 201 or image block 203 of image 201 via input 202, such as images in a sequence of images forming a video or video sequence. Image block 203 may also be referred to as the current image block or the image block to be encoded, and image 201 may be referred to as the current image or the image to be encoded (especially in video encoding when distinguishing the current image from other images, such as previously encoded and / or decoded images in the same video sequence, i.e., including the current image).

[0120] Embodiments of encoder 20 may include a segmentation unit ( Figure 2 (Not shown in the image) is used to segment image 201 into multiple blocks, such as image blocks 203, typically into multiple non-overlapping blocks. The segmentation unit can be used to apply the same block size and corresponding raster defining the block size to all images in a video sequence, or to change the block size between images, subsets, or groups of images, and segment each image into corresponding blocks.

[0121] In one instance, the prediction processing unit 260 of encoder 20 can be used to perform any combination of the segmentation techniques described above.

[0122] As with image 201, image patch 203 is also, or can be considered as, a two-dimensional array or matrix of sampled points with sampled values, although its size is smaller than that of image 201. In other words, image patch 203 may include, for example, a single sampled array (e.g., a luminance array in the case of black and white image 201) or three sampled arrays (e.g., a luminance array and two chrominance arrays in the case of a color image) or any other number and / or type of array depending on the applied color format. The number of sampled points in the horizontal and vertical directions (or axes) of image patch 203 defines the size of image patch 203.

[0123] like Figure 2 The encoder 20 shown is used to encode the image 201 block by block, for example, to perform encoding and prediction for each image block 203.

[0124] The residual calculation unit 204 is used to calculate the residual block 205 based on the image block 203 and the prediction block 265 (other details of the prediction block 265 are provided below). For example, the residual block 205 is obtained in the sample domain by subtracting the sample value of the prediction block 265 from the sample value of the image block 203 on a sample-by-sample (pixel-by-pixel) basis.

[0125] The transform processing unit 206 applies a transform, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), to the sample values ​​of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 can also be called transform residual coefficients and represent the residual block 205 in the transform domain.

[0126] Transform processing unit 206 can be used to apply integer approximations of DCT / DST, such as those specified for HEVC / H.265. These integer approximations are typically scaled by a factor compared to orthogonal DCT transforms. To maintain the norm of the residual blocks processed by the forward and inverse transforms, an additional scaling factor is applied as part of the transform process. The scaling factor is typically chosen based on certain constraints, such as a power of 2 for the shift operation, the bit depth of the transform coefficients, accuracy, and implementation cost. For example, a specific scaling factor can be specified on the decoder 30 side via, for example, inverse transform processing unit 212 (and on the encoder 20 side via, for example, inverse transform processing unit 212 for the corresponding inverse transform), and correspondingly, a corresponding scaling factor can be specified on the encoder 20 side via transform processing unit 206 for the forward transform.

[0127] Quantization unit 208 is used to quantize the transform coefficients 207, for example, by applying scalar quantization or vector quantization, to obtain quantized transform coefficients 209. Quantized transform coefficients 209 can also be referred to as quantized residual coefficients 209. The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scales can be applied to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. The appropriate quantization step size can be indicated by the quantization parameter (QP). For example, the quantization parameter can be an index of a predefined set of appropriate quantization steps. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization can involve division by the quantization step size and a corresponding quantization or inverse quantization, performed, for example, by inverse quantization 210, or it can involve multiplication by the quantization step size. The quantization step size can be determined using quantization parameters according to some standard embodiments, such as HEVC. Generally, the quantization step size can be calculated based on the quantization parameters using a fixed-point approximation of an equation involving division. Additional scaling factors can be introduced for quantization and inverse quantization to recover the norm of the residual block, which may have been modified due to the scaling used in the fixed-point approximation of the equations used for the quantization step size and quantization parameters. In one example implementation, the scaling of the inverse transform and inverse quantization can be combined. Alternatively, a custom quantization table can be used and transmitted from the encoder to the decoder via a signal, for example, in a bitstream. Quantization is a lossy operation, where the loss increases with the quantization step size.

[0128] The inverse quantization unit 210 is used to apply the inverse quantization of the quantization unit 208 to the quantized coefficients to obtain the dequantized coefficients 211. For example, it applies the inverse quantization scheme of the quantization scheme applied by the quantization unit 208 based on or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 can also be called the dequantized residual coefficients 211, corresponding to the transform coefficients 207, although the loss due to quantization is usually different from that of the transform coefficients.

[0129] The inverse transform processing unit 212 is used to apply the inverse transform of the transform applied by the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), to obtain the inverse transform block 213 in the sample domain. The inverse transform block 213 can also be referred to as the inverse transform dequantized block 213 or the inverse transform residual block 213.

[0130] Reconstruction unit 214 (e.g., summer 214) is used to add inverse transform block 213 (i.e. reconstructed residual block 213) to prediction block 265 to obtain reconstructed block 215 in the sample domain, for example, by adding the sample value of reconstructed residual block 213 to the sample value of prediction block 265.

[0131] Optionally, for example, buffer unit 216 (or simply "buffer" 216) of line buffer 216 is used to buffer or store reconstructed block 215 and corresponding sample values ​​for, for example, intra-frame prediction. In other embodiments, the encoder can be used to perform any type of estimation and / or prediction, such as intra-frame prediction, using the unfiltered reconstructed block and / or corresponding sample values ​​stored in buffer unit 216.

[0132] For example, embodiments of encoder 20 can be configured such that buffer unit 216 is used not only to store the reconstructed block 215 for intra-frame prediction 254, but also for loop filter unit 220 (in Figure 2 (not shown in the image), and / or, for example, such that buffer unit 216 and decoded image buffer unit 230 form a buffer. Other embodiments may be used to filter block 221 and / or blocks or samples from decoded image buffer 230 (not shown in the image), and / or, for example, such that buffer unit 216 and decoded image buffer unit 230 form a buffer. Figure 2 (Not shown in the image) is used as input or basis for intra-frame prediction 254.

[0133] Loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, thereby facilitating pixel transformation or improving video quality. Loop filter unit 220 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters such as a bilateral filter, an adaptive loop filter (ALF), or a sharpening or smoothing filter, or a co-filter. Although loop filter unit 220 is used in... Figure 2 The image shown is an in-loop filter, but in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be referred to as the filtered reconstructed block 221. The decoded image buffer 230 can store the reconstructed encoded block after the loop filter unit 220 performs the filtering operation on the reconstructed encoded block.

[0134] An embodiment of encoder 20 (correspondingly, loop filter unit 220) can be used to output loop filter parameters (e.g., sample adaptive offset information), for example, directly output or entropy encoded by entropy coding unit 270 or any other entropy coding unit, such that decoder 30 can receive and apply the same loop filter parameters for decoding.

[0135] The decoded picture buffer (DPB) 230 can be a reference picture memory for storing reference picture data for the encoder 20 to encode video data. The DPB 230 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM)) or other types of memory devices. The DPB 230 and buffer 216 can be provided by the same memory device or separate memory devices. In one example, the decoded picture buffer (DPB) 230 is used to store filtered block 221. The decoded picture buffer 230 can be further used to store other previously filtered blocks of the same current picture or different pictures, such as previously reconstructed pictures, for example, previously reconstructed and filtered blocks 221, and can provide complete previously reconstructed, i.e., decoded pictures (and corresponding reference blocks and samples) and / or partially reconstructed current pictures (and corresponding reference blocks and samples), for example, for inter-frame prediction. In one instance, if reconstructed block 215 can be reconstructed without in-loop filtering, the decoded picture buffer (DPB) 230 is used to store reconstructed block 215.

[0136] The prediction processing unit 260, also known as the block prediction processing unit 260, is used to receive or acquire image block 203 (current image block 203 of current image 201) and reconstructed image data, such as reference samples of the same (current) image from buffer 216 and / or reference image data 231 of one or more previously decoded images from decoded image buffer 230, and to process such data to make predictions, i.e., to provide prediction block 265 which can be inter-frame prediction block 245 or intra-frame prediction block 255.

[0137] The mode selection unit 262 can be used to select a prediction mode (e.g., intra-frame or inter-frame prediction mode) and / or the corresponding prediction block 245 or 255 used as prediction block 265 to compute residual block 205 and reconstructed block 215.

[0138] Embodiments of the mode selection unit 262 can be used to select a prediction mode (e.g., from those prediction modes supported by the prediction processing unit 260), which provides the best match or minimum residual (minimum residual means better compression in transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression in transmission or storage), or considers or balances both. The mode selection unit 262 can be used to determine the prediction mode based on rate distortion optimization (RDO), i.e., selecting the prediction mode that provides minimum RDO optimization, or selecting the prediction mode whose relevant rate distortion at least meets the prediction mode selection criteria.

[0139] The following will explain in detail the prediction processing performed by instances of encoder 20 (e.g., by prediction processing unit 260) and the mode selection performed (e.g., by mode selection unit 262).

[0140] As described above, encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra-frame prediction modes and / or inter-frame prediction modes.

[0141] The intra-prediction mode set can include 35 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in H.265, or it can include 67 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in the developing H.266.

[0142] In possible implementations, the set of inter-frame prediction modes depends on the available reference image (i.e., at least a portion of the decoded image stored in the DBP 230 as described above) and other inter-frame prediction parameters, such as whether to use the entire reference image or only a portion of the reference image, such as a search window region surrounding the current block, to search for the best matching reference block, and / or whether to apply pixel interpolation such as half-pixel and / or quarter-pixel interpolation. The set of inter-frame prediction modes may include, for example, an Advanced Motion Vector Prediction (AMVP) mode and a merge mode. In specific implementations, the set of inter-frame prediction modes may include an improved control-point-based AMVP mode according to embodiments of the present invention, and an improved control-point-based merge mode. In one instance, the intra-frame prediction unit 254 may be used to perform any combination of the inter-frame prediction techniques described below.

[0143] In addition to the prediction modes described above, embodiments of the present invention may also apply skip mode and / or direct mode.

[0144] The prediction processing unit 260 can be further used to segment the image block 203 into smaller block partitions or sub-blocks, for example, by iteratively using quad-tree (QT) segmentation, binary-tree (BT) segmentation, or triple-tree (TT) segmentation, or any combination thereof, and to perform prediction for each of the block partitions or sub-blocks, for example, wherein mode selection includes selecting the tree structure of the segmented image block 203 and selecting a prediction mode to be applied to each of the block partitions or sub-blocks.

[0145] Inter-frame prediction unit 244 may include motion estimation (ME) units. Figure 2 (not shown in the image) and motion compensation (MC) unit ( Figure 2 (Not shown in the image). The motion estimation unit is used to receive or acquire image block 203 (current image block 203 of current image 201) and decoded image 231, or at least one or more previously reconstructed blocks, such as reconstructed blocks of one or more other / different previously decoded images 231, to perform motion estimation. For example, a video sequence may include the current image and the previously decoded image 31, or in other words, the current image and the previously decoded image 31 may be part of an image sequence that forms a video sequence, or form that image sequence.

[0146] For example, encoder 20 can be used to select a reference block from multiple reference blocks of the same or different images in multiple other images, and send it to the motion estimation unit (…). Figure 2(Not shown in the image) Provides a reference image and / or provides the offset (spatial offset) between the position (X, Y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter. This offset is also called a motion vector (MV).

[0147] The motion compensation unit is used to acquire inter-frame prediction parameters and perform inter-frame prediction based on or using these parameters to obtain inter-frame prediction block 245. (The motion compensation unit...) Figure 2 Motion compensation performed (not shown) can involve extracting or generating prediction blocks based on motion / block vectors determined through motion estimation (possibly performing interpolation with subpixel accuracy). Interpolation filtering can generate additional pixel samples from known pixel samples, potentially increasing the number of candidate prediction blocks available for encoding picture blocks. Once the motion vector of the PU for the current picture block is received, the motion compensation unit 246 can locate the prediction block pointed to by the motion vector in a list of reference pictures. The motion compensation unit 246 can also generate syntax elements associated with blocks and video stripes for the decoder 30 to use when decoding picture blocks of the video stripe.

[0148] Specifically, the inter-frame prediction unit 244 can transmit syntax elements to the entropy coding unit 270. These syntax elements include inter-frame prediction parameters (e.g., indication information for selecting the inter-frame prediction mode for the current block prediction after traversing multiple inter-frame prediction modes). In some application scenarios, if there is only one inter-frame prediction mode, the inter-frame prediction parameters may not be carried in the syntax elements; in this case, the decoding end 30 can directly use the default prediction mode for decoding. It is understood that the inter-frame prediction unit 244 can be used to perform any combination of inter-frame prediction techniques.

[0149] Intra-prediction unit 254 is used to acquire, for example, image block 203 (current image block) of the same image and one or more previously reconstructed blocks, such as reconstructed adjacent blocks, for intra-prediction. For example, encoder 20 can be used to select an intra-prediction mode from a plurality of (predetermined) intra-prediction modes.

[0150] An embodiment of encoder 20 can be used to select an intra-prediction mode based on optimization criteria, such as based on minimum residual (e.g., providing an intra-prediction mode that most closely resembles the current image block 203) or minimum bitrate distortion.

[0151] Intra-prediction unit 254 is further configured to determine intra-prediction block 255 based on intra-prediction parameters as selected for the intra-prediction mode. In any case, after selecting the intra-prediction mode for the block, intra-prediction unit 254 is also configured to provide intra-prediction parameters to entropy coding unit 270, i.e., to provide information indicating the selected intra-prediction mode for the block. In one instance, intra-prediction unit 254 can be used to perform any combination of intra-prediction techniques.

[0152] Specifically, the intra-prediction unit 254 can transmit syntax elements to the entropy coding unit 270. These syntax elements include intra-prediction parameters (e.g., indication information for selecting the intra-prediction mode for the current block prediction after traversing multiple intra-prediction modes). In some application scenarios, if there is only one intra-prediction mode, the intra-prediction parameters may not be carried in the syntax elements; in this case, the decoder 30 can directly use the default prediction mode for decoding.

[0153] Entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (e.g., variable length coding (VLC), context adaptive VLC (CAVLC), arithmetic coding, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to one or all (or not to apply) of the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, and / or loop filter parameters to obtain encoded picture data 21 that can be output as, for example, an encoded bitstream 21, through output 272. The encoded bitstream can be transmitted to video decoder 30, or archived for later transmission or retrieval by video decoder 30. Entropy coding unit 270 can also be used to entropy code other syntax elements of the current video strip being encoded.

[0154] Other structural variations of the video encoder 20 can be used to encode video streams. For example, a non-transform-based encoder 20 can directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another embodiment, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0155] Specifically, in this embodiment of the invention, the encoder 20 can be used to implement the inter-frame prediction method described in the following embodiments.

[0156] It should be understood that other structural variations of the video encoder 20 can be used to encode the video stream. For example, for certain image blocks or frames, the video encoder 20 can directly quantize the residual signal without processing by the transform processing unit 206, and correspondingly, without processing by the inverse transform processing unit 212; or, for certain image blocks or frames, the video encoder 20 does not generate residual data, and correspondingly, without processing by the transform processing unit 206, the quantization unit 208, the inverse quantization unit 210, and the inverse transform processing unit 212; or, the video encoder 20 can directly store the reconstructed image block as a reference block without processing by the filter 220; or, the quantization unit 208 and the inverse quantization unit 210 in the video encoder 20 can be combined. The loop filter 220 is optional, and for lossless compression coding, the transform processing unit 206, the quantization unit 208, the inverse quantization unit 210, and the inverse transform processing unit 212 are optional. It should be understood that, depending on the application scenario, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 may be selectively enabled.

[0157] See Figure 3 , Figure 3 A schematic / conceptual block diagram illustrating an example of a decoder 30 for implementing embodiments of the present invention is shown. The video decoder 30 is configured to receive, for example, encoded image data (e.g., encoded bitstream) 21 encoded by encoder 20, to obtain a decoded image 231. During the decoding process, the video decoder 30 receives video data from the video encoder 20, such as encoded video bitstreams representing image blocks of encoded video stripes and associated syntax elements.

[0158] exist Figure 3 In one example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a buffer 316, a loop filter 320, a decoded image buffer 330, and a prediction processing unit 360. Prediction processing unit 360 may include an inter-frame prediction unit 344, an intra-frame prediction unit 354, and a mode selection unit 362. In some instances, video decoder 30 can perform substantially the same functions as the referenced unit. Figure 2 The video encoder 20 describes the encoding passes as the inverse of the decoding passes.

[0159] Entropy decoding unit 304 is used to perform entropy decoding on encoded image data 21 to obtain, for example, quantization coefficients 309 and / or decoded encoded parameters. Figure 3(Not shown in the image), for example, any one or all of the (decoded) inter-frame prediction, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 is further configured to forward the inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the prediction processing unit 360. The video decoder 30 may receive syntax elements at the video strip level and / or video block level.

[0160] The inverse quantization unit 310 can function the same as the inverse quantization unit 110, the inverse transform processing unit 312 can function the same as the inverse transform processing unit 212, the reconstruction unit 314 can function the same as the reconstruction unit 214, the buffer 316 can function the same as the buffer 216, the loop filter 320 can function the same as the loop filter 220, and the decoded image buffer 330 can function the same as the decoded image buffer 230.

[0161] The prediction processing unit 360 may include an inter-frame prediction unit 344 and an intra-frame prediction unit 354, wherein the inter-frame prediction unit 344 may be functionally similar to the inter-frame prediction unit 244, and the intra-frame prediction unit 354 may be functionally similar to the intra-frame prediction unit 254. The prediction processing unit 360 is typically used to perform block prediction and / or obtain prediction blocks 365 from encoded data 21, and to receive or obtain prediction-related parameters and / or information about the selected prediction mode from, for example, an entropy decoding unit 304 (explicitly or implicitly).

[0162] When a video strip is encoded as an intra-coded (I) strip, the intra-prediction unit 354 of the prediction processing unit 360 generates a prediction block 365 for a picture block of the current video strip based on the intra-prediction mode of the signal representation and data from previously decoded blocks of the current frame or picture. When a video frame is encoded as an inter-coded (i.e., B or P) strip, the inter-prediction unit 344 (e.g., a motion compensation unit) of the prediction processing unit 360 generates a prediction block 365 for a video block of the current video strip based on motion vectors and other syntax elements received from the entropy decoding unit 304. For inter-frame prediction, a prediction block can be generated from a reference picture within a list of reference pictures. The video decoder 30 can construct a list of reference frames, list 0 and list 1, based on reference pictures stored in the DPB 330 using a default construction technique.

[0163] The prediction processing unit 360 is configured to determine prediction information for video blocks of the current video slice by parsing motion vectors and other syntax elements, and to use the prediction information to generate prediction blocks for the current video slice to be properly decoded. In one embodiment of the invention, the prediction processing unit 360 uses some received syntax elements to determine the prediction mode (e.g., intra-frame or inter-frame prediction), inter-frame prediction slice type (e.g., B-slice, P-slice, or GPB-slice) for the video blocks used to encode the video slice, the construction information of one or more of the reference picture lists for the slice, the motion vector for each inter-frame coded video block of the slice, the inter-frame prediction state of each inter-frame coded video block of the slice, and other information to decode the video blocks of the current video slice. In another embodiment of the present disclosure, the syntax elements received by the video decoder 30 from the bitstream include syntax elements from one or more of the adaptive parameter set (APS), sequence parameter set (SPS), picture parameter set (PPS), or slice header.

[0164] The inverse quantization unit 310 can be used to inverse quantize (i.e., dequantize) the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 304. The inverse quantization process may involve using quantization parameters calculated by the video encoder 20 for each video block in the video strip to determine the degree of quantization to be applied and similarly determine the degree of inverse quantization to be applied.

[0165] The inverse transform processing unit 312 is used to apply an inverse transform (e.g., inverse DCT, inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to generate a residual block in the pixel domain.

[0166] Reconstruction unit 314 (e.g., summer 314) is used to add inverse transform block 313 (i.e. reconstructed residual block 313) to prediction block 365 to obtain reconstructed block 315 in the sample domain, for example by adding the sample values ​​of reconstructed residual block 313 to the sample values ​​of prediction block 365.

[0167] Loop filter unit 320 (during or after the encoding loop) is used to filter the reconstructed block 315 to obtain the filtered block 321, thereby facilitating pixel transformation or improving video quality. In one instance, loop filter unit 320 can be used to perform any combination of the filtering techniques described below. Loop filter unit 320 is intended to represent one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or other filters such as a bilateral filter, an adaptive loop filter (ALF), or a sharpening or smoothing filter, or a co-filter. Although loop filter unit 320 is used in... Figure 3 The loop filter unit 320 is shown as an in-loop filter, but in other configurations, it can be implemented as a post-loop filter.

[0168] The decoded video block 321 in the given frame or image is then stored in the decoded image buffer 330, which stores a reference image for subsequent motion compensation.

[0169] Decoder 30 is used, for example, to output decoded image 31 via output 332 for presentation to or viewing by the user.

[0170] Other variations of the video decoder 30 can be used to decode compressed bitstreams. For example, the decoder 30 can generate an output video stream without the loop filter unit 320. For example, the non-transform-based decoder 30 can directly inverse quantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another embodiment, the video decoder 30 may have an inverse quantization unit 310 and an inverse transform processing unit 312 combined into a single unit.

[0171] Specifically, in this embodiment of the invention, the decoder 30 is used to implement the inter-frame prediction method described in the following embodiments.

[0172] It should be understood that other structural variations of the video decoder 30 can be used to decode the encoded video bitstream. For example, the video decoder 30 may generate an output video stream without processing by filter 320; or, for certain image blocks or frames, the entropy decoding unit 304 of the video decoder 30 may not decode the quantized coefficients, and accordingly, processing by the inverse quantization unit 310 and the inverse transform processing unit 312 is unnecessary. The loop filter 320 is optional; and for lossless compression, the inverse quantization unit 310 and the inverse transform processing unit 312 are optional. It should be understood that, depending on the application scenario, the inter-frame prediction unit and the intra-frame prediction unit may be selectively enabled.

[0173] It should be understood that in the encoder 20 and decoder 30 of this application, the processing result of a certain stage can be further processed and output to the next stage. For example, after the interpolation filtering, motion vector derivation or loop filtering stages, the processing result of the corresponding stage can be further clipped or shifted.

[0174] For example, the motion vectors of the control points of the current image block derived from the motion vectors of adjacent affine coded blocks, or the motion vectors of the sub-blocks of the current image block derived from them, can be further processed, and this application does not limit this. For example, the range of motion vector values ​​can be constrained to a certain bit width. Assuming the allowed bit width of the motion vector is bitDepth, the range of the motion vector is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where the "^" symbol represents a power. If bitDepth is 16, the value range is -32768 to 32767. If bitDepth is 18, the value range is -131072 to 131071. As another example, the values ​​of the motion vectors (e.g., the motion vectors MV of four 4x4 sub-blocks within an 8x8 image block) can be constrained so that the maximum difference between the integer parts of the four 4x4 sub-blocks MV does not exceed N pixels, for example, not more than one pixel.

[0175] The following two methods can be used to constrain it within a certain bit width:

[0176] Method 1: Remove the high-order bits of the overflowing motion vector:

[0177] ux = (vx + 2) bitDepth )%2 bitDepth

[0178] vx = (ux >= 2) bitDepth-1 )? (ux-2 bitDepth ):ux

[0179] uy=(vy+2 bitDepth )%2 bitDepth

[0180] vy=(uy>=2) bitDepth-1 )? (uy-2 bitDepth ):uy

[0181] Where vx is the horizontal component of the motion vector of the image block or its sub-blocks, vy is the vertical component of the motion vector of the image block or its sub-blocks, ux and uy are intermediate values; bitDepth represents the bit width.

[0182] For example, if the value of vx is -32769, the result obtained using the above formula is 32767. This is because in computers, values ​​are stored in two's complement form. The two's complement of -32769 is 1,0111,1111,1111,1111 (17 bits). The computer discards the high-order bits when overflowing, so the value of vx becomes 0111,1111,1111,1111,1111, which is 32767, consistent with the result obtained using the formula.

[0183] Method 2 involves clipping the motion vectors, as shown in the following formula:

[0184] vx = Clip3(-2) bitDepth-1 ,2 bitDepth-1 -1,vx)

[0185] vy = Clip3(-2) bitDepth-1 ,2 bitDepth-1 -1,vy)

[0186] Where vx is the horizontal component of the motion vector of the image block or its sub-blocks, and vy is the vertical component of the motion vector of the image block or its sub-blocks; where x, y, and z correspond to the three input values ​​of the MV clamping process Clip3, and Clip3 is defined as clamping the value of z to the interval [x, y].

[0187]

[0188] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a video decoding device 400 (e.g., a video encoding device 400 or a video decoding device 400) provided in an embodiment of the present invention. The video decoding device 400 is suitable for implementing the embodiments described herein. In one embodiment, the video decoding device 400 may be a video decoder (e.g., ... Figure 1A decoder 30) or video encoder (e.g. Figure 1A The encoder 20). In another embodiment, the video decoding device 400 may be the one described above. Figure 1A decoder 30 or Figure 1A One or more components in encoder 20.

[0189] The video decoding device 400 includes: an input port 410 and a receiving unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; a transmitter unit (Tx) 440 and an output port 450 for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include photoelectric conversion components and electro-optical (EO) components coupled to the input port 410, receiver unit 420, transmitter unit 440, and output port 450 for the input or output of optical or electrical signals.

[0190] Processor 430 is implemented in both hardware and software. Processor 430 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 430 communicates with ingress port 410, receiver unit 420, transmitter unit 440, egress port 450, and memory 460. Processor 430 includes a decoding module 470 (e.g., encoding module 470 or decoding module 470). Encoding / decoding module 470 implements the embodiments disclosed herein to implement the chroma block prediction method provided by embodiments of the present invention. For example, encoding / decoding module 470 implements, processes, or provides various encoding operations. Therefore, encoding / decoding module 470 provides a substantial improvement to the functionality of video decoding device 400 and affects the transitions of video decoding device 400 to different states. Alternatively, encoding / decoding module 470 can be implemented with instructions stored in memory 460 and executed by processor 430.

[0191] Memory 460 includes one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store programs while they are selectively executed, and to store instructions and data read during program execution. Memory 460 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random access memory (SRAM).

[0192] See Figure 5 , Figure 5 It is available as an exemplary embodiment. Figure 1A A simplified block diagram of device 500 comprising one or both of the source device 12 and the destination device 14. Device 500 can implement the technology of this application. In other words, Figure 5This is a schematic block diagram illustrating one implementation of an encoding or decoding device (referred to as decoding device 500) according to an embodiment of this application. The decoding device 500 may include a processor 510, a memory 530, and a bus system 550. The processor and memory are connected via the bus system. The memory stores instructions, and the processor executes the instructions stored in the memory. The memory of the decoding device stores program code, and the processor can call the program code stored in the memory to execute various video encoding or decoding methods described in this application. To avoid repetition, further details are omitted here.

[0193] In this embodiment, the processor 510 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0194] The memory 530 may include a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may also be used as memory 530. Memory 530 may include code and data 531 accessed by processor 510 using bus 550. Memory 530 may further include an operating system 533 and an application program 535, which includes at least one program that allows processor 510 to execute the video encoding or decoding methods described in this application (especially the inter-frame prediction methods described in this application). For example, application program 535 may include applications 1 to N, which further include a video encoding or decoding application (hereinafter referred to as a video decoding application) that executes the video encoding or decoding methods described in this application.

[0195] In addition to the data bus, the bus system 550 may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 550 in the diagram.

[0196] Optionally, the decoding device 500 may also include one or more output devices, such as a display 570. In one example, the display 570 may be a haptic display that combines a display with a haptic unit capable of operatively sensing touch input. The display 570 may be connected to the processor 510 via a bus 550.

[0197] The following details the solutions of the embodiments of this application:

[0198] Video coding mainly includes intra-frame prediction, inter-frame prediction, transform, quantization, entropy encoding, and in-loop filtering (primarily de-blocking filtering). After dividing the image into coding blocks, intra-frame or inter-frame prediction is performed, and transform quantization is applied after obtaining the residuals. Finally, entropy encoding is performed to output the bitstream. Here, the coding block is an array of pixels (M can be equal to N or not), and the pixel value at each pixel position is known.

[0199] Intra-frame prediction refers to using the pixel values ​​of pixels in the reconstructed region of the current image to predict the pixel values ​​of pixels in the current coding block.

[0200] Inter-frame prediction involves finding a matching reference block for the current coded block in the reconstructed image to obtain the motion information of the current coded block. Then, based on the motion information, the predicted information or predicted value of the pixel values ​​in the current coded block is calculated (the distinction between information and value will not be made hereafter). The process of calculating motion information is called motion estimation (ME), and the process of calculating the predicted value of the pixel values ​​in the current coded block is called motion compensation (MC).

[0201] It should be noted that the motion information of the current coded block includes the indication information of the prediction direction (usually forward prediction, backward prediction, or bidirectional prediction), one or two motion vectors (MV) pointing to the reference block, and the indication information of the image where the reference block is located (usually denoted as the reference frame index).

[0202] Forward prediction refers to selecting a reference image from the forward reference image set to obtain a reference block for the current coding block. Backward prediction refers to selecting a reference image from the backward reference image set to obtain a reference block for the current coding block. Bidirectional prediction refers to selecting a reference image from both the forward and backward reference image sets to obtain a reference block. When using the bidirectional prediction method, the current coding block will have two reference blocks. Each reference block requires a motion vector and a reference frame index for indication. Then, the predicted pixel values ​​of the pixels in the current block are determined based on the pixel values ​​of the pixels in the two reference blocks.

[0203] The motion estimation process requires trying multiple reference blocks in the reference image for the current coded block. The final choice of which reference block(s) to use for prediction is determined using rate-distortion optimization (RDO) or other methods.

[0204] After obtaining prediction information using intra-frame or inter-frame prediction, the pixel value of each pixel in the current coding block is subtracted from the corresponding prediction information to obtain residual information. This residual information is then transformed using methods such as Discrete Cosine Transform (DCT), and finally encoded using quantization entropy to obtain the bitstream. The predicted signal plus the reconstructed residual signal requires further filtering to obtain the reconstructed signal, which is then used as the reference signal for subsequent coding.

[0205] Decoding is essentially the reverse of encoding. For example, entropy decoding is first used to inverse quantize and transform to obtain residual information. The decoded bitstream then determines whether the current coded block uses intra-frame prediction or inter-frame prediction. If it's intra-frame prediction, the pixel values ​​of surrounding reconstructed pixels are used to construct prediction information according to the intra-frame prediction method used. If it's inter-frame prediction, motion information needs to be parsed, and the parsed motion information is used to determine a reference block in the reconstructed image. The pixel values ​​of the pixels within this block are then used as prediction information; this process is called motion compensation (MC). The reconstructed information is obtained by adding the prediction information and residual information to the filter.

[0206] In HEVC, two inter-frame prediction modes are used: Advanced Motion Vector Prediction (AMVP) mode and Merge mode.

[0207] For AMVP mode, a candidate motion vector list is first constructed using motion information from adjacent coded blocks in the spatial or temporal domains of the current coded block. Then, the optimal motion vector is selected from this list as the motion vector predictor (MVP) for the current coded block. The rate-distortion cost is calculated using the formula J = SAD + λ, where J is the rate-distortion cost (RD Cost), SAD is the sum of absolute differences (SAD) between the predicted pixel value and the original pixel value obtained after motion estimation using the candidate motion vector predictor, R is the bit rate, and λ is the Lagrange multiplier. The encoder passes the index of the selected motion vector predictor in the candidate motion vector list and the reference frame index to the decoder. Further, motion search is performed within the neighborhood centered on the MVP to obtain the actual motion vector of the current coded block. The encoder passes the difference between the MVP and the actual motion vector (Motion vector difference) to the decoder.

[0208] For Merge mode, a candidate motion information list is first constructed using motion information from adjacent coded blocks in the spatial or temporal domains of the current coded block. Then, the optimal motion information is determined from the candidate motion information list using rate-distortion cost and used as the motion information for the current coded block. Finally, the index value of the optimal motion information in the candidate motion information list (denoted as the merge index, hereinafter the same) is passed to the decoder. The spatial and temporal candidate motion information of the current coded block is as follows: Figure 6 As shown, the spatial candidate motion information comes from five spatially adjacent blocks (A0, A1, B0, B1, and B2). If an adjacent block is unavailable or is in intra-frame coding mode, it is not added to the candidate motion information list. The temporal candidate motion information of the current coded block is obtained by scaling the MV of the corresponding block in the reference frame based on the picture order count (POC) of the reference frame and the current frame. First, it is determined whether the block at position T in the reference frame is available. If it is not available, the block at position C is selected.

[0209] In HEVC's inter-frame prediction, all pixels within a coded block use the same motion information, and then motion compensation is performed based on the motion information to obtain the predicted values ​​of the pixels in the coded block.

[0210] A video sequence contains a certain number of images—usually called frames. Adjacent images are often very similar, meaning they contain a lot of redundancy. The purpose of motion compensation is to improve the compression ratio by eliminating this redundancy between adjacent frames. Motion compensation is a method for describing the differences between adjacent frames (adjacent here means adjacent in terms of coding relationship, but not necessarily adjacent in terms of playback order), and it is part of the inter-frame prediction process. Before motion compensation, the motion information of the coded block has been obtained through motion estimation or bitstream decoding. This motion information mainly includes: (1) the prediction direction of the coded block: including forward, backward and bidirectional prediction. Forward prediction indicates that the coded block is predicted by the preceding coded frame, backward prediction indicates that the coded block is predicted by the following coded frame, and bidirectional prediction indicates that the coded block is predicted by combining forward and backward coded frames; (2) the reference frame index of the coded block, indicating the frame in which the reference block of the current coded block is located; (3) the motion vector MV of the coded block, representing the motion displacement of the coded block relative to the reference block. MV includes a horizontal component (denoted as MV). x ) and vertical component (denoted as MV) y ), representing the horizontal and vertical displacements of the coded block relative to the reference block, respectively. There is only one MV when the coded block is predicted forward or backward, and two MVs when the coded block is predicted bidirectionally. Figure 7 The above motion information is explained. Figure 7 In the following descriptions of motion and prediction information, 0 represents forward and 1 represents backward. For example, Ref0 represents the forward reference frame, Ref1 represents the backward reference frame, MV0 represents the forward motion vector, and MV1 represents the backward motion vector. A, B, and C represent the forward reference block, the current coded block, and the backward reference block, respectively. Cur is the current coded frame, and the dashed line represents the motion trajectory of B. Motion compensation is the process of finding the reference block based on the motion information and processing the reference block to obtain the prediction block of the coded block.

[0211] The basic process of motion compensation for forward prediction is as follows: Figure 7 As shown, the current coded block is block B in the diagram, with width W and height H. Based on the motion information, the forward reference frame for the current coded block B is frame Ref0, and the forward motion vector MV0 of the current coded block B is (MV0 = ...). x ,MV0 yWhen encoding block B in the Cur frame, firstly, based on the coordinates (i,j) of the top-left corner of B in the Cur frame, find the same coordinate point in the Ref0 frame. Based on the width and height of block B, block B' in Ref0 can be obtained. Then, based on the MV0 of block B', B' is moved to block A. Finally, block A is interpolated to obtain the prediction block of the current encoding block B. The pixel values ​​of each pixel in the prediction block of the current encoding block B are called the predicted values ​​of the corresponding pixels in block B. The motion compensation process for backward prediction is the same as that for forward prediction, only the reference direction is different. It should be noted that the prediction blocks obtained by motion compensation for backward prediction and forward prediction are called the forward prediction block and backward prediction block, respectively. When the encoding block is not bidirectionally predicted, the forward prediction block and backward prediction block obtained at this time are the prediction blocks of the current encoding block.

[0212] For bidirectional prediction, the forward prediction block and the backward prediction block are first obtained by performing motion compensation processes according to the motion information, respectively. Then, the pixel values ​​with the same position in the forward prediction block and the backward prediction block are obtained by weighted prediction or by bidirectional optical flow (BIO or BDOF) based on bidirectional prediction.

[0213] The weighted prediction method, when calculating the predicted value of the current coding block, simply involves sequentially weighting and summing the pixel values ​​of the preceding prediction block and the corresponding pixel values ​​of the following prediction block.

[0214] PredB(i,j)=ω0PredA(i,j)+ω1PredC(i,j) (1)

[0215] In formula (1), PredB(i,j), PredA(i,j), and PredC(i,j) are the predicted values ​​of the current coded block, the forward predicted block, and the backward predicted block at coordinate (i,j), respectively. ω0 and ω1 (0<=ω0<=1, 0<=ω1<=1, and ω0+ω1=1) are weighting coefficients, which may have different specific values ​​for different encoders. Typically, ω0 and ω1 are both 1 / 2.

[0216] Figure 8 An example is given to derive the prediction block of the current coded block by weighted summation. Figure 8 In this code, PredB, PredA, and PredC represent the prediction block, forward prediction block, and backward prediction block of the current coding block, respectively, each with a size of 4x4. The values ​​of the smaller blocks within each prediction block are the predicted values ​​at a specific point. PredB, PredA, and PredC each establish a coordinate system with their top-left corner as the origin. For example, the predicted value of PredB at coordinates (0,0) is:

[0217] PredB(0,0)=ω0PredA(0,0)+ω1PredC(0,0)

[0218] =ω0a 0,0 +ω1c 0,0

[0219] The PredB prediction value at coordinate (0,1) is:

[0220] PredB(0,1)=ω0PredA(0,1)+ω1PredC(0,1)

[0221] =ω0a 0,1 +ω1c 0,1

[0222] The remaining points are calculated in turn, without further explanation.

[0223] It can be seen that the bidirectional prediction weighted prediction technique is computationally simple, but this block-based motion compensation method is very crude, especially for images with complex textures, the prediction effect is poor and the compression efficiency is not high.

[0224] After performing bidirectional prediction motion compensation in the current CU, BIO obtains forward and backward prediction blocks. Based on the forward and backward prediction values, it derives the corrected motion vector for each 4x4 sub-block in the current CU. Finally, it performs compensation on each pixel within the current coding block to obtain the prediction block of the current CU.

[0225] The corrected motion vector (v) for each 4x4 sub-block x ,v y The predicted values ​​of L0 and L1 are obtained by applying BIO to a 6x6 window Ω around the sub-block. Specifically, (v x ,v y The derivation is done through formulas.

[0226]

[0227] in, The floor function (floor(A) represents the largest integer not greater than A); th' BIO =2 13-BD This is a threshold used to prevent excessively large correction motion vectors from causing error propagation. 2,m =S2>>12, S 2,s =S2&(2 12 -1). BD is the current pixel bit width. The ? operation means that if S1 > 0, then... Otherwise, v x =0. & is the bitwise AND operation.

[0228] S1, S2, S3, S5, and S6 are calculated according to the following formula:

[0229]

[0230] in,

[0231]

[0232] Among them, I (k) (i,j) is the predicted value of the pixel position (i,j) in the current CU (k equals 0 or 1, 0 represents forward, 1 represents backward, the same below); and The horizontal gradient value and the numerical gradient value at pixel position (i,j) are respectively obtained by the following formula:

[0233]

[0234] After obtaining the corrected motion vector according to formula (2), the final predicted value of each pixel in the current block is determined according to the following formula:

[0235]

[0236] shift and o offset The values ​​are 15-BD and 1<<(14-BD)+2·(1<<13). rnd(.) is the rounding function.

[0237] To calculate the corrected motion vector of a 4x4 sub-block, the forward and backward prediction values ​​I of its 6x6 region are required. (k) (x,y), the horizontal and vertical gradient values ​​in the forward and backward directions. and Calculating the gradient value of a 6x6 region requires the prediction value of an 8x8 region. Therefore, when obtaining forward and backward prediction values ​​through the interpolation filter, it is necessary to expand the prediction pixel block by 2 rows and 2 columns in each direction to obtain a prediction pixel block of size (W+4)*(H+4) before calculating the gradient value of (W+2)*(H+2), where W is the width of the current CU and H is the height of the current CU.

[0238] To reduce the complexity of BIO, existing technologies have implemented special handling for the boundaries of the CU.

[0239] First, the predicted value of the W*H region is obtained by using an 8-tap filter. Then, the region is expanded by 1 row and 1 column in all directions and the predicted value of the expanded region is obtained by using a bilinear filter, thus obtaining the predicted pixel value of the (W+2)*(H+2) region.

[0240] Next, based on the predicted pixel values ​​of the (W+2)*(H+2) region, the gradient value of the W*H region can be calculated according to formula (5).

[0241] Finally, following the padding method, the gradient values ​​in the W*H region are expanded outwards to obtain the gradient values ​​in the (W+2)*(H+2) region; the predicted values ​​in the W*H region are also expanded outwards to obtain the predicted values ​​in the (W+2)*(H+2) region. Padding is as follows. Figure 9 As shown, the pixel values ​​of the edge are assigned to the extended area.

[0242] The specific implementation process of BIO is as follows:

[0243] Step 1: Determine the motion information of the current CU

[0244] The motion information of the current CU can be determined by Merge mode, AMVP mode (as described in the background art), or other modes, which are not limited here.

[0245] It should be noted that other methods for determining motion information can also be applied to this invention, which will not be elaborated here.

[0246] Step 2: Determine if the current CU meets the conditions for using BIO.

[0247] If the current CU uses bidirectional prediction, and the relationship between the forward reference frame number POC_L0, the backward reference frame number POC_L1, and the current frame number POC_Cur satisfies the following formula, then the current CU meets the conditions for using BIO:

[0248] (POC_L0–POC_Cur)*(POC_L1–POC_Cur)<0

[0249] It should be noted that BIO can also be determined by checking if the current CU's size is greater than a preset threshold. For example, BIO can only be used if the current CU's height W>=8 and width H>=8.

[0250] It should be noted that other BIO usage conditions can also be applied to this invention, which will not be elaborated here.

[0251] If the current CU meets the conditions for using BIO, proceed to step 3; otherwise, perform motion compensation in other ways.

[0252] Step 3: Calculate the forward and backward prediction values ​​of the current CU.

[0253] Using motion information, motion compensation is performed to obtain forward and backward predicted values ​​I. (k)(i,j), where i=-1..cuW,j=-1..cuH (then we get the prediction matrix of (cuW+2)*(cuH+2)).

[0254] Among them, I (k) In (i,j), i = 0..cuW-1, j = 0..cuH-1, the values ​​are obtained by interpolation using an 8-tap interpolation filter, and the predicted values ​​at other positions (extended by 1 row and 1 column) are obtained by interpolation using a bilinear interpolation filter.

[0255] It should be noted that the predicted value of the extended region can also be obtained by other methods, such as using the same 8-tap interpolation filter, or directly using the reference pixel at the integer pixel position, which is not limited here.

[0256] It should be noted that, when calculating the SAD between the forward and backward predicted values, it is determined whether the SAD is less than the threshold TH_CU. If it is less, BIO is not executed. Otherwise, BIO is executed. Other determination methods can also be applied to this invention, which will not be elaborated here.

[0257] The formula for calculating SAD is as follows:

[0258]

[0259] The threshold TH_CU can be set to (1<<(BD–8+shift))*cuW*cuH, where shift is Max(2,14-BD).

[0260] Step 4: Calculate the horizontal and vertical gradient values ​​of the current CU's forward and backward predictions.

[0261] The horizontal and vertical gradient values ​​are calculated according to formula (4) based on the forward and backward predicted values. and Where i = 0..cuW-1, j = 0..cuH-1 (then the prediction matrix cuW*cuH is obtained).

[0262] Step 5: Padding the forward and backward predicted values ​​of the current CU, as well as the gradient values ​​in the horizontal and vertical directions.

[0263] like Figure 9 Padding is performed using the method to obtain... Where i = -1..cuW, j = -1..cuH (then we obtain the prediction matrix, horizontal gradient matrix, and vertical gradient matrix of (cuW+2)*(cuH+2)).

[0264] Step 6: Derive the corrected motion vector for each 4x4 sub-block, and then perform weighted summation.

[0265] For each 4x4 sub-block, vx and vy are obtained according to formula (2), and finally weighted according to formula (6) to obtain the predicted value of each 4x4 sub-block.

[0266] It should be noted that the SAD between the forward and backward prediction values ​​of each 4x4 sub-block can be calculated to determine whether it is less than the threshold TH_SCU. If it is less, a weighted average is directly performed; otherwise, vx and vy are obtained according to formula (2), and then weighted according to formula (6). Other judgment methods can also be applied to this invention, which will not be elaborated here. TU_SCU can be set to 1<<(BD–3+shift).

[0267] Virtual pipeline data units (VPDUs) are non-overlapping MxM luminance / NxN chrominance processing units. In a hardware decoder, consecutive VPDUs are processed simultaneously in different pipeline stages. Different pipeline stages process different VPDUs at the same time.

[0268] The criteria for classifying VPDUs are as follows:

[0269] 1) If the VPDU includes one or more CUs, the CUs are completely contained in the VPDU.

[0270] 2) If the CU includes one or more VPDUs, the VPDUs are completely contained within the CU.

[0271] In the existing technology, the size of the VPDU is 64x64, such as Figure 10 As shown, the dashed line represents the boundary of the VPDU, and the solid line represents the boundary of the CU. Figure 11 The image shows an illegal CU partition.

[0272] When a CU includes multiple VPDUs, the hardware decoder processes them by splitting them into consecutive VPDUs. For example, if the CU size is 128x128 and the VPDU size is 64x64, then four VPDUs will be processed consecutively.

[0273] The technical problem this invention aims to solve is that when the CU uses BIO technology for motion compensation, the processing method for CU boundary pixels differs from that for pixels inside the CU. If there is a VPDU partition boundary inside the CU, during BIO prediction, in order to ensure that the VPDU processing results are consistent with the CU processing results, this boundary needs to be processed in the same way as the pixels inside the CU, which increases the implementation complexity.

[0274] See Figure 12 This application provides an inter-frame prediction method, which can be applied to... Figure 2 The inter-frame prediction unit 244 in the encoder shown may be applied to Figure 3 In the inter-frame prediction unit 344 of the decoder shown, the method can be a bidirectional inter-frame prediction method, including:

[0275] Step 101: Select the smaller width from the preset image segmentation width Width and the width cuW of the image block to be processed, denoted as blkW, as the width of the first image block; select the smaller height from the preset image segmentation height High and the height cuH of the image block to be processed, denoted as blkH, as the height of the first image block.

[0276] When the method in this embodiment is applied to an encoder, the encoder divides the image into image blocks to be processed when encoding the image. In this step, the image blocks to be processed are obtained, and then the minimum width blkW = min(Width, cuW) is selected and the minimum width blkW is used as the width of the first image block, and the minimum height blkH = min(Hight, cuH) is selected and the minimum height blkH is used as the height of the first image block.

[0277] In this embodiment, when the method is applied to a decoder, the decoder receives a video stream from the encoder, which includes image blocks to be processed. In this step, the image blocks to be processed are extracted from the video stream, and then a minimum width blkW = min(Width, cuW) is selected and the minimum width blkW is used as the width of the first image block, and a minimum height blkH = min(Hight, cuH) is selected and the minimum height blkH is used as the height of the first image block.

[0278] The preset image segment width (Width) and preset image segment height (Hight) can be equal to the width and height of the VPDU, respectively. Alternatively, the preset image segment width (Width) can be 64, 32, or 16, and the preset image segment height (Hight) can also be 64, 32, or 16. For example, Width = 64, Height = 64; or Width = 32, Height = 32; or Width = 16, Height = 16; or Width = 64, Height = 32; or Width = 32, Height = 64; or Width = 64, Height = 16; or Width = 16, Height = 64; or Width = 32, Height = 16; or Width = 16, Height = 32, etc.

[0279] Step 102: Determine multiple first image blocks in the image block to be processed based on the width blkW and height blkH of the first image block.

[0280] It should be understood that, in one feasible implementation, the width and height of the image patch to be processed are the same as the width and height of the first image patch, respectively; that is, the image patch to be processed contains only one first image patch. Obtaining the predicted value of the first image patch is equivalent to obtaining the predicted value of the image patch to be processed.

[0281] For any first image patch, obtain the predicted value of the first image patch by performing the following steps 103 to 107.

[0282] Step 103: Obtain the first prediction block of the first image block based on the motion information of the image block to be processed. The width of the first prediction block is greater than the width of the first image block, and the height of the first prediction block is greater than the height of the first image block.

[0283] The motion information of the image block to be processed includes the motion information of the first image block, which includes information such as a reference image and motion vectors. The inter-frame prediction in this embodiment adopts a bidirectional prediction method based on optical flow (i.e., the BIO or BDOF correlation technology mentioned above), so the motion information of the first image block includes information such as a forward reference image, a backward reference image, a forward motion vector, and a backward motion vector.

[0284] When the method of this embodiment is applied to an encoder, the encoder can determine the motion information of the image block to be processed using Merge mode, AMVP mode, or other modes. The motion information of the image block to be processed includes the motion information of each first image block in the image block to be processed. In this step, the motion information of the image block to be processed determined by the encoder is obtained, and the motion information of the first image block is obtained from the motion information of the image block to be processed.

[0285] In this embodiment, when the method is applied to a decoder, the decoder receives motion information of image blocks to be processed from a video stream from the encoder. The motion information of the image blocks to be processed includes motion information of each first image block within the image blocks to be processed. In this step, the motion information of the image blocks to be processed is extracted from the video stream, and the motion information of the first image blocks is obtained from the motion information of the image blocks to be processed.

[0286] The first prediction block of the first image block includes a first forward prediction block and a first backward prediction block. In this step, the first forward prediction block and the first backward prediction block of the first image block can be obtained through the following steps (1) to (8). Steps (1) to (8) can be as follows:

[0287] (1): Based on the first position and motion information of the first image block in the image block to be processed, a first forward region is determined in the forward reference image. The width of the first forward region is blkW+2 and the height is blkH+2.

[0288] For example, see Figure 13 The motion information of the first image block B includes a forward reference image Ref0, a backward reference image Ref1, a forward motion vector MV0, and a backward motion vector MV1. Based on the first position of the first image block B, a second forward region B11 is determined in the forward reference image Ref0. The width of the second forward region B11 is blkW, and the height is blkH. Based on the positions of the forward motion vector MV0 and the second forward region B11, a third forward region B12 is determined. The width of the third forward region B12 is blkW, and the height is blkH. A first forward region A1, including the third forward region B12, is determined. The width of the first forward region A1 is blkW+2, and the height is blkH+2. The center of the third forward region B12 coincides with the center of the first forward region A1.

[0289] (2): In the forward reference image, determine whether the vertex position of the first forward region coincides with the pixel position in the forward reference image. If it coincides with the pixel position in the forward reference image, obtain the image block located in the first forward region from the forward reference image as the first forward prediction block of the first image block; if it does not coincide with the pixel position in the forward reference image, execute step (3).

[0290] For example, see Figure 13 Taking the top-left vertex of the first forward region A1 as an example, assuming the vertex position of the top-left vertex of the first forward region A1 in the forward reference image Ref0 is (15, 16), then the vertex position of this top-left vertex coincides with the position of a pixel in the forward reference image Ref0, which is the pixel at position (15, 16) in the forward reference image Ref0. As another example, assuming the vertex position of the top-left vertex of the first forward region A1 in the forward reference image Ref0 is (15.3, 16.2), then the vertex position of this top-left vertex does not coincide with the position of a pixel in the forward reference image Ref0, meaning there is no pixel at position (15.3, 16.2) in the forward reference image Ref0.

[0291] (3): In the forward reference image, determine the pixel closest to the vertex position of the first forward region, and use this pixel as the vertex to determine the fourth forward region. The width of the fourth forward region is blkW+2 and the height is blkH+2.

[0292] For any vertex position of the first forward region, taking the top left vertex position of the first forward region as an example, in the forward reference image, determine the pixel closest to the top left vertex position, and use this pixel as the top left vertex to determine the fourth forward region. The width of the fourth forward region is blkW+2 and the height is blkH+2.

[0293] For example, see Figure 13 The top left vertex of the first forward region A1 is located at (15.3, 16.2). In the forward reference image Ref0, the position of the pixel closest to the vertex (15.3, 16.2) is determined as (15, 16). The fourth forward region A2 is determined with the pixel at position (15, 16) as the top left vertex. The width of the fourth forward region A2 is blkW+2 and the height is blkH+2.

[0294] (4): Determine the fifth forward region, which includes the fourth forward region. The center of the fourth forward region and the center of the fifth forward region coincide. The width of the fifth forward region is blkW+n+1 and the height is blkH+n+1. Obtain the image block located in the fifth forward region in the forward reference image. Use the interpolation filter to interpolate and filter the image block to obtain the first forward prediction block of the first image block. The width of the first forward prediction block is blkW+2 and the height is blkH+2. n is the number of taps of the interpolation filter.

[0295] For example, see the example of an 8-tap interpolation filter. Figure 13 A fifth forward region A3 is determined, including the fourth forward region A2. The center of the fourth forward region A2 and the center of the fifth forward region A3 coincide. The width of the fifth forward region A3 is blkW+9 and the height is blkH+9. An image block located in the fifth forward region A3 is obtained in the forward reference image Ref0. An interpolation filter is used to interpolate and filter the image block to obtain the first forward prediction block of the first image block B. The width of the first forward prediction block is blkW+2 and the height is blkH+2.

[0296] (5): Based on the first position and motion information of the first image block, a first backward region is determined in the backward reference image. The width of the first backward region is blkW+2 and the height is blkH+2.

[0297] For example, see Figure 13 Based on the first position of the first image block B, a second backward region C11 is determined in the backward reference image Ref1. The width of the second backward region C11 is blkW, and the height is blkH. Based on the backward motion vector MV1 and the position of the second backward region C11, a third backward region C12 is determined. The width of the third backward region C12 is blkW, and the height is blkH. A first backward region D1, including the third backward region C12, is determined. The width of the first backward region D1 is blkW+2, and the height is blkH+2. The center of the third backward region C12 and the center of the first backward region D1 can coincide.

[0298] (6): In the backward reference image, determine whether the vertex position of the first backward region coincides with the pixel position in the backward reference image. If it coincides with the pixel position in the backward reference image, obtain the image block located in the first backward region from the backward reference image as the first backward prediction block of the first image block; if it does not coincide with the pixel position in the backward reference image, execute step (7).

[0299] For example, see Figure 13 Taking the top-left vertex of the first backward region A1 as an example, assuming the vertex position of the top-left vertex of the first backward region D1 in the backward reference image Ref0 is (5, 6), then the vertex position of this top-left vertex coincides with the position of a pixel in the backward reference image Ref0, which is the pixel at position (5, 6) in the backward reference image Ref0. As another example, assuming the vertex position of the top-left vertex of the first backward region D1 in the backward reference image Ref0 is (5.3, 6.2), then the vertex position of this top-left vertex does not coincide with the position of a pixel in the backward reference image Ref0, meaning there is no pixel at position (5.3, 6.2) in the backward reference image Ref0.

[0300] (7): In the backward reference image, determine the pixel closest to the vertex position of the first backward region, and use this pixel as the vertex to determine the fourth backward region. The width of the fourth backward region is blkW+2 and the height is blkH+2.

[0301] For any vertex position of the first backward region, taking the top left vertex position of the first backward region as an example, in the backward reference image, determine the pixel closest to the top left vertex position, and use this pixel as the top left vertex to determine the fourth backward region. The width of the fourth backward region is blkW+2 and the height is blkH+2.

[0302] For example, see Figure 13 The top left vertex of the first backward region D1 is located at (5.3, 6.2). In the backward reference image Ref1, the position of the pixel closest to the vertex (5.3, 6.2) is determined to be (5, 6). The fourth backward region D2 is determined with the pixel at position (5, 6) as the top left vertex. The width of the fourth backward region D2 is blkW+2 and the height is blkH+2.

[0303] (8): Determine the fifth backward region including the fourth backward region. The center of the fourth backward region and the center of the fifth backward region coincide. The width of the fifth backward region is blkW+n+1 and the height is blkH+n+1. Obtain the image block located in the fifth backward region in the backward reference image. Use the interpolation filter to interpolate and filter the image block to obtain the first backward prediction block of the first image block. The width of the first backward prediction block is blkW+2 and the height is blkH+2.

[0304] For example, see the example of an 8-tap interpolation filter. Figure 13 A fifth backward region D3 is determined, including the fourth backward region D2. The center of the fourth backward region D2 and the center of the fifth backward region D3 coincide. The width of the fifth backward region D3 is blkW+9 and the height is blkH+9. An image block located in the fifth backward region D3 is obtained in the backward reference image Ref1. An interpolation filter is used to interpolate and filter the image block to obtain the first backward prediction block of the first image block B. The width of the first backward prediction block is blkW+2 and the height is blkH+2.

[0305] The number of taps n in the interpolation filter can be 6, 8, or 10, etc.

[0306] During this step, it can also be determined whether to use BIO (Block I / O) for inter-frame prediction based on the motion information of the image patch to be processed. If it is determined that BIO will be used for inter-frame prediction, this step begins. The determination process can be as follows:

[0307] Determine if the frame number of the image block to be processed, the frame number of the forward reference image, and the frame number of the backward reference image meet the preset BIO usage conditions. If the preset BIO usage conditions are met, determine whether to use the BIO method for inter-frame prediction and begin this step. If the preset BIO usage conditions are not met, determine whether to use a method other than BIO for inter-frame prediction. The implementation process of other methods will not be described in detail here.

[0308] The default conditions for using BIO can be those shown in the first formula below.

[0309] The first formula is (POC_L0-POC_Cur)*(POC_L1-POC_Cur)<0;

[0310] In the first formula, POC_L0 is the frame number of the forward reference image, POC_Cur is the frame number of the image block to be processed, POC_L1 is the frame number of the backward reference image, and * represents a multiplication operation.

[0311] In this step, it can also be determined whether to use the BIO method for inter-frame prediction based on the first forward prediction block and the first backward prediction block of the first image block. If it is determined that the BIO method should be used for inter-frame prediction, the following step 104 is executed. This determination process can be as follows:

[0312] Based on the first forward prediction block and the first backward prediction block of the first image block, SAD is calculated using the following second formula. If SAD exceeds a preset threshold TH_CU, it is determined that the BIO method will be used for inter-frame prediction, and the operation in step 103 below begins. If SAD does not exceed the preset threshold TH_CU, it is determined that a method other than the BIO method will be used for inter-frame prediction. The implementation process of other methods is not described in detail here.

[0313] The second formula is:

[0314] In the second formula, I (1) (i,j) represents the predicted value of the pixel in the i-th row and j-th column of the first backward prediction block. (0) (i,j) represents the predicted value of the pixel in the i-th row and j-th column of the first forward prediction block.

[0315] TH_CU = (1 << (BD - 8 + shift)) * blkW * blkH, shift = Max(2, 14 - BD), BD is the current pixel width, abs() is the absolute value operation, and << is the left shift operation.

[0316] Step 104: Perform gradient calculation on the first prediction block of the first image block to obtain the first gradient value matrix of the first image block. The width of the first gradient value matrix is ​​blkW and the height is blkH.

[0317] The first gradient value matrix includes a first forward horizontal gradient value matrix, a first forward vertical gradient value matrix, a first backward horizontal gradient value matrix, and a first backward vertical gradient value matrix.

[0318] In this step, based on the predicted values ​​of each pixel in the first prediction block, the horizontal gradient value and the vertical gradient value are calculated using the following third formula; each calculated horizontal gradient value corresponds to a row number and a column number, and each calculated vertical gradient value corresponds to a row number and a column number; based on the row number and column number corresponding to each calculated horizontal gradient value, the calculated horizontal gradient values ​​are combined to form the first horizontal gradient matrix of the first image block; based on the row number and column number of each calculated vertical gradient value, the calculated vertical gradient values ​​are combined to form the first vertical gradient matrix of the first image block.

[0319] Specifically, when calculating the gradient value of a certain row or column in the gradient value matrix, the predicted values ​​of two pixels are obtained from the first prediction block according to the row number and the column number. A horizontal gradient value or a vertical gradient value is calculated according to the predicted values ​​of the two pixels using the following third formula. The horizontal gradient value corresponds to the row number and the column number respectively, or the vertical gradient value corresponds to the row number and the column number respectively.

[0320] The first prediction block includes a first forward prediction block and a first backward prediction block. Based on the first forward prediction block, the forward horizontal gradient value and the forward vertical gradient value are calculated using the following third formula. Each calculated forward horizontal gradient value corresponds to a row number and a column number, and each calculated forward vertical gradient value corresponds to a row number and a column number. Based on the row number and column number corresponding to each calculated forward horizontal gradient value, the calculated forward horizontal gradient values ​​are combined to form the first forward horizontal gradient matrix of the first image block. Based on the row number and column number of each calculated forward vertical gradient value, the calculated forward vertical gradient values ​​are combined to form the first forward vertical gradient matrix of the first image block.

[0321] Based on the first backward prediction block, the backward horizontal gradient value and the backward vertical gradient value are calculated using the following third formula; each calculated backward horizontal gradient value corresponds to a row number and a column number, and each calculated backward vertical gradient value corresponds to a row number and a column number; based on the row number and column number corresponding to each calculated backward horizontal gradient value, the calculated backward horizontal gradient values ​​are combined to form the first backward horizontal gradient matrix of the first image block; based on the row number and column number corresponding to each calculated backward vertical gradient value, the calculated backward vertical gradient values ​​are combined to form the first backward vertical gradient matrix of the first image block.

[0322] The third formula is:

[0323] In the third formula: the value of k can be 0 or 1, where 0 represents forward and 1 represents backward. Let k be the horizontal gradient value in the i-th row and j-th column. When k = 0, Let k be the forward horizontal gradient value in the i-th row and j-th column. When k = 1, Let be the backward horizontal gradient value in the i-th row and j-th column, and >> is the right shift operation.

[0324] I (k) (i+1, j) represents the predicted value of the pixel in the (i+1)th row and jth column of the first prediction block. When k = 0, I (k) (i+1, j) represents the predicted value of the pixel in the (i+1)th row and jth column of the first forward prediction block. When k = 1, I (k) (i+1, j) represents the predicted value of the pixel in the (i+1)th row and jth column of the first backward prediction block. (k)(i-1, j) represents the predicted value of the pixel in the (i-1)th row and jth column of the first prediction block. When k = 0, I (k) (i-1, j) represents the predicted value of the pixel in the (i-1)th row and jth column of the first forward prediction block. When k = 1, I (k) (i-1, j) is the predicted value of the pixel in the (i-1)th row and jth column of the first backward prediction block.

[0325] I (k) (i, j+1) represents the predicted value of the pixel in the i-th row and j+1-th column of the first prediction block. When k = 0, I (k) (i, j+1) represents the predicted value of the pixel in the i-th row and j+1-th column of the first forward prediction block. When k = 1, I (k) (i, j+1) represents the predicted value of the pixel in the i-th row and j+1-th column of the first backward prediction block. (k) (i, j-1) represents the predicted value of the pixel in the i-th row and j-1-th column of the first prediction block. When k = 0, I (k) (i, j-1) represents the predicted value of the pixel in the i-th row and j-1-th column of the first forward prediction block. When k = 1, I (k) (i, j-1) is the predicted value of the pixel in the i-th row and j-1-th column of the first backward prediction block.

[0326] It should be noted that for a first prediction block with a width of blkW+2 and a height of blkH+2, the first gradient value matrix with a width of blkW and a height of blkH can be obtained using the third formula mentioned above. This first gradient value matrix includes a first horizontal gradient value matrix with a width of blkW and a height of blkH, and a first vertical gradient value matrix with a width of blkW and a height of blkH. In other words, for a first forward prediction block with a width of blkW+2 and a height of blkH+2, the first forward horizontal gradient value matrix with a width of blkW and a height of blkH, as well as the first forward vertical gradient value matrix with a width of blkW and a height of blkH, can be obtained using the third formula mentioned above. Similarly, for a first backward prediction block with a width of blkW+2 and a height of blkH+2, the first backward horizontal gradient value matrix with a width of blkW and a height of blkH, as well as the first backward vertical gradient value matrix with a width of blkW and a height of blkH, can be obtained using the third formula mentioned above.

[0327] Step 105: Based on the gradient values ​​at the edge positions of the first gradient value matrix, perform a first expansion on the width and height of the first gradient value matrix so that the width and height of the first expanded first gradient value matrix are respectively 2 pixels larger than the width and height of the first image block.

[0328] The width and height of the first expanded gradient value matrix are equal to the width and height of the first prediction block, respectively. The width of the first prediction block is blkW+2 and the height is blkH+2, and the width and height of the first gradient value matrix are also blkW+2 and blkH+2, respectively.

[0329] In this step, the width and height of the first forward horizontal gradient value matrix, the first forward vertical gradient value matrix, the first backward horizontal gradient value matrix, and the first backward vertical gradient value matrix are expanded so that the width of the first forward horizontal gradient value matrix, the first forward vertical gradient value matrix, the first backward horizontal gradient value matrix, and the first backward vertical gradient value matrix are all blkkW+2 and the height is all blkH+2.

[0330] In this step, the first gradient value matrix includes four edges. For the gradient value at the left edge of the first gradient value matrix, a column of gradient values ​​is extended to the left of the first gradient value matrix based on the gradient value at the left edge. For the gradient value at the right edge of the first gradient value matrix, a column of gradient values ​​is extended to the right of the first gradient value matrix based on the gradient value at the right edge. For the gradient value at the top edge of the first gradient value matrix, a row of gradient values ​​is extended to the top of the first gradient value matrix based on the gradient value at the top of the first gradient value matrix. For the gradient value at the bottom edge of the first gradient value matrix, a row of gradient values ​​is extended to the bottom of the first gradient value matrix based on the gradient value at the bottom of the first gradient value matrix. This results in the width and height of the first extended first gradient value matrix being 2 pixels larger than the width and height of the first image patch, respectively.

[0331] Step 106: Calculate the motion information correction amount for each basic processing unit in the first image block based on the first prediction block and the first gradient value matrix.

[0332] The width and height of a basic processing unit can both be M, meaning a basic processing unit is an image block consisting of M*M pixels. The value of M can be 2, 3, or 4, etc.

[0333] The motion information correction amount of the basic processing unit includes horizontal motion information correction amount and vertical motion information correction amount.

[0334] This step can be achieved through operations 1061 to 1064, which can be as follows:

[0335] 1061: Divide the first image block to obtain each basic processing unit included in the first image block, and each basic processing unit is an image block of size M*M.

[0336] 1062: For any basic processing unit, based on the position of the basic processing unit, determine the basic prediction block of the basic processing unit from the first prediction block. The width of the basic prediction block is M+2 and the height is M+2.

[0337] Assuming the basic processing unit is located in rows 1 to M and columns 1 to M in the first image block, the image block in rows 0 to M+1 and columns 0 to M+1 in the first prediction block is used as the basic prediction block of the basic processing unit.

[0338] The basic prediction block of the basic processing unit includes a forward basic prediction block and a backward basic prediction block. That is, in the first forward prediction block, the image blocks in rows 0 to M+1 and columns 0 to M+1 are used as the forward basic prediction blocks of the basic processing unit, and in the first backward prediction block, the image blocks in rows 0 to M+1 and columns 0 to M+1 are used as the backward basic prediction blocks of the basic processing unit.

[0339] 1063: Based on the position of the basic processing unit, determine the basic gradient value matrix of the basic processing unit from the first gradient value matrix. The width of the basic gradient value matrix is ​​M+2 and the height is M+2.

[0340] Assuming the basic processing unit is located in rows 1 to M and columns 1 to M in the first image block, the matrix in rows 0 to M+1 and columns 0 to M+1 in the first gradient value matrix is ​​used as the basic gradient value matrix of the basic processing unit.

[0341] The basic gradient value matrix of the basic processing unit includes a forward horizontal basic gradient value matrix, a forward vertical basic gradient value matrix, a backward horizontal basic gradient value matrix, and a backward vertical basic gradient value matrix. Specifically, the matrix in rows 0 to M+1 and columns 0 to M+1 of the first forward horizontal gradient value matrix is ​​used as the forward horizontal basic gradient value matrix of the basic processing unit; the matrix in rows 0 to M+1 and columns 0 to M+1 of the first forward vertical gradient value matrix is ​​used as the forward vertical basic gradient value matrix of the basic processing unit; the matrix in rows 0 to M+1 and columns 0 to M+1 of the first backward horizontal gradient value matrix is ​​used as the backward horizontal basic gradient value matrix of the basic processing unit; and the matrix in rows 0 to M+1 and columns 0 to M+1 of the first backward vertical gradient value matrix is ​​used as the backward vertical basic gradient value matrix of the basic processing unit.

[0342] 1064: Calculate the motion information correction amount of the basic processing unit based on the basic prediction block and basic gradient value matrix of the basic processing unit.

[0343] In 1064, based on the forward basic prediction block, backward basic prediction block, forward horizontal basic gradient value matrix, forward vertical horizontal basic gradient value matrix, backward horizontal basic gradient value matrix, and backward vertical basic gradient value matrix of the basic processing unit, the horizontal motion information correction amount and the vertical motion information correction amount of the basic processing unit are calculated by the following fourth and fifth formulas.

[0344] The fourth formula is:

[0345] The fifth formula is:

[0346]

[0347] In the fourth formula above, (i,j)∈Ω represents i=0,1…………M+1, j=0,1……M+1. In the fifth formula above, v x v is the horizontal motion information correction amount for this basic processing unit. y This is the correction amount for the vertical motion information of this basic processing unit. BIO =2 13-BD , The floor function (floor(A) represents the largest integer not greater than A); S 2,m =S2>>12, S 2,s =S2&(2 12 -1).

[0348] By repeating the operations 1062 to 1064 above, the motion information correction amount of each basic processing unit included in the first image block can be obtained.

[0349] Step 107: Obtain the predicted value of the first image block based on the motion information correction amount of each basic processing unit included in the first image block.

[0350] The predicted value of the first image block includes the predicted value of each pixel in each basic processing unit of the first image block.

[0351] For any basic processing unit included in the first image block, the predicted value of each pixel included in the basic processing unit is calculated using the following sixth formula based on the forward basic prediction block, backward basic prediction block, forward horizontal basic gradient value matrix, forward vertical basic gradient value matrix, backward horizontal basic gradient value matrix, and backward vertical basic gradient value matrix of the basic processing unit.

[0352] The sixth formula is:

[0353]

[0354] In the sixth formula, pred BIO(i,j) represents the predicted value of the pixel in the i-th row and j-th column of this basic processing unit. shift = 15 - BD, o offset = 1 << (14-BD) + 2 * (1 << 13). rnd() is for rounding.

[0355] Repeat steps 103 to 107 above to obtain the predicted value of each first image block in the image block to be processed.

[0356] Step 108: Combine the predicted values ​​of multiple first image blocks included in the image block to be processed to obtain the predicted value of the image block to be processed.

[0357] Regarding the above Figure 12 The inter-frame prediction method shown can be summarized as steps 1 to 6 below. These steps 1 to 6 can be respectively:

[0358] Step 1: Determine the motion information of the current CU

[0359] The motion information of the current CU can be determined by Merge mode, AMVP mode (as described in the background art), or other modes, which are not limited here.

[0360] It should be noted that other methods for determining motion information can also be applied to this invention, which will not be elaborated here.

[0361] Step 2: Determine if the current CU meets the conditions for using BIO.

[0362] If the current CU uses bidirectional prediction, and the relationship between the forward reference frame number POC_L0, the backward reference frame number POC_L1, and the current frame number POC_Cur satisfies the following formula, then the current CU meets the conditions for using BIO:

[0363] (POC_L0–POC_Cur)*(POC_L1–POC_Cur)<0

[0364] It should be noted that BIO can also be determined by checking if the current CU's size is greater than a preset threshold. For example, BIO can only be used if the current CU's height W>=8 and width H>=8.

[0365] It should be noted that other BIO usage conditions can also be applied to this invention, which will not be elaborated here.

[0366] If the current CU meets the conditions for using BIO, proceed to step 3; otherwise, perform motion compensation in other ways.

[0367] Obtain the dimensions of the VPDU, VPDU_X and VPDU_Y, and set the parameters blkW and blkH.

[0368] blkW=Min(cuW,VPDU_X)

[0369] blkH = Min(cuH, VPDU_Y)

[0370] The Min function represents taking the minimum value.

[0371] For example, if the CU size is 128x128 and the VPDU size is 64x64, then blkW is 64 and blkH is 64.

[0372] For example, if the CU size is 128x128 and the VPDU size is 128x32, then blkW is 128 and blkH is 32.

[0373] For example, if the CU size is 128x128 and the VPDU size is 32x128, then blkW is 32 and blkH is 128.

[0374] Optionally, if the size of the maximum inter-frame prediction processing unit is smaller than the VPDU size, blkW and blkH can be set according to the following formula.

[0375] blkW=Min(cuW,MAX_MC_X)

[0376] blkH = Min(cuH, MAX_MC_Y)

[0377] For example, if the CU size is 128x128 and the size of the largest inter-frame prediction processing unit is 32x32, then blkW is 32 and blkH is 32.

[0378] For each CU, partition it according to blkW and blkH, and perform BIO:

[0379] Step 3: Calculate the forward and backward prediction values ​​of the current CU.

[0380] Using motion information, motion compensation is performed to obtain forward and backward predicted values ​​I. (k) (i,j), where i=-1..blkW,j=-1..blkH (then we get the prediction matrix of (blkW+2)*(blkH+2)).

[0381] Among them, I (k) In (i,j), i = 0..blkW-1, j = 0..blkH-1, the values ​​are obtained by interpolation using an 8-tap interpolation filter, and the predicted values ​​at other positions (extended by 1 row and 1 column) are obtained by interpolation using a bilinear interpolation filter.

[0382] It should be understood that the predicted value can be obtained by using the size of the VPDU as the smallest unit, or by using a block size smaller than the VPDU as the smallest unit; there is no limitation on this.

[0383] It should be noted that the predicted value of the extended region can also be obtained by other methods, such as using the same 8-tap interpolation filter, or directly using the reference pixel at the integer pixel position, which is not limited here.

[0384] It should be noted that, when calculating the SAD between the forward and backward predicted values, it is determined whether the SAD is less than the threshold TH_CU. If it is less, BIO is not executed. Otherwise, BIO is executed. Other determination methods can also be applied to this invention, which will not be elaborated here.

[0385] The formula for calculating SAD is as follows:

[0386]

[0387] The threshold TH_CU can be set to (1<<(BD–8+shift))*blkW*blkH, where shift is Max(2,14-BD).

[0388] Step 4: Calculate the horizontal and vertical gradient values ​​of the current CU's forward and backward predictions.

[0389] The horizontal and vertical gradient values ​​are calculated according to formula (4) based on the forward and backward predicted values. and Where i = 0..blkW-1, j = 0..blkH-1 (then the gradient matrix of blkW*blkH is obtained).

[0390] Step 5: Padding the forward and backward predicted values ​​of the current CU, as well as the gradient values ​​in the horizontal and vertical directions.

[0391] like Figure 9 Padding is performed using the method to obtain I (k) (i,j) and Where i = -1..blkW, j = -1..blkH (then we obtain the prediction matrix, horizontal gradient matrix, and vertical gradient matrix of (blkW+2)*(blkH+2)).

[0392] Step 6: Derive the corrected motion vector for each 4x4 sub-block, and then perform weighted summation.

[0393] For each 4x4 sub-block, vx and vy are obtained according to formula (2), and finally weighted according to formula (6) to obtain the predicted value of each 4x4 sub-block.

[0394] It should be noted that the SAD between the forward and backward prediction values ​​of each 4x4 sub-block can be calculated to determine whether it is less than the threshold TH_SCU. If it is less, a weighted average is directly performed; otherwise, vx and vy are obtained according to formula (2), and then weighted according to formula (6). Other judgment methods can also be applied to this invention, which will not be elaborated here. TU_SCU can be set to 1<<(BD–3+shift).

[0395] In this embodiment, since the smaller of the preset image segmentation width Width and the width cuW of the image block to be processed is selected, denoted as blkW, and the smaller of the preset image segmentation height High and the height cuH of the image block to be processed is selected, denoted as blkH, the first image block included in the image block to be processed is determined based on blkW and blkH. This avoids the large area of ​​each determined first image block. When performing inter-frame prediction processing on each first image block, it reduces the consumption of a large amount of hardware resources such as memory space, reduces the implementation complexity, and improves the processing efficiency of inter-frame prediction.

[0396] See Figure 14 This application provides an inter-frame prediction method, which can be applied to... Figure 2 The inter-frame prediction unit 244 in the encoder shown may be applied to Figure 3 In the inter-frame prediction unit 344 of the decoder shown, the method can be a bidirectional inter-frame prediction method, including:

[0397] Steps 201-202 are the same as steps 101-102, and will not be described in detail here.

[0398] Step 203: Obtain the first prediction block of the first image block based on the motion information of the image block to be processed. The width of the first prediction block is equal to the width of the first image block, and the height of the first prediction block is equal to the height of the first image block.

[0399] The motion information of the first image block includes information such as a reference image and motion vectors. The inter-frame prediction in this embodiment adopts a bidirectional prediction method based on optical flow, so the motion information of the first image block includes information such as a forward reference image, a backward reference image, a forward motion vector, and a backward motion vector.

[0400] When the method of this embodiment is applied to an encoder, the encoder can determine the motion information of the image block to be processed using Merge mode, AMVP mode, or other modes. The motion information of the image block to be processed includes the motion information of each first image block in the image block to be processed. In this step, the motion information of the image block to be processed determined by the encoder is obtained, and the motion information of the first image block is obtained from the motion information of the image block to be processed.

[0401] In this embodiment, when the method is applied to a decoder, the decoder receives motion information of image blocks to be processed from a video stream from the encoder. The motion information of the image blocks to be processed includes motion information of each first image block within the image blocks to be processed. In this step, the motion information of the image blocks to be processed is extracted from the video stream, and the motion information of the first image blocks is obtained from the motion information of the image blocks to be processed.

[0402] The first prediction block of the first image block includes a first forward prediction block and a first backward prediction block. In this step, the first forward prediction block and the first backward prediction block of the first image block can be obtained through the following steps (1) to (8). Steps (1) to (8) can be as follows:

[0403] (1): Based on the first position and motion information of the first image block, a first forward region is determined in the forward reference image. The width of the first forward region is equal to blkW and the height is equal to blkH.

[0404] For example, see Figure 15 The motion information of the first image block B includes a forward reference image Ref0, a backward reference image Ref1, a forward motion vector MV0, and a backward motion vector MV1. Based on the first position of the first image block B, a second forward region B11 is determined in the forward reference image Ref0. The width of the second forward region B11 is blkW, and the height is blkH. Based on the positions of the forward motion vector MV0 and the second forward region B11, a first forward region B12 is determined. The width of the first forward region B12 is blkW, and the height is blkH.

[0405] (2): In the forward reference image, determine whether the vertex position of the first forward region coincides with the pixel position in the forward reference image. If it coincides with the pixel position in the forward reference image, obtain the image block located in the first forward region from the forward reference image as the first forward prediction block of the first image block; if it does not coincide with the pixel position in the forward reference image, execute step (3).

[0406] For example, see Figure 15Taking the top-left vertex of the first forward region B12 as an example, assuming that the vertex position of the top-left vertex of the first forward region B12 in the forward reference image Ref0 is (15, 16), then the vertex position of this top-left vertex coincides with the position of a pixel in the forward reference image Ref0, which is the pixel at position (15, 16) in the forward reference image Ref0. As another example, assuming that the vertex position of the top-left vertex of the first forward region B12 in the forward reference image Ref0 is (15.3, 16.2), then the vertex position of this top-left vertex does not coincide with the position of a pixel in the forward reference image Ref0, meaning that there is no pixel at position (15.3, 16.2) in the forward reference image Ref0.

[0407] (3): In the forward reference image, determine the pixel closest to the vertex position of the first forward region, and use this pixel as the vertex to determine the third forward region. The width of the third forward region is blkW and the height is blkH.

[0408] For any vertex position of the first forward region, assuming the top-left vertex position of the first forward region is used, determine the pixel closest to the top-left vertex position in the forward reference image, and use this pixel as the top-left vertex to determine the third forward region. The width of the third forward region is blkW and the height is blkH.

[0409] For example, see Figure 15 The top left vertex of the first forward region B12 is located at (15.3, 16.2). In the forward reference image Ref0, the position of the pixel closest to the vertex (15.3, 16.2) is determined as (15, 16). The third forward region A1 is determined with the pixel at position (15, 16) as the top left vertex. The width of the third forward region A1 is blkW and the height is blkH.

[0410] (4): Determine the fourth forward region including the third forward region. The center of the third forward region and the center of the fourth forward region coincide. The width of the fourth forward region is blkW+n-1 and the height is blkH+n-1. Obtain the image block located in the fourth forward region in the forward reference image. Use the interpolation filter to interpolate and filter the image block to obtain the first forward prediction block of the first image block. The width of the first forward prediction block is blkW and the height is blkH. n is the number of taps of the interpolation filter.

[0411] For example, see the example of an 8-tap interpolation filter. Figure 15A fourth forward region A2 is determined, including the third forward region A1. The center of the third forward region A1 and the center of the fourth forward region A2 coincide. The width of the fourth forward region A2 is blkW+7 and the height is blkH+7. An image block located in the fourth forward region A2 is obtained in the forward reference image Ref0. An interpolation filter is used to interpolate and filter the image block to obtain the first forward prediction block of the first image block B. The width of the first forward prediction block is blkW and the height is blkH.

[0412] (5): Based on the first position and motion information of the first image block, a first backward region is determined in the backward reference image. The width of the first backward region is blkW and the height is blkH.

[0413] For example, see Figure 15 Based on the first position of the first image block B, a second backward region C11 is determined in the backward reference image Ref1. The width of the second backward region C11 is blkW, and the height is blkH. Based on the positions of the backward motion vector MV1 and the second backward region C12, a first backward region C12 is determined. The width of the first backward region C12 is blkW, and the height is blkH.

[0414] (6): In the backward reference image, determine whether the vertex position of the first backward region coincides with the pixel position in the backward reference image. If it coincides with the pixel position in the backward reference image, obtain the image block located in the first backward region from the backward reference image as the first backward prediction block of the first image block; if it does not coincide with the pixel position in the backward reference image, execute step (7).

[0415] For example, see Figure 15 Taking the top-left vertex of the first backward region C12 as an example, assuming that the vertex position of the top-left vertex of the first backward region C12 in the backward reference image Ref1 is (5, 6), then the vertex position of this top-left vertex coincides with the position of a pixel in the backward reference image Ref1, which is the pixel at position (5, 6) in the backward reference image Ref1. As another example, assuming that the vertex position of the top-left vertex of the first backward region C12 in the backward reference image Ref1 is (5.3, 6.2), then the vertex position of this top-left vertex does not coincide with the position of a pixel in the backward reference image Ref0, meaning that there is no pixel at position (5.3, 6.2) in the backward reference image Ref0.

[0416] (7): In the backward reference image, determine the pixel closest to the vertex position of the first backward region, and use this pixel as the vertex to determine the third backward region. The width of the third backward region is blkW and the height is blkH.

[0417] For any vertex position of the first backward region, assuming that the top-left vertex position of the first backward region is used, the pixel closest to the top-left vertex position in the backward reference image is determined, and the third backward region is determined with this pixel as the top-left vertex. The width of the third backward region is blkW and the height is blkH.

[0418] For example, see Figure 15 The top left vertex of the first backward region C12 is located at (5.3, 6.2). In the backward reference image Ref1, the position of the pixel closest to the vertex (5.3, 6.2) is determined to be (5, 6). The third backward region D1 is determined with the pixel at position (5, 6) as the top left vertex. The width of the third backward region D1 is blkW and the height is blkH.

[0419] (8): Determine a fourth backward region including the third backward region. The center of the third backward region and the center of the fourth backward region coincide. The width of the fourth backward region is blkW+n-1 and the height is blkH+n-1. Obtain the image block located in the fourth backward region in the backward reference image. Use an interpolation filter to interpolate and filter the image block to obtain the first backward prediction block of the first image block. The width of the first backward prediction block is blkW and the height is blkH.

[0420] For example, see the example of an 8-tap interpolation filter. Figure 15 A fourth backward region D2 is determined, including the third backward region D1. The center of the third backward region D1 and the center of the fourth backward region D2 coincide. The width of the fourth backward region D2 is blkW+7 and the height is blkH+7. An image block located in the fourth backward region D2 is obtained in the backward reference image Ref1. An interpolation filter is used to interpolate and filter the image block to obtain the first backward prediction block of the first image block B. The width of the first backward prediction block is blkW and the height is blkH.

[0421] During this step, it can also be determined whether to use BIO (Block I / O) for inter-frame prediction based on the motion information of the image patch to be processed. If it is determined that BIO will be used for inter-frame prediction, this step is initiated. The determination process can be found in [link to relevant documentation]. Figure 12 The details of step 103 in the illustrated embodiment will not be described in detail here.

[0422] In this step, it can also be determined whether to use the BIO method for inter-frame prediction based on the first forward prediction block and the first backward prediction block of the first image block. If it is determined that the BIO method should be used for inter-frame prediction, step 204 is executed. This determination process can be found in [reference needed]. Figure 12 The details of step 103 in the illustrated embodiment will not be described in detail here.

[0423] Step 204: Perform gradient operation on the first prediction block of the first image block to obtain the first gradient value matrix of the first image block. The width of the first gradient value matrix is ​​blkW-2 and the height of the first gradient value matrix is ​​blkH-2.

[0424] The first gradient value matrix includes a first forward horizontal gradient value matrix, a first forward vertical gradient value matrix, a first backward horizontal gradient value matrix, and a first backward vertical gradient value matrix.

[0425] The widths of the first forward horizontal gradient matrix, the first forward vertical gradient matrix, the first backward horizontal gradient matrix, and the first backward vertical gradient matrix can all be blkW-2, and the heights of the first forward horizontal gradient matrix, the first forward vertical gradient matrix, the first backward horizontal gradient matrix, and the first backward vertical gradient matrix can all be blkH-2.

[0426] In this step, the detailed implementation process of performing gradient calculation on the first prediction block of the first image patch can be found in [link to relevant documentation]. Figure 12 The details of step 104 in the illustrated embodiment will not be described in detail here.

[0427] Step 205: Based on the gradient values ​​at the edge positions of the first gradient value matrix, perform a first expansion on the width and height of the first gradient value matrix so that the width and height of the first expanded first gradient value matrix are respectively 2 pixels larger than the width and height of the first image block.

[0428] The width and height of the first expanded gradient value matrix are respectively equal to the width blkW+2 and the height blkH+2 of the first prediction block.

[0429] In this step, the width and height of the first forward horizontal gradient value matrix, the first forward vertical gradient value matrix, the first backward horizontal gradient value matrix, and the first backward vertical gradient value matrix are expanded so that the width of the first forward horizontal gradient value matrix, the first forward vertical gradient value matrix, the first backward horizontal gradient value matrix, and the first backward vertical gradient value matrix are all blkkW+2 and the height is all blkH+2.

[0430] The method for performing the first expansion on the first gradient value matrix can be found in [reference needed]. Figure 12 The details of step 205 in the illustrated embodiment will not be described in detail here.

[0431] Step 206: Copy the pixel values ​​at the block edge positions of the first prediction block to perform a second expansion of the width and height of the first prediction block. The width of the first prediction block after the second expansion is blkW+2 and the height is blkH+2.

[0432] In this step, the pixel values ​​at the block edge positions of the first forward prediction block and the first backward prediction block are copied to perform a second expansion of the width and height of the first forward prediction block and the first backward prediction block. Specifically, in this step, the width and height of the second-expanded first forward prediction block are blkW+2 and blkH+2, respectively, and the width and height of the second-expanded first backward prediction block are blkW+2 and blkH+2, respectively.

[0433] In this step, the pixel values ​​of the block edge region of the first prediction block can also be interpolated and filtered to further expand the width and height of the first prediction block.

[0434] Optionally, in step 203, if the image block with width blkW and height blkH in the reference image is directly used as the first predicted block of the first image block, i.e., see [link to relevant documentation] Figure 15 In the forward reference image Ref0, the image block in the first forward region B12 is used as the first forward prediction block, and in the backward reference image Ref1, the image block in the first backward region C12 is used as the first backward prediction block. In this case, the first prediction block is an image block in the reference image. Then, a ring of pixels surrounding the first prediction block and closest to it is selected in the reference image. The selected ring of pixels and the first prediction block are combined to form a second extended first prediction block with a width of blkW+2 and a height of blkH+2.

[0435] Optionally, in step 203, a first predicted block of the first image block is obtained through an interpolation filter. In this case, the first predicted block is not an image block in the reference image. For any pixel on any edge of the first predicted block, for ease of explanation, this edge is referred to as the first edge. Based on the first position of each pixel on the first edge in the reference image, the second position of each pixel included in the second edge is obtained. The second edge is located outside the first predicted block, and the second edge differs from the first edge by one pixel. The second edge includes blkW+2 pixels or blkH+2 pixels. For each pixel on the second edge, in the reference image, the second position of the pixel is located between two adjacent pixels or between four adjacent pixels. An interpolation filter is used to interpolate the two adjacent pixels or the four adjacent pixels to obtain the pixel. The second edge corresponding to each edge of the first predicted block is obtained in the above manner. Each obtained second edge and the first predicted block are combined to form a second expanded first predicted block with a width of blkW+2 and a height of blkH+2.

[0436] Step 206 can also be performed before step 204. This allows gradient value calculations to be performed on the second expanded first prediction block when the second expanded first prediction block is obtained, resulting in the first gradient value matrix of the first image block. Since the width of the second expanded first prediction block is blkW+2 and the height is blkH+2, the width of the first gradient value matrix is ​​blkW and the height is blkH. Then, based on the gradient values ​​at the edge positions of the first gradient value matrix, the width and height of the first gradient value matrix are first expanded so that the width and height of the first expanded first gradient value matrix are respectively 2 pixels larger than the width and height of the first image block.

[0437] Steps 207-209 are the same as steps 106-108, and will not be described in detail here.

[0438] Regarding the above Figure 16 The inter-frame prediction method shown can be summarized as steps 1 to 6 below. These steps 1 to 6 can be respectively:

[0439] Step 1: Determine the motion information of the current CU

[0440] The motion information of the current CU can be determined by Merge mode, AMVP mode (as described in the background art), or other modes, which are not limited here.

[0441] It should be noted that other methods for determining motion information can also be applied to this invention, which will not be elaborated here.

[0442] Step 2: Determine if the current CU meets the conditions for using BIO.

[0443] If the current CU uses bidirectional prediction, and the relationship between the forward reference frame number POC_L0, the backward reference frame number POC_L1, and the current frame number POC_Cur satisfies the following formula, then the current CU meets the conditions for using BIO:

[0444] (POC_L0–POC_Cur)*(POC_L1–POC_Cur)<0

[0445] It should be noted that BIO can also be determined by checking if the current CU's size is greater than a preset threshold. For example, BIO can only be used if the current CU's height W>=8 and width H>=8.

[0446] It should be noted that other BIO usage conditions can also be applied to this invention, which will not be elaborated here.

[0447] If the current CU meets the conditions for using BIO, proceed to step 3; otherwise, perform motion compensation in other ways.

[0448] Obtain the dimensions of the VPDU, VPDU_X and VPDU_Y, and set the parameters blkW and blkH.

[0449] blkW=Min(cuW,VPDU_X)

[0450] blkH = Min(cuH, VPDU_Y)

[0451] For example, if the CU size is 128x128 and the VPDU size is 64x64, then blkW is 64 and blkH is 64.

[0452] For example, if the CU size is 128x128 and the VPDU size is 128x32, then blkW is 128 and blkH is 32.

[0453] For example, if the CU size is 128x128 and the VPDU size is 32x128, then blkW is 32 and blkH is 128.

[0454] Optionally, if the size of the maximum inter-frame prediction processing unit is smaller than the VPDU size, blkW and blkH can be set according to the following formula.

[0455] blkW=Min(cuW,MAX_MC_X)

[0456] blkH = Min(cuH, MAX_MC_Y)

[0457] For example, if the CU size is 128x128 and the size of the largest inter-frame prediction processing unit is 32x32, then blkW is 32 and blkH is 32.

[0458] For each CU, partition it according to blkW and blkH, and perform BIO:

[0459] Step 3: Calculate the forward and backward prediction values ​​of the current CU.

[0460] Using motion information, motion compensation is performed to obtain forward and backward predicted values ​​I. (k) (i,j), where i=0..blkW-1,j=0..blkH-1 (then the prediction matrix blkW*blkH is obtained).

[0461] It should be understood that the predicted value can be obtained by using the size of the VPDU as the smallest unit, or by using a block size smaller than the VPDU as the smallest unit; there is no limitation on this.

[0462] Step 4: Calculate the horizontal and vertical gradient values ​​of the current CU's forward and backward predictions.

[0463] The horizontal and vertical gradient values ​​are calculated according to formula (4) based on the forward and backward predicted values. and Where i = 1..blkW-2, j = 1..blkH-2 (then we get the gradient matrix of (blkW-2)*(blkH-2)).

[0464] Step 5: Padding the forward and backward predicted values ​​of the current CU, as well as the gradient values ​​in the horizontal and vertical directions.

[0465] like Figure 9 Padding is performed using the method to obtain I (k) (i,j) and Where i = -1..blkW, j = -1..blkH (padding the (blkW-2)*(blkH-2) matrix 2 rows and 2 columns to obtain the prediction matrix, horizontal gradient matrix, and vertical gradient matrix of (blkW+2)*(blkH+2)).

[0466] Step 6: Derive the corrected motion vector for each 4x4 sub-block, and then perform weighted summation.

[0467] For each 4x4 sub-block, vx and vy are obtained according to formula (2), and finally weighted according to formula (6) to obtain the predicted value of each 4x4 sub-block.

[0468] In this embodiment, the smaller width (denoted as blkW) is selected from the preset image segmentation width Width and the width cuW of the image block to be processed, and the smaller height (denoted as blkH) is selected from the preset image segmentation height High and the height cuH of the image block to be processed, and the first image block included in the image block to be processed is determined based on blkW and blkH, thus avoiding the large area of ​​each determined first image block. This reduces the consumption of a large amount of memory space when performing inter-frame prediction processing on each first image block. Furthermore, the first prediction block of the first image block is obtained based on the motion information of the first image block. Since the width and height of the first prediction block are equal to the width and height of the first image block, the first prediction block can be smaller. This reduces the CPU and memory resources consumed in obtaining the first prediction block, lowers the implementation complexity, and improves processing efficiency.

[0469] See Figure 16 This application provides an inter-frame prediction method, which can be applied to... Figure 2 The inter-frame prediction unit 244 in the encoder shown may be applied to Figure 3 In the inter-frame prediction unit 344 of the decoder shown, the method can be a bidirectional inter-frame prediction method, including:

[0470] Step 301: Compare the width cuW of the first image block with the preset image segmentation width Width, and compare the height cuH of the first image block with the preset image segmentation height High. If cuW is greater than or equal to Width and / or cuH is greater than or equal to High, proceed to step 302. If cuW is less than Width and cuH is less than High, proceed to step 305.

[0471] In this embodiment, when the method is applied to an encoder, the encoder divides the image into first image blocks when encoding the image. Prior to this step, the first image blocks are obtained from the encoder.

[0472] In this embodiment, when the method is applied to a decoder, the decoder receives a video stream from an encoder, the video stream including a first image block. Prior to this step, the first image block is extracted from the video stream.

[0473] During this step, the motion information of the first image block can be used to determine whether to use the BIO method for inter-frame prediction. If it is determined that the BIO method will be used for inter-frame prediction, this step begins. For detailed implementation details, please refer to [link to relevant documentation]. Figure 12 The details of step 103 in the illustrated embodiment will not be described in detail here.

[0474] Step 302: Obtain the second prediction block of the first image block based on the motion information of the first image block. The width of the second prediction block is cuW+4 and the height is cuH+4.

[0475] The motion information of the first image block includes information such as a reference image and motion vectors. The inter-frame prediction in this embodiment adopts a bidirectional prediction method based on optical flow, so the motion information of the first image block includes information such as a forward reference image, a backward reference image, a forward motion vector, and a backward motion vector.

[0476] When the method of this embodiment is applied to an encoder, the encoder can use Merge mode, AMVP mode, or other modes to determine the motion information of the first image block. In this step, the motion information of the first image block determined by the encoder is obtained.

[0477] In this embodiment, when the method is applied to a decoder, the decoder receives motion information from the video stream of the encoder, including a first image block. In this step, the motion information of the first image block is extracted from the video stream.

[0478] The second prediction block of the first image block includes a second forward prediction block and a second backward prediction block. In this step, the second forward prediction block and the second backward prediction block of the first image block can be obtained through the following steps (1) to (8). Steps (1) to (8) can be as follows:

[0479] (1): Based on the first position and motion information of the first image block in the image block to be processed, a first forward region is determined in the forward reference image. The width of the first forward region is blkW+4 and the height is blkH+4.

[0480] For example, see Figure 13 The motion information of the first image block B includes a forward reference image Ref0, a backward reference image Ref1, a forward motion vector MV0, and a backward motion vector MV1. Based on the first position of the first image block B, a second forward region B11 is determined in the forward reference image Ref0. The width of the second forward region B11 is blkW, and the height is blkH. Based on the positions of the forward motion vector MV0 and the second forward region B11, a third forward region B12 is determined. The width of the third forward region B12 is blkW, and the height is blkH. A first forward region A1, including the third forward region B12, is determined. The width of the first forward region A1 is blkW+4, and the height is blkH+4. The center of the third forward region B12 coincides with the center of the first forward region A1.

[0481] (2): In the forward reference image, determine whether the vertex position of the first forward region coincides with the pixel position in the forward reference image. If it coincides with the pixel position in the forward reference image, obtain the image block located in the first forward region from the forward reference image as the second forward prediction block of the first image block; if it does not coincide with the pixel position in the forward reference image, execute step (3).

[0482] For example, see Figure 13 Taking the top-left vertex of the first forward region A1 as an example, assuming the vertex position of the top-left vertex of the first forward region A1 in the forward reference image Ref0 is (15, 16), then the vertex position of this top-left vertex coincides with the position of a pixel in the forward reference image Ref0, which is the pixel at position (15, 16) in the forward reference image Ref0. As another example, assuming the vertex position of the top-left vertex of the first forward region A1 in the forward reference image Ref0 is (15.3, 16.2), then the vertex position of this top-left vertex does not coincide with the position of a pixel in the forward reference image Ref0, meaning there is no pixel at position (15.3, 16.2) in the forward reference image Ref0.

[0483] (3): In the forward reference image, determine the pixel closest to the vertex position of the first forward region, and use this pixel as the vertex to determine the fourth forward region. The width of the fourth forward region is blkW+4 and the height is blkH+4.

[0484] For any vertex position of the first forward region, assuming the top-left vertex position of the first forward region is used, determine the pixel closest to the top-left vertex position in the forward reference image, and use this pixel as the top-left vertex to determine the fourth forward region. The width of the fourth forward region is blkW+4 and the height is blkH+4.

[0485] For example, see Figure 13 The top left vertex of the first forward region A1 is located at (15.3, 16.2). In the forward reference image Ref0, the position of the pixel closest to the vertex (15.3, 16.2) is determined to be (15, 16). The fourth forward region A2 is determined with the pixel at position (15, 16) as the top left vertex. The width of the fourth forward region A2 is blkW+4 and the height is blkH+4.

[0486] (4): Determine the fifth forward region, which includes the fourth forward region. The center of the fourth forward region and the center of the fifth forward region coincide. The width of the fifth forward region is blkW+n+3 and the height is blkH+n+3. Obtain the image block located in the fifth forward region in the forward reference image. Use the interpolation filter to interpolate and filter the image block to obtain the second forward prediction block of the first image block. The width of the second forward prediction block is blkW+4 and the height is blkH+4. n is the number of taps of the interpolation filter.

[0487] For example, see the example of an 8-tap interpolation filter. Figure 13 A fifth forward region A3 is determined, including the fourth forward region A2. The center of the fourth forward region A2 and the center of the fifth forward region A3 coincide. The width of the fifth forward region A3 is blkW+11 and the height is blkH+11. An image block located in the fifth forward region A3 is obtained in the forward reference image Ref0. An interpolation filter is used to interpolate and filter the image block to obtain the second forward prediction block of the first image block B. The width of the second forward prediction block is blkW+4 and the height is blkH+4.

[0488] (5): Based on the first position and motion information of the first image block, a first backward region is determined in the backward reference image. The width of the first backward region is blkW+4 and the height is blkH+4.

[0489] For example, see Figure 13Based on the first position of the first image block B, a second backward region C11 is determined in the backward reference image Ref1. The width of the second backward region C11 is blkW, and the height is blkH. Based on the backward motion vector MV1 and the position of the second backward region C11, a third backward region C12 is determined. The width of the third backward region C12 is blkW, and the height is blkH. A first backward region D1, including the third backward region C12, is determined. The width of the first backward region D1 is blkW+4, and the height is blkH+4. The center of the third backward region C12 and the center of the first backward region D1 can coincide.

[0490] (6): In the backward reference image, determine whether the vertex position of the first backward region coincides with the pixel position in the backward reference image. If it coincides with the pixel position in the backward reference image, obtain the image block located in the first backward region from the backward reference image as the second backward prediction block of the first image block; if it does not coincide with the pixel position in the backward reference image, execute step (7).

[0491] For example, see Figure 13 Taking the top-left vertex of the first backward region A1 as an example, assuming that the vertex position of the top-left vertex of the first backward region A1 in the backward reference image Ref0 is (5, 6), then the vertex position of this top-left vertex coincides with the position of a pixel in the backward reference image Ref0, which is the pixel at position (5, 6) in the backward reference image Ref0. As another example, assuming that the vertex position of the top-left vertex of the first backward region D1 in the backward reference image Ref0 is (5.3, 6.2), then the vertex position of this top-left vertex does not coincide with the position of a pixel in the backward reference image Ref0, meaning that there is no pixel at position (5.3, 6.2) in the backward reference image Ref0.

[0492] (7): In the backward reference image, determine the pixel closest to the vertex position of the first backward region, and use this pixel as the vertex to determine the fourth backward region. The width of the fourth backward region is blkW+4 and the height is blkH+4.

[0493] For any vertex position of the first backward region, assuming that the top-left vertex position of the first backward region is used, determine the pixel closest to the top-left vertex position in the backward reference image, and use this pixel as the top-left vertex to determine the fourth backward region. The width of the fourth backward region is blkW+4 and the height is blkH+4.

[0494] For example, see Figure 13The top left vertex of the first backward region D1 is located at (5.3, 6.2). In the backward reference image Ref1, the position of the pixel closest to the vertex (5.3, 6.2) is determined to be (5, 6). The fourth backward region D2 is determined with the pixel at position (5, 6) as the top left vertex. The width of the fourth backward region D2 is blkW+4 and the height is blkH+4.

[0495] (8): Determine the fifth backward region, which includes the fourth backward region. The center of the fourth backward region and the center of the fifth backward region coincide. The width of the fifth backward region is blkW+n+3 and the height is blkH+n+3. Obtain the image block located in the fifth backward region in the backward reference image. Use the interpolation filter to interpolate and filter the image block to obtain the second backward prediction block of the first image block. The width of the second backward prediction block is blkW+4 and the height is blkH+4.

[0496] For example, see the example of an 8-tap interpolation filter. Figure 13 A fifth backward region D3 is determined, including the fourth backward region D2. The center of the fourth backward region D2 and the center of the fifth backward region D3 coincide. The width of the fifth backward region D3 is blkW+11 and the height is blkH+11. An image block located in the fifth backward region D3 is obtained in the backward reference image Ref1. An interpolation filter is used to interpolate and filter this image block to obtain the second backward prediction block of the first image block B. The width of the second backward prediction block is blkW+4 and the height is blkH+4. n is the number of taps of the interpolation filter.

[0497] Step 303: Perform gradient operation on the second prediction block of the first image block to obtain the first gradient value matrix of the first image block. The width of the first gradient value matrix is ​​cuW+2 and the height is cuH+2.

[0498] The first gradient value matrix includes a first forward horizontal gradient value matrix, a first forward vertical gradient value matrix, a first backward horizontal gradient value matrix, and a first backward vertical gradient value matrix.

[0499] In this step, the detailed implementation process of performing gradient calculations on the second prediction block of the first image patch to obtain the first gradient value matrix can be found in [link to relevant documentation]. Figure 12 The detailed process of obtaining the first gradient value matrix in step 104 of the illustrated embodiment will not be described in detail here.

[0500] The first prediction block includes a second forward prediction block and a second backward prediction block. Based on the second forward prediction block, a second forward horizontal gradient matrix with a width of cuW+2 and a height of cuH+2, and a second forward vertical gradient matrix with a width of cuW+2 and a height of cuH+2, can be obtained. Based on the second backward prediction block, a second backward horizontal gradient matrix with a width of cuW+2 and a height of cuH+2, and a second backward vertical gradient matrix with a width of cuW+2 and a height of cuH+2, can be obtained.

[0501] Step 304: Determine the first prediction block of the first image block in the second prediction block. The width of the first prediction block is cuW+2 and the height is cuH+2. Then proceed to step 308.

[0502] The center of the first prediction block coincides with the center of the second prediction block.

[0503] The first prediction block includes a first forward prediction block and a first backward prediction block.

[0504] In this step, a first forward prediction block with a width of cuW+2 and a height of cuH+2 is determined in the second forward prediction block; a first backward prediction block with a width of cuW+2 and a height of cuH+2 is determined in the second backward prediction block.

[0505] Step 305: Obtain the first prediction block of the first image block based on the motion information of the first image block. The width of the first prediction block is cuW+2 and the height of the first prediction block is cuH+2.

[0506] For details on obtaining the first prediction block in this step, please refer to [link / reference]. Figure 12 The details of step 103 in the illustrated embodiment will not be described in detail here.

[0507] Step 306: Perform gradient operation on the first prediction block of the first image block to obtain the first gradient value matrix of the first image block. The width of the first gradient value matrix is ​​cuW and the height is cuH.

[0508] The first gradient value matrix includes a first forward horizontal gradient value matrix, a first forward vertical gradient value matrix, a first backward horizontal gradient value matrix, and a first backward vertical gradient value matrix.

[0509] For a detailed explanation of this step, please refer to [link / reference]. Figure 12 The details of step 104 in the illustrated embodiment will not be described in detail here.

[0510] Step 307: Based on the gradient values ​​at the edge positions of the first gradient value matrix, perform a first expansion on the width and height of the first gradient value matrix so that the width and height of the first expanded first gradient value matrix are respectively 2 pixels larger than the width and height of the first image block.

[0511] For a detailed explanation of this step, please refer to [link / reference]. Figure 12 The details of step 105 in the illustrated embodiment will not be described in detail here.

[0512] Steps 308-310 are the same as steps 106-108, and will not be described in detail here.

[0513] Regarding the above Figure 15 The inter-frame prediction method shown can be summarized as steps 1 to 6 below. These steps 1 to 6 can be respectively:

[0514] Step 1: Determine the motion information of the current CU

[0515] The motion information of the current CU can be determined by Merge mode, AMVP mode (as described in the background art), or other modes, which are not limited here.

[0516] It should be noted that other methods for determining motion information can also be applied to this invention, which will not be elaborated here.

[0517] Step 2: Determine if the current CU meets the conditions for using BIO.

[0518] If the current CU uses bidirectional prediction, and the relationship between the forward reference frame number POC_L0, the backward reference frame number POC_L1, and the current frame number POC_Cur satisfies the following formula, then the current CU meets the conditions for using BIO:

[0519] (POC_L0–POC_Cur)*(POC_L1–POC_Cur)<0

[0520] It should be noted that other BIO usage conditions can also be applied to this invention, which will not be elaborated here.

[0521] If the current CU meets the conditions for using BIO, proceed to step 3; otherwise, perform motion compensation in other ways.

[0522] Step 3: Calculate the forward and backward prediction values ​​of the current CU.

[0523] If cuW is greater than or equal to VPDU_X or cuH is greater than or equal to VPDU_Y, motion compensation is performed using motion information to obtain the forward and backward predicted values ​​I. (k)(i,j), where i=-2..cuW+1,j=-2..cuH+1 (then the same interpolation filter is used to obtain the prediction matrix (cuW+4)*(cuH+4);

[0524] Otherwise, motion information is used to perform motion compensation to obtain the forward and backward predicted values ​​I. (k) (i,j), where i=-1..cuW,j=-1..cuH (then we get the prediction matrix of (cuW+2)*(cuH+2)).

[0525] Among them, I (k) In (i,j), i = 0..cuW-1, j = 0..cuH-1, the values ​​are obtained by interpolation using an 8-tap interpolation filter, and the predicted values ​​at other positions (extended by 1 row and 1 column) are obtained by interpolation using a bilinear interpolation filter.

[0526] It should be understood that the predicted value can be obtained by using the size of the VPDU as the smallest unit, or by using a block size smaller than the VPDU as the smallest unit; there is no limitation on this.

[0527] It should be noted that the predicted value of the extended region can also be obtained by other methods, such as using the same 8-tap interpolation filter, or directly using the reference pixel at the integer pixel position, which is not limited here.

[0528] It should be noted that, when calculating the SAD between the forward and backward predicted values, it is determined whether the SAD is less than the threshold TH_CU. If it is less, BIO is not executed. Otherwise, BIO is executed. Other determination methods can also be applied to this invention, which will not be elaborated here.

[0529] The formula for calculating SAD is as follows:

[0530]

[0531] The threshold TH_CU can be set to (1<<(BD–8+shift))*cuW*cuH, where shift is Max(2,14-BD).

[0532] Step 4: Calculate the horizontal and vertical gradient values ​​of the current CU's forward and backward predictions.

[0533] If cuW is greater than or equal to VPDU_X or cuH is greater than or equal to VPDU_Y, the horizontal and vertical gradient values ​​are calculated according to formula (4). and Where i = -1..cuW, j = -1..cuH (then we obtain the gradient matrix of (cuW+2)*(cuH+2)).

[0534] Otherwise, the horizontal and vertical gradient values ​​are calculated according to formula (4) based on the forward and backward predicted values. and Where i = 0..cuW-1, j = 0..cuH-1 (then the gradient matrix of cuW*cuH is obtained).

[0535] Step 5: If cuW is less than VPDU_X and cuH is less than VPDU_Y, perform padding on the forward and backward predicted values ​​of the current CU, as well as the gradient values ​​in the horizontal and vertical directions.

[0536] like Figure 9 Padding is performed using the method to obtain... and Where i = -1..cuW, j = -1..cuH (then we obtain the prediction matrix, horizontal gradient matrix, and vertical gradient matrix of (cuW+2)*(cuH+2)).

[0537] Step 6: Derive the corrected motion vector for each 4x4 sub-block, and then perform weighted summation.

[0538] For each 4x4 sub-block, vx and vy are obtained according to formula (2), and finally weighted according to formula (6) to obtain the predicted value of each 4x4 sub-block.

[0539] It should be noted that the SAD between the forward and backward prediction values ​​of each 4x4 sub-block can be calculated to determine whether it is less than the threshold TH_SCU. If it is less, a weighted average is directly performed; otherwise, vx and vy are obtained according to formula (2), and then weighted according to formula (6). Other judgment methods can also be applied to this invention, which will not be elaborated here. TU_SCU can be set to 1<<(BD–3+shift).

[0540] In this embodiment of the invention, the boundary of the VPDU is predicted in the same way as the boundary of the CU. When the CU contains multiple VPDUs, the implementation complexity of motion compensation prediction is reduced.

[0541] In this embodiment, when cuW is greater than or equal to Width and / or cuH is greater than or equal to Height, the second prediction block of the first image block is obtained based on the motion information of the first image block. Since the width of the second prediction block is cuW+4 and the height is cuH+4, gradient calculation is performed on the second prediction block of the first image block to obtain a first gradient value matrix with a width of cuW+2 and a height of cuH+2. This eliminates the need to expand the edges of the first gradient value matrix and improves the efficiency of inter-frame prediction.

[0542] Figure 17As shown in the illustrative flowchart of an embodiment of this application, an inter-frame prediction method is provided, including:

[0543] S1201. Obtain motion information of the image block to be processed, wherein the image block to be processed includes multiple virtual pipeline data units, and the virtual pipeline data unit includes at least one basic processing unit.

[0544] S1202. Based on the motion information, obtain the prediction value matrix for each of the virtual flow data units;

[0545] S1203. Calculate the horizontal prediction gradient matrix and the vertical prediction gradient matrix for each of the virtual pipeline data units based on each of the prediction value matrices.

[0546] S1204. Based on the predicted value matrix, the horizontal prediction gradient matrix, and the vertical prediction gradient matrix, calculate the motion information correction amount of each basic processing unit in each virtual pipeline data unit.

[0547] In one feasible implementation, obtaining the prediction value matrix for each virtual pipeline data unit based on the motion information includes: obtaining an initial prediction matrix for each virtual pipeline data unit based on the motion information, wherein the initial prediction matrix is ​​of the same size as the virtual pipeline data unit; and using the initial prediction matrix as the prediction value matrix.

[0548] In one feasible implementation, after obtaining the initial prediction matrix for each of the virtual pipeline data units, the method further includes: extending the edges of the initial prediction matrix by pixels to obtain an extended prediction matrix, wherein the size of the extended prediction matrix is ​​larger than the size of the initial prediction matrix; correspondingly, using the initial prediction matrix as the prediction value matrix includes: using the extended prediction matrix as the prediction value matrix.

[0549] In one feasible implementation, the pixel-point expansion of the edge of the initial prediction matrix includes: obtaining the pixel value of the pixel outside the initial prediction matrix by interpolation based on the pixel value of the pixel inside the initial prediction matrix, or using the pixel value of the pixel outside the initial prediction matrix that is adjacent to the edge as the pixel value of the pixel outside the initial prediction matrix.

[0550] In one feasible implementation, the virtual pipeline data unit includes multiple motion compensation units, and obtaining the prediction value matrix of each virtual pipeline data unit based on the motion information includes: obtaining the compensation value matrix of each motion compensation unit based on the motion information; and combining the compensation value matrices of the multiple motion compensation units to obtain the prediction value matrix.

[0551] In one feasible implementation, the step of calculating the horizontal prediction gradient matrix and the vertical prediction gradient matrix of each virtual pipeline data unit based on each prediction value matrix includes: performing horizontal gradient calculation and vertical gradient calculation on the prediction value matrix to obtain the horizontal prediction gradient matrix and the vertical prediction gradient matrix, respectively.

[0552] In one feasible implementation, before calculating the motion information correction amount for each basic processing unit in each virtual pipeline data unit based on the predicted value matrix, the horizontal predicted gradient matrix, and the vertical predicted gradient matrix, the method further includes: extending the edges of the predicted value matrix by pixels to obtain a filled predicted matrix, the filled predicted matrix having a preset size; extending the edges of the horizontal predicted gradient matrix and the vertical predicted gradient matrix by gradient values ​​respectively to obtain a filled horizontal gradient matrix and a filled vertical gradient matrix, the filled horizontal gradient matrix and the filled vertical gradient matrix each having the preset size; correspondingly, calculating the motion information correction amount for each basic processing unit in each virtual pipeline data unit based on the predicted value matrix, the horizontal predicted gradient matrix, and the vertical predicted gradient matrix includes: calculating the motion information correction amount for each basic processing unit in each virtual pipeline data unit based on the filled predicted matrix, the filled horizontal gradient matrix, and the filled vertical gradient matrix.

[0553] In one feasible implementation, before extending the edges of the predicted value matrix by pixels, the method further includes: determining that the size of the predicted value matrix is ​​smaller than the preset size.

[0554] In one feasible implementation, before performing gradient value expansion on the edges of the horizontal prediction gradient matrix and the edges of the vertical prediction gradient matrix, the method further includes: determining that the size of the horizontal prediction gradient matrix and / or the size of the vertical prediction gradient matrix is ​​smaller than the preset size.

[0555] In one feasible implementation, after calculating the motion information correction amount of each basic processing unit in each virtual pipeline data unit, the method further includes: obtaining the predicted value of each basic processing unit based on the predicted value matrix of the virtual pipeline data unit and the motion correction amount of each basic processing unit in the virtual pipeline data unit.

[0556] In one feasible implementation, the method is used for bidirectional prediction; correspondingly, the motion information includes motion information of a first reference frame list and motion information of a second reference frame list; the prediction value matrix includes a first prediction value matrix and a second prediction value matrix, wherein the first prediction value matrix is ​​obtained based on the motion information of the first reference frame list, and the second prediction value matrix is ​​obtained based on the motion information of the second reference frame list; the horizontal prediction gradient matrix includes a first horizontal prediction gradient matrix and a second horizontal prediction gradient matrix, wherein the first horizontal prediction gradient matrix is ​​calculated based on the first prediction value matrix, and the second horizontal prediction gradient matrix is ​​calculated based on the second prediction value matrix; the vertical prediction gradient matrix includes a first vertical prediction gradient matrix and a second vertical prediction gradient matrix, wherein the first vertical prediction gradient matrix is ​​calculated based on the first prediction value matrix, and the second vertical prediction gradient matrix is ​​calculated based on the second prediction value matrix; the motion information correction amount includes a first reference frame list motion information correction amount and a second reference frame list motion information correction amount, wherein the first reference frame list motion information correction amount is calculated based on the first prediction value matrix, the first horizontal prediction gradient matrix, and the second vertical prediction gradient matrix.

[0557] In one feasible implementation, before performing pixel-level expansion on the edges of the initial prediction matrix, the method further includes: determining that the temporal location of the image frame containing the image block to be processed is between the first reference frame indicated by the motion information of the first reference frame list and the second reference frame indicated by the motion information of the second reference frame list.

[0558] In one feasible implementation, after obtaining the prediction value matrix for each of the virtual pipeline data units, the method further includes: determining that the difference between the first prediction value matrix and the second prediction value matrix is ​​less than a first threshold.

[0559] In one feasible implementation, the motion information correction amount of the basic processing unit corresponds to a basic prediction value matrix in the prediction value matrix. Before calculating the motion information correction amount of each basic processing unit in each virtual pipeline data unit based on the prediction value matrix, the horizontal prediction gradient matrix, and the vertical prediction gradient matrix, the method further includes: determining that the difference between the first basic prediction value matrix and the second basic prediction value matrix is ​​less than a second threshold.

[0560] In one feasible implementation, the basic processing unit has a size of 4x4.

[0561] In one feasible implementation, the width of the virtual pipeline data unit is W, the height is H, the size of the extended prediction matrix is ​​(W+n+2)x(H+n+2), and correspondingly, the size of the horizontal prediction gradient matrix is ​​(W+n)x(H+n), and the size of the vertical prediction gradient matrix is ​​(W+n)x(H+n), where W and H are positive integers, and n is an even number.

[0562] In one feasible implementation, n is 0, 2, or -2.

[0563] In one feasible implementation, before acquiring the motion information of the image block to be processed, the method further includes: determining that the image block to be processed includes the plurality of virtual pipeline data units.

[0564] Figure 18 As shown in the illustrative flowchart of an embodiment of this application, an apparatus for inter-frame prediction is provided, comprising:

[0565] The acquisition module 1301 is used to acquire motion information of an image block to be processed, wherein the image block to be processed includes multiple virtual pipeline data units, and the virtual pipeline data unit includes at least one basic processing unit.

[0566] The compensation module 1302 is used to obtain the prediction value matrix of each of the virtual flow data units based on the motion information.

[0567] The calculation module 1303 is used to calculate the horizontal prediction gradient matrix and the vertical prediction gradient matrix of each of the virtual pipeline data units based on each of the prediction value matrices.

[0568] The correction module 1304 is used to calculate the motion information correction amount of each basic processing unit in each virtual pipeline data unit based on the prediction value matrix, the horizontal prediction gradient matrix and the vertical prediction gradient matrix.

[0569] In one feasible implementation, the compensation module 1302 is specifically used to: obtain an initial prediction matrix for each virtual pipeline data unit based on the motion information, wherein the initial prediction matrix is ​​equal in size to the virtual pipeline data unit; and use the initial prediction matrix as the prediction value matrix.

[0570] In one feasible implementation, the compensation module 1302 is specifically used to: extend the edges of the initial prediction matrix by pixels to obtain an extended prediction matrix, the size of which is larger than the size of the initial prediction matrix; and use the extended prediction matrix as the prediction value matrix.

[0571] In one feasible implementation, the compensation module 1302 is specifically used to: obtain the pixel value of the pixel outside the initial prediction matrix by interpolation based on the pixel value of the pixel within the initial prediction matrix, or to use the pixel value of the pixel at the edge of the initial prediction matrix as the pixel value of the pixel outside the initial prediction matrix adjacent to the edge.

[0572] In one feasible implementation, the virtual pipeline data unit includes multiple motion compensation units, and the compensation module is specifically used to: obtain a compensation value matrix for each motion compensation unit based on the motion information; and combine the compensation value matrices of the multiple motion compensation units to obtain the prediction value matrix.

[0573] In one feasible implementation, the calculation module 1303 is specifically used to: perform horizontal gradient calculation and vertical gradient calculation on the predicted value matrix to obtain the horizontal prediction gradient matrix and the vertical prediction gradient matrix, respectively.

[0574] In one feasible implementation, a filling module 1305 is further included, configured to: extend the edges of the prediction value matrix by pixels to obtain a filled prediction matrix, the filled prediction matrix having a preset size; extend the edges of the horizontal prediction gradient matrix and the vertical prediction gradient matrix by gradient values ​​respectively to obtain a filled horizontal gradient matrix and a filled vertical gradient matrix, the filled horizontal gradient matrix and the filled vertical gradient matrix having the preset size respectively; and calculate the motion information correction amount of each basic processing unit in each of the virtual pipeline data units based on the filled prediction matrix, the filled horizontal gradient matrix and the filled vertical gradient matrix.

[0575] In one feasible implementation, a judgment module 1306 is further included, used to: determine that the size of the predicted value matrix is ​​less than the preset size.

[0576] In one feasible implementation, the determination module 1306 is further configured to: determine that the size of the horizontal prediction gradient matrix and / or the size of the vertical prediction gradient matrix is ​​less than the preset size.

[0577] In one feasible implementation, the correction module 1304 is further configured to: obtain the predicted value of each basic processing unit based on the predicted value matrix of the virtual pipeline data unit and the motion correction amount of each basic processing unit in the virtual pipeline data unit.

[0578] In one feasible implementation, the device is used for bidirectional prediction; correspondingly, the motion information includes motion information of a first reference frame list and motion information of a second reference frame list; the prediction value matrix includes a first prediction value matrix and a second prediction value matrix, wherein the first prediction value matrix is ​​obtained based on the motion information of the first reference frame list, and the second prediction value matrix is ​​obtained based on the motion information of the second reference frame list; the horizontal prediction gradient matrix includes a first horizontal prediction gradient matrix and a second horizontal prediction gradient matrix, wherein the first horizontal prediction gradient matrix is ​​calculated based on the first prediction value matrix, and the second horizontal prediction gradient matrix is ​​calculated based on the second prediction value matrix; the vertical prediction gradient matrix includes a first vertical prediction gradient matrix and a second vertical prediction gradient matrix, wherein the first vertical prediction gradient matrix is ​​calculated based on the first prediction value matrix, and the second vertical prediction gradient matrix is ​​calculated based on the second prediction value matrix; the motion information correction amount includes a first reference frame list motion information correction amount and a second reference frame list motion information correction amount, wherein the first reference frame list motion information correction amount is calculated based on the first prediction value matrix, the first horizontal prediction gradient matrix, and the second vertical prediction gradient matrix.

[0579] In one feasible implementation, the determination module 1306 is further configured to: determine that the temporal position of the image frame in which the image block to be processed is located is between the first reference frame indicated by the motion information of the first reference frame list and the second reference frame indicated by the motion information of the second reference frame list.

[0580] In one feasible implementation, the judgment module 1306 is further configured to: determine that the difference between the first predicted value matrix and the second predicted value matrix is ​​less than a first threshold.

[0581] In one feasible implementation, the judgment module 1306 is further configured to: determine that the difference between the first basic prediction value matrix and the second basic prediction value matrix is ​​less than a second threshold.

[0582] In one feasible implementation, the basic processing unit has a size of 4x4.

[0583] In one feasible implementation, the width of the virtual pipeline data unit is W, the height is H, the size of the extended prediction matrix is ​​(W+n+2)x(H+n+2), and correspondingly, the size of the horizontal prediction gradient matrix is ​​(W+n)x(H+n), and the size of the vertical prediction gradient matrix is ​​(W+n)x(H+n), where W and H are positive integers, and n is an even number.

[0584] In one feasible implementation, n is 0, 2, or -2.

[0585] In one feasible implementation, the determination module 1306 is further configured to: determine that the image block to be processed includes the plurality of virtual pipeline data units.

[0586] See Figure 19 The figure shows an illustrative flowchart of an embodiment of this application, illustrating an inter-frame prediction apparatus 1400, comprising:

[0587] The determining module 1401 is used to determine a plurality of first image blocks in the image block to be processed based on the preset image division width, the preset image division height, and the width and height of the image block to be processed;

[0588] The prediction module 1402 is used to perform bidirectional prediction based on optical flow on the plurality of first image blocks respectively to obtain the prediction value of each first image block;

[0589] The combination module 1403 is used to combine the predicted values ​​of the plurality of first image blocks to obtain the predicted value of the image block to be processed.

[0590] In one feasible implementation, the determining module 1401 is configured to:

[0591] The width of the first image block is determined by comparing the preset image segmentation width with the width of the image block to be processed.

[0592] The height of the first image block is determined by comparing the preset image segmentation height with the height of the image block to be processed.

[0593] Based on the width and height of the first image block, a plurality of first image blocks are determined in the image block to be processed.

[0594] In one feasible implementation, the width of the first image block is the minimum value between the preset image segmentation width and the width of the image block to be processed, and the height of the first image block is the minimum value between the preset image segmentation height and the height of the image block to be processed.

[0595] In one feasible implementation, the prediction module 1402 is used for:

[0596] Based on the motion information of the image block to be processed, a first prediction block of the first image block is obtained;

[0597] Gradient calculation is performed on the first prediction block to obtain the first gradient value matrix of the first image block;

[0598] Based on the first prediction block and the first gradient value matrix, calculate the motion information correction amount for each basic processing unit in the first image block;

[0599] Based on the motion information correction amount of each basic processing unit, the predicted value of the first image block is obtained.

[0600] In one possible implementation, the device 1400 further includes a first expansion module 1404:

[0601] The first expansion module is configured to, based on the pixel values ​​at the block edge positions of the first prediction block, perform a first expansion on the width and height of the first prediction block, such that the width and height of the first expanded first prediction block are respectively larger than the width and height of the first image block by 2 pixel points; and / or, based on the gradient values ​​at the matrix edge positions of the first gradient value matrix, perform a first expansion on the width and height of the first gradient value matrix, such that the width and height of the first expanded first gradient value matrix are respectively larger than the width and height of the first image block by 2 pixel points;

[0602] Correspondingly, the prediction module 1402 is used to calculate the motion information correction amount of each basic processing unit in the first image block based on the first expanded first prediction block and / or the first expanded first gradient value matrix.

[0603] In one possible implementation, the device further includes a second expansion module 1405:

[0604] The second extension module is used to perform interpolation filtering on the pixel values ​​of the block edge region of the first prediction block, or to copy the pixel values ​​of the block edge position of the first prediction block to perform a second extension on the width and height of the first prediction block;

[0605] Correspondingly, the prediction module 1402 is used to perform gradient calculation on the second extended first prediction block.

[0606] In one feasible implementation, the first prediction block includes a forward prediction block and a backward prediction block, and the first gradient value matrix includes a forward horizontal gradient value matrix, a forward vertical gradient value matrix, a backward horizontal gradient value matrix, and a backward vertical gradient value matrix.

[0607] In one feasible implementation, the preset image segmentation width is 64, 32, or 16, and the preset image segmentation height is 64, 32, or 16.

[0608] In one feasible implementation, the basic processing unit is a 4x4 pixel matrix.

[0609] In this embodiment, since the determining module determines multiple first image blocks in the image block to be processed based on the preset image segmentation width, preset image segmentation height, and the width and height of the image block to be processed, the size of the first image block is constrained by the preset image segmentation width and preset image segmentation height, which prevents the area of ​​each determined first image block from being too large. This reduces the consumption of hardware resources such as memory, lowers the implementation complexity of inter-frame prediction, and improves processing efficiency.

[0610] Those skilled in the art will appreciate that the functionality described in conjunction with the various illustrative logic blocks, modules, and algorithmic steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality described by the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium that includes any medium facilitating the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this way, the computer-readable medium may substantially correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may comprise a computer-readable medium.

[0611] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are specifically addressed to non-temporary tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The combination of the above items should also be included in the scope of computer-readable media.

[0612] Instructions can be executed by one or more processors, such as digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the various illustrative logic blocks, modules, and steps described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques can be fully implemented within one or more circuit or logic elements.

[0613] The technology of this application can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The various components, modules, or units described in this application are intended to emphasize functional aspects of the apparatus for performing the disclosed technology, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units can be combined with suitable software and / or firmware within a codec hardware unit, or provided via interoperable hardware units (containing one or more processors as described above).

[0614] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0615] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for inter-frame prediction, the method being applied to a coding device, characterized in that, include: Obtain an image block to be processed, wherein the image block to be processed is obtained by dividing the image, and the image block to be processed is a coding unit (CU); Based on a preset image segmentation width, a preset image segmentation height, and the width and height of the image block to be processed, a plurality of first image blocks are determined in the image block to be processed, wherein the preset image segmentation width is 16 and the preset image segmentation height is 16; wherein the width of the first image block is equal to the minimum value between the preset image segmentation width and the width of the image block to be processed, and the height of the first image block is equal to the minimum value between the preset image segmentation height and the height of the image block to be processed; Bidirectional optical flow-based prediction is performed on each of the plurality of first image blocks to obtain a prediction value for each first image block; wherein, each first image block includes a plurality of basic processing units, and the prediction value of the first image block includes the prediction value of each pixel in each basic processing unit in the first image block, wherein the prediction value of each pixel in the basic processing unit is calculated based on the forward basic prediction block, the backward basic prediction block, the forward horizontal basic gradient matrix, the forward vertical basic gradient matrix, the backward horizontal basic gradient matrix, and the backward vertical basic gradient matrix of the basic processing unit; The predicted values ​​of the plurality of first image blocks are combined to obtain the predicted value of the image block to be processed.

2. The method according to claim 1, characterized in that, The step of determining multiple first image blocks within the image block to be processed based on a preset image segmentation width, a preset image segmentation height, and the width and height of the image block to be processed includes: The width of the first image block is determined by comparing the preset image segmentation width with the width of the image block to be processed. The height of the first image block is determined by comparing the preset image segmentation height with the height of the image block to be processed. The plurality of first image blocks are determined in the image block to be processed based on the width and height of the first image block.

3. The method according to claim 1 or 2, characterized in that, The basic processing unit is a 4x4 pixel matrix.

4. An apparatus for inter-frame prediction, the apparatus being used for encoding, characterized in that, include: A module for acquiring image blocks to be processed, wherein the image blocks to be processed are obtained by dividing the image, and the image blocks to be processed are encoding units (CUs). The determining module is used to determine a plurality of first image blocks in the image block to be processed based on a preset image segmentation width, a preset image segmentation height, and the width and height of the image block to be processed. The preset image segmentation width is 16, and the preset image segmentation height is 16. The width of the first image block is equal to the minimum value between the preset image segmentation width and the width of the image block to be processed, and the height of the first image block is equal to the minimum value between the preset image segmentation height and the height of the image block to be processed. A prediction module is used to perform bidirectional optical flow-based prediction on the plurality of first image blocks to obtain a prediction value for each first image block; wherein, the first image block includes a plurality of basic processing units, and the prediction value of the first image block includes the prediction value of each pixel in each basic processing unit in the first image block, wherein the prediction value of each pixel in the basic processing unit is calculated based on the forward basic prediction block, the backward basic prediction block, the forward horizontal basic gradient matrix, the forward vertical basic gradient matrix, the backward horizontal basic gradient matrix, and the backward vertical basic gradient matrix of the basic processing unit; The combination module is used to combine the predicted values ​​of the multiple first image blocks to obtain the predicted value of the image block to be processed.

5. The apparatus according to claim 4, characterized in that, The determining module is used for: The width of the first image block is determined by comparing the preset image segmentation width with the width of the image block to be processed. The height of the first image block is determined by comparing the preset image segmentation height with the height of the image block to be processed. The plurality of first image blocks are determined in the image block to be processed based on the width and height of the first image block.

6. The apparatus according to claim 4 or 5, characterized in that, The basic processing unit is a 4x4 pixel matrix.

7. A device for inter-frame prediction, characterized in that, include: A non-volatile memory and a processor are coupled together, the processor calling program code stored in the memory to perform the method described in any one of claims 1-3.

Images