Decoding method, decoding device, electronic device, and machine-readable storage medium
The method addresses the limitation of conventional prediction compensation by determining pixel-specific gradient and offset vectors to adaptively adjust predictions, enhancing decoding flexibility and accuracy for image blocks with diverse motion vectors.
Patent Information
- Application Number
- JP2024028586
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-09-23
- Filing Date
- 2024-02-28
- Publication Date
- 2024-12-09
- Estimated Expiration
- 2040-09-16
AI Technical Summary
Conventional prediction compensation adjustment methods are limited to image blocks with identical motion vectors between sub-blocks and pixel points, failing to account for variations in motion vectors within sub-blocks.
A method that determines a predicted value for each pixel point, calculates a gradient value and offset vector, and then uses these to derive a predicted compensation value for each pixel point, allowing for flexible prediction compensation adjustment across various motion vector configurations.
Expands the applicability of prediction compensation adjustment to include image blocks with varying motion vectors, improving decoding efficiency and accuracy.
Smart Images

Figure 0007596574000033 
Figure 0007596574000034 
Figure 0007596574000035
Abstract
Description
[Technical field]
[0001] The present application relates to video codec technology, and in particular to a decoding method, a decoding device, an electronic device, and a machine-readable storage medium. [Background technology]
[0002] Previously, the Joint Video Experts Team (JVET) conference proposed a prediction compensation adjustment method, which obtains an initial predicted value of each sub-block based on initial motion information of each sub-block of a current image block, obtains a compensation value of each sub-block based on the initial predicted value of each sub-block, and further obtains a final predicted value of the current image block based on the compensation value of each sub-block and the initial predicted value. Summary of the Invention [Problem to be solved by the invention]
[0003] However, in reality, the conventional prediction compensation adjustment method can only be applied to image blocks that use bidirectional prediction mode and in which the motion vectors of each sub-block are the same as the motion vectors of each pixel point of the corresponding sub-block, and prediction compensation adjustment cannot be performed for image blocks in which the motion vector of a sub-block differs from the motion vector of the pixel point of the sub-block. [Means for solving the problem]
[0004] In view of this, the present application provides a codec method, apparatus and device.
[0005] Specifically, the present application is realized by the following technical solutions.
[0006] According to a first aspect of the present invention, determining a predicted value for each pixel point of the current image block; determining a gradient value for each pixel point of the current image block based on the predicted value for each pixel point of the current image block; determining an offset vector for each pixel point of the current image block; determining a predicted compensation value of each pixel point of the current image block according to the gradient value and the offset vector of each pixel point of the current image block; determining a final predicted value for each pixel point of the current image block according to the predicted value and the predicted compensation value of each pixel point of the current image block; The present invention provides a codec method including the steps of:
[0007] According to a second aspect of the present embodiment, a first determining means configured to determine a predicted value for each pixel point of a current image block; and a second determining means configured to determine a gradient value for each pixel point of the current image block based on the predicted value for each pixel point of the current image block; third determining means configured to determine an offset vector of each pixel point of the current image block; a fourth determining means configured to determine a predicted compensation value for each pixel point of the current image block according to a gradient value and an offset vector of each pixel point of the current image block; and a fifth determining means configured to determine a final predicted value for each pixel point of the current image block according to a predicted value and a predicted compensation value of each pixel point of the current image block. The present invention provides a codec device having the following configuration.
[0008] According to a third aspect of the present invention, A coding device is provided that includes a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to achieve the following steps: Determine the predicted value of each pixel point of the current image block; determining a gradient value for each pixel point of the current image block according to the predicted value for each pixel point of the current image block; determining an offset vector for each pixel point of the current image block; Determine a predicted compensation value of each pixel point of the current image block according to the gradient value and the offset vector of each pixel point of the current image block; A final predicted value of each pixel point of the current image block is determined according to the predicted value and the predicted compensation value of each pixel point of the current image block.
[0009] According to a fourth aspect of the present embodiment, A decoding device is provided, comprising a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to achieve the steps of: determining a predicted value for each pixel point of a current image block; determining a gradient value for each pixel point of the current image block according to the predicted value for each pixel point of the current image block; determining an offset vector for each pixel point of the current image block; Determine a predicted compensation value of each pixel point of the current image block according to the gradient value and the offset vector of each pixel point of the current image block; A final predicted value of each pixel point of the current image block is determined according to the predicted value and the predicted compensation value of each pixel point of the current image block.
[0010] In the codec method of the embodiment of the present application, after determining the predicted value of each pixel point of the current image block, the prediction compensation value of each pixel point of the current image block is determined according to the gradient value and offset vector of each pixel point of the current image block.Furthermore, the final predicted value of each pixel point of the current image block is determined according to the predicted value and the prediction compensation value of each pixel point of the current image block.Prediction compensation adjustment is not limited to the image block that uses bidirectional prediction mode, and is not limited to the image block that the motion vector of each sub-block is the same as the motion vector of each pixel point of the corresponding sub-block.Therefore, the application scope of prediction compensation adjustment is expanded. [Brief description of the drawings]
[0011] [Figure 1A] FIG. 2 is a schematic diagram illustrating block division according to an exemplary embodiment of the present application; [Figure 1B] FIG. 2 is a schematic diagram illustrating block division according to an exemplary embodiment of the present application; [Diagram 2] FIG. 2 is a schematic diagram illustrating 8-tap interpolation according to an exemplary embodiment of the present application. [Diagram 3] FIG. 3 is a flow chart illustrating a codec method according to an exemplary embodiment of the present application. [Figure 4] FIG. 4 is a schematic diagram showing motion vectors of control points in an affine motion mode according to an exemplary embodiment of the present application. [Figure 5A] FIG. 5A is a schematic diagram illustrating an integer pixel block according to an exemplary embodiment of the present application. [Figure 5B] FIG. 5B is a schematic diagram illustrating one row / column filling of integer pixel points according to an exemplary embodiment of the present application. [Figure 5C] FIG. 5C is a schematic diagram illustrating two-row / column filling of integer pixel points according to an exemplary embodiment of the present application. [Figure 5D] FIG. 5D is a schematic diagram illustrating two-row / column filling of integer pixel points according to another exemplary embodiment of the present application. [Figure 6] FIG. 6 is a flow chart illustrating a codec method according to an exemplary embodiment of the present application. [Figure 7A] FIG. 2 is a schematic diagram illustrating one row / column filling of integer pixel points according to an exemplary embodiment of the present application; [Figure 7B] FIG. 2 is a schematic diagram illustrating one row / column filling of integer pixel points according to an exemplary embodiment of the present application; [Figure 7C] FIG. 2 is a schematic diagram illustrating one row / column filling of integer pixel points according to an exemplary embodiment of the present application; [Figure 7D] FIG. 2 is a schematic diagram illustrating one row / column filling of integer pixel points according to an exemplary embodiment of the present application; [Figure 7E] FIG. 2 is a schematic diagram illustrating one row / column filling of integer pixel points according to an exemplary embodiment of the present application; [Figure 8]FIG. 8 is a flow chart illustrating a method for selecting candidate motion vectors according to an exemplary embodiment of the present application. [Figure 9] FIG. 9 is a flow chart illustrating a method for selecting candidate motion vectors according to an exemplary embodiment of the present application. [Figure 10] FIG. 10 is a flowchart illustrating a method for selecting a prediction mode according to an exemplary embodiment of the present application. [Figure 11] FIG. 11 is a flowchart illustrating a method for selecting a prediction mode according to an exemplary embodiment of the present application. [Figure 12] FIG. 12 is a schematic diagram showing the configuration of a codec device according to an exemplary embodiment of the present application. [Figure 13] FIG. 13 is a schematic diagram showing a hardware configuration of an encoding device according to an exemplary embodiment of the present application. [Figure 14] FIG. 14 is a schematic diagram showing a hardware configuration of a decoding device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Exemplary embodiments will now be described in detail, examples of which are illustrated in the drawings. Where the following description refers to the drawings, the same reference numerals in different drawings refer to the same or similar elements unless otherwise specified. The embodiments described in the following exemplary embodiments do not include all embodiments consistent with the present application. On the contrary, they are merely examples of apparatuses and methods consistent with parts of the present application, as described in the claims.
[0013] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this application and the claims, the words "a," "the," "said," and "the" may be used in the plural as well as in the singular, unless the context clearly indicates otherwise.
[0014] In order to help those skilled in the art better understand the technical solutions provided in the embodiments of the present application, a brief description of the block partitioning technology, the intra sub-block partitioning scheme in the video coding standard, and some technical terms related to the embodiments of the present application will be first given.
[0015] I. Block division technology in video coding standards
[0016] In HEVC (High Efficiency Video Coding), one coding tree unit (CTU) is divided into CUs (Coding Units) by quad-tree recursion. At the CU level of the leaf node, it is determined whether intra coding or inter coding is used. The CU is further divided into two or four prediction units (PUs), and the same prediction information is used in the same PU. After obtaining residual information after prediction is completed, one CU is further divided into multiple transform units (TUs) by four branches. For example, the current image block of this application is one PU.
[0017] However, the block partitioning technology in the latest proposed versatile video coding (VVC) has changed significantly. A partitioning configuration that combines binary trees, ternary trees, and quad trees replaces the original partitioning mode, and the distinction between the original concepts of CU, PU, and TU disappears, supporting a more flexible CU partitioning method. Here, a CU is a positive It can be divided into squares or rectangles. First, a CTU is divided into quadtrees. Then, the leaf nodes of the quadtree are further divided into binary trees and ternary trees. As shown in FIG. 1A, there are five division types of CUs: quadtree division, horizontal binary tree division, vertical binary tree division, horizontal ternary tree division, and vertical ternary tree division. As shown in FIG. 1B, CU division within one CTU can be performed by any combination of the above five division types. From the above, it can be seen that the shape of each PU is different depending on the division method. For example, squares and rectangles of different sizes.
[0018] 2. Technical terminology
[0019] A prediction pixel (Prediction Signal) is a pixel value derived from an already coded pixel, and a residual is obtained as the difference between the original pixel and the prediction pixel, and then the residual is transformed and quantized and coefficient coded.
[0020] For example, an inter-predicted pixel is a pixel value derived from a reference picture (reconstructed pixel) of a current image block. Since the pixel positions are discrete, the final predicted pixel is obtained by an interpolation operation. The closer the predicted pixel is to the original pixel, the smaller the residual energy obtained by subtracting the two is, which improves the compression function of the encoding.
[0021] As for the motion vector (abbreviated as MV), in inter-coding, the relative displacement between the current coding block and the best matching block in a reference image is represented by the MV. Each divided block (also called sub-block) has a corresponding motion vector that needs to be transferred to the decoding side. If the MV of each sub-block is coded and transferred independently, a significant amount of bits needs to be consumed, especially for the sub-blocks that are divided into small sizes. In order to reduce the amount of bits used for coding, the video coding utilizes the spatial domain correlation between adjacent image blocks to predict the MV of the block currently being coded by the MV of the adjacent coded block, and codes the prediction difference. In this way, the amount of bits to represent the MV can be efficiently reduced. As a result, in the process of encoding the MV of the current image block, the MV of the current image block is generally predicted using the MV of adjacent encoded blocks, and the difference between the predicted MV value (Motion Vector Prediction, abbreviated as MVP) and the true estimated value of the motion vector, i.e., the motion vector residual (Motion Vector Difference, abbreviated as MVD), is encoded, thereby efficiently reducing the amount of bits required for encoding the MV.
[0022] In the motion information, since the MV is configured to display the relative displacement between the current block and the best matching block in a reference image, in addition to the MV information, in order to accurately obtain the information pointing to the image block, it is necessary to indicate which reference image is used in the reference image index information. In the video coding technology, a reference image list is generally created for the current image based on a certain principle, and the reference image index information indicates which reference image in the reference image list is used by the current image block. In addition, since many coding technologies support multiple reference image lists, an index value indicating which reference image list is used is also required. The index value can be called a reference direction. In the video coding, the coding information related to the motion, such as the MV, the reference picture index, and the reference direction, is collectively called motion information.
[0023] Interpolation: If the current pixel is a non-integer pixel, the existing pixel value cannot be directly copied from the corresponding reference picture, so the desired pixel value can only be obtained by interpolation.
[0024] As shown in Figure 2, when it is necessary to obtain a pixel value Y1 / 2 with an offset of 1 / 2 pixel, it is necessary to obtain it by interpolating the surrounding existing pixel value X. If an interpolation filter with N taps is used, it is necessary to obtain it by interpolating N integer pixels around. If the number of taps is 8, JPEG0007596574000001.jpg523 and JPEG0007596574000002.jpg45 are the filter coefficients, i.e., the weighting coefficients.
[0025] Motion compensation is the process of obtaining all predicted pixel values for the current image block by interpolation (or copying).
[0026] The TMVP mode (Temporal Motion Vector Prediction) is a mode that uses motion vectors in a temporal domain reference picture.
[0027] BDOF mode (Bi-directional Optical Flow), also called BIO mode, is a mode in which a motion compensation value is adjusted by an optical flow method based on the motion compensation values of two reference pictures.
[0028] The DMVR mode (Decoder-side Motion Vector Refinement) is a mode in which a motion vector difference is obtained by searching based on the motion compensation values of two reference pictures.
[0029] SMVD mode (Symmetric Motion Vector Difference) is a mode in which a motion vector difference in one direction in a bidirectional prediction mode is estimated and obtained based on the difference with the motion vector in the other direction, and does not require coding.
[0030] Exemplarily, it can be obtained by scaling the motion vector difference in one direction, and the scaling factor depends on the distance between the two reference pictures and the picture to which the current image block belongs.
[0031] In order to make the above objectives, features and advantages of the embodiments of the present application more comprehensible, the technical solutions of the embodiments of the present application are described in more detail with reference to the drawings.
[0032] It should be noted that the codec method applied to the prediction compensation adjustment described in this application may be applied to an encoding device or a decoding device. When applied to an encoding device, the image block described in this application represents an encoding block, and when applied to a decoding device, the image block described in this application represents a decoding block, and the description thereof will be omitted hereinafter.
[0033] 3 is a flow chart of a codec method applied to prediction compensation adjustment according to an embodiment of the present application. As shown in FIG. 3, the codec method applied to prediction compensation adjustment includes the following steps:
[0034] In step S300, a predicted value for each pixel point of the current image block is determined.
[0035] In an embodiment of the present application, based on the motion information of the current image block, The predicted value of each pixel point is determined.
[0036] Illustratively, the current image block is divided into a number of sub-blocks, and the predicted value of the pixel point of each sub-block is determined based on the sub-blocks.
[0037] Exemplarily, the sub-block may be a 4*4 sub-block, or may be an 8*8 sub-block.
[0038] For example, an 8*8 image block is divided into four 4*4 sub-blocks, and the predicted values of each pixel point of each 4*4 sub-block are determined respectively.
[0039] Also, for example, a 16*16 image block is divided into four 8*8 sub-blocks, and a predicted value of each pixel point of each 8*8 sub-block is determined.
[0040] In step S310, the gradient value of each pixel point of the current image block is determined based on the predicted value of each pixel point of the current image block.
[0041] In step S320, the offset vector of each pixel point of the current image block is determined.
[0042] In the embodiment of the present application, the gradient value of each pixel point of the motion image block can be determined based on the predicted value of each pixel point of the current image block to realize prediction compensation.
[0043] Also, the offset vector of each pixel point of the current image block can be determined.
[0044] It should be noted that in the embodiment of the present application, there is no necessary timing relationship between determining the gradient value of each pixel point of the current image block and determining the offset vector of each pixel point of the current image block. That is, as shown in Fig. 3, the gradient value of each pixel point of the current image block may be determined first, and then the offset vector of each pixel point of the current image block may be determined, or the offset vector of each pixel point of the current image block may be determined first, and then the gradient value of each pixel point of the current image block may be determined, or the gradient value and the offset vector of each pixel point of the current image block may be determined in parallel. Here, the specific realization of this is omitted.
[0045] In step S330, a predicted compensation value of each pixel point of the current image block is determined according to the gradient value and offset vector of each pixel point of the current image block.
[0046] In an embodiment of the present application, after determining the gradient value and offset vector of each pixel point of the current image block, a predicted compensation value of each pixel point of the current image block can be determined according to the gradient value and offset vector of each pixel point of the current image block.
[0047] In step S340, a final predicted value of each pixel point of the current image block is determined according to the predicted value and the predicted compensation value of each pixel point of the current image block.
[0048] In an embodiment of the present application, after determining the predicted compensation value of each pixel point of the current image block, the final predicted value of each pixel point of the current image block can be determined according to the predicted value and the predicted compensation value of each pixel point of the current image block.
[0049] In the method shown in FIG. 3, after determining the predicted value of each pixel point of the current image block, the current image block is calculated based on the gradient value and offset vector of each pixel point of the current image block. The prediction compensation value of each pixel point of the current image block is determined. Then, the final prediction value of each pixel point of the current image block is determined according to the prediction value and the prediction compensation value of each pixel point of the current image block. The prediction compensation adjustment is not limited to the image block using the bidirectional prediction mode, nor is it limited to the image block in which the motion vector of each sub-block is the same as the motion vector of each pixel point of the corresponding sub-block. Therefore, the application range of the prediction compensation adjustment is expanded.
[0050] In the present embodiment, the prediction compensation adjustment is not limited to image blocks using bidirectional prediction modes, ie, it can be applied to image blocks using either unidirectional prediction modes or bidirectional prediction modes.
[0051] As a possible embodiment, if the current image block uses a unidirectional prediction mode, in the above step S300, determining a prediction value for each pixel point of the current image block may include determining a unidirectional prediction value for each pixel point of the current image block.
[0052] In the above step S310, determining a gradient value of each pixel point of the current image block based on a predicted value of each pixel point of the current image block may include determining a gradient value of each pixel point of the current image block in a single direction based on a predicted value of each pixel point of the current image block in that direction.
[0053] In the above step S320, determining the offset vector of each pixel point of the current image block may include determining an offset vector of each pixel point of the current image block in the corresponding direction.
[0054] In the above step S330, determining a predicted compensation value of each pixel point of the current image block based on a gradient value and an offset vector of each pixel point of the current image block may include determining a predicted compensation value of each pixel point of the current image block in a corresponding direction based on a gradient value and an offset vector of each pixel point of the current image block in a corresponding direction.
[0055] In the above step S340, determining a final predicted value of each pixel point of the current image block based on the predicted value and the predicted compensation value of each pixel point of the current image block may include determining a final predicted value of each pixel point of the current image block in a corresponding direction based on the predicted value and the predicted compensation value of each pixel point of the current image block in a corresponding direction.
[0056] For example, if the current image block uses a unidirectional prediction mode, determining the predicted value of each pixel point of the current image block can determine the unidirectional predicted value of each pixel point of the current image block.
[0057] For example, if the current image block uses a forward prediction mode, a forward prediction value of each pixel point of the current image block can be determined, and if the current image block uses a backward prediction mode, a backward prediction value of each pixel point of the current image block can be determined.
[0058] For ease of explanation and understanding, the unidirectional prediction mode used by the current image block is illustrated as a forward prediction mode.
[0059] After determining the forward predicted value of each pixel point of the current image block, a forward gradient value of each pixel point of the current image block is determined based on the forward predicted value of each pixel point of the current image block, and a forward offset vector of each pixel point of the current image block is determined.
[0060] Furthermore, a forward prediction compensation value of each pixel point of the current image block is determined according to the forward gradient value and offset vector of each pixel point of the current image block, and a forward final prediction value of each pixel point of the current image block is determined according to the forward prediction value and prediction compensation value of each pixel point of the current image block.
[0061] As another possible embodiment, when the current image block uses a bidirectional prediction mode, in the above step S300, determining a predicted value of each pixel point of the current image block may include respectively determining forward and backward predicted values of each pixel point of the current image block.
[0062] In the above step S310, determining the gradient value of each pixel point of the current image block based on the predicted value of each pixel point of the current image block may include respectively determining the forward and backward gradient values of each pixel point of the current image block based on the forward and backward predicted values of each pixel point of the current image block.
[0063] In the above step S320, determining the offset vector of each pixel point of the current image block may include respectively determining the forward and backward offset vectors of each pixel point of the current image block.
[0064] In the above step S330, determining the prediction compensation value of each pixel point of the current image block according to the gradient value and offset vector of each pixel point of the current image block may include respectively determining the forward and backward prediction compensation values of each pixel point of the current image block according to the forward and backward gradient values and offset vectors of each pixel point of the current image block.
[0065] In the above step S340, determining a final predicted value of each pixel point of the current image block based on the predicted value and predicted compensation value of each pixel point of the current image block may include respectively determining final forward and backward predicted values of each pixel point of the current image block based on the forward and backward predicted values and predicted compensation values of each pixel point of the current image block, and determining a final predicted value of each pixel point of the current image block based on the forward and backward final predicted values of each pixel point of the current image block.
[0066] For example, when the current image block uses a bidirectional prediction mode, it is necessary to respectively determine the forward and backward predicted values and the prediction compensation value of each pixel point of the current image block, thereby determining the forward and backward final predicted values of each pixel point of the current image block, and then determining the final predicted value of each pixel point of the current image block.
[0067] For example, when the current image block uses a bidirectional prediction mode, for the method of determining the final predicted value in the forward direction or the final predicted value in the backward direction of each pixel point of the current image block, refer to the implementation method when the current image block uses unidirectional prediction.
[0068] After determining the final forward predicted value and the final backward predicted value of each pixel point of the current image block, the final predicted value of each pixel point of the current image block is determined based on the final forward and backward predicted values of each pixel point of the current image block.
[0069] For example, the final predicted value of each pixel point of the current image block is obtained by weighting the final predicted values of the forward and backward directions of the pixel point of the current image block.
[0070] As a possible embodiment, in the above step S320, determining the offset vector of each pixel point of the current image block can be: For any one of the sub-blocks of the current image block, determine an offset vector of a designated pixel point in the sub-block; Based on the offset vector of the designated pixel point in the sub-block, offset vectors of the remaining pixel points in the sub-block are determined.
[0071] For example, for any one of the sub-blocks of the current image block, to determine the offset vectors of each pixel point in the sub-block, first determine the offset vector of any one of the pixel points (referred to as a designated pixel point in this application) in the sub-block, and then determine the offset vectors of the remaining pixel points in the sub-block based on the offset vector of the designated pixel point in the sub-block.
[0072] As an example, the current image block uses an affine motion mode. The offset vector of the specified pixel point is determined as follows: This may include determining a vector offset of the designated pixel point in the sub-block based on an offset between the designated pixel point in the sub-block and a center position of the sub-block, and an affine parameter.
[0073] Exemplarily, if the current image block uses an affine motion mode, a vector offset of a designated pixel point in the sub-block is determined based on an offset between the designated pixel point in the sub-block and a center position of the sub-block, and an affine parameter.
[0074] In one example, determining a vector offset of the designated pixel point of the sub-block based on an offset between the designated pixel point of the sub-block and a center position of the sub-block and an affine parameter includes: The method may also include determining a horizontal component of a vector offset of a designated pixel point in the sub-block based on a horizontal offset between a designated pixel point in the sub-block and a center position of the sub-block, and a vertical offset between a designated pixel point in the sub-block and a center position of the sub-block, a first affine parameter and a second affine parameter, and determining a vertical component of the vector offset of the designated pixel point in the sub-block based on a horizontal offset between a designated pixel point in the sub-block and a center position of the sub-block, and a vertical offset between a designated pixel point in the sub-block and a center position of the sub-block, a third affine parameter and a fourth affine parameter.
[0075] For example, for a four-parameter affine model, the first affine parameter is the same as the fourth affine parameter, both of which are a ratio of a first numerical value to the width of the sub-block, the second affine parameter is the opposite of the third affine parameter, and the third affine parameter is a ratio of a second numerical value to the width of the sub-block.
[0076] For a six-parameter affine model, the first affine parameter is the ratio of a first numerical value to the width of the sub-block, the second affine parameter is the ratio of a third numerical value to the height of the sub-block, the third affine parameter is the ratio of a second numerical value to the width of the sub-block, and the fourth affine parameter is the ratio of a fourth numerical value to the height of the sub-block.
[0077] The first value is a difference between a horizontal component of the motion vector of the upper right control point of the subblock and a horizontal component of the motion vector of the upper left control point of the subblock, the second value is a difference between a vertical component of the motion vector of the upper right control point of the subblock and a vertical component of the motion vector of the upper left control point of the subblock, and the third value is a difference between a horizontal component of the motion vector of the upper right control point of the subblock and a vertical component of the motion vector of the upper left control point of the subblock. The first value is the difference between the horizontal component of the motion vector of the lower left control point and the horizontal component of the motion vector of the upper left control point of the sub-block, and the fourth value is the difference between the vertical component of the motion vector of the lower left control point and the vertical component of the motion vector of the upper left control point of the sub-block.
[0078] For example, as shown in FIG. 4, the motion vectors of the upper left control point, the upper right control point, and the lower left control point are respectively (v 0x ,v 0y ), (v 1x ,v 1y ) and (v 2x ,v 2y ), the first number is v 1x -v 0x and the second number is v 1y -v 0y and the third number is v 2x -v 0y and the fourth number is v 2x -v 0y The first to fourth affine parameters (referred to as c to f, respectively) in the four-parameter affine model and the six-parameter affine model are expressed as Equation 1 and Equation 2, respectively.
[0079]
number
[0080]
number
[0081] In one example, determining the vector offset of the designated pixel point in the sub-block based on the offset between the designated pixel point in the sub-block and the center position of the sub-block and the affine parameters is realized by the following formula:
[0082]
number
[0083] For example, Δv x (x, y) is the horizontal component of the offset vector of the specified pixel in the sub-block, and Δv v (x, y) is the vertical component of the offset vector of a specified pixel in the subblock, and (Δx, Δy) is the offset between the specified pixel point and the center position of the subblock.
[0084] In one example, determining the offset vectors of the remaining pixel points in the sub-block based on the offset vector of the designated pixel point in the sub-block includes determining horizontal components of offset vectors of the remaining pixel points in the same row as the designated pixel point in the sub-block based on a horizontal component of the offset vector of the designated pixel point in the sub-block and a first affine parameter; determining vertical components of offset vectors of remaining pixel points in the same row as the designated pixel point in the sub-block based on a vertical component of the offset vector of the designated pixel point in the sub-block and the third affine parameter; For any one pixel point in the row in which the specified pixel point is located in the sub-block, the method may include determining horizontal components of offset vectors of remaining pixel points in the same column as the pixel point in the sub-block based on the horizontal component of the offset vector of the pixel point and the second affine parameter, and determining vertical components of offset vectors of remaining pixel points in the same column as the pixel point in the sub-block based on the vertical component of the offset vector of the pixel point and the fourth affine parameter.
[0085] For example, when the current image block uses an affine motion mode, after determining the offset vector of a specified pixel point in a sub-block, the offset vectors of the remaining pixel points in the same row as the specified pixel point in the sub-block are respectively determined based on the offset vector of the sub-block, the first affine parameter and the third affine parameter.
[0086] After determining the offset vectors of each pixel point in the row in which the specified pixel point is located in the sub-block, for any one pixel point in the row in which the specified pixel point is located in the sub-block, the offset vectors of the remaining pixel points in the same column as the pixel point in the sub-block are determined based on the offset vector of the pixel point, the second affine parameter, and the fourth affine parameter.
[0087] For example, for any one pixel point in the sub-block, the horizontal component of the offset vector of the pixel point is the sum of the horizontal component of the offset vector of the adjacent pixel point on the left side of the pixel point (if there is one) and the first affine parameter, or the difference between the horizontal component of the offset vector of the adjacent pixel point on the right side of the pixel point (if there is one) and the first affine parameter, and the vertical component of the offset vector of the pixel point is the sum of the vertical component of the offset vector of the adjacent pixel point on the left side of the pixel point (if there is one) and the third affine parameter, or the difference between the vertical component of the offset vector of the adjacent pixel point on the right side of the pixel point (if there is one) and the third affine parameter.
[0088] Furthermore, for any one pixel point in the sub-block, the horizontal component of the offset vector of the pixel point is the sum of the horizontal component of the offset vector of the adjacent pixel point above the pixel point (if there is one) and the second affine parameter, or the difference between the horizontal component of the offset vector of the adjacent pixel point below the pixel point (if there is one) and the second affine parameter, and the vertical component of the offset vector of the pixel point is the sum of the vertical component of the offset vector of the adjacent pixel point above the pixel point (if there is one) and the fourth affine parameter, or the difference between the vertical component of the offset vector of the adjacent pixel point below the pixel point (if there is one) and the fourth affine parameter.
[0089] As a possible embodiment, in the above step S310, determining the gradient value of each pixel point of the current image block based on the predicted value of each pixel point of the current image block can be: For any one of the sub-blocks of the current image block, fill the upper, lower, left, and right edges of the sub-block with N rows / columns (N rows / columns means N rows and N columns) of integer pixel points, respectively; This may include determining a gradient value for each pixel point of the sub-block based on the predicted value for each pixel point of the sub-block and the pixel values of the filled integer pixel points.
[0090] For example, when determining the horizontal component of the gradient value of a pixel point, the pixel value of the pixel point on the left side of the pixel point and the pixel value of the pixel point on the right side of the pixel point need to be used, and when determining the vertical component of the gradient value of a pixel point, the pixel value of the pixel point above the pixel point and the pixel value of the pixel point below the pixel point need to be used. However, there is no pixel point on at least one side of the pixel point on the edge of the sub-block.
[0091] For example, for a pixel point on the top edge of a subblock, there is no pixel point above it in that subblock, and for a pixel point on the left edge of a subblock, there is no pixel point to the left of it in that subblock.
[0092] Therefore, in order to determine the gradient value of each pixel point in a subblock, it is necessary to fill the top, bottom, left, and right edges of the subblock with N (N is a positive integer) rows / columns of integer pixel points (where pixels are integers), and the gradient value of each pixel point in the subblock is determined based on the predicted value of each pixel point in the subblock and the pixel value of the filled integer pixel point.
[0093] Illustratively, to reduce the bandwidth increase, pixel values of integer pixel points of the filling row / column are obtained by copying from neighboring integer pixel points of the reference pixel.
[0094] As an embodiment, filling the upper, lower, left and right edges of the sub-block with integer pixel points of N rows / columns is as follows: determining an integer pixel block of the same size as the sub-block in the reference picture that is closest to the sub-block; This may include setting pixel values of integer pixel points in the N rows / columns closest to the integer pixel block as filling values for the top, bottom, left, right and bottom edges of the sub-block, respectively.
[0095] For example, to achieve filling of N rows / columns of integer pixel points on the top, bottom, left and right edges of the sub-block, first determine an integer pixel block (a block of integer pixel points) of the same size as the sub-block in the reference picture that is closest to the sub-block.
[0096] For example, as shown in FIG. 5A, if the sub-block is a 4*4 sub-block, determine the integer pixel block that is the closest 4*4 sub-block to the sub-block in the reference picture.
[0097] When the integer pixel block of the same size closest to the sub-block in the reference picture is determined, the pixel values of integer pixel points of the adjacent N rows / columns around the integer pixel block can be used as the filling values for the top, bottom, left, and right edges of the sub-block, respectively.
[0098] For example, schematic diagrams of the filling points for N=1 and N=2 are shown in Figures 5B and 5C, respectively.
[0099] In one embodiment, the filling of the upper, lower, left and right edges of the sub-block with N rows / columns of integer pixel points is This may include filling N rows / columns of integer pixel points directly above, below, to the left, and to the right of the integer pixel block.
[0100] For example, after filling the top, bottom, left and right edges of a sub-block with N rows / columns of integer pixel points, a block is obtained whose width and height are both increased by 2N compared to the original block. When calculating the gradient value of each pixel point in the original sub-block based on the pixel value of each pixel point of the block, the corner points of the pixel points of each layer among the pixel points of the outer N layers of the block are not used.
[0101] Therefore, in order to reduce the amount of filling work and improve filling efficiency, when filling N rows / columns of integer pixel points on the top, bottom, left, and right edges of a subblock, N rows / columns of integer pixel points can be filled directly above, directly below, directly to the left, and directly to the right of the integer pixel block of the same size that is closest to the subblock in the reference picture.
[0102] For example, taking N=2 as an example, the final determined filling point is shown in FIG. 5D.
[0103] In one embodiment, when N=1, filling the upper, lower, left, and right edges of the sub-block with N rows / columns of integer pixel points respectively is as follows: if the sub-pixel vertical component of the prediction of the pixel point in the sub-block is greater than half a pixel, filling the top edge of the sub-block with the integer pixel point closest above, and filling the bottom edge of the sub-block with the integer pixel point next closest below; if the sub-pixel vertical component of the prediction of the pixel point in the sub-block is less than half a pixel, filling the top edge of the sub-block with the next nearest integer pixel point above, and filling the bottom edge of the sub-block with the nearest integer pixel point below; if the sub-pixel horizontal component of the prediction of the pixel point in the sub-block is greater than half a pixel, filling the left edge of the sub-block with the integer pixel point closest to the left and filling the right edge of the sub-block with the integer pixel point next closest to the right; This may include filling the left edge of the subblock with the next nearest integer pixel point on the left, and filling the right edge of the subblock with the nearest integer pixel point on the right, if the subpixel horizontal component of the predicted value of the pixel point in the subblock is less than half a pixel.
[0104] Take determining the horizontal component of the gradient value of a pixel point as an example. When determining the horizontal component of the gradient value of a pixel point, it is necessary to use the pixel value of the pixel point on the left side of the pixel point, which has a relative displacement of 1 pixel with the pixel point, and the pixel value of the pixel point on the right side, which has a relative displacement of 1 pixel with the pixel point. In addition, when filling pixel points in a subblock, since the filled row / column is an integer pixel point, if the predicted value of the pixel point of the subblock is a non-integer pixel, the filled integer pixel point is not displaced by 1 pixel relative to the pixel point on the edge of the subblock (the relative displacement between the non-integer pixel point and the integer pixel point is a non-integer pixel).
[0105] In order to reduce the error of the gradient value of the determined pixel point and improve the accuracy of the gradient value of the determined pixel point, when filling one row / column of integer pixel points on the top, bottom, left, right, and right edges of the subblock, the position of the integer pixel point to be filled is selected based on whether the vertical and horizontal components of the subpixels (i.e., the parts other than integer pixels, for example, when the pixel value is 5.4, the subpixels are 0.4) of the predicted value of the pixel point of the subblock are larger than half a pixel.
[0106] Illustrates filling of integer pixel points on top and bottom edges.
[0107] The sub-pixel vertical component of the predicted value of the pixel point of the sub-block is obtained, and the sub-pixel vertical component is compared with the half pixel.
[0108] Exemplarily, the sub-pixel of the predicted value of a pixel point of a sub-block is the sub-pixel of the pixel point in the sub-block at the position where the predicted value corresponds to the reference picture.
[0109] If the vertical component of the subpixel is greater than half a pixel, then for any pixel point on the top edge of the subblock, the remaining integer pixel points above the pixel point are filled with the nearest integer pixel point above the pixel point, since the relative displacement between the pixel point and the nearest integer pixel point above the pixel point is closest to one pixel.
[0110] For any one pixel point on the bottom edge of the sub-block, for the remaining integer pixel points below that pixel point, the relative displacement of that pixel point with respect to the next nearest integer pixel point below that pixel point is closest to one pixel.
[0111] For example, if the sub-pixel vertical component of a pixel point on the lower edge of the sub-block is 0.8, the relative displacement between the pixel point and the next nearest integer pixel point below the pixel point is 1.2. , the relative displacement between the pixel point and the remaining integer pixel points below the pixel point is greater than 1.2. Obviously, the relative displacement between the pixel point and the next closest integer pixel point below the pixel point is closest to 1 pixel.
[0112] Therefore, the next nearest integer pixel point below the pixel point is filled.
[0113] Similarly, if the sub-pixel vertical component of the predicted value of the pixel point in the sub-block is less than half a pixel, fill the top edge of the sub-block with the next nearest integer pixel point above, and fill the bottom edge of the sub-block with the nearest integer pixel point below; if the sub-pixel horizontal component of the prediction at the pixel point of the sub-block is greater than half a pixel, filling the left edge of the sub-block with the integer pixel point closest to the left and filling the right edge of the sub-block with the integer pixel point next closest to the right; This may include filling the left edge of the subblock with the next nearest integer pixel point on the left, and filling the right edge of the subblock with the nearest integer pixel point on the right, if the subpixel horizontal component of the prediction value at the pixel point of the subblock is less than half a pixel.
[0114] In addition, when the sub-pixel horizontal component of the predicted value of the pixel point is half a pixel, the process may be performed as if the sub-pixel horizontal component of the predicted value of the pixel point is larger than half a pixel, or the process may be performed as if the sub-pixel horizontal component of the predicted value of the pixel point is smaller than half a pixel.
[0115] Similarly, if the sub-pixel vertical component of the predicted value of a pixel point is half a pixel, it may be processed as if the sub-pixel vertical component of the predicted value of the pixel point was greater than half a pixel, or it may be processed as if the sub-pixel vertical component of the predicted value of the pixel point was less than half a pixel.
[0116] In one example, filling the upper, lower, left, and right edges of the sub-block with integer pixel points of one row / column each is: This includes filling integer pixel points other than the four corner points in a filling area consisting of rows / columns to be filled on the top, bottom, left, right, and right edges of the subblock.
[0117] For example, after filling the top, bottom, left, and right edges of a sub-block with one row / column of integer pixel points, a block is obtained whose width and height are both increased by 2 compared to the original block. When calculating the gradient value of each pixel point of the original sub-block based on the pixel value of each pixel point of the block, the four corner points of the block are not used.
[0118] For example, if the original sub-block is a 4*4 sub-block, a 6*6 block is obtained by filling one row / column of integer pixel points on the top, bottom, left, and right edges of the sub-block, respectively, and when calculating the gradient value of each pixel point of the original 4*4 block, the four corner points of the 6*6 block are not used.
[0119] Therefore, in order to reduce the amount of filling work and improve filling efficiency, when filling one row / column of integer pixel points on the top, bottom, left, and right edges of a subblock, it is possible to fill integer pixel points other than the four corner points in the filling area consisting of the rows / columns to be filled on the top, bottom, left, and right edges of the subblock.
[0120] In one embodiment, determining a gradient value for each pixel point of the sub-block based on the predicted value of each pixel point of the sub-block and the pixel value of the filled integer pixel point includes:
[0121] For any pixel point in the subblock, determine the horizontal component of the gradient value of the pixel point based on the pixel values of the N adjacent pixel points to the left of the pixel point and the pixel values of the N adjacent pixel points to the right of the pixel point, and determine the horizontal component of the gradient value of the pixel point based on the pixel values of the N adjacent pixel points above the pixel point and the pixel values of the N adjacent pixel points below the pixel point. This may include determining a vertical component of the gradient value for the pixel point based on the raw value.
[0122] For example, for any pixel point in the subblock, a horizontal component of the gradient value of the pixel point is determined based on the pixel value of a pixel point to the left of the pixel point and the pixel value of a pixel point to the right of the pixel point, respectively, and a vertical component of the gradient value of the pixel point is determined based on the pixel value of a pixel point above the pixel point and the pixel value of a pixel point below the pixel point.
[0123] For pixel values on the edge of the sub-block, when determining the horizontal and / or vertical components of the gradient value of the pixel point, it is necessary to use the pixel values of the filled integer pixel points adjacent to the pixel point.
[0124] For example, for a pixel point on the top edge of the sub-block, when determining the vertical component of the gradient value of that pixel point, the pixel value of the adjacent filled integer pixel point above that pixel point (the closest integer pixel point or the next closest integer pixel point) must be used, and for a pixel point on the left edge of the sub-block, when determining the horizontal component of the gradient value of that pixel point, the pixel value of the adjacent filled integer pixel point to the left of that pixel point must be used.
[0125] For example, taking N=1 as an example, for any pixel point of a sub-block, the gradient value of the pixel point can be determined by the following formula:
[0126]
number
[0127]
number
[0128] For example, g x (i,j) is the horizontal component of the gradient value of the pixel point at the ith row and jth column of the subblock, g v where (i,j) is the vertical component of the gradient value of the pixel point at the i-th column and j-th row of the sub-block, and I(i,j) is the pixel value of the pixel point at the i-th column and j-th row of the sub-block.
[0129] On the encoding side, the pixel values of the pixel points in the sub-block are predicted values and the pixel values of the integer pixel points to be filled are initial values, and on the decoding side, the pixel values of the pixel points in the sub-block are predicted values and the pixel values of the integer pixel points to be filled are reconstructed values.
[0130] In order to ensure the effectiveness of the predictive compensation adjustment scheme provided by the embodiments of the present application and to avoid applying the scheme when the predictive compensation adjustment scheme provided by the embodiments of the present application cannot be applied, conditions under which the predictive compensation adjustment scheme provided by the embodiments of the present application can be applied (referred to as predictive compensation adjustment application conditions in the present application) can be set in advance.
[0131] In a possible embodiment, before determining the gradient value of each pixel point of the current image block based on the predicted value of each pixel point of the current image block, further Determining whether the current image block satisfies a prediction compensation adjustment application condition; If yes, performing a gradient value determination for each pixel point of the current image block according to the predicted value of each pixel point of the current image block. In one example, the prediction compensation adjustment application conditions include a specified prediction mode used by the current image block, in which the motion vector of a sub-block is not completely the same as the motion vector of each pixel point of the sub-block.
[0132] For example, the motion vector of the subblock is completely the same as the motion vector of each pixel point of the subblock. Considering the case of a prediction mode that is not the same as the current image block (referred to as a designated prediction mode in this application), prediction compensation adjustment cannot be performed based on the conventional prediction compensation adjustment proposal, so the prediction compensation adjustment application condition is set to that the current image block uses the designated prediction mode.
[0133] After determining the predicted value of each pixel point of the current image block, determine whether to apply the prediction compensation adjustment scheme provided in the embodiment of the present application according to the prediction mode of the current image block.
[0134] When applied, ie, when the current image block uses the specified prediction mode, the current image block is subjected to prediction compensation adjustment as in the above embodiment.
[0135] Illustratively, the specified prediction mode includes an affine motion mode.
[0136] In another example, the above prediction compensation adjustment application conditions further include that the currently predicted component is a luminance component, i.e., the luminance component of the current image block using the specified prediction mode is prediction compensation adjusted with the prediction compensation adjustment proposal provided by the embodiment of the present application.
[0137] In another example, the above prediction compensation adjustment application condition further includes that the currently predicted component is a chrominance component, i.e., the chrominance component of the current image block using the specified prediction mode is prediction compensation adjusted using the prediction compensation adjustment proposal provided by the embodiment of the present application.
[0138] In order to make those skilled in the art more familiar with the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application will be described below with reference to specific examples.
[0139] Example 1 6 is a flow chart of a coding method according to an embodiment of the present application, in which a decoding process and an Affine mode are taken as an example, as shown in FIG. 6, the coding method includes the following steps:
[0140] In step S600, a predicted value for each pixel point of a decoded block is determined.
[0141] As an example, a decoding block is divided into 4*4 sub-blocks, and each sub-block is decoded.
[0142] For any one 4*4 sub-block, the predicted value of each pixel point of the sub-block is determined based on the Affine mode.
[0143] For example, for a pixel point with coordinates (i,j) in a subblock, its pixel value is I(i,j).
[0144] In step S610, it is determined whether the decoded block satisfies the prediction compensation adjustment application condition. If it does, the process proceeds to step S620; if it does not, the current process ends.
[0145] For example, after determining the predicted value of each pixel point of the decoded block, it is determined whether the decoded block satisfies a prediction compensation adjustment application condition.
[0146] If the decoded block does not satisfy the conditions for applying prediction compensation adjustment, it is not necessary to perform prediction compensation adjustment according to the prediction compensation adjustment scheme provided in the embodiment of the present application, in which the predicted value of each pixel point of the decoded block determined in step S600 is determined as the final predicted value of each pixel point, or the prediction compensation adjustment scheme may be performed by other methods, and the specific implementation thereof is not limited.
[0147] In step S620, for any one of the sub-blocks, the gradient value of each pixel point of the sub-block is determined based on the predicted value of each pixel point of the sub-block.
[0148] In step S630, the offset vector of each pixel point of the sub-block is determined.
[0149] For example, when it is determined that the decoded block satisfies the prediction compensation adjustment application condition, for any 4*4 sub-block, the gradient value of each pixel point of the 4*4 sub-block can be determined based on the predicted value of each pixel point of the 4*4 sub-block.
[0150] Also, the offset vector of each pixel point of the 4*4 sub-block can be determined.
[0151] In step S640, a predicted compensation value of each pixel point of the sub-block is determined according to the gradient value and offset vector of each pixel point of the sub-block.
[0152] For example, for a 4*4 sub-block, after determining the gradient value and offset vector of each pixel point of the 4*4 sub-block, the predicted compensation value of each pixel point of the current image block can be determined based on the gradient value and offset vector of each pixel point of the current image block.
[0153] In step S650, a final predicted value of each pixel point of the decoded block is determined according to the predicted value and the predicted compensation value of each pixel point of the decoded block.
[0154] For example, when using a unidirectional prediction mode, the prediction value and prediction compensation value of each pixel point of the decoded block in the corresponding direction can be determined as in steps S600 to S640 above, and the final prediction value of each pixel point of the decoded block is obtained.
[0155] When the bidirectional prediction mode is used, first, the forward and backward predicted values and prediction compensation values of each pixel point of the decoded block can be determined as in steps S600 to S640 above, and then the final predicted value of each pixel point of the decoded block is obtained.
[0156] Example 2 In one example, the application conditions of the codec method process included in the above steps S600 to S650 are as follows: The current image block uses an affine motion mode.
[0157] Example 3 In one example, the application conditions of the codec method process included in the above steps S600 to S650 are as follows: the current image block uses an affine motion mode; The component of the current prediction is a luma component.
[0158] Example 4 In one example, the application conditions of the codec method process included in the above steps S600 to S650 are as follows: the current image block uses an affine motion mode; The components of the current prediction are chrominance components.
[0159] Example 5 In one example, on any one of the above embodiments, in step S620, determining the gradient value of each pixel point of the sub-block includes the following processes.
[0160] For a 1.4*4 block, fill in one row / column of integer pixel points on the top, bottom, left, and right edges respectively to obtain a corresponding 6*6 block. Illustratively, the pixel values of integer pixel points filling the top, bottom, left and right edges of a 4*4 block are obtained from the reconstructed values of adjacent integer pixel points of a reference picture.
[0161] 2. Calculate the gradient value of each pixel point of the 4*4 sub-block based on the pixel value of each pixel point of the 6*6 block. For example, for any pixel point in a 4*4 sub-block, the gradient value of the pixel point (the horizontal component of the gradient value of the pixel point is mm_gradx, and the vertical component of the gradient value of the pixel point is mm_grady) can be determined based on Equation 4 and Equation 5.
[0162] Example 6 For example, in the above embodiment 5, for a 4*4 sub-block, filling one row / column of integer pixel points on the top, bottom, left, and right edges respectively includes the following embodiments.
[0163] As an example, integer pixel points to be filled in the top, bottom, left and right edges of a 4*4 subblock are selected based on whether the horizontal and vertical components of the subpixels of the predicted values of the pixel points of the 4*4 subblock are larger than half a pixel.
[0164] As shown in FIG. 7A, if both the horizontal and vertical components of the sub-pixel of the predicted value of a pixel point are larger than half a pixel, the top edge of the 4*4 sub-block is filled with the integer pixel point that is closest from above, the bottom edge of the 4*4 sub-block is filled with the next closest integer pixel point from below, the left edge of the 4*4 sub-block is filled with the integer pixel point that is closest from the left, and the right edge of the 4*4 sub-block is filled with the next closest integer pixel point from the right.
[0165] As shown in FIG. 7B, if the horizontal and vertical components of the sub-pixel of the predicted value of a pixel point are both smaller than half a pixel, the top edge of the 4*4 subblock is filled with the next closest integer pixel point from above, the bottom edge of the 4*4 subblock is filled with the closest integer pixel point from below, the left edge of the 4*4 subblock is filled with the next closest integer pixel point from the left, and the right edge of the 4*4 subblock is filled with the closest integer pixel point from the right.
[0166] As shown in FIG. 7C, if the horizontal component of the sub-pixel of the predicted value of a pixel point is greater than half a pixel and the vertical component is less than half a pixel, the top edge of the 4*4 sub-block is filled with the next closest integer pixel point from above, the bottom edge of the 4*4 sub-block is filled with the closest integer pixel point from below, the left edge of the 4*4 sub-block is filled with the closest integer pixel point on the left, and the right edge of the 4*4 sub-block is filled with the next closest integer pixel point on the right.
[0167] As shown in FIG. 7D, if the horizontal component of the sub-pixel of the predicted value of a pixel point is smaller than half a pixel and the vertical component is larger than half a pixel, the top edge of the 4*4 sub-block is filled with the integer pixel point that is closest from above, the bottom edge of the 4*4 sub-block is filled with the next closest integer pixel point from below, the left edge of the 4*4 sub-block is filled with the next closest integer pixel point from the left, and the right edge of the 4*4 sub-block is filled with the integer pixel point that is closest from the right.
[0168] For example, in FIG. 7A to FIG. 7D, the triangle is a prediction value of each pixel point of a 4*4 sub-block. , the circles are the reconstructed values of integer pixel points of the reference picture, and the shaded circles are the reconstructed values of integer pixel points in the selected reference picture to fill.
[0169] In addition, when the sub-pixel horizontal component of the predicted value of the pixel point is half a pixel, the process may be performed as if the sub-pixel horizontal component of the predicted value of the pixel point is larger than half a pixel, or the process may be performed as if the sub-pixel horizontal component of the predicted value of the pixel point is smaller than half a pixel.
[0170] Similarly, if the sub-pixel vertical component of the predicted value of a pixel point is half a pixel, it may be processed as if the sub-pixel vertical component of the predicted value of the pixel point was greater than half a pixel, or it may be processed as if the sub-pixel vertical component of the predicted value of the pixel point was less than half a pixel.
[0171] Example 7 As an example, in Example 6, the pixel values of the four corner points of the filling area are not used, so when filling one row / column of integer pixel points on the top, bottom, left, and right edges of a 4*4 sub-block, the four corner points are not filled, and instead, four points each on the top, bottom, left, and right (a total of 16 points) are filled.
[0172] Taking the case where both the horizontal and vertical components of the sub-pixel of the predicted value of a pixel point are larger than half a pixel as an example, a schematic diagram after filling one row / column on the top, bottom, left, right and bottom edges of a 4*4 sub-block is shown in FIG. 7E.
[0173] For other cases, similar results are obtained.
[0174] Example 8 In one example, in any one of the above embodiments, in the above step S630, determining the offset vector of each pixel point of the sub-block includes the following process.
[0175] 1. Calculate the affine parameters c (i.e., the first affine parameter above), d (i.e., the second affine parameter above), e (i.e., the third affine parameter above), and f (i.e., the fourth affine parameter above). For example, the affine parameters c, d, e, and f can be determined by Equations 1 and 2.
[0176] 2. Determine an offset vector of the first pixel point of the 4*4 sub-block according to the affine parameters c, d, e, f and the offset between the first pixel point of the 4*4 sub-block and the center position of the 4*4 sub-block. For the sake of illustration, the specified pixel point is the first pixel point.
[0177] Based on the coordinates of the first pixel point of the 4*4 sub-block and the coordinates of the center position of the 4*4 sub-block, an offset (assumed to be (Δx, Δy)) between the first pixel point of the 4*4 sub-block and the center position of the 4*4 sub-block is determined.
[0178] This allows the offset vector of the first pixel point of the 4*4 sub-block to be determined by the following formula:
[0179]
number
[0180] 3. Based on the offset vector of the first pixel point of the 4*4 sub-block, determine the offset vectors of the remaining pixel points of the 4*4 sub-block.
[0181] For example, after determining the offset vector of the first pixel point of the 4*4 sub-block according to Equation 6, the offset vectors of the remaining pixel points in the first row of the 4*4 sub-block can be determined by the following equation:
[0182]
number
[0183] For example, Δv x (w,0) is the horizontal component of the offset vector of the pixel point in the w+1th column and 1st row of the 4*4 subblock, and Δv y (w,0) is the vertical component of the offset vector of the pixel point in the w+1th column and 1st row of the 4*4 sub-block, where 1≦w≦3. After determining the offset vector of the pixel point in the 1st row of the 4*4 sub-block based on Equation 7, the offset vectors of the remaining pixel points in the 4*4 sub-block can be determined by the following equation:
[0184]
number
[0185] For example, Δv x (w,h) is the horizontal component of the offset vector of the pixel point in the w+1th column and h+1th row of the 4*4 subblock, and Δv y (w,h) is the vertical component of the offset vector of the pixel point in the w+1th column and h+1th row of the 4*4 subblock, where 1≦h≦3.
[0186] In addition, since the affine mode parameters and the offset between the pixel point at the same position in each sub-block and the center position of the sub-block are all the same, after calculating the offset vector of each pixel point in one sub-block, the remaining sub-blocks in the decoding block can use the offset vector of each pixel point of that sub-block.
[0187] Example 9 In one example, in Example 9, in order to ensure accuracy and improve hardware compatibility, when calculating the affine parameters c, d, e, and f, a "shift left then shift right" is used instead of division.
[0188] For c and d, first shift left by (MAX_CU_DEPTH-Log2[width]), then shift right by MAX_CU_DEPTH. For d and f in the 6-parameter model, first shift left by (MAX_CU_DEPTH-Log2[height]), then shift right by MAX_CU_DEPTH.
[0189] For example, width is the width of the current image block, height is the height of the current image block, and MAX_CU_DEPTH is the maximum image block width or height depth, which is 7 by default (i.e., the corresponding image block width or height is 128).
[0190] In one example, taking MAX_CU_DEPTH=7, c, d, e, and f can be determined by the following formula:
[0191]
number
[0192]
number
[0193] For a six-parameter model, this can be determined by the following formula:
[0194]
number
[0195]
number
[0196] Illustratively, << represents a shift to the left.
[0197] For the four-parameter model, d=-e and f=c.
[0198] Example 10 In one example, in order to improve the accuracy of the offset vectors, in any one of the above embodiments, when determining the offset vectors of each pixel point of a 4*4 sub-block, the offset vectors of each pixel point can be enlarged.
[0199] For example, in the process of calculating the offset vector, the offset vector of each pixel point of a 4*4 sub-block is enlarged by a factor of N1 (N1 is a positive integer, such as 4 or 8), and then the offset vector of the first pixel point of the 4*4 sub-block is determined by the following formula:
[0200]
number
[0201] After determining the offset vector of the first pixel point of the 4*4 sub-block according to Equation 13, the offset vectors of the remaining pixel points in the first row of the 4*4 sub-block can be determined by the following equation:
[0202]
number
[0203] After determining the offset vector of the pixel point in the first row of the 4*4 sub-block according to Equation 14, the offset vectors of the remaining pixel points in the 4*4 sub-block can be determined by the following equation:
[0204]
number
[0205] Based on Equations 13 to 15, after determining the offset vectors enlarged by N1 times for each pixel point of the 4*4 sub-block, the offset vectors enlarged by N1 times for each pixel point are reduced by N2 times.
[0206] Exemplarily, N2 may be the same as N1 or different from N1.
[0207] For example, if N1 is 4, N2 may be 4 or N2 may be 2 (i.e., the finally determined offset vector is the offset vector enlarged by 2 times).
[0208] In one example, the offset vector determined by the following equation is reduced.
[0209]
Number
[0210]
Number
[0211]
Number
[0212] Exemplarily, mvx is the horizontal component of the offset vector of a pixel point where there is a certain value, and mvy is the vertical component of the offset vector of the pixel point. >> represents shifting to the right, and << represents shifting to the left. mvx>=0 means that when mvx is 0 or more, set mvx = 1, and in the opposite case, set mvx = 0. mvy>=0 means that when mvy is 0 or more, set mvy = 1, and in the opposite case, set mvy = 0.
[0213] Exemplarily, limit mvx and mvy within the range of [-(1<<bdlimit), (1<<bdlimit)-1]. bdlimit is a value based on experience.
[0214] In one example, bdlimit=max(6,bitdepth-6), where bitdpth is the bit depth, i.e. the bit width required for the luminance values, typically 10 or 8.
[0215] In another example, bdlimit=max(5,bitdepth-7).
[0216] Example 11 In one example, in any one of the eighth to tenth embodiments, if the offset between the first pixel point of a 4*4 subblock and the center position of the 4*4 subblock is (-1.5, -1.5), then Δv x (0,0)=-1.5c-1.5d, Δv y (0,0)=-1.5e-1.5f, and the horizontal and vertical components of the offset vector scaled four times are Δv x (0,0)=-6c-6d, Δv y (0,0)=-6e-6f.
[0217] Example 12 In one example, in any one of the eighth to tenth embodiments, if the offset between the first pixel point of a 4*4 subblock and the center position of the 4*4 subblock is (-2, -2), then Δv x (0,0)=-2c-2d, Δv y (0,0)=-2e-2f, and the horizontal and vertical components of the offset vector scaled four times are Δv x (0,0)=-8c-8d, Δv y (0,0)=-8e-8f.
[0218] Example 13 In one example, in any one of the first to seventh embodiments, the offset vector of each pixel point of the sub-block can be determined by the following formula:
[0219]
number
[0220]
number
[0221] Illustratively, N3 is the magnification factor of the offset vector, e.g., 4. posOffsetX=N4*c+N4*d, posOffsetY=N4*e+N4*f, where N4 is 6 or 8.
[0222] Example 14
[0223] In one example, in any one of the above embodiments, determining the predicted compensation value of each pixel point of the sub-block according to the gradient value and offset vector of each pixel point of the sub-block can be realized by the following formula:
[0224]
number
[0225] For example, ΔI(i,j) is the predicted compensation value of the pixel point in the i-th column and j-th row of the subblock, and g x (i,j) is the horizontal component of the gradient value of the pixel point at the i-th column and j-th row of the sub-block, and g v (i,j) is the vertical component of the gradient value of the pixel point at the i-th column and j-th row of the sub-block, and Δv x (i,j) is the horizontal component of the offset vector of the pixel point in the i-th column and j-th row of the subblock, and Δv v (i,j) is the vertical component of the offset vector of the pixel point in the i-th column and j-th row of the sub-block.
[0226] Example 15 In one example, in any one of the above embodiments, if the decoded block uses a unidirectional prediction mode, the unidirectional (forward or backward) prediction compensation adjustment can be realized by the following formula:
[0227]
number
[0228]
number
[0229]
number
[0230] For example, mm_dI2 is a unidirectional prediction compensation value of a pixel point, mm_dI3 is the final predicted value of the pixel point in the corresponding direction, mvx is the horizontal component of the unidirectional offset vector of the pixel point, mvy is the vertical component of the offset vector of the pixel point in the corresponding direction, mm_gradx is the horizontal component of the gradient value of the pixel point in the corresponding direction, mm_grady is the vertical component of the gradient value of the pixel point in the corresponding direction, mm_src is the predicted value of the pixel point in the corresponding direction, and mm_w is the weight value of the pixel point in the corresponding direction, which is 1 in the unidirectional prediction mode.
[0231] Example 16 In one example, in any one of the above embodiments, if the decoded block uses a bidirectional prediction mode, forward and backward prediction compensation adjustments are respectively implemented to implement the prediction compensation adjustment of the decoded block.
[0232] For example, forward prediction compensation adjustment can be realized by the following equation:
[0233]
number
[0234]
number
[0235]
number
[0236] For example, mm_dI02 is the forward prediction compensation value of a pixel point, mm_dI03 is the final forward prediction value of the pixel point, mvx0 is the horizontal component of the forward offset vector of the pixel point, mvy0 is the vertical component of the forward offset vector of the pixel point, mm_gradx0 is the horizontal component of the forward gradient value of the pixel point, mm_grady0 is the vertical component of the forward gradient value of the pixel point, mm_src0 is the forward prediction value of the pixel point, and mm_w0 is the forward weight value of the pixel point.
[0237] For example, the backward prediction compensation adjustment can be realized by the following equation:
[0238]
number
[0239]
number
[0240]
number
[0241] For example, mm_dI12 is a backward prediction compensation value of a pixel point, mm_dI13 is a backward final prediction value of the pixel point, mvx1 is the horizontal component of the backward offset vector of the pixel point, mvy1 is the vertical component of the backward offset vector of the pixel point, mm_gradx1 is the horizontal component of the backward gradient value of the pixel point, mm_grady1 is the vertical component of the backward gradient value of the pixel point, mm_src1 is a backward prediction value of the pixel point, and mm_w1 is a backward weight value of the pixel point.
[0242] Based on the final forward and backward predicted values of each pixel point of the decoded block, a weighting process (the sum of the final forward predicted value and the final backward predicted value is divided by the sum of the forward weight value and the backward weight value) is completed to obtain the final predicted value of each pixel point of the decoded block.
[0243] Example 17 8 is a flow chart of a method for selecting a candidate motion vector according to an embodiment of the present application. As shown in FIG. 8, the method for selecting a candidate motion vector includes the following steps:
[0244] In step S800, determine whether the current image block satisfies the condition for using a temporal domain reference picture, if so, proceed to step S810, if not, proceed to step S820.
[0245] In step S810, the motion vector of the time-domain reference picture is allowed to be used for the encoding / decoding of the current image block.
[0246] In step S820, reject using the motion vector of the time-domain reference picture for encoding / decoding the current image block.
[0247] In an embodiment of the present application, considering that in some specific situations, the probability that a motion vector of a time-domain reference picture will be finally selected is low, in order to improve codec performance, a condition on whether to allow the use of a motion vector of a time-domain reference picture (referred to as a condition for using a time-domain reference picture in the present application) is set in advance, so as to avoid selecting a motion vector of a time-domain reference picture as a candidate motion vector when a motion vector of a time-domain reference picture is not applied.
[0248] Correspondingly, when encoding / decoding a current image block, it is possible to determine whether the current image block meets the conditions for using a time-domain reference picture, and determine whether to allow the use of a motion vector of the time-domain reference picture for encoding / decoding the current image block, or to reject the use of a motion vector of the time-domain reference picture for encoding / decoding the current image block.
[0249] In one example, the above condition for using the time-domain reference picture is: The size of the reference picture is the same as the size of the picture to which the current image block belongs.
[0250] For example, when the size of the reference picture is different from the size of the picture to which the current image block belongs, the probability that the motion vector of the time-domain reference picture is finally selected is low. Therefore, in order to improve the codec performance, the size of the reference picture may be changed to the size of the current picture. The condition for using a time-domain reference picture is that the size of the reference picture is the same as the size of the picture to which the current image block belongs, i.e., if the size of the reference picture is the same as the size of the picture to which the current image block belongs, the motion vector of the time-domain reference picture is allowed to be used for encoding / decoding the current image block, and if the size of the reference picture is different from the size of the picture to which the current image block belongs, the motion vector of the time-domain reference picture is refused to be used for encoding / decoding the current image block.
[0251] In another example, the above-mentioned time-domain reference picture usage condition further includes: This includes that the reference picture is not a long-term reference picture (LTRP).
[0252] Exemplarily, a long-term reference picture is a reference picture that is close to the picture to which the current image block belongs, and generally, a long-term reference picture is far away from the picture to which the current image block belongs.
[0253] For example, considering that the probability that the motion vector of the time domain reference picture will finally be selected is low when the size of the reference picture is the same as the size of the picture to which the current image block belongs but the reference picture is a long-term picture, the condition for using the time domain reference picture is that the reference picture is not a long-term reference picture, i.e., when the size of the reference picture is the same as the size of the picture to which the current image block belongs and the reference picture is not a long-term reference picture, the motion vector of the time domain reference picture is allowed to be used for encoding / decoding the current image block, and when the size of the reference picture is different from the size of the picture to which the current image block belongs and / or the reference picture is a long-term reference picture, the motion vector of the time domain reference picture is refused to be used for encoding / decoding the current image block.
[0254] In one possible embodiment, after allowing to use the motion vectors of the time-domain reference picture for the encoding / decoding of the current image block, further Determining whether the current image block satisfies a condition for using scaling of motion vectors of a time-domain reference picture; If yes, allowing the motion vector of the time-domain reference picture to be scaled for encoding / decoding the current image block; If not, refusing to scale the motion vector of the time-domain reference picture for encoding / decoding the current image block.
[0255] For example, considering that in some specific situations, the probability that the motion vector of the scaled time domain reference picture will be finally selected is low, in order to improve codec performance, a condition on whether to allow the use of the motion vector of the scaled time domain reference picture (referred to in this application as a condition for using the scaling of the motion vector of the time domain reference picture) is set in advance, so as to avoid the motion vector of the scaled time domain reference picture being selected as a candidate motion vector when the motion vector of the scaled time domain reference picture is not applied.
[0256] In one example, the condition for using the motion vector scaling of the time-domain reference picture further includes: These include that the size of the reference picture is the same as the size of the picture to which the current image block belongs, and that the reference picture is not a long-term reference picture.
[0257] For example, when the size of the reference picture is different from the size of the picture to which the current image block belongs and / or the reference picture is a long-term picture, the time after scaling Considering the low probability that the motion vector of the regional reference picture will ultimately be selected, the conditions for using scaling of the motion vector of the temporal domain reference picture are that the size of the reference picture is the same as the size of the picture to which the current image block belongs, and that the reference picture is not a long-term reference picture.
[0258] If the size of the reference picture is the same as the size of the picture to which the current image block belongs and the reference picture is not a long-term reference picture, the motion vector of the scaled time-domain reference picture is allowed to be used for encoding / decoding of the current image block, and if the size of the reference picture is different from the size of the picture to which the current image block belongs and / or the reference picture is a long-term picture, the motion vector of the scaled time-domain reference picture is refused to be used for encoding / decoding of the current image block.
[0259] In one possible embodiment, after determining whether the current image block satisfies the condition for using a time-domain reference picture, further comprising: Allowing the use of a first type of prediction mode when a condition for using a time domain reference picture is satisfied; and rejecting the use of the first type of prediction mode and the second type of prediction mode when a condition for using the time-domain reference picture is not met.
[0260] In one example, the first type of prediction modes include modes related to time domain motion vectors, such as TMVP mode, BDOF mode, DMVR mode, etc., and the second type of prediction modes include modes related to scaling of time domain motion vectors, such as SMVD mode.
[0261] In addition, in one example, if the condition for using a temporal domain reference picture is satisfied, the second type of prediction mode may be allowed to be used.
[0262] In one possible embodiment, after determining whether the condition for using the time-domain reference picture is satisfied, further comprising: permitting the use of a first type of prediction mode and a second type of prediction mode when a condition for using scaling of a motion vector of a time domain reference picture is satisfied; and rejecting the use of the second type of prediction mode when a condition for using scaling of the motion vector of the time-domain reference picture is not met.
[0263] In one example, if the condition for using the scaling of the motion vector of the time-domain reference picture is not met, the use of the first type of prediction mode may be further rejected.
[0264] Example 18 9 is a flow chart of a method for selecting a candidate motion vector according to an embodiment of the present application. As shown in FIG. 9, the method for selecting a candidate motion vector includes the following steps:
[0265] In step S900, judge whether the current image block satisfies the use condition of motion vector scaling of the temporal domain reference picture; if yes, proceed to step S910; if not, proceed to step S920.
[0266] In step S910, the motion vector of the time-domain reference picture is allowed to be scaled for the encoding / decoding of the current image block.
[0267] In step S920, reject scaling the motion vector of the time-domain reference picture for encoding / decoding the current image block.
[0268] In the embodiment of the present application, taking into consideration that in some specific situations, the probability that the motion vector of the scaled time domain reference picture will be finally selected is low, in order to improve codec performance, a condition on whether to allow the use of the motion vector of the scaled time domain reference picture (referred to in the present application as a condition for using the scaling of the motion vector of the time domain reference picture) is set in advance, so as to avoid the motion vector of the scaled time domain reference picture being selected as a candidate motion vector when the motion vector of the scaled time domain reference picture is not applied.
[0269] In one example, the above-mentioned condition for using the motion vector scaling of the time-domain reference picture further includes: These include that the size of the reference picture is the same as the size of the picture to which the current image block belongs, and that the reference picture is not a long-term reference picture.
[0270] For example, considering that when the size of the reference picture is different from the size of the picture to which the current image block belongs and / or the reference picture is a long-term picture, the probability that the motion vector of the time-domain reference picture after scaling will be finally selected is low, the conditions for using scaling of the motion vector of the time-domain reference picture are that the size of the reference picture is the same as the size of the picture to which the current image block belongs and that the reference picture is not a long-term reference picture.
[0271] If the size of the reference picture is the same as the size of the picture to which the current image block belongs and the reference picture is not a long-term reference picture, the motion vector of the scaled time-domain reference picture is allowed to be used for encoding / decoding of the current image block, and if the size of the reference picture is different from the size of the picture to which the current image block belongs and / or the reference picture is a long-term picture, the motion vector of the scaled time-domain reference picture is rejected to be used for encoding / decoding of the current image block.
[0272] In one possible embodiment, after determining whether the condition for using the time-domain reference picture is satisfied, further comprising: permitting the use of a first type of prediction mode and a second type of prediction mode when a condition for using scaling of a motion vector of a time domain reference picture is satisfied; and refusing to use the second type of prediction mode if a condition for using scaling of the motion vector of the time-domain reference picture is not met.
[0273] In one example, the first type of prediction modes include modes related to time domain motion vectors, such as TMVP mode, BDOF mode, DMVR mode, etc., and the second type of prediction modes include modes related to scaling of time domain motion vectors, such as SMVD mode.
[0274] In one example, if the condition for using the scaling of the motion vector of the time-domain reference picture is not met, the use of the first type of prediction mode may be further rejected.
[0275] Example 19
[0276] 10 is a flow chart of a prediction mode selection method according to an embodiment of the present application. As shown in FIG. 10, the prediction mode selection method includes the following steps:
[0277] In step S1000, it is determined whether the current image block satisfies the application condition for the designated prediction mode. If it does, the process proceeds to step S1010. If it does not, the process proceeds to step S1020. Proceed to step S1020.
[0278] In step S1010, application of the designated prediction mode is permitted.
[0279] In step S1020, application of the designated prediction mode is rejected.
[0280] In an embodiment of the present application, taking into consideration that in some specific situations, use of a specified prediction mode may degrade codec performance, in order to improve codec performance, conditions for whether or not to allow application of a specified prediction mode (referred to in the present application as application conditions for a specified prediction mode) are set in advance so as to avoid setting the specified prediction mode as a candidate prediction mode when the specified prediction mode is not applied.
[0281] In one possible embodiment, the specified prediction modes include prediction modes of a first type.
[0282] Exemplarily, the first type of prediction mode includes a mode related to a time domain motion vector, such as a TMVP mode, a BDOF mode, a DMVR mode, etc.
[0283] In one example, the application condition of the specified prediction mode is The size of the reference picture is the same as the size of the picture to which the current image block belongs.
[0284] For example, considering that the use of the first type of prediction mode reduces codec performance when the size of the reference picture is different from the size of the picture to which the current image block belongs, in order to improve codec performance, the condition for applying the designated prediction mode is that the size of the reference picture is the same as the size of the picture to which the current image block belongs, i.e., if the size of the reference picture is the same as the size of the picture to which the current image block belongs, application of the designated prediction mode is allowed, and if the size of the reference picture is different from the size of the picture to which the current image block belongs, application of the designated prediction mode is rejected.
[0285] In another example, the application condition of the specified prediction mode may further include: Including that the reference picture is not a long-term reference picture.
[0286] For example, considering that the use of the first type of prediction mode degrades codec performance when the size of the reference picture is the same as the size of the picture to which the current image block belongs but the reference picture is a long-term reference picture, the condition for applying the designated prediction mode is that the reference picture is not a long-term reference picture, i.e., if the size of the reference picture is the same as the size of the picture to which the current image block belongs and the reference picture is not a long-term reference picture, application of the designated prediction mode is allowed, and if the size of the reference picture is different from the size of the picture to which the current image block belongs and / or the reference picture is a long-term reference picture, application of the designated prediction mode is rejected.
[0287] In another possible embodiment, the specified prediction modes include a first type of prediction mode and a second type of prediction mode.
[0288] Exemplarily, the second type of prediction mode includes a mode related to scaling a time domain motion vector, for example, an SMVD mode.
[0289] In one example, the application condition of the specified prediction mode is These include that the size of the reference picture is the same as the size of the picture to which the current image block belongs, and that the reference picture is not a long-term reference picture.
[0290] For example, in consideration of the fact that the use of the first type of prediction mode and the second type of prediction mode may degrade codec performance when the size of the reference picture is different from the size of the picture to which the current image block belongs and / or the reference picture is a long-term picture, the conditions for applying the specified prediction mode are that the size of the reference picture is the same as the size of the picture to which the current image block belongs and that the reference picture is not a long-term reference picture.
[0291] If the size of the reference picture is the same as the size of the picture to which the current image block belongs and the reference picture is not a long-term reference picture, application of the specified prediction mode is allowed, and if the size of the reference picture is different from the size of the picture to which the current image block belongs and / or the reference picture is a long-term reference picture, application of the specified prediction mode is rejected.
[0292] In a possible embodiment, the specified prediction mode includes a second type of prediction mode.
[0293] In one example, the application condition of the specified prediction mode is These include that the size of the reference picture is the same as the size of the picture to which the current image block belongs, and that the reference picture is not a long-term reference picture.
[0294] For example, in consideration of the fact that the use of the second type of prediction mode may degrade codec performance when the size of the reference picture is different from the size of the picture to which the current image block belongs and / or the reference picture is a long-term picture, the conditions for applying the specified prediction mode are that the size of the reference picture is the same as the size of the picture to which the current image block belongs and that the reference picture is not a long-term reference picture.
[0295] If the size of the reference picture is the same as the size of the picture to which the current image block belongs and the reference picture is not a long-term reference picture, application of the specified prediction mode is allowed, and if the size of the reference picture is different from the size of the picture to which the current image block belongs and / or the reference picture is a long-term reference picture, application of the specified prediction mode is rejected.
[0296] Example 20 11 is a flow chart of a prediction mode selection method according to an embodiment of the present application. As shown in FIG. 11, the prediction mode selection method includes the following steps:
[0297] In step S1100, it is determined whether the current image block satisfies the application condition of the DMVR mode. If so, the process proceeds to step S1110; if not, the process proceeds to step S1120.
[0298] In step S1110, application of the DMVR mode is permitted.
[0299] In step S1120, application of the DMVR mode is rejected.
[0300] In the embodiments of the present application, taking into consideration that in some specific situations, codec efficiency is low when using DMVR mode, in order to improve codec performance, conditions for whether or not to allow application of DMVR mode (referred to in the present application as application conditions for DMVR mode) are set in advance so as to avoid executing DMVR mode when DMVR mode is not applied.
[0301] Correspondingly, when decoding the current image block, it is determined whether the current image block satisfies the conditions for applying the DMVR mode, and if so, the application of the DMVR mode is permitted; if not, the application of the DMVR mode is rejected.
[0302] In one example, the application conditions of the above DMVR mode are: Includes the current mode being regular merge / skip mode.
[0303] Exemplarily, the regular merge / skip modes are a general merge mode and a general skip mode. In the general merge mode or the general skip mode, one motion information is selected from the candidate motion information list, and a prediction value of the current image block is generated based on the selected motion information.
[0304] The candidate motion information list includes spatial domain adjacent candidate motion information, temporal domain adjacent candidate motion information, spatial domain non-adjacent candidate motion information, motion information obtained by combining existing motion information, and / or default motion information.
[0305] For example, if it is determined that the current image block is in regular merge / skip mode, the application of the DMVR mode is permitted, and conversely, the application of the DMVR mode is rejected.
[0306] In another example, the application conditions of the above DMVR mode are: the value of the sequence level control DMVR mode switch is a first value, and the value of the picture header control DMVR mode switch is a second value; The current mode is regular merge / skip mode, The current image block uses a bidirectional prediction mode, and the display order of one of the two reference pictures is located before the picture to which the current image block belongs, and the display order of the other is located after the picture to which the current image block belongs, and the distances of the two reference pictures to the picture to which the current image block belongs are the same; The weights of the two reference pictures are the same, The size of the current image block satisfies the constraints, and The length and width of the two reference pictures are respectively the same as the length and width of the picture to which the current image block belongs.
[0307] In one example, the first value is 1.
[0308] In one example, the size of the current image block satisfies the restriction conditions including that the width of the current image block is greater than or equal to 8, the height of the current image block is greater than or equal to 8, and the area of the current image block is greater than or equal to 128.
[0309] In the embodiment of the present application, after determining the predicted value of each pixel point of the current image block, the predicted compensation value of each pixel point of the current image block is determined according to the gradient value and offset vector of each pixel point of the current image block.Furthermore, the final predicted value of each pixel point of the current image block is determined according to the predicted value and the predicted compensation value of each pixel point of the current image block.The prediction compensation adjustment is not limited to the image block using bidirectional prediction mode, and is not limited to the image block in which the motion vector of each sub-block is the same as the motion vector of each pixel point of the corresponding sub-block.The application scope of the prediction compensation adjustment is expanded.
[0310] Having described the method according to the present application, the apparatus according to the present application will now be described.
[0311] FIG. 12 is a schematic diagram of the configuration of a codec device according to the present application. As shown in FIG. The deck device is a first determining means 1210 configured to determine a predicted value for each pixel point of the current image block; second determining means 1220 configured to determine a gradient value of each pixel point of the current image block based on the predicted value of each pixel point of the current image block; a third determining means 1230 configured to determine an offset vector of each pixel point of the current image block; a fourth determining means 1240 configured to determine a predicted compensation value of each pixel point of the current image block according to a gradient value and an offset vector of each pixel point of the current image block; The method further comprises a fifth determining means 1250 configured to determine a final predicted value for each pixel point of the current image block according to the predicted value and the predicted compensation value of each pixel point of the current image block.
[0312] In one possible embodiment, if the current image block uses a unidirectional prediction mode, The first determining means 1210 is specifically configured to determine a unidirectional predicted value of each pixel point of the current image block; The second determining means 1220 is specifically configured to determine, according to a unidirectional predicted value of each pixel point of the current image block, a gradient value of the corresponding direction of each pixel point of the current image block; The third determining means 1230 is specifically configured to determine an offset vector in the corresponding direction of each pixel point of the current image block; The fourth determining means 1240 is specifically configured to determine a predicted compensation value of each pixel point of the current image block in a corresponding direction according to a gradient value and an offset vector of each pixel point of the current image block in a corresponding direction; The fifth determination means 1250 is specifically configured to determine a final predicted value of each pixel point of the current image block in the corresponding direction according to the predicted value and the predicted compensation value of each pixel point of the current image block in the corresponding direction.
[0313] In one possible embodiment, when the current image block uses a bi-directional prediction mode, The first determining means 1210 is specifically configured to respectively determine forward and backward predicted values of each pixel point of the current image block; The second determining means 1220 is specifically configured to respectively determine forward and backward gradient values of each pixel point of the current image block according to forward and backward predicted values of each pixel point of the current image block; The third determining means 1230 is specifically configured to respectively determine the forward and backward offset vectors of each pixel point of the current image block; The fourth determining means 1240 is specifically configured to respectively determine forward and backward prediction compensation values of each pixel point of the current image block according to forward and backward gradient values and offset vectors of each pixel point of the current image block; The fifth determination means 1250 is specifically configured to respectively determine, based on the forward and backward predicted values and prediction compensation value of each pixel point of the current image block, the forward and backward final predicted values of each pixel point of the current image block, and determine, based on the forward and backward final predicted values of each pixel point of the current image block, the final predicted value of each pixel point of the current image block.
[0314] In one possible embodiment, the third determining means 1230 specifically: For any one of the sub-blocks of the current image block, determine an offset vector of a designated pixel point in the sub-block; Based on the offset vector of the designated pixel point in the sub-block, The offset vectors of the remaining pixel points in the block are determined.
[0315] In a possible embodiment, the third determination means 1230 is specifically configured to determine a vector offset of the designated pixel point of the sub-block based on an offset between the designated pixel point in the sub-block and a center position of the sub-block, and an affine parameter.
[0316] In one possible embodiment, the third determining means 1230 specifically: determining a horizontal component of a vector offset of the designated pixel point of the sub-block based on a horizontal offset between the designated pixel point in the sub-block and a center position of the sub-block, a vertical offset between the designated pixel point in the sub-block and a center position of the sub-block, a first affine parameter, and a second affine parameter; The vertical component of the vector offset of the designated pixel point of the sub-block is determined based on a horizontal offset between the designated pixel point in the sub-block and a center position of the sub-block, a vertical offset between the designated pixel point in the sub-block and a center position of the sub-block, a third affine parameter, and a fourth affine parameter.
[0317] In one possible embodiment, for a four-parameter affine model, the first affine parameter is the same as the fourth affine parameter, both of which are a ratio of a first numerical value to the width of the sub-block, the second affine parameter is the opposite of the third affine parameter, both of which are a ratio of a second numerical value to the width of the sub-block, The first value is the difference between the horizontal component of the motion vector of the top right control point of the sub-block and the horizontal component of the motion vector of the top left control point, and the second value is the difference between the vertical component of the motion vector of the top right control point of the sub-block and the vertical component of the motion vector of the top left control point.
[0318] In one possible embodiment, for a six-parameter affine model, the first affine parameter is a ratio of the first numerical value to a width of the sub-block, the second affine parameter is a ratio of a third numerical value to a height of the sub-block, the third affine parameter is a ratio of the second numerical value to a width of the sub-block, and the fourth affine parameter is a ratio of a fourth numerical value to a height of the sub-block; The first numerical value is the difference between the horizontal component of the motion vector of the top right control point of the sub-block and the horizontal component of the motion vector of the top left control point, the second numerical value is the difference between the vertical component of the motion vector of the top right control point of the sub-block and the vertical component of the motion vector of the top left control point, the third numerical value is the difference between the horizontal component of the motion vector of the bottom left control point of the sub-block and the horizontal component of the motion vector of the top left control point, and the fourth numerical value is the difference between the vertical component of the motion vector of the bottom left control point of the sub-block and the vertical component of the motion vector of the top left control point.
[0319] In one possible embodiment, the third determining means 1230 specifically: determining horizontal components of offset vectors of remaining pixel points in the same row as the designated pixel point in the sub-block based on the horizontal component of the offset vector of the designated pixel point in the sub-block and the first affine parameter; determining vertical components of offset vectors of remaining pixel points in the same row as the designated pixel point in the sub-block based on the vertical component of the offset vector of the designated pixel point in the sub-block and the third affine parameter; For any one pixel point in the row in which the specified pixel point is located in the sub-block, the horizontal components of the offset vectors of the remaining pixel points in the same column as the pixel point in the sub-block are determined based on the horizontal component of the offset vector of the pixel point and the second affine parameter, and the vertical components of the offset vectors of the remaining pixel points in the same column as the pixel point in the sub-block are determined based on the vertical component of the offset vector of the pixel point and the fourth affine parameter.
[0320] In one possible embodiment, the second determining means 1220 specifically determines, for any sub-block of the current image block, to fill N rows / columns of integer pixel points on the top, bottom, left, right and bottom edges of the sub-block, respectively, where N is a positive integer; The method is configured to determine a gradient value for each pixel point of the sub-block based on the predicted value of each pixel point of the sub-block and the pixel values of the filled integer pixel points.
[0321] In one possible embodiment, the second determining means 1220 specifically: determining a nearest integer pixel block of the same size to the sub-block in the reference picture; The pixel values of integer pixel points in the N rows / columns adjacent to the integer pixel block are set as filling values for the top, bottom, left, and right edges of the sub-block.
[0322] In one possible embodiment, the second determining means 1220 specifically: The integer pixel block is configured to be filled with N rows / columns of integer pixel points directly above, directly below, directly to the left, and directly to the right, respectively.
[0323] In one possible embodiment, when N=1, the second determining means 1220 specifically includes the following: if the sub-pixel vertical component of the predicted value of the pixel point in the sub-block is greater than half a pixel, fill the upper edge of the sub-block with the integer pixel point closest to the upper side, and fill the lower edge of the sub-block with the integer pixel point next closest to the lower side; if the sub-pixel vertical component of the prediction of the pixel point in the sub-block is less than half a pixel, filling the top edge of the sub-block with the next nearest integer pixel point above, and filling the bottom edge of the sub-block with the nearest integer pixel point below; if the sub-pixel horizontal component of the prediction of the pixel point in the sub-block is greater than half a pixel, filling the left edge of the sub-block with the integer pixel point closest to the left and filling the right edge of the sub-block with the integer pixel point next closest to the right; If the sub-pixel horizontal component of the predicted value of a pixel point in the sub-block is smaller than half a pixel, the left edge of the sub-block is filled with the next nearest integer pixel point on the left, and the right edge of the sub-block is filled with the nearest integer pixel point on the right.
[0324] In one possible embodiment, the second determining means 1220 specifically: For any one pixel point in the sub-block, the horizontal component of the gradient value of the pixel point is determined based on the pixel values of the N adjacent pixel points to the left of the pixel point and the pixel values of the N adjacent pixel points to the right of the pixel point, and the vertical component of the gradient value of the pixel point is determined based on the pixel values of the N adjacent pixel points above the pixel point and the pixel values of the N adjacent pixel points below the pixel point.
[0325] In one possible embodiment, the second determining means 1220 further comprises: determining whether the current image block satisfies a prediction compensation adjustment application condition; If so, it is configured to perform a step of determining a gradient value for each pixel point of the current image block based on the predicted value for each pixel point of the current image block.
[0326] In one possible embodiment, the prediction compensation adjustment application condition is that the current image block uses a specified prediction mode, and that the motion vector of a sub-block is supported by the specified prediction mode. This includes the fact that the motion vectors of each pixel point in the block are not completely homologous.
[0327] In one possible embodiment, the prediction compensation adjustment application condition further comprises that the current predicting component is a luminance component.
[0328] In one possible embodiment, the prediction compensation adjustment application condition further includes that the currently predicted component is a chrominance component.
[0329] In one possible embodiment, the specified prediction mode includes an affine motion mode.
[0330] 13 is a schematic diagram of a hardware configuration of an encoding device according to an embodiment of the present application. The encoding device includes a processor 1301 and a machine-readable storage medium 1302 that stores machine-executable instructions. The processor 1301 and the machine-readable storage medium 1302 can communicate with each other via a system bus 1303. The processor 1301 can execute the above-described codec method by reading and executing the machine-executable instructions corresponding to the codec control logic in the machine-readable storage medium 1302.
[0331] The machine-readable storage medium 1302 of the present disclosure may be any electronic, magnetic, optical, or other physical storage device that contains or stores information, such as executable instructions or data. For example, the machine-readable storage medium may be a random access memory (RAM), a volatile memory, a non-volatile memory, a flash memory, a storage drive (e.g., a hard drive), a solid-state drive, any type of storage disk (e.g., an optical disk, a DVD, etc.), or a similar storage medium, or a combination thereof.
[0332] 14 is a schematic diagram of a hardware configuration of a decoding device according to an embodiment of the present application. The decoding device includes a processor 1401 and a machine-readable storage medium 1402 storing machine-executable instructions. The processor 1401 and the machine-readable storage medium 1402 can communicate with each other via a system bus 1403. The processor 1401 can execute the codec method described above by reading and executing the machine-executable instructions corresponding to the codec control logic in the machine-readable storage medium 1402.
[0333] The machine-readable storage medium 1402 of the present disclosure may be any electronic, magnetic, optical, or other physical storage device that contains or stores information, such as executable instructions or data. For example, the machine-readable storage medium may be RAM, volatile memory, non-volatile memory, flash memory, a storage drive (e.g., a hard drive), a solid-state drive, any type of storage disk (e.g., optical disk, DVD, etc.), or a similar readable storage medium, or a combination thereof.
[0334] It should be noted that, in this specification, relational terms such as first, second, etc., are merely used to distinguish one element or process from another element or process, and do not necessarily require or imply any such actual relationship or order between the elements or processes. Also, the terms "comprise", "comprises", or other variations imply a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a set of elements not only includes those elements, but also further includes other elements not expressly listed, or further includes the inherent elements of such process, method, article, or apparatus. Unless otherwise specifically limited, an element defined by the phrase "comprises a ..." does not exclude that the process, method, article, or apparatus that includes the element further includes other identical elements.
[0335] The above are merely preferred embodiments of the present application, and are not intended to limit the present application. Any amendments, equivalent replacements, improvements, etc. made within the scope of the principles are included in the scope of the claims of this application.
Claims
1. Determining whether the current image block satisfies the application condition of a bidirectional optical flow mode; If satisfied, allowing the application of bidirectional optical flow mode; If not, refusing to apply the bidirectional optical flow mode; When the current image block applies a bidirectional optical flow mode, the application conditions satisfied by the current image block include: a size of a reference picture of the current image block is the same as a size of a picture to which the current image block belongs, and the reference picture is not a long-term reference picture; The current image block is any one of prediction units, and the prediction unit is obtained by dividing the current image block into quad tree partitions, horizontal binary tree partitions, vertical binary tree partitions, horizontal ternary tree partitions, and vertical ternary tree partitions, or by any combination thereof; If the current image block is applied to a bidirectional optical flow mode, the current image block is decoded in a bidirectional optical flow mode, the decoding step including: determining a predicted value for each pixel point of the current image block; determining a gradient value for each pixel point of the current image block based on the predicted value for each pixel point of the current image block; determining an offset vector for each pixel point of the current image block; determining a predicted compensation value of each pixel point of the current image block according to the gradient value and the offset vector of each pixel point of the current image block; determining a final predicted value of each pixel point of the current image block according to the predicted value and the predicted compensation value of each pixel point of the current image block; 13. A decoding method comprising:
2. The current image block is a bidirectionally predicted block, and the step of decoding the current image block in a bidirectional optical flow mode includes: respectively determining forward and backward gradient values of each pixel point of the current image block according to the motion information of the current image block; determining forward and backward offset vectors of each pixel point of the current image block; respectively determining forward and backward prediction compensation values of each pixel point of the current image block according to the forward and backward gradient values of each pixel point of the current image block and the forward and backward offset vectors of each pixel point of the current image block; According to the forward and backward predicted values of each pixel point of the current image block and the forward and backward predicted compensation values of each pixel point of the current image block, respectively determining the forward and backward final predicted values of each pixel point of the current image block; determining a final predicted value of each pixel point of the current image block according to the final predicted values of the forward and backward directions of each pixel point of the current image block; 2. The method according to claim 1 .
3. The current image block includes a plurality of 4*4 sub-blocks, and the process of decoding each of the 4*4 sub-blocks includes: determining a predicted value for each pixel point in the 4*4 sub-block; According to the prediction value of each pixel point in the 4*4 sub-block, determining a gradient value of each pixel point in the 4*4 sub-block and an offset vector of each pixel point in the 4*4 sub-block; determining a predicted compensation value of each pixel point in the 4*4 sub-block according to the gradient value and offset vector of each pixel point in the 4*4 sub-block; determining a final predicted value of the 4*4 sub-block according to the predicted value of each pixel point in the 4*4 sub-block and the predicted compensation value of each pixel point in the 4*4 sub-block; 3. The method according to claim 1 or 2, characterized in that
4. refusing to apply a bidirectional optical flow mode to a current image block if the size of a reference picture of the current image block is different from the size of a picture to which the current image block belongs; rejecting the application of a bidirectional optical flow mode to the current image block if two reference pictures of the current image block are long-term reference pictures; 2. The method according to claim 1 .
5. The shape of the prediction unit is rectangular or square.
2. The method according to claim 1 .
6. A decoding device comprising a processor and a machine-readable storage medium, the machine-readable storage medium stores machine-executable instructions executable by the processor; The processor executes the machine-executable instructions, Determining whether the current image block satisfies the application condition of a bidirectional optical flow mode; If satisfied, allowing the application of bidirectional optical flow mode; If not satisfied, rejecting the application of the bidirectional optical flow mode; When the current image block applies a bidirectional optical flow mode, the application conditions satisfied by the current image block include: a size of a reference picture of the current image block is the same as a size of a picture to which the current image block belongs, and the reference picture is not a long-term reference picture; The current image block is any one of prediction units, and the prediction unit is obtained by dividing the current image block into quad tree partitions, horizontal binary tree partitions, vertical binary tree partitions, horizontal ternary tree partitions, and vertical ternary tree partitions, or by any combination thereof; If the current image block applies a bidirectional optical flow mode, the processor executes machine-executable instructions to perform decoding in a bidirectional optical flow mode for the current image block; The decoding process is as follows: determining a predicted value for each pixel point of the current image block; determining a gradient value for each pixel point of the current image block based on the predicted value for each pixel point of the current image block; determining an offset vector for each pixel point of the current image block; determining a predicted compensation value of each pixel point of the current image block according to the gradient value and the offset vector of each pixel point of the current image block; determining a final predicted value of each pixel point of the current image block according to the predicted value and the predicted compensation value of each pixel point of the current image block; A decoding device comprising:
7. The current image block is a bidirectionally predicted block, and the step of decoding the current image block in a bidirectional optical flow mode includes: respectively determining forward and backward gradient values of each pixel point of the current image block according to the motion information of the current image block; determining forward and backward offset vectors of each pixel point of the current image block; respectively determining forward and backward prediction compensation values of each pixel point of the current image block according to the forward and backward gradient values of each pixel point of the current image block and the forward and backward offset vectors of each pixel point of the current image block; According to the forward and backward predicted values of each pixel point of the current image block and the forward and backward predicted compensation values of each pixel point of the current image block, respectively determining the forward and backward final predicted values of each pixel point of the current image block; determining a final predicted value of each pixel point of the current image block according to the final predicted values of the forward and backward directions of each pixel point of the current image block; A decoding device according to claim 6, characterized in that it is
8. The current image block includes a plurality of 4*4 sub-blocks, and the process of decoding each of the 4*4 sub-blocks includes: determining a predicted value for each pixel point in the 4*4 sub-block; According to the prediction value of each pixel point in the 4*4 sub-block, determining a gradient value of each pixel point in the 4*4 sub-block and an offset vector of each pixel point in the 4*4 sub-block; determining a predicted compensation value of each pixel point in the 4*4 sub-block according to the gradient value and offset vector of each pixel point in the 4*4 sub-block; determining a final predicted value of the 4*4 sub-block according to the predicted value of each pixel point in the 4*4 sub-block and the predicted compensation value of each pixel point in the 4*4 sub-block; A decoding device according to claim 6 or 7, characterized in that it is
9. refusing to apply a bidirectional optical flow mode to a current image block if the size of a reference picture of the current image block is different from the size of a picture to which the current image block belongs; rejecting the application of a bidirectional optical flow mode to the current image block if two reference pictures of the current image block are long-term reference pictures; A decoding device according to claim 6, characterized in that it is
10. 1. An electronic device comprising a processor and a memory for storing processor-executable instructions, An electronic device, characterized in that the processor is configured to execute the decoding method according to any one of claims 1 to 5.
11. A machine-readable storage medium having instructions stored thereon, comprising: A machine-readable storage medium, characterized in that, when the instructions are executed by a processor, the medium implements the decoding method according to any one of claims 1 to 5.
