Method and apparatus for predictive refinement using optical flow on affine-coded blocks

Abstract

To provide an apparatus, an encoder, a decoder and corresponding methods for prediction refinement with optical flow (PROF) for an affine coded block.SOLUTION: A method includes, when a plurality of optical flow decision conditions are fulfilled for an affine coded block, performing a PROF process for a current sub-block of the affine coded block to obtain refined prediction sample values of the current sub-block of the affine coded block. After the sub-block based affine motion compensation is performed, a prediction sample value of the current sample of the current sub-block is refined by adding a delta prediction value. Thus, it allows for a better trade-off between coding complexity and prediction accuracy.SELECTED DRAWING: Figure 11A

Description

[Technical Field]

[0001] Cross-reference of related applications This patent application claims priority under U.S. Provisional Patent Application No. 62 / 821,440, filed on 20 March 2019, and under U.S. Provisional Patent Application No. 62 / 839,765, filed on 28 April 2019. The disclosures of the aforementioned patent applications are incorporated herein by reference in their entirety.

[0002] Embodiments of this disclosure generally relate to the field of picture processing, and more specifically to a method for refining subblock-based affine motion-compensated predictions using optical flow when one or more constraints are required. [Background technology]

[0003] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as digital television broadcasting, video transmission over the internet and mobile networks, real-time conversation applications like video chat, video conferencing, DVD and Blu-ray® discs, video content acquisition and editing systems, and camcorders for security applications.

[0004] Even relatively short videos can require a considerable amount of video data to render, which can pose challenges when data is streamed or otherwise transmitted over communication networks with limited bandwidth. Therefore, video data is generally compressed before being transmitted over modern telecommunication networks. Video size can also be a concern when videos are stored on storage devices, as memory resources can be limited. Video compression devices often encode video data at the source using software and / or hardware before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device that decodes the video data. Given limited network resources and the ever-increasing demand for higher video quality, improved compression and decompression techniques that improve compression ratios with little to no sacrifice of picture quality are desirable.

[0005] Recently, affine tools have been introduced to Versatile Video Coding, and theoretically, affine motion model parameters can be used to derive the motion vector for each sample in a coding block. However, since generating sample-based affine motion-compensated predictions is very complex, a sub-block-based affine motion compensation method is used. In this method, the coding block is divided into sub-blocks, and each sub-block is assigned a motion vector (MV) derived from the affine motion model parameters. However, this sub-block-based prediction loses accuracy. Therefore, a good trade-off must be achieved between coding complexity and prediction accuracy. [Overview of the project] [Means for solving the problem]

[0006] Embodiments of this application provide apparatus and methods for encoding and decoding according to independent claims. Embodiments of this application provide apparatus and methods for predictive refinement (PROF) using optical flow on affine-coded blocks, such that a good trade-off between complexity and the accuracy of subblock-based affine prediction can be achieved.

[0007] Embodiments are defined by the features of the independent claim and by more advantageous implementations of the embodiments according to the features of the dependent claim.

[0008] Specific embodiments are outlined in the attached independent claims, and other embodiments are outlined in the dependent claims.

[0009] The aforementioned and other objectives are achieved by the subject matter of the independent claim. Further implementations are evident from the dependent claims, description, and drawings.

[0010] According to a first aspect, the present invention relates to a method for predictive refinement with optical flow (PROF) on affine-coded blocks (i.e., blocks encoded or decoded using affine tools). to The method is applied to a subblock of a sample within an affine-coded block. The method is performed by an encoding or decoding device. The current subblock of an affine-coded block (each sub B The process may include a step of performing a PROF process on the current subblock of an affine-coded block to obtain refined predicted sample values ​​(i.e., final predicted sample values) for the block, and multiple constraints for applying PROF are not met or satisfied for the affine-coded block. The step of performing a PROF process on the current subblock of an affine-coded block includes the step of performing an optical flow process on the current subblock to obtain a delta prediction value for the current sample of the current subblock, and the step of obtaining a refined predicted sample value for the current sample based on the delta prediction value for the current sample and the predicted sample value for the current subblock (the step of performing an optical flow process on the current subblock to obtain a delta prediction value for the current subblock, and the step of obtaining a refined predicted sample value for the current subblock based on the delta prediction value for the current subblock and the predicted sample value for the current subblock). When a refined predicted sample value is generated for each subblock of an affine-coded block, it can be understood that a refined predicted sample value for the affine-coded block is generated naturally.

[0011] Therefore, an improved method is provided that enables a better trade-off between coding complexity and prediction accuracy. To refine subblock-based affine motion-compensated predictions using optical flow at pixel / sample level granularity, a Prediction Refinement (PROF) process using optical flow is conditionally performed. These conditions ensure that computations involving PROF occur only when they can improve prediction accuracy, thereby reducing unnecessary increases in computational complexity. Thus, the beneficial effect achieved by the techniques disclosed herein is an improvement in the overall compression performance of the coding method.

[0012] Note that the terms "block", "coding block", or "image block" used in the present disclosure may include a transform unit (TU), a prediction unit (PU), a coding unit (CU), etc. In VVC, the transform unit and the coding unit are generally aligned except for a few scenarios where the tilt of the TU or sub-block transform (SBT) is used. The terms "block", "image block", "coding block", and "picture block" are In this specification, they may be used interchangeably. understood. The terms "affine block", "affine picture block", "affine-coded block", and "affine motion block" may be used interchangeably herein. The terms "sample" and "pixel" may be used interchangeably with each other in the present disclosure. The terms "predicted sample value" and "predicted pixel value" may be used interchangeably with each other in the present disclosure. The terms "sample position" and "pixel position" may be used interchangeably with each other in the present disclosure.

[0013] In a possible implementation form of the method according to the first aspect itself, before performing the PROF process on the current sub-block of the affine-coded block, the method further includes the step of determining that a plurality of constraint conditions for applying the PROF are not satisfied for the affine-coded block.

[0014] In a possible implementation of the method according to the first aspect itself, the first indication information indicates that a plurality of constraint conditions for applying PROF are invalid for a picture including an affinity-coded block, or the first indication information indicates that the plurality of constraint conditions for applying PROF are invalid for a slice associated with a picture including an affinity-coded block, and the second indication information indicates no division of the affinity-coded blocks, that is, the variable fallbackModeTriggered is set to 1. When the variable fallbackModeTriggered is set to 1, no division of the affinity-coded blocks is required, that is, it can be understood that each sub-block of the affinity-coded block has the same motion vector. This indicates that the affinity-coded block has only translational motion. When the variable fallbackModeTriggered is set to 0, division of the affinity-coded blocks is required, that is, each sub-block of the affinity-coded block has its own motion vector. This indicates that the affinity-coded block has non-translational motion.

[0015] In the present disclosure, in some cases or situations for affinity-coded blocks, it is allowed that PROF is not applied. According to the constraints for applying PROF, those cases or situations are determined. Thus, a better trade-off can be achieved between coding complexity and prediction accuracy.

[0016] In any previous implementation of the first aspect or in a possible implementation of the method according to the first aspect itself, the step of performing optical flow processing on the current sub-block to obtain the delta prediction value of the current sample of the current sub-block A step of obtaining a second prediction matrix (in one example, the second prediction matrix is ​​generated based on a first prediction matrix corresponding to the predicted sample values ​​of the current subblock, where the predicted sample values ​​of the current subblock can be obtained by performing subblock-based affine motion compensation for the current subblock), wherein the size of the second prediction matrix is ​​larger than the size of the first prediction matrix (for example, the first prediction matrix has the size sbWidth*sbHeight, and the second prediction matrix has the size (sbWidth+2)*(sbHeight+2), where the variables sbWidth and sbHeight represent the width and height of the current subblock, respectively), that is, the step of obtaining the second prediction matrix generates a first prediction matrix based on motion information of the current subblock, where the elements of the first prediction matrix correspond to the predicted sample values ​​of the current subblock, and the step of obtaining the second prediction matrix further generates a second prediction matrix based on the first prediction matrix, or generates a second prediction matrix based on motion information of the current subblock, A step of generating a horizontal prediction gradient matrix and a vertical prediction gradient matrix based on a second prediction matrix, wherein the size of the second prediction matrix is ​​greater than or equal to the size of the horizontal prediction gradient matrix and the vertical prediction gradient matrix (for example, the horizontal prediction gradient matrix or the vertical prediction gradient matrix has a size of sbWidth*sbHeight, and the second prediction matrix has a size of (sbWidth+2)*(sbHeight+2)), and The process includes the step of calculating the delta prediction value (ΔI(i,j)) of the current sample in the current subblock based on the horizontal prediction gradient value of the current sample in the horizontal prediction gradient matrix, the vertical prediction gradient value of the current sample in the vertical prediction gradient matrix, and the difference (MVD) between the motion vector of the current sample in the current subblock and the motion vector of the center sample in the subblock. It can be understood that the MVD has horizontal and vertical components. The horizontal prediction gradient value of the current sample in the horizontal prediction gradient matrix corresponds to the horizontal component of the MVD, and the vertical prediction gradient value of the current sample in the vertical prediction gradient matrix corresponds to the vertical component of the MVD.

[0017] Note that an affine block can be a coding or decoding block of a picture in a video signal. The current subblock of an affine-coded block is, for example, a 4x4 block. The luma position (xCb, yCb) represents the position of the top-left sample of the affine-coded block relative to the top-left sample of the current picture. Samples in the current subblock can be referred to using the absolute position of the sample with respect to (or relative to) the top-left sample of the picture, e.g., (x, y), or the relative position of the sample with respect to the top-left sample of the subblock (in combination with other coordinates), e.g., (xSb+i, ySb+j), where (xSb, ySb) is the top-left of the picture. sample Top left of the subblock related to sample These are the coordinates.

[0018] The first prediction matrix may be a two-dimensional array containing rows and columns, and the elements of the array may be referenced using (i,j), where i is the horizontal / row index and j is the vertical / column index. The ranges of i and j can be, for example, i = 0..sbwidth-1 and j = 0..sbHeight-1, where sbWidth represents the width of the subblock and sbHeight represents the height of the subblock. In some examples, the size of the first prediction matrix is ​​the same as the size of the current block. For example, the size of the first prediction matrix may be 4x4, and the current block has a size of 4x4.

[0019] The second prediction matrix may be a two-dimensional array containing rows and columns, and the elements of the array may be referenced using (i,j), where i is the horizontal / row index and j is the vertical / column index. The ranges of i and j can be, for example, i = -1..sbwidth and j = -1..sbHeight, where sbWidth represents the width of the subblock and sbHeight represents the height of the subblock. In some examples, the size of the second prediction matrix is ​​larger than the size of the first prediction matrix; that is, the size of the second prediction matrix may be larger than the size of the current block. For example, the size of the second prediction matrix may be (sbWidth+2)*(sbHeight+2), while the current block has a size of sbWidth*sbHeight. For example, the size of the second prediction matrix may be 6x6, while the current block has a size of 4x4.

[0020] The horizontal and vertical gradient matrices may be arbitrary two-dimensional arrays containing rows and columns, and the elements of the array may be referenced using (i,j), where x is the horizontal / row index and y is the vertical / column index. The range of i and j may be, for example, i=0..sbWidth-1 and j=0..sbHeight-1, where sbWidth represents the width of the subblock and sbHeight represents the height of the subblock. In some examples, the size of the horizontal and vertical gradient matrices is the same as the size of the current block. For example, the size of the horizontal and vertical gradient matrices may be 4x4, and the current block has a size of 4x4.

[0021] If the position (x,y) of an element in the horizontal prediction gradient matrix is ​​the same as the position (p,q) of an element in the vertical prediction gradient matrix, i.e., (x,y)=(p,q), then the elements of the horizontal prediction gradient matrix correspond to the elements of the vertical prediction gradient matrix.

[0022] Therefore, the PROF process can refine subblock-based affine motion-compensated predictions using optical flow at sample-level granularity without increasing memory access bandwidth (by having the second prediction matrix based on the first prediction matrix or the (original) predicted sample values ​​of the current subblock), thereby achieving a higher granularity of motion compensation.

[0023] In any preceding implementation of the first embodiment or a possible implementation of the method according to the first embodiment itself, the motion vector difference between the motion vector of the current sample unit containing the current sample (e.g., a 2x2 sample block) and the motion vector of the central sample of a subblock is used as the difference between the motion vector of the current sample of the current subblock and the motion vector of the central sample of the subblock. Here, the motion vector of the central sample of a subblock can be understood as the MV of the subblock to which the current sample (i,j) belongs (i.e., the subblock MV). By using a sample unit such as a 2x2 sample block to calculate the motion vector difference, it becomes possible to balance processing overhead with prediction accuracy. In any preceding implementation of the first embodiment or a possible implementation of the method according to the first embodiment itself, the elements of the second prediction matrix are represented by I1(p,q), where the range of values ​​for p is [-1,sbW] and the range of values ​​for q is [-1,sbH]. The elements of the horizontal prediction gradient matrix are represented by X(i,j), which corresponds to the sample (i,j) of the current subblock within the affine-coded block, where the range of i is [0, sbW-1] and the range of j is [0, sbH-1]. The elements of the vertical prediction gradient matrix are represented by Y(i,j), which corresponds to the sample (i,j) of the current subblock within the affine-coded block, where the range of i is [0, sbW-1] and the range of j is [0, sbH-1]. sbW represents the width of the current subblock within the affine-coded block, and sbH represents the height of the current subblock within the affine-coded block. In an alternative representation, the elements of the second prediction matrix are represented by I1(p,q), where the range of p is [0,subW+1] and the range of q is [0,subH+1]. The elements of the horizontal prediction gradient matrix are represented by X(i,j), which corresponds to the sample (i,j) of the current subblock within the affine-coded block, where the range of i is [1, sbW] and the range of j is [1, sbH]. The elements of the vertical prediction gradient matrix are represented by Y(i,j), which corresponds to the sample (i,j) of the current subblock within the affine-coded block, where the range of i is [1, sbW] and the range of j is [1, sbH]. sbW represents the width of the current subblock within the affine-coded block, and sbH represents the height of the current subblock within the affine-coded block. Since p has a value from [0, subW+1] and q has a value from [0, subH+1], it can be understood that the top-left sample (or the origin of the coordinate system) is located at (1,1). On the other hand, since p has a value from [-1, subW] and q has a value from [-1, subH], it can be understood that the top-left sample (or the origin of the coordinate system) is located at (0,0).

[0024] In any preceding implementation of the first embodiment or a possible implementation of the method according to the first embodiment itself, before performing a PROF process on the current subblock of the affine-coded block, the method further includes the step of performing subblock-based affine motion compensation on the current subblock of the affine-coded block in order to obtain predicted sample values ​​(original or refined) of the current subblock.

[0025] The According to the second embodiment, This invention A method for predictive refinement (PROF) using optical flow on affine-coded blocks Regarding. This delicious, The process includes the step of performing a PROF process on the current subblock of an affine-coded block to obtain the refined predicted sample value (i.e., the final predicted sample value) of the current subblock of the affine-coded block, wherein multiple optical flow determination conditions are met for the affine-coded block, where the meeting of multiple optical flow determination conditions means that not all constraints for applying PROF are met. The step of performing a PROF process on the current subblock of an affine-coded block includes the steps of performing an optical flow process on the current subblock to obtain a delta prediction value for the current sample of the current subblock, and obtaining a refined predicted sample value for the current sample based on the delta prediction value for the current sample and the predicted sample value (original or refined) for the current sample of the current subblock.

[0026] Therefore, an improved method is provided that enables the achievement of a good trade-off between coding complexity and prediction accuracy. To refine subblock-based affine motion-compensated predictions using optical flow at pixel / sample level granularity, a Prediction Refinement (PROF) process using optical flow is conditionally performed. These conditions ensure that computations involving PROF occur only when they can improve prediction accuracy, thereby reducing unnecessary increases in computational complexity. Thus, the beneficial effect achieved by the techniques disclosed herein is an improvement in the overall compression performance of the coding method.

[0027] In a possible implementation of the method according to the second aspect itself, before performing the PROF process on the current subblock of the affine-coded block, the method further includes the step of determining that a plurality of optical flow determination conditions are satisfied for the affine-coded block.

[0028] In a possible implementation of the method according to the second embodiment, the optical flow determination conditions include: first directional information indicating that PROF is valid for a picture containing an affine-coded block, or first directional information indicating that PROF is valid for a slice associated with a picture containing an affine-coded block; and second directional information indicating a division of the affine-coded block, such as the variable fallbackModeTriggered being set to equal to 0. When the variable fallbackModeTriggered is set to equal to 0, a division of the affine-coded block is required, which can be understood to mean that each subblock of the affine-coded block has its own motion vector, indicating that the affine-coded block has non-translational motion.

[0029] When all constraints for applying PROF are not met according to the design of the constraints for applying PROF, it is permissible for PROF to be applicable. Thus, a trade-off between coding complexity and prediction accuracy becomes possible.

[0030] In any preceding implementation of the second embodiment or a possible implementation of the method according to the second embodiment itself, the step of performing optical flow processing on the current subblock to obtain a delta prediction value for the current sample of the current subblock is: A step of obtaining a second prediction matrix, wherein the elements of the second prediction matrix are based on the predicted sample values ​​of the current subblock, and in some examples, the step of obtaining the second prediction matrix includes a step of generating a first prediction matrix based on the motion information of the current subblock, wherein the elements of the first prediction matrix correspond to the predicted sample values ​​of the current subblock, and a step of generating a second prediction matrix based on the first prediction matrix, or a step of generating a second prediction matrix based on the motion information of the current subblock. A step of generating a horizontal prediction gradient matrix and a vertical prediction gradient matrix based on a second prediction matrix, wherein the size of the second prediction matrix is ​​greater than or equal to the size of the horizontal prediction gradient matrix and the vertical prediction gradient matrix. The process includes the step of calculating the delta prediction value (ΔI(i,j)) of the current sample in the current subblock based on the horizontal predicted gradient value of the current sample in the horizontal predicted gradient matrix, the vertical predicted gradient value of the current sample in the vertical predicted gradient matrix, and the difference between the motion vector of the current sample in the current subblock and the motion vector of the center sample in the subblock.

[0031] In any prior implementation of the second embodiment or a possible implementation of the method according to the second embodiment itself, The method is, The process further includes the step of performing subblock-based affine motion compensation for the current subblock of an affine-coded block in order to obtain the (original) predicted sample value of the current subblock of the affine-coded block.

[0032] In any preceding implementation of the second embodiment or in any possible implementation of the method according to the second embodiment itself, the motion vector difference between the motion vector of the current sample unit (e.g., a 2x2 sample block) to which the current sample belongs and the motion vector of the central sample of the subblock is used as the difference between the motion vector of the current sample of the current subblock and the motion vector of the central sample of the subblock.

[0033] In any prior implementation of the second embodiment or a possible implementation of the method according to the second embodiment itself, The elements of the second prediction matrix are represented by I1(p,q), where the range of p is [-1,sbW] and the range of q is [-1,sbH]. The elements of the horizontal prediction gradient matrix are represented by X(i,j), which corresponds to the sample (i,j) of the current subblock within the affine-coded block, where the range of i is [0, sbW-1] and the range of j is [0, sbH-1]. The elements of the vertical prediction gradient matrix are represented by Y(i,j), which corresponds to the sample (i,j) of the current subblock within the affine-coded block, where the range of i is [0, sbW-1] and the range of j is [0, sbH-1]. sbW represents the width of the current subblock within the affine-coded block, and sbH represents the height of the current subblock within the affine-coded block.

[0034] According to a third aspect, the present invention provides an apparatus for predictive refinement (PROF) using optical flow on affine-coded blocks (i.e., blocks encoded or decoded using affine tools). ru. The device corresponds to an encoding device or a decoding device. The device is A decision unit configured to determine that multiple constraints for applying PROF are not satisfied for an affine-coded block, The current subblock of an affine-coded block (each sub B It may include a prediction processing unit configured to perform a PROF process on the current subblock of an affine-coded block in order to obtain refined predicted sample values ​​(i.e., final predicted sample values) of the block, and multiple constraints for applying PROF may be not met or satisfied for the affine-coded block. The prediction processing unit performs optical flow processing on the current subblock to obtain the delta prediction value of the current sample in the current subblock, and obtains a refined predicted sample value of the current sample based on the delta prediction value of the current sample and the predicted sample value of the current sample in the current subblock. and) It is constructed for this purpose. When refined predicted sample values ​​are generated for each subblock of an affine-coded block, it can be understood that refined predicted sample values ​​for the affine-coded block are naturally generated.

[0035] In possible implementations of the device according to the third embodiment, several constraints for applying PROF include: first directional information indicating that PROF is invalid for pictures containing affine-coded blocks, or first directional information indicating that PROF is invalid for slices associated with a picture containing affine-coded blocks, and second directional information indicating no affine-coded block segmentation, i.e., the variable fallbackModeTriggered is set to 1. When the variable fallbackModeTriggered is set to 1, it can be understood that no affine-coded block segmentation is required, i.e., each subblock of the affine-coded block has the same motion vector. This indicates that the affine-coded block has only translational motion. When the variable fallbackModeTriggered is set to 0, no affine-coded block segmentation is required, i.e., each subblock of the affine-coded block has its own motion vector. This indicates that the affine-coded block has non-translational motion.

[0036] In any preceding implementation of the third embodiment or in any possible implementation of the device according to the third embodiment itself, the prediction processing unit is configured to obtain a second prediction matrix (in one example, the second prediction matrix is ​​generated based on a first prediction matrix corresponding to the predicted sample values ​​of the current subblock, which can be obtained by performing subblock-based affine motion compensation for the current subblock), and the size of the second prediction matrix is ​​larger than the size of the first prediction matrix (for example, the size of the first prediction matrix is ​​sbWidth*sbHeight). The second prediction matrix has the size (sbWidth+2)*(sbHeight+2), where the variables sbWidth and sbHeight represent the width and height of the current subblock, respectively. That is, the step of obtaining the second prediction matrix generates the first prediction matrix based on the motion information of the current subblock, where the elements of the first prediction matrix correspond to the predicted sample values ​​of the current subblock, and the step of obtaining the second prediction matrix further generates the second prediction matrix based on the first prediction matrix, or generates the second prediction matrix based on the motion information of the current subblock, A step of generating a horizontal prediction gradient matrix and a vertical prediction gradient matrix based on a second prediction matrix, wherein the size of the second prediction matrix is ​​greater than or equal to the size of the horizontal prediction gradient matrix and the vertical prediction gradient matrix (for example, the horizontal prediction gradient matrix or the vertical prediction gradient matrix has a size of sbWidth*sbHeight, and the second prediction matrix has a size of (sbWidth+2)*(sbHeight+2)), and The process includes the step of calculating the delta prediction value (ΔI(i,j)) of the current sample in the current subblock based on the horizontal predicted gradient value of the current sample in the horizontal predicted gradient matrix, the vertical predicted gradient value of the current sample in the vertical predicted gradient matrix, and the difference between the motion vector of the current sample in the current subblock and the motion vector of the center sample in the subblock.

[0037] In any preceding implementation of the third embodiment or in any possible implementation of the device according to the third embodiment itself, the motion vector difference between the motion vector of the current sample unit containing the current sample (e.g., a 2x2 sample block) and the motion vector of the central sample of a subblock is used as the difference between the motion vector of the current sample of the current subblock and the motion vector of the central sample of the subblock. Here, the motion vector of the central sample of a subblock can be understood as the MV of the subblock to which the current sample (i,j) belongs (i.e., the subblock MV). By using a sample unit such as a 2x2 sample block to calculate the motion vector difference, it becomes possible to balance processing overhead with prediction accuracy. In any preceding implementation of the third embodiment or in any possible implementation of the device according to the third embodiment itself, the elements of the second prediction matrix are represented by I1(p,q), where the range of values ​​for p is [-1,sbW] and the range of values ​​for q is [-1,sbH]. The elements of the horizontal prediction gradient matrix are represented by X(i,j), which corresponds to the sample (i,j) of the current subblock within the affine-coded block, where the range of i is [0, sbW-1] and the range of j is [0, sbH-1]. The elements of the vertical prediction gradient matrix are represented by Y(i,j), which corresponds to the sample (i,j) of the current subblock within the affine-coded block, where the range of i is [0, sbW-1] and the range of j is [0, sbH-1]. sbW represents the width of the current subblock within the affine-coded block, and sbH represents the height of the current subblock within the affine-coded block.

[0038] In any preceding implementation of the third embodiment or in any possible implementation of the device according to the third embodiment itself, the prediction processing unit 1503 is configured to perform subblock-based affine motion compensation for the current subblock of an affine-coded block in order to obtain predicted sample values ​​(original or to be refined) of the current subblock.

[0039] The According to aspect 4, This invention Apparatus for predictive refinement (PROF) using optical flow on affine-coded blocks Regarding. This device, A decision unit configured to determine that multiple optical flow decision conditions are satisfied for an affine-coded block, where the satisfaction of multiple optical flow decision conditions means that all constraints for applying PROF are not satisfied, and The system may also include a prediction processing unit configured to perform a PROF process on the current subblock of an affine-coded block in order to obtain a refined predicted sample value (i.e., final predicted sample value) of the current subblock of the affine-coded block, wherein a plurality of optical flow determination conditions are satisfied for the affine-coded block, and the prediction processing unit is configured to perform an optical flow process on the current subblock in order to obtain a delta predicted value of the current sample of the current subblock, and to obtain a refined predicted sample value of the current sample based on the delta predicted value of the current sample and the (original or refined) predicted sample value of the current sample of the current subblock.

[0040] In a possible implementation of the device according to the fourth embodiment itself, the optical flow determination conditions include a first instruction indicating that PROF is enabled for a picture containing an affine-coded block, or that PROF is enabled for a slice associated with a picture containing an affine-coded block, and a second instruction indicating a division of the affine-coded block, such as the variable fallbackModeTriggered being set to equal to 0. When the variable fallbackModeTriggered is set to equal to 0, a division of the affine-coded block is required, which can be understood to mean that each subblock of the affine-coded block has its own motion vector, indicating that the affine-coded block has non-translational motion.

[0041] In any preceding implementation of the fourth embodiment or in any possible implementation of the device according to the fourth embodiment itself, the prediction processing unit is: Obtaining a second prediction matrix, wherein the elements of the second prediction matrix are based on the predicted sample values ​​of the current subblock, and in some examples, the step of obtaining the second prediction matrix includes the step of generating a first prediction matrix based on the motion information of the current subblock, wherein the elements of the first prediction matrix correspond to the predicted sample values ​​of the current subblock, and the step of generating a second prediction matrix based on the first prediction matrix, or the step of generating a second prediction matrix based on the motion information of the current subblock. The process involves generating a horizontal and vertical gradient prediction matrix based on a second prediction matrix, wherein the size of the second prediction matrix is ​​greater than or equal to the size of the horizontal and vertical gradient prediction matrices. It is configured to calculate the delta prediction value (ΔI(i,j)) of the current sample in the current subblock, based on the horizontal prediction gradient value of the current sample in the horizontal prediction gradient matrix, the vertical prediction gradient value of the current sample in the vertical prediction gradient matrix, and the difference between the motion vector of the current sample in the current subblock and the motion vector of the central sample in the subblock.

[0042] In any preceding implementation of the fourth aspect or in any possible implementation of the device according to the fourth aspect itself, the prediction processing unit is configured to perform subblock-based affine motion compensation for the current subblock of the affine-coded block in order to obtain the (original) predicted sample value of the current subblock of the affine-coded block.

[0043] In any preceding implementation of the fourth aspect or in any possible implementation of the device according to the fourth aspect itself, the motion vector difference between the motion vector of the current sample unit (e.g., a 2x2 sample block) to which the current sample belongs and the motion vector of the central sample of the subblock is used as the difference between the motion vector of the current sample of the current subblock and the motion vector of the central sample of the subblock.

[0044] In any preceding implementation of the fourth aspect or in any possible implementation of the device according to the fourth aspect itself, the elements of the second prediction matrix are represented by I1(p,q), where the range of values ​​for p is [-1,sbW] and the range of values ​​for q is [-1,sbH]. The elements of the horizontal prediction gradient matrix are represented by X(i,j), which corresponds to the sample (i,j) of the current subblock within the affine-coded block, where the range of i is [0, sbW-1] and the range of j is [0, sbH-1]. The elements of the vertical prediction gradient matrix are represented by Y(i,j), which corresponds to the sample (i,j) of the current subblock within the affine-coded block, where the range of i is [0, sbW-1] and the range of j is [0, sbH-1]. sbW represents the width of the current subblock within the affine-coded block, and sbH represents the height of the current subblock within the affine-coded block.

[0045] A method according to a first aspect of the present invention may be carried out by an apparatus according to a third aspect of the present invention. Further features and implementations of the apparatus according to the third aspect of the present invention correspond to features and implementations of the method according to the first aspect of the present invention.

[0046] A method according to a second aspect of the present invention may be carried out by an apparatus according to a fourth aspect of the present invention. Further features and implementations of the apparatus according to the fourth aspect of the present invention correspond to features and implementations of the method according to the second aspect of the present invention.

[0047] According to a fifth aspect, the present invention relates to an encoder (20) comprising a processing circuit for performing a method according to the first or second aspect itself or an implementation thereof.

[0048] According to a sixth aspect, the present invention relates to a decoder (30) comprising a processing circuit for performing a method according to the first or second aspect itself or an implementation thereof.

[0049] According to a seventh aspect, the present invention relates to a decoder. The decoder is One or more processors, The system comprises a non-temporary computer-readable storage medium coupled to a processor and storing a program for execution by the processor, wherein the program, when executed by the processor, configures the decoder to perform a first embodiment itself or a method according to its implementation.

[0050] According to an eighth aspect, the present invention relates to an encoder. The encoder is One or more processors, The system comprises a non-temporary computer-readable storage medium coupled to a processor and storing a program for execution by the processor, wherein the program, when executed by the processor, configures the encoder to perform a first embodiment or a method according to its implementation.

[0051] According to a ninth aspect, the present invention relates to an apparatus for encoding a video stream, comprising a processor and memory. The memory stores instructions causing the processor to perform the method according to a second aspect.

[0052] According to a tenth aspect, the present invention , Includes a processor and memory. a device for decoding video streams Regarding this, memory stores instructions that cause the processor to perform a method according to a first aspect.

[0053] According to the eleventh aspect, the present invention relates to a computer program that, when executed on a computer, includes program code for performing a method according to the first or second aspect or any possible embodiment of the first or second aspect.

[0054] According to the 12th aspect, The present invention When executed 、1 or more processors This relates to a computer-readable storage medium that stores instructions for coding video data. The instruction causes one or more processors to perform a method according to the first or second embodiment or any possible embodiment of the first or second embodiment.

[0055] In a further embodiment, a video picture encoding method is provided, which includes the steps of determining instruction information, which is used to indicate whether a picture block to be encoded will be encoded according to a target inter prediction method, the target inter prediction method including an inter prediction method according to a first or second embodiment or any possible embodiment of the first or second embodiment, and encoding the instruction information into a bitstream.

[0056] In a further embodiment, a video picture decoding method is provided, comprising the steps of: parsing a bitstream to obtain instruction information, the instruction information being used to indicate whether a picture block to be decoded will be processed according to a target inter-prediction method, the target inter-prediction method comprising an inter-prediction method according to a first or second embodiment or any possible embodiment of the first or second embodiment; and processing the picture block to be decoded according to the target inter-prediction method when the instruction information indicates that processing will be performed according to the target inter-prediction method.

[0057] Details of one or more embodiments are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the description, drawings, and claims.

[0058] The following embodiments of the present invention will be described in more detail with reference to the accompanying drawings and illustrations. [Brief explanation of the drawing]

[0059] [Figure 1A] This is a block diagram showing an example of a video coding system configured to implement embodiments of the present invention. [Figure 1B] This is a block diagram showing another example of a video coding system configured to implement embodiments of the present invention. [Figure 2] This is a block diagram showing an example of a video encoder configured to implement an embodiment of the present invention. [Figure 3] This is a block diagram illustrating an exemplary structure of a video decoder configured to implement embodiments of the present invention. [Figure 4] This is a block diagram showing examples of encoding or decoding devices. [Figure 5] This is a block diagram showing another example of an encoding or decoding device. [Figure 6]This figure shows the current block's candidate spatial and temporal movement information. [Figure 7] This figure shows the current affine-coded block and the adjacent affine-coded block where A1 is located. [Figure 8A] This figure shows an example illustrating the constructed control point motion vector prediction method. [Figure 8B] This figure shows an example illustrating the constructed control point motion vector prediction method. [Figure 9A] This is a flowchart showing the process of a decoding method according to one embodiment of this application. [Figure 9B] This figure shows the constructed method for predicting the motion vector of control points. [Figure 9C] This figure shows a sample or pixel of the current affine-coded block, along with the motion vectors of the top-left and top-right control points. [Figure 9D] This figure shows a 6x6 prediction signal window for calculating or generating the horizontal prediction gradient matrix, the vertical prediction gradient matrix, and the 4x4 subblock. [Figure 9E] This figure shows a horizontal prediction gradient matrix, a vertical prediction gradient matrix, and an 18x18 prediction signal window for calculating or generating a 16x16 block. [Figure 10] This figure shows the difference Δv(i,j) (red arrow) between the sample MV calculated for a sample position (i,j) denoted by v(i,j) and the subblock MV(VSB) of the subblock to which the sample (i,j) belongs. [Figure 11A] This figure shows a method for predictive refinement (PROF) using optical flow on an affine-coded block according to one embodiment of the present disclosure. [Figure 11B] This figure shows another method for predictive refinement (PROF) using optical flow on an affine-coded block, according to another embodiment of the present disclosure. [Figure 12]This figure shows a PROF process according to one embodiment of the present disclosure. [Figure 13] This figure shows the peripheral and inner regions of a (M+2)*(N+2) prediction block according to one embodiment of the present disclosure. [Figure 14] This figure shows the peripheral and inner regions of a (M+2)*(N+2) prediction block according to another embodiment of the present disclosure. [Figure 15] This block diagram shows an exemplary structure of an apparatus for predictive refinement (PROF) using optical flow on affine-coded blocks of video signals, according to some embodiments of the present disclosure. [Figure 16] This is a block diagram illustrating an example structure of a content supply system that enables content distribution services. [Figure 17] This is a block diagram showing the structure of an example terminal device. [Modes for carrying out the invention]

[0060] In the following, unless otherwise explicitly specified, the same reference numeral refers to the same or at least functionally equivalent feature.

[0061] The following description includes references to the accompanying drawings, which form part of the present disclosure and, as illustrative examples, illustrate specific embodiments of the present invention or specific ways in which embodiments of the present invention may be used. It is understood that embodiments of the present invention may be used in other embodiments and may include structural or logical variations not shown in the drawings. Therefore, the following detailed description should not be taken as limiting, and the scope of the invention is defined by the appended claims.

[0062] For example, disclosures relating to a described method may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, if one or more specific method steps are described, a corresponding device may include one or more units, such as functional units (e.g., one unit that performs one or more steps, or multiple units that each perform one or more of the steps), even if such one or more units are not explicitly described or illustrated in the drawings, in order to perform the described one or more method steps. Conversely, if a particular device is described based on one or more units, such as functional units, a corresponding method may include one step for performing the function of one or more units (e.g., one step that performs the function of one or more units, or multiple steps that each perform one or more of the functions of multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Furthermore, it is understood that the various exemplary embodiments and / or features of the aspects described herein may be combined with each other unless otherwise specifically stated.

[0063] Video coding typically refers to the processing of a sequence of pictures that make up a video or video sequence. Instead of the term "picture," the terms "frame" or "image" may be used synonymously in the field of video coding. Video coding (or coding in general) consists of two parts: video encoding and video decoding. Video encoding is performed on the source side and typically involves processing the original video picture (e.g., by compression) to reduce the amount of data required to represent the video picture (for more efficient storage and / or transmission). Video decoding is performed on the destination side and typically involves the reverse processing compared to the encoder to reconstruct the video picture. Embodiments referring to "coding" of a video picture (or picture in general) should be understood as relating to the "encoding" or "decoding" of the video picture or each video sequence. The combination of the encoding and decoding parts is also called a codec (Coding and Decoding).

[0064] In lossless video coding, the original video picture can be reconstructed, meaning the reconstructed video picture will have the same quality as the original video picture (assuming there is no transmission loss or other data loss during storage or transmission). In lossy video coding, for example, quantization is used to further compress the video picture, reducing the amount of data that represents the video picture, which means it cannot be fully reconstructed in the decoder, meaning the quality of the reconstructed video picture will be lower or worse compared to the quality of the original video picture.

[0065] Several video coding standards belong to the group of “lossy hybrid video codecs” (i.e., combining spatial and temporal prediction in the sample domain with 2D transform coding to apply quantization in the transform domain). Each picture in a video sequence is typically divided into a set of non-overlapping blocks, and coding is typically performed at the block level. In other words, in an encoder, video is typically processed, or coded, at the block (video block) level, for example, by using spatial (in-picture) and / or temporal (inter-picture) predictions to generate prediction blocks, subtracting the prediction blocks from the current block (the block currently being processed / will be processed) to obtain residual blocks, and transforming the residual blocks to reduce (compress) the amount of data that will be transmitted by quantizing the residual blocks in the transform domain. In a decoder, however, the reverse processing compared to the encoder is applied to the coded or compressed blocks to reconstruct the current block for representation. Furthermore, the encoder replicates the decoder's processing loop so that both the encoder and decoder generate the same predictions (e.g., intra-predictions and inter-predictions) and / or reconstructions for processing subsequent blocks, i.e., coding.

[0066] Hereinafter, embodiments of the video coding system 10, video encoder 20, and video decoder 30 will be described with reference to Figures 1 to 3.

[0067] Figure 1A is a schematic block diagram showing an exemplary coding system 10, for example, a video coding system 10 (or simply coding system 10) that may utilize the techniques of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) of the video coding system 10 represent examples of devices that may be configured to perform the various illustrative techniques described in this application.

[0068] As shown in Figure 1A, the coding system 10, for example, encodes picture data 21 The system includes a source device 12 configured to provide encoded picture data 21 to a destination device 14 for decoding.

[0069] The source device 12 includes an encoder 20 and may additionally, i.e., optionally, include a picture source 16, a preprocessor (or preprocessing unit) 18, for example, a picture preprocessor 18, and a communication interface or communication unit 22.

[0070] The picture source 16 may comprise, or may comprise, any type of picture capture device, e.g., a camera for capturing real-world pictures, and / or any type of picture generation device, e.g., a computer graphics processor for generating computer-animated pictures, or any other type of device for acquiring and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). The picture source may also comprise any type of memory or storage for storing any of the aforementioned pictures.

[0071] To distinguish it from the processing performed by the preprocessor 18 and the preprocessing unit 18, the picture or picture data 17 may also be called the raw picture or raw picture data 17.

[0072] The preprocessor 18 is configured to receive (raw) picture data 17 and perform preprocessing on the picture data 17 to obtain a preprocessed picture 19 or preprocessed picture data 19. The preprocessing performed by the preprocessor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or denoising. It can be understood that the preprocessing unit 18 may be an optional component.

[0073] The video encoder 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21 (further details are described below, for example, based on Figure 2).

[0074] The communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) to another device, such as the destination device 14 or any other device, via the communication channel 13 for storage or direct reconstruction.

[0075] The destination device 14 includes a decoder 30 (for example, a video decoder 30) and may additionally, or optionally, include a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0076] The communication interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof) directly from, for example, the source device 12 or any other source, for example, a storage device, for example, an encoded picture data storage device, and to provide the encoded picture data 21 to the decoder 30.

[0077] Communication interfaces 22 and 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, for example, via a direct wired or wireless connection, or via any type of network, for example, a wired network or a wireless network or any combination thereof, or any type of private and public network or any combination thereof.

[0078] The communication interface 22 may be configured, for example, to package the encoded picture data 21 into an appropriate format, such as a packet, and / or to process the encoded picture data using any kind of transmit encoding or processing for transmission over a communication link or communication network.

[0079] A communication interface 28, which is the counterpart to communication interface 22, may be configured, for example, to receive transmitted data and process the transmitted data using any kind of corresponding decoding or processing and / or depackaging of the transmission to obtain encoded picture data 21.

[0080] Both communication interfaces 22 and 28 may be configured as unidirectional communication interfaces, as indicated by the arrows to the communication channel 13 in Figure 1A pointing from the source device 12 to the destination device 14, or as bidirectional communication interfaces, and may be configured to send and receive messages, for example, to set up a connection, to acknowledge, and to exchange any other information regarding the communication link and / or data transmission, such as the transmission of encoded picture data.

[0081] The decoder 30 is configured to receive the encoded picture data 21 and provide the decoded picture data 31 or the decoded picture 31 (further details are described below, for example, based on Figure 3 or Figure 5).

[0082] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), for example, the decoded picture 31, in order to obtain the post-processed picture data 33, for example, the post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (for example, from YCbCr to RGB), color correction, cropping, or resampling, or any other processing to prepare the decoded picture data 31 for display by, for example, the display device 34.

[0083] The display device 34 of the destination device 14 is configured to receive post-processed picture data 33 for displaying the picture to, for example, a user or viewer. The display device 34 may be any type of display for representing the reconstructed picture, such as an integrated or external display or monitor, or may comprise such a display. The display may comprise, for example, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a microLED display, a liquid crystal on silicon (LCoS), a digital optical processor (DLP), or any other type of display.

[0084] Figure 1A shows the source device 12 and the destination device 14 as separate devices, but the device embodiment may also have the functions of both or both of these, namely the source device 12 or its corresponding function and the destination device 14 or its corresponding function. In such embodiments, the source device 12 or its corresponding function and the destination device 14 or its corresponding function may be implemented using the same hardware and / or software, by separate hardware and / or software, or in any combination thereof.

[0085] As will become apparent to those skilled in the art based on the description, the presence and (strict) division of functions of various units or functions within the source device 12 and / or destination device 14, as shown in Figure 1A, may vary depending on the actual device and application.

[0086] An encoder 20 (e.g., a video encoder 20) or a decoder 30 (e.g., a video decoder 30), or both an encoder 20 and a decoder 30, may be implemented via processing circuits, such as those shown in Figure 1B, including one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic circuits, hardware, dedicated to video coding, or any combination thereof. The encoder 20 may be implemented via processing circuits 46 for embodying various modules, as discussed with respect to the encoder 20 in Figure 2 and / or any other encoder systems or subsystems described herein. The decoder 30 may be implemented via processing circuits 46 for embodying various modules, as discussed with respect to the decoder 30 in Figure 3 and / or any other decoder systems or subsystems described herein. The processing circuits may be configured to perform various operations, as will be discussed later. As shown in Figure 5, if the technique is partially implemented in software, the device may store instructions for the software in a suitable non-temporary computer-readable storage medium, and may use one or more processors to execute the instructions in hardware to perform the technique of the present disclosure. Either the video encoder 20 or the video decoder 30 may be integrated as part of a synthesized encoder / decoder (codec) in a single device, for example, as shown in Figure 1B.

[0087] The source device 12 and destination device 14 may comprise any type of handheld or stationary device, including a wide range of devices such as notebook or laptop computers, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (such as content service servers or content distribution servers), broadcast receiver devices, and broadcast transmitter devices, and may or may not use an operating system. In some cases, the source device 12 and destination device 14 may support wireless communication. Therefore, the source device 12 and destination device 14 may be wireless communication devices.

[0088] In some cases, the video coding system 10 shown in Figure 1A is merely an example, and the techniques of this application may be applied to video coding configurations (e.g., video coding or video decoding) that do not necessarily involve any data communication between the coding device and the decoding device. In other examples, data may be retrieved from local memory, streamed over a network, etc. The video coding device may code the data and store it in memory, and / or the video decoding device may retrieve the data from memory and decode it. In some examples, coding and decoding are performed by devices that do not communicate with each other, but simply code the data into memory and / or retrieve the data from memory and decode it.

[0089] For convenience of explanation, embodiments of the present invention are described herein with reference to, for example, High-Efficiency Video Coding (HEVC) or to reference software for Versatile Video Coding (VVC), a next-generation video coding standard developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of the present invention are not limited to HEVC or VVC.

[0090] Encoder and encoding method Figure 2 shows a schematic block diagram of an exemplary video encoder 20 configured to implement the technique of the present application. In the example of Figure 2, the video encoder 20 comprises an input 201 (or input interface 201), a residual calculation unit 204, a transformation unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transformation unit 212, a reconstruction unit 214, a loop filter unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output 272 (or output interface 272). The mode selection unit 260 may include an inter-prediction unit 244, an intra-prediction unit 254, and a segmentation unit 262. The inter-prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 as shown in Figure 2 may also be called a hybrid video encoder or a video encoder according to a hybrid video codec.

[0091] The residual calculation unit 204, the conversion processing unit 206, the quantization unit 208, and the mode selection unit 260 are sometimes referred to as forming the forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse conversion processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoding picture buffer (DPB) 230, the inter-prediction unit 244, and the intra-prediction unit 254 are sometimes referred to as forming the reverse signal path of the video encoder 20, and the reverse signal path of the video encoder 20 corresponds to the signal path of the decoder (see video decoder 30 in Figure 3). The inverse quantization unit 210, the inverse conversion processing unit 212, the reconstruction unit 214, the loop filter 220, the decoding picture buffer (DPB) 230, the inter-prediction unit 244, and the intra-prediction unit 254 are also referred to as forming the “built-in decoder” of the video encoder 20.

[0092] Pictures and picture segmentation (pictures and blocks) The encoder 20 may be configured to receive, for example, a picture 17 (or picture data 17) via input 201, for example, a picture from a sequence of pictures that make up a video or video sequence. The received picture or picture data may also be a pre-processed picture 19 (or pre-processed picture data 19). For brevity, the following description will refer to picture 17. Picture 17 may also be called the current picture or the picture to be coded (specifically, in video coding, to distinguish the current picture from other pictures, for example, previously coded and / or decoded pictures in the same video sequence, i.e., the video sequence that also contains the current picture).

[0093] A (digital) picture is, or can be considered as, a two-dimensional array or matrix of samples with intensity values. A sample in the array may also be called a pixel (short for picture element) or pel. The number of samples in the horizontal and vertical (or axis) directions of the array or picture determines the size and / or resolution of the picture. For color representation, typically three color components are used; that is, a picture may be represented by, or contain, three sample arrays. RGB In a format or color space, a picture contains corresponding red, green, and blue sample arrays. However, in video coding, each pixel is typically represented in a luminance and chrominance format or color space, such as YCbCr, which contains a luminance component represented by Y (sometimes L is used instead) and two chrominance components represented by Cb and Cr. The luminance (or abbreviated as luma) component Y represents brightness or gray level intensity (for example, as in a grayscale picture), while the two chrominance (or abbreviated as chroma) components Cb and Cr represent chromaticity or color information components. Thus, a picture in YCbCr format contains a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). A picture in RGB format may be converted to or from YCbCr format, and vice versa, a process also known as color conversion or color transformation. If a picture is monochrome, it may only have a luminance sample array. Therefore, a picture could be, for example, a luminance sample array in a monochrome format, or two corresponding arrays of luminance samples and chromance samples in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0094] Embodiments of the video encoder 20 may include a picture partitioning unit (not shown in Figure 2) configured to partition a picture 17 into multiple (typically non-overlapping) picture blocks 203. These blocks may also be called root blocks, macroblocks (H.264 / AVC), or coding tree blocks (CTBs) or coding tree units (CTUs) (H.265 / HEVC and VVC). The picture partitioning unit may be configured to use the same block size for all pictures in the video sequence and a corresponding grid that defines that block size, or to change the block size between pictures or between subsets or groups of pictures, partitioning each picture into a corresponding block.

[0095] In a further embodiment, the video encoder may be configured to directly receive a block 203 of picture 17, for example, one, some, or all of the blocks that make up picture 17. The picture block 203 may also be called the current picture block or the picture block to be coded.

[0096] Like picture 17, picture block 203 is also a two-dimensional array or matrix of samples with intensity values ​​(sample values), but smaller in dimensions than picture 17, or can be considered as such. In other words, block 203 may consist of, for example, one sample array (e.g., a lumar array in the case of monochrome picture 17, or a lumar array or chromar array in the case of a color picture) or three sample arrays (e.g., a lumar array and two chromar arrays in the case of color picture 17) or any other number and / or type of arrays depending on the color format applied. The number of samples in the horizontal and vertical (or axis) directions of block 203 determines the size of block 203. Thus, the block may be, for example, an M×N (M columns vs N rows) array of samples, or an M×N array of conversion coefficients.

[0097] An embodiment of the video encoder 20, as shown in Figure 2, may be configured to encode the picture 17 block by block, for example, encoding and prediction may be performed for each block 203.

[0098] An embodiment of the video encoder 20 shown in Figure 2 may further be configured to divide and / or encode a picture by using slices (also called video slices), wherein the picture may be divided into one or more slices (usually non-overlapping) or encoded using them, and each slice may comprise one or more blocks (e.g., CTUs).

[0099] Embodiments of the video encoder 20 as shown in Figure 2 may further be configured to divide and / or encode a picture by using tile groups (also called video tile groups) and / or tiles (also called video tiles), wherein a picture may be divided into one or more tile groups (usually non-overlapping) or encoded using them, each tile group may comprise, for example, one or more blocks (e.g., CTUs) or one or more tiles, each tile may be, for example, rectangular in shape and comprise one or more blocks (e.g., CTUs), for example, a complete block or a partial block.

[0100] Residual calculation The residual calculation unit 204 may be configured to calculate the residual block 205 (also called residual 205) based on the picture block 203 and the predicted block 265, for example, by subtracting the sample values ​​of the predicted block 265 (further details of the predicted block 265 will be given later) from the sample values ​​of the picture block 203 on a sample-by-sample basis (per pixel) in order to obtain the residual block 205 in the sample region.

[0101] conversion The transformation processing unit 206 may be configured to apply a transformation, such as a discrete cosine transform (DCT) or discrete sine transform (DST), to the sample values ​​of the residual block 205 in order to obtain transformation coefficients 207 in the transformation domain. The transformation coefficients 207 may also be called transformation residual coefficients and may represent the residual block 205 in the transformation domain.

[0102] The conversion processing unit 206 may be configured to apply an integer approximation of DCT / DST, such as the conversion specified for H.265 / AVC. Compared to the orthogonal DCT conversion, such an integer approximation is typically scaled by a certain coefficient. An additional scaling coefficient is applied as part of the conversion process to preserve the norm of the residual blocks processed by the forward and inverse conversions. The scaling coefficient is typically chosen based on several constraints, such as the scaling coefficient being a power of 2 for the shift operation, the bit depth of the conversion coefficient, and the trade-off between accuracy and implementation cost. For example, a specific scaling coefficient may be specified for the inverse conversion by, for example, the inverse conversion processing unit 212 (and the corresponding inverse conversion by, for example, the inverse conversion processing unit 312 in the video decoder 30), and the corresponding scaling coefficient for the forward conversion by, for example, the conversion processing unit 206 in the encoder 20 may be specified accordingly.

[0103] Embodiments of the video encoder 20 (each a conversion processing unit 206) may be configured to output conversion parameters, for example, one or more conversion types, either directly or in an encoded or compressed state via the entropy coding unit 270, so that, for example, the video decoder 30 can receive and use these conversion parameters for decoding.

[0104] Quantization The quantization unit 208 may be configured to quantize the transformation coefficient 207 to obtain a quantized coefficient 209, for example by applying scalar quantization or vector quantization. The quantized coefficient 209 may also be called the quantized transformation coefficient 209 or the quantized residual coefficient 209.

[0105] The quantization process can reduce the bit depth associated with some or all of the 207 conversion coefficients. For example, n-bit conversion coefficients may be rounded to m-bit conversion coefficients during quantization, where n is greater than m. The degree of quantization may be modified by adjusting the quantization parameter (QP). For example, in scalar quantization, different scalings may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, while larger quantization step sizes correspond to coarser quantization. Applicable quantization step sizes can be indicated by the quantization parameter (QP). The quantization parameter may be, for example, an index to a predetermined set of applicable quantization step sizes. For example, a small quantization parameter may correspond to finer quantization (smaller quantization step size), a large quantization parameter may correspond to coarser quantization (larger quantization step size), or vice versa. Quantization may involve division by the quantization step size, and the corresponding and / or dequantization, for example by the inverse quantization unit 210, may involve multiplication by the quantization step size. Some standards, such as the HEVC embodiment, may be configured to use quantization parameters to determine the quantization step size. Generally, the quantization step size can be calculated based on the quantization parameters using a fixed-point approximation of the equations involving division. Additional scaling factors may be introduced for quantization and dequantization to restore the norm of the residual block, which may be modified by the scaling used in the fixed-point approximations of the equations for the quantization step size and quantization parameters. In one exemplary implementation, the scaling of the inverse and dequantization may be combined. Alternatively, a customized quantization table may be used to signal from the encoder to the decoder, for example, in a bitstream. Quantization is a lossy operation, and the loss increases with increasing quantization step size.

[0106] Embodiments of the video encoder 20 (each a quantization unit 208) may be configured to output quantization parameters (QP) either directly or encoded via, for example, the entropy coding unit 270, so that, for example, the video decoder 30 can receive and apply these quantization parameters for decoding.

[0107] inverse quantization The inverse quantization unit 210 is configured to apply the inverse quantization of the quantization unit 208 to the quantized coefficients in order to obtain the dequantized coefficients 211, for example, by applying the inverse of the quantization scheme applied by the quantization unit 208, based on or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 may also be called the dequantized residual coefficients 211, and may correspond to the transformation coefficients 207, although they are usually not identical to the transformation coefficients due to the loss due to quantization.

[0108] Inverse Transform The inverse transform processing unit 212 is configured to apply the inverse transform of the transform applied by the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST) or other inverse transform, in order to obtain the reconstructed residual block 213 (or the corresponding dequantized coefficient 213) in the sample region. The reconstructed residual block 213 may also be called the transform block 213.

[0109] Rebuild The reconstruction unit 214 (for example, an adder or summer 214) is configured to add the transformed block 213 (i.e., the reconstructed residual block 213) to the predicted block 265 in order to obtain the reconstructed block 215 in the sample region by, for example, adding the sample values ​​of the reconstructed residual block 213 and the sample values ​​of the predicted block 265 sample by sample.

[0110] filtering The loop filter unit 220 (or simply "loop filter" 220) is configured to filter the reconstructed block 215 to obtain the filtered block 221, or more generally, to filter the reconstructed sample to obtain the filtered sample. The loop filter unit is configured, for example, to smooth pixel transitions or to improve video quality in other ways. The loop filter unit 220 may comprise one or more loop filters such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a bilateral filter, an adaptive loop filter (ALF), a sharpening filter, a smoothing filter, or a co-filter, or any combination thereof. Although the loop filter unit 220 is shown as a loop filter in Figure 2, in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be called the filtered reconstructed block 221.

[0111] Embodiments of the video encoder 20 (each a loop filter unit 220) may be configured to output loop filter parameters (such as sample adaptive offset information) either directly or encoded via, for example, the entropy coding unit 270, so that, for example, the decoder 30 can receive and apply the same loop filter parameters or the respective loop filters for decoding.

[0112] Decode picture buffer The decoded picture buffer (DPB) 230 may be a memory that stores a reference picture, or more generally, reference picture data, for encoding video data by the video encoder 20. The DPB 230 may be formed by any of various memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer (DPB) 230 may be configured to store one or more filtered blocks 221. The decoded picture buffer 230 may further be configured to store other previously filtered blocks, e.g., previously reconstructed and filtered blocks 221, of the same current picture or a different picture, e.g., a previously reconstructed picture, e.g., a previously reconstructed picture, e.g., a fully reconstructed, i.e., decoded picture (and corresponding reference blocks and samples) and / or a partially reconstructed current picture (and corresponding reference blocks and samples), e.g., for interpretation. The decoded picture buffer (DPB) 230 may also be configured to store, for example, one or more unfiltered reconstructed blocks 215, or generally unfiltered reconstructed samples, if the reconstructed blocks 215 have not been filtered by the loop filter unit 220, or to store any other further processed versions of the reconstructed blocks or samples.

[0113] Mode selection (categorization and prediction) The mode selection unit 260 comprises a segmentation unit 262, an inter-prediction unit 244, and an intra-prediction unit 254, and is configured to receive or acquire original picture data, for example, the original block 203 (the current block 203 of the current picture 17), and reconstructed picture data, for example, filtered and / or unfiltered reconstructed samples or blocks of the same (current) picture, and / or from one or more previously decoded pictures, for example, from the decoded picture buffer 230 or other buffers (for example, line buffers not shown). The reconstructed picture data is used as reference picture data for predictions, such as inter-prediction or intra-prediction, in order to acquire prediction blocks 265 or predictors 265.

[0114] The mode selection unit 260 may be configured to determine or select a segmentation (including no segmentation) and prediction mode (e.g., intra-prediction mode or inter-prediction mode) for the current block prediction mode, and to generate a corresponding prediction block 265, which is used for calculating the residual block 205 and for reconstructing the reconstructed block 215.

[0115] Embodiments of the mode selection unit 260 may be configured to select a segmentation and prediction mode (for example, from those supported by or available to the mode selection unit 260) that yields the best match, or in other words, the smallest residual (the smallest residual means better compression for transmission or storage), or the smallest signaling overhead (the smallest signaling overhead means better compression for transmission or storage), or considers or balances both. The mode selection unit 260 may be configured to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., to select a prediction mode that yields the smallest rate distortion. In this context, terms such as “best,” “smallest,” and “optimal” do not necessarily refer to an overall “best,” “smallest,” and “optimal,” but may also refer to the satisfaction of termination or selection criteria, such as a value being above or below a certain threshold, or other constraints that reduce complexity and processing time, which may lead to a “suboptimal choice.”

[0116] In other words, the partitioning unit 262 may be configured to partition block 203 into smaller block partitions or subblocks (which still form blocks) by repeatedly using, for example, quadtree partitioning (QT), binary tree partitioning (BT), ternary tree partitioning (TT), or any combination thereof, and to perform predictions for each of the block partitions or subblocks, for example, mode selection comprising selection of the tree structure of the partitioned block 203, and prediction modes applied to each of the block partitions or subblocks.

[0117] The following describes in more detail the segmentation (by the segmentation unit 260, for example) and prediction (by the inter-prediction unit 244 and intra-prediction unit 254) processes performed by the exemplary video encoder 20.

[0118] compartmentalization The partitioning unit 262 may partition (or split) the current block 203 into smaller partitions, for example, smaller blocks of square or rectangular size. These smaller blocks (which may also be called subblocks) may be further partitioned into even smaller partitions. This is also called tree partitioning or hierarchical tree partitioning, and may, for example, the root block at root tree level 0 (hierarchy level 0, depth 0) be recursively partitioned into, for example, two or more blocks at the next lowest tree level, for example, a node at tree level 1 (hierarchy level 1, depth 1), and these blocks may be further partitioned into, for example, two or more blocks at the next lowest level, for example, tree level 2 (hierarchy level 2, depth 2), and so on, until partitioning ends, for example, when a termination criterion is met, for example, when the maximum tree depth or minimum block size is reached. Blocks that are not further partitioned are also called tree leaf blocks or leaf nodes. A tree that uses partitioning into two divisions is called a binary tree (BT), a tree that uses partitioning into three divisions is called a ternary tree (TT), and a tree that uses partitioning into four divisions is called a quaternary tree (QT).

[0119] As previously mentioned, the term “block” as used herein can refer to a portion of a picture, particularly a square or rectangular portion. For example, with reference to HEVC and VVC, a block may be a coding tree unit (CTU), coding unit (CU), prediction unit (PU), and transformation unit (TU), and / or a corresponding block, such as a coding tree block (CTB), coding block (CB), transformation block (TB), or prediction block (PB), or they may correspond to these.

[0120] For example, a coding tree unit (CTU) may be a CTB for a luminous sample, two corresponding CTBs for a chroma sample of a picture having three sample arrays, or a CTB for a monochrome picture or a sample of a picture coded using three separate color planes and syntax structures used to code the sample. Correspondingly, a coding tree block (CTB) may be an N×N block of samples for some value of N such that the division of components into the CTB is a partitioning. A coding unit (CU) may be a coding block for a luminous sample, two corresponding coding blocks for a chroma sample of a picture having three sample arrays, or a coding block for a monochrome picture or a sample of a picture coded using three separate color planes and syntax structures used to code the sample. Correspondingly, a coding block (CB) may be an M×N block of samples for some values ​​of M and N such that the division of the CTB into the coding block is a partitioning.

[0121] In embodiments, for example, according to HEVC, a coding tree unit (CTU) may be divided into CUs by using a quadtree structure that can be represented as a coding tree. The decision of whether to code a picture area using interpicture (time) prediction or intrapicture (spatial) prediction is made at the CU level. Each CU may further be divided into one, two, or four PUs according to the PU division type. Within a single PU, the same prediction process is applied, and the relevant information is sent to the decoder for each PU. After obtaining residual blocks by applying the prediction process based on the PU division type, the CU may be divided into transformation units (TUs) according to another quadtree structure similar to a coding tree for the CU.

[0122] In embodiments, for example, according to the latest video coding standard currently under development called Versatile Video Coding (VVC), composite quadtree and binary tree (QTBT) segmentation is used to segment coding blocks. In the QTBT block structure, CUs can have either a square or rectangular shape. For example, a coding tree unit (CTU) is first segmented by a quadtree structure. The quadtree leaf nodes are further segmented by a binary or ternary (or triple) tree structure. The segmented tree leaf nodes are called coding units (CUs), and their segmentation is used for prediction and transformation processing without further segmentation. This means that CUs, PUs, and TUs have the same block size in the QTBT coding block structure. In parallel, multiple segmentations, such as triple tree segmentations, may be used with the QTBT block structure.

[0123] In one example, the mode selection unit 260 of the video encoder 20 may be configured to perform any combination of the segmentation techniques described herein.

[0124] As described above, the video encoder 20 is configured to determine or select the best or optimal prediction mode from a set of prediction modes (for example, a predetermined set). The set of prediction modes may include, for example, an intra-prediction mode and / or an inter-prediction mode.

[0125] Intra Prediction The set of intra-predictive modes may comprise 35 different intra-predictive modes, such as DC (or average) mode and non-directional modes like planar mode, or directional modes like those defined in HEVC, or it may comprise 67 different intra-predictive modes, such as DC (or average) mode and non-directional modes like planar mode, or directional modes like those defined for VVC.

[0126] The intra-prediction unit 254 is configured to use reconstructed samples of adjacent blocks of the same current picture to generate an intra-prediction block 265 according to one of the intra-prediction modes in a set of intra-prediction modes.

[0127] The intra-prediction unit 254 (or generally the mode selection unit 260) is further configured to output intra-prediction parameters (or generally information indicating the selected intra-prediction mode for a block) in the form of syntax elements 266 to the entropy coding unit 270 for inclusion in the encoded picture data 21, so that, for example, the video decoder 30 can receive and use the prediction parameters for decoding.

[0128] Interpretation The set of interpretation modes (or possible interpretation modes) is defined by the available reference picture (i.e., for example). DPB It depends on the previously decoded picture (at least partially decoded) stored in 230, and other interpretation parameters, such as whether the entire reference picture is used to find the best matching reference block, or only a portion of the reference picture, such as the search window area around the current block, and / or whether pixel interpolation is applied, such as half / semipel and / or quarterpel interpolation.

[0129] In addition to the prediction modes described above, skip mode and / or direct mode may be applied.

[0130] The interpretation unit 244 may include motion estimation (ME) units and motion compensation (MC) units (neither of which are shown in Figure 2). The motion estimation unit may be configured to receive or acquire picture block 203 (the current picture block 203 of the current picture 17) and decoded picture 231, or at least one or more previously reconstructed blocks, e.g., one or more reconstructed blocks of one or more other / different previously decoded pictures 231, for motion estimation. For example, a video sequence may comprise the current picture and the previously decoded picture 231, or in other words, the current picture and the previously decoded picture 231 may be part of, or may form, a sequence of pictures that make up the video sequence.

[0131] The encoder 20 may be configured, for example, to select a reference block from multiple reference blocks of the same or different pictures of multiple other pictures, and provide the motion estimation unit with the reference picture (or reference picture index) and / or the offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as interpretation parameters. This offset is also called the motion vector (MV).

[0132] The motion compensation unit is configured to obtain, for example, receive interprediction parameters and to perform interprediction based on or using the interprediction parameters to obtain interprediction block 265. Motion compensation performed by the motion compensation unit may involve fetching or generating prediction blocks based on motion / block vectors determined by motion estimation, and possibly performing interpolation to subsample accuracy. Interpolation filtering may generate additional samples from known samples, thus increasing the number of prediction block candidates that can be used to code picture blocks. Upon receiving the motion vector for the current picture block's PU, the motion compensation unit may find the prediction block pointed to by the motion vector in one of the reference picture lists.

[0133] The motion compensation unit may also generate syntax elements associated with blocks and video slices for use by the video decoder 30 when decoding picture blocks of video slices. In addition to slices and their respective syntax elements, tile groups and / or tiles and their respective syntax elements may be generated or used, or as an alternative.

[0134] Entropy coding The entropy coding unit 270 is configured to apply, for example, an entropy coding algorithm or scheme (e.g., variable-length coding (VLC) scheme, context-adaptive VLC scheme (CAVLC), arithmetic coding scheme, binarization, context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), stochastic interval-partitioned entropy (PIPE) coding, or another entropy coding method or technique) or bypass (no compression) to quantized coefficients 209, inter-prediction parameters, intra-prediction parameters, loop filter parameters, and / or other syntax elements to obtain encoded picture data 21, which can be output via output 272 in the form of an encoded bitstream 21, for example, so that a video decoder 30 can receive and use its parameters for decoding. The encoded bitstream 21 can be transmitted to the video decoder 30 or stored in memory for later transmission or retrieval by the video decoder 30.

[0135] Other structural variations of the video encoder 20 may be used to encode a video stream. For example, a non-conversion-based encoder 20 can directly quantize the residual signal for some blocks or frames without a conversion processing unit 206. In another implementation, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 that are combined into a single unit.

[0136] Decoder and decoding method Figure 3 shows an example of a video decoder 30 configured to implement the technique of the present application. The video decoder 30 is configured to receive encoded picture data 21 (e.g., encoded bitstream 21), which is encoded by, for example, an encoder 20, in order to obtain a decoded picture 331. The encoded picture data or bitstream contains information for decoding the encoded picture data, for example, data representing picture blocks and associated syntax elements of an encoded video slice (and / or tile group or tile).

[0137] In the example in Figure 3, the decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transformation processing unit 312, a reconstruction unit 314 (for example, an adder 314), a loop filter 320, and a decoding picture buffer ( DPB The video decoder 30 comprises a mode application unit 360, an interpretation unit 344, and an intraprediction unit 354. The interpretation unit 344 may be a motion compensation unit or may include one. In some examples, the video decoder 30 may perform a decoding path that is generally the reverse of the encoding path described with respect to the video encoder 100 from Figure 2.

[0138] As described with respect to encoder 20, the inverse quantization unit 210, inverse processing unit 212, reconstruction unit 214, loop filter 220, decoding picture buffer (DPB) 230, inter-prediction unit 344, and intra-prediction unit 354 are also referred to as forming the “built-in decoder” of video encoder 20. Thus, the inverse quantization unit 310 may be functionally identical to the inverse quantization unit 110, the inverse processing unit 312 may be functionally identical to the inverse processing unit 212, the reconstruction unit 314 may be functionally identical to the reconstruction unit 214, the loop filter 320 may be functionally identical to the root block 220, and the decoding picture buffer 330 may be functionally identical to the decoding picture buffer 230. Therefore, the descriptions given for each unit and function of video encoder 20 apply accordingly to each unit and function of video decoder 30.

[0139] Entropy decoding The entropy decoding unit 304 is configured to analyze the bitstream 21 (or generally the encoded picture data 21) and, for example, perform entropy decoding on the encoded picture data 21 to obtain, for example, quantized coefficients 309 and / or decoded coding parameters (not shown in Figure 3), such as inter-prediction parameters (e.g., reference picture index and motion vector), intra-prediction parameters (e.g., intra-prediction mode or index), transformation parameters, quantization parameters, loop filter parameters, and / or other syntax elements, or any or all of them. The entropy decoding unit 304 may be configured to apply a decoding algorithm or scheme corresponding to an encoding scheme such as those described with respect to the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 may further be configured to provide the inter-prediction parameters, intra-prediction parameters, and / or other syntax elements to the mode application unit 360 and other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice level and / or video block level. In addition to slices and their respective syntax elements, tile groups and / or tiles and their respective syntax elements may be received and / or used as alternatives.

[0140] inverse quantization The inverse quantization unit 310 may be configured to receive quantization parameters (QP) (or generally, information about inverse quantization) and quantized coefficients from encoded picture data 21 (for example, by analysis and / or decoding by the entropy decoding unit 304), and, based on the quantization parameters, apply inverse quantization to the decoded quantized coefficients 309 to obtain dequantized coefficients 311, which may also be called transformed coefficients 311. The inverse quantization process may include the use of quantization parameters determined by the video encoder 20 for each video block in a video slice (or tile or tile group) to determine the degree of quantization to be applied, and similarly the degree of inverse quantization.

[0141] Inverse Transform The inverse transformation processing unit 312 may be configured to receive the dequantized coefficients 311, also called the transformation coefficients 311, and apply a transformation to the dequantized coefficients 311 to obtain the reconstructed residual block 213 in the sample region. The reconstructed residual block 213 may also be called the transformation block 313. The transformation may be an inverse transformation, such as an inverse DCT, inverse DST, inverse integer transformation, or a conceptually similar inverse transformation process. The inverse transformation processing unit 312 may further be configured to receive transformation parameters or corresponding information from the encoded picture data 21 (for example, by analysis and / or decoding by the entropy decoding unit 304) to determine the transformation to be applied to the dequantized coefficients 311.

[0142] Rebuild The reconstruction unit 314 (for example, an adder or summer 314) may be configured to add the reconstructed residual block 313 to the predictive block 365 in order to obtain the reconstructed block 315 in the sample region, for example, by adding the sample values ​​of the reconstructed residual block 313 to the sample values ​​of the predictive block 365.

[0143] filtering The loop filter unit 320 (either within or after the coding loop) is configured to filter the reconstructed block 315 to obtain a filtered block 321, for example, to smooth pixel transitions or to improve video quality in another way. The loop filter unit 320 may comprise one or more loop filters such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as a bilateral filter, an adaptive loop filter (ALF), a sharpening filter, a smoothing filter, or a co-filter, or any combination thereof. Although the loop filter unit 320 is shown as a loop filter in Figure 3, in other configurations, the loop filter unit 320 may be implemented as a post-loop filter.

[0144] Decode picture buffer The decoded video block 321 of the picture is then stored in the decoded picture buffer 330, which stores the decoded picture 331 as a reference picture for display to subsequent motion compensation and / or output to other pictures, respectively.

[0145] The decoder 30 is configured to output the decoded picture 311, for example via output 312, for presentation to the user or for viewing by the user.

[0146] prediction The inter-prediction unit 344 may be identical to the inter-prediction unit 244 (specifically, the motion compensation unit), and the intra-prediction unit 354 may be functionally identical to the inter-prediction unit 254, and perform division or division determination and prediction based on division and / or prediction parameters, or each piece of information received from the encoded picture data 21 (for example, by analysis and / or decoding by the entropy decoding unit 304). The mode application unit 360 may be configured to perform block-by-block predictions (intra-predictions or inter-predictions) based on the reconstructed picture, block, or each sample (filtered or unfiltered) in order to obtain prediction blocks 365.

[0147] When a video slice is coded as an intra-coded (I) slice, the intra-prediction unit 354 of the mode application unit 360 is configured to generate a prediction block 365 for the picture block of the current video slice based on the signaled intra-prediction mode and data from previously decoded blocks of the current picture. When a video picture is coded as an inter-coded (i.e., B or P) slice, the inter-prediction unit 344 (e.g., motion compensation unit) of the mode application unit 360 is configured to generate a prediction block 365 for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 304. For inter-prediction, the prediction block may be generated from one of the reference pictures in one of the reference picture lists. The video decoder 30 may construct reference frame lists, namely List 0 and List 1, using default construction techniques based on the reference pictures stored in the DPB 330. The same or similar may apply to tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles) in addition to slices (e.g., video slices), or instead, depending on the embodiment using them, for example, video may be coded using I, P, or B tile groups and / or tiles.

[0148] The mode-applying unit 360 is configured to determine prediction information for a video block in the current video slice by analyzing motion vectors or related information and other syntax elements, and uses the prediction information to generate a prediction block for the current video block being decoded. For example, the mode-applying unit 360 uses some of the received syntax elements to determine the prediction mode (e.g., intra-prediction or inter-prediction) used to code the video blocks in the video slice, the inter-prediction slice type (e.g., B-slice, P-slice, or GPB-slice), construction information for one or more of the reference picture lists for the slice, motion vectors for each intercoded video block in the slice, the inter-prediction status for each intercoded video block in the slice, and other information for decoding the video blocks in the current video slice. The same or similar may apply to embodiments that use tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles) in addition to slices (e.g., video slices), or by embodiments that use them, for example, video may be coded using I, P, or B tile groups and / or tiles.

[0149] An embodiment of the video decoder 30, as shown in Figure 3, may be configured to divide and / or decode a picture by using slices (also called video slices), where the picture is divided into one or more slices (usually non-overlapping) or decoded using them, and each slice may contain one or more blocks (e.g., CTUs).

[0150] Embodiments of the video decoder 30, as shown in Figure 3, may be configured to segment and / or decode a picture by using tile groups (also called video tile groups) and / or tiles (also called video tiles), wherein a picture may be segmented into one or more tile groups (usually non-overlapping) or decoded using them, each tile group may comprise, for example, one or more blocks (e.g., CTUs) or one or more tiles, each tile may be, for example, rectangular in shape and comprise one or more blocks (e.g., CTUs), for example, a complete block or a partial block.

[0151] Other variations of the video decoder 30 may be used to decode the encoded picture data 21. For example, the decoder 30 can generate an output video stream without a loop filtering unit 320. For example, a non-transformation-based decoder 30 can directly dequantize the residual signal for some blocks or frames without an inverse transformation processing unit 312. In another implementation, the video decoder 30 may have an inverse quantization unit 310 and an inverse transformation processing unit 312 that are combined into a single unit.

[0152] It should be understood that in encoder 20 and decoder 30, the processing result of the current step may be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations such as clipping or shifting may be performed on the processing result of interpolation filtering, motion vector derivation, or loop filtering.

[0153] Further operations may be applied to the derived motion vectors of the current block (including, but not limited to, affine mode control point motion vectors, affine mode, planar mode, ATMVP mode subblock motion vectors, time motion vectors, etc.). For example, the value of a motion vector is constrained to a predetermined range according to the bit that represents it. If the bit representing the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" means exponent. For example, if bitDepth is set to equal 16, the range is -32768 to 32767, and if bitDepth is set to equal 18, the range is -131072 to 131071. For example, the value of a derived motion vector (e.g., the MV of four 4x4 subblocks in one 8x8 block) is constrained so that the maximum difference between the integer parts of the four 4x4 subblock MVs is not greater than N samples, such that the maximum difference between the integer parts is not greater than 1 sample.

[0154] Figure 4 is a schematic diagram of a video coding device 400 according to one embodiment of the present disclosure. The video coding device 400 is suitable for implementing the disclosed embodiments as described herein. In one embodiment, the video coding device 400 may be a decoder, such as the video decoder 30 in Figure 1A, or an encoder, such as the video encoder 20 in Figure 1A.

[0155] The video coding device 400 comprises an inlet port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 for processing data, a transmitter unit (Tx) 440 and an exit port 450 (or output port 450) for transmitting data, and memory 460 for storing data. The video coding device 400 may also comprise optical-electrical (OE) components and electrical-optical (EO) components coupled to the inlet port 410, receiver unit 420, transmitter unit 440, and exit port 450 for the input and output of optical or electrical signals.

[0156] The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGAs, ASICs, and DSPs. The processor 430 communicates with an inlet port 410, a receiver unit 420, a transmitter unit 440, an exit port 450, and memory 460. The processor 430 includes a coding module 470. The coding module 470 implements the disclosed embodiments described above. For example, the coding module 470 performs, processes, prepares, or provides various coding operations. Thus, including the coding module 470 results in a considerable improvement to the functionality of the video coding device 400 and causes the video coding device 400 to be transformed into different states. Alternatively, the coding module 470 is implemented as instructions stored in memory 460 and executed by the processor 430.

[0157] Memory 460 may comprise one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device to store such programs when they are selected for execution, and to store instructions and data read during program execution. Memory 460 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random-access memory (RAM), tri-level associative memory (TCAM), and / or static random-access memory (SRAM).

[0158] Figure 5 shows an exemplary embodiment. 1A This is a simplified block diagram of the device 500, which can be used as either or both of the source device 12 and the destination device 14.

[0159] The processor 502 in the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device, or multiple devices, that are currently existing or will be developed in the future, that are capable of manipulating or processing information. The disclosed implementation may be practiced using a single processor, for example, processor 502, as shown, but benefits in speed and efficiency can be obtained by using one or more processors.

[0160] In one implementation, the memory 504 of the device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 504. Memory 504 may include code and data 506 accessed by the processor 502 using the bus 512. Memory 504 may further include an operating system 508 and an application program 510, the application program 510 including at least one program that enables the processor 502 to perform the method described herein. For example, the application program 510 may include applications 1 through N, which further include video coding applications that perform the method described herein.

[0161] The device 500 may also include one or more output devices, such as a display 518. The display 518 may, in one example, be a touch-sensitive display, which combines the display with a touch-sensing element that is operable to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0162] Although illustrated here as a single bus, the bus 512 of device 500 may consist of multiple buses. Furthermore, the secondary storage 514 may be directly coupled to other components of device 500, or it may be accessed via a network, and may comprise a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 500 can be implemented in a wide range of configurations.

[0163] The following will first explain the concepts used in this application.

[0164] 1. Interpretation Mode HEVC uses two interpretation modes: advanced motion vector prediction (AMVP) mode and merge mode.

[0165] In AMVP mode, the currently active block's spatially or temporally adjacent encoded blocks (referred to as adjacent blocks) are scanned first. A motion vector candidate list (which may also be called a motion information candidate list) is constructed based on the motion information of the adjacent blocks, and then the optimal motion vector is determined from the motion vector candidate list based on the rate-distortion cost. The motion information candidate with the minimum rate-distortion cost is used as the motion vector predictor (MVP) for the current block. Both the positions of the adjacent blocks and their scanning order are predetermined. The rate-distortion cost is calculated according to equation (1), where J represents the rate-distortion cost (RD cost), SAD is the sum of absolute differences (SAD) between the original sample values ​​and the predicted sample values ​​obtained through motion estimation using the motion vector candidate predictor, R represents the bit rate, and λ represents the Lagrangian multiplier. The encoder communicates the index value and reference frame index value of the selected motion vector predictor in the motion vector candidate list to the decoder. Furthermore, a motion search is performed in the neighborhood in the MVP to obtain the actual motion vector for the current block. The encoder transfers the difference between the MVP and the actual motion vector (motion vector difference) to the decoder. J = SAD + λR (1)

[0166] In merge mode, the motion vector candidate list is first constructed based on the motion information of spatially or temporally adjacent encoded blocks of the current block. Then, the optimal motion information is determined from the motion vector candidate list as the motion information of the current block based on rate-distortion cost. The index value of the position of the optimal motion information in the motion vector candidate list (hereinafter referred to as the merge index) is communicated to the decoder. The spatial and temporal motion information candidates for the current block are shown in Figure 6. The spatial motion information candidates are from five spatially adjacent blocks (A0, A1, B0, B1, and B2). If an adjacent block is unavailable (the adjacent block does not exist, or the adjacent block is not encoded, or the prediction mode used for the adjacent block is not inter-prediction mode), the motion information of this adjacent block is not added to the motion vector candidate list. The temporal motion information candidate for the current block is obtained by scaling the block's MV at the corresponding position in the reference frame, based on the picture order count (POC) of the reference frame and the current frame. First, it is determined whether the block at position T in the reference frame is available. If the block is unavailable, the block at position C is selected.

[0167] Similar to AMVP mode, in merge mode, both the positions of adjacent blocks and their traversal order are predetermined. In addition, the positions of adjacent blocks and their traversal order may differ in different modes.

[0168] It can be seen that the motion vector candidate list (also called the candidate list, which can be abbreviated as the candidate list) needs to be maintained in both AMVP mode and merge mode. Before new motion information is added to the candidate list each time, it is first checked whether the same motion information already exists in the list. If the same motion information already exists, the motion information is not added to the list. This checking process is called pruning of the motion vector candidate list. The purpose of pruning the list is to avoid including the same motion information in the list, thereby avoiding the calculation of redundant rate strain costs.

[0169] In HEVC interpretation, the same motion information is used for all samples in a coding block, and motion compensation is performed based on this motion information to obtain predictors for the samples in the coding block. However, not all samples in a coding block have the same motion characteristics. Using the same motion information for the coding block can result in inaccurate motion-compensated predictions and more residual information.

[0170] Existing video coding standards apply block-matching motion estimation based on translational motion models, assuming that the motion of all samples within a block is consistent. However, in the real world, there are various types of motion. Many objects, such as rotating objects, roller coasters rotating in various directions, fireworks displays, and stunts in movies, especially moving objects in User Generated Content (UGC) scenarios, exhibit non-translational motion. When block motion compensation techniques based on translational motion models in existing coding standards are used for coding these moving objects, coding efficiency can be significantly affected. Therefore, non-translational motion models, such as affine motion models, are introduced to further improve coding efficiency.

[0171] Based on this, AMVP modes may be classified into translational model-based AMVP modes and non-translational model-based AMVP modes (e.g., affine model-based AMVP modes) depending on the different motion models, and merge modes may be classified into translational model-based merge modes and non-translational model-based merge modes (e.g., affine model-based merge modes).

[0172] 2. Non-translational motion model Prediction based on a non-translational motion model involves using the same motion model on both the encoder and decoder sides to derive motion information for each sub-block within the current block (also called sub-motion compensation units or basic motion compensation units), and then performing motion compensation based on the sub-block motion information to obtain the predicted block, thereby improving prediction efficiency. Common non-translational motion models include 4-parameter affine motion models and 6-parameter affine motion models.

[0173] In this embodiment of the present application, the sub-motion compensation unit (also called a sub-block) may be a sample or an N1 × N2 sample block obtained based on a particular segmentation method, where both N1 and N2 are positive integers, and N1 may be equal to or not equal to N2.

[0174] The 4-parameter affine motion model is expressed as equation (2).

[0175]

number

[0176] A four-parameter affine motion model can be represented by the motion vectors of two samples and their coordinates relative to the top-left sample of the current block. The samples used to represent the motion model parameters are called control points. sample (0,0) and the upper right sampleIf (W,0) is used as a control point, then the left of the current block The above Control point and right The above First, the motion vectors (vx0, vy0) and (vx1, vy1) for each control point are determined. Then, motion information for each sub-motion compensation unit of the current block is obtained according to equation (3), where (x, y) is the coordinate of the sub-motion compensation unit relative to the top-left sample of the current block (e.g., the coordinate of the top-left sample), and W represents the width of the current block. It should be understood that other control points may be used instead. For example, samples at positions (2, 2) and (W+2, 2), or (-2, -2) and (W-2, -2) may be used as control points. The selection of control points is not limited to the examples enumerated herein.

[0177]

number

[0178] The 6-parameter affine motion model is expressed as equation (4).

[0179]

number

[0180] A 6-parameter affine motion model can be represented by the motion vectors of three samples and their coordinates relative to the top-left sample of the current block. If the top-left sample (0,0), top-right sample (W,0), and bottom-left sample (0,H) of the current block are used as control points, then the left of the current block The above Control point, right The above Control point, and left BelowFirst, the motion vectors (vx0,vy0), (vx1,vy1), and (vx2,vy2) for each control point are determined. Then, the motion information for each sub-motion compensation unit of the current block is obtained according to equation (5), where (x,y) is the coordinate of the sub-motion compensation unit relative to the top-left sample of the current block, and W and H represent the width and height of the current block, respectively. It should be understood that other control points may be used instead. For example, samples at positions (2,2), (W+2,2), and (2,H+2), or (-2,-2), (W-2,-2), and (-2,H-2) may be used as control points. These examples are not limiting.

[0181]

number

[0182] Coding blocks predicted using an affine motion model are called affine-coded blocks.

[0183] Generally, motion information for the control points of an affine-coded block can be obtained by using an affine motion model-based advanced motion vector prediction (AMVP) mode or an affine motion model-based merge mode.

[0184] The motion information of the control points in the current coding block can be obtained by using an inherited control point motion vector prediction method or a constructed control point motion vector prediction method.

[0185] 3. Inherited control point motion vector prediction method The inherited control point motion vector prediction method refers to using the motion model of an adjacent encoded affine-coded block to determine the candidate control point motion vectors for the current block.

[0186] The current block shown in Figure 7 is used as an example. To find the affine-coded block located in the adjacent positions of the current block and to obtain the control point motion information of the affine-coded block, the adjacent blocks around the current block are scanned in a specified order, for example, A1→B1→B0→A0→B2. Furthermore, the control point motion vector (for merge mode) or control point motion vector predictor (for AMVP mode) of the current block is derived by using a motion model constructed based on the control point motion information of the affine-coded block. The order A1→B1→B0→A0→B2 mentioned above is used only as an example and should not be interpreted as limiting. Other orders may be used. In addition, the adjacent blocks are not limited to A1, B1, B0, A0, and B2, and various blocks in adjacent positions may be used.

[0187] Blocks in adjacent locations are pre-defined size samples or sample blocks obtained based on a specific division method. It could be For example, a sample block could be a 4x4 sample block, a 4x2 sample block, or a sample block of another size. These block sizes are illustrative and should not be interpreted as limiting.

[0188] The following explains the decision-making process using A1 as an example, and a similar process can be applied to other cases.

[0189] As shown in Figure 7, if the coding block in which A1 is located is a 4-parameter affine-coded block, then the upper left of the affine-coded block sample (x4, y4) motion vector (vx4, vy4) and upper right sample The motion vector (vx5, vy5) at (x5, y5) is obtained. Top left of the current affine-coded block. sampleThe motion vector (vx0,vy0) of (x0,y0) is calculated according to equation (6) and is in the upper right of the current affine-coded block. sample The motion vector (vx1, vy1) of (x1, y1) is calculated according to equation (7).

[0190]

number

[0191] The top left of the current block is obtained based on the affine-coded block in which A1 is located. sample (x0, y0) motion vector (vx0, vy0) and upper right sample The combination of motion vectors (vx1, vy1) for (x1, y1) represents the candidate motion vectors for the current block's control points.

[0192] If the coding block in which A1 is located is a 6-parameter affine-coded block, then the top left of the affine-coded block. sample Motion vector (vx4, vy4) of (x4, y4), upper right sample (x5, y5) motion vector (vx5, vy5) and bottom left sample The motion vector (vx6, vy6) for (x6, y6) is obtained. Top left of the current block. sample The motion vector (vx0, vy0) of (x0, y0) is calculated according to equation (8). sample The motion vector (vx1, vy1) of (x1, y1) is calculated according to equation (9). (Bottom left of the current block) sample The motion vector (vx2, vy2) of (x2, y2) is calculated according to equation (10).

[0193]

number

[0194] The top left of the current block is obtained based on the affine-coded block in which A1 is located. sample(x0, y0) motion vector (vx0, vy0), upper right sample The motion vector (vx1, vy1) of (x1, y1), and the lower left corner. sample The combination of motion vectors (vx2, vy2) for (x2, y2) represents the candidate motion vectors for the current block's control points.

[0195] It should be noted that other motion models, position candidates, and search and scan sequences are also applicable to this application. Details are not described in this embodiment of the application.

[0196] It should be noted that methods used to represent the motion models of adjacent coding blocks and the current coding block for other control points are also applicable to this application. Further details are not described herein.

[0197] 4. Constructed Control Point Motion Vector Prediction Method 1 The constructed control point motion vector prediction method involves combining the motion vectors of adjacent encoded blocks around the control points of the current block as the control point motion vector of the current affine-coded block, without considering whether those adjacent encoded blocks are affine-coded blocks.

[0198] Top left of the current block sample and upper right sample The motion vector is determined by using the motion information of adjacent coded blocks around the current coding block. Figure 8A is used as an example to illustrate the constructed control point motion vector prediction method. Note that Figure 8A is merely an example and should not be interpreted as limiting.

[0199] As shown in Figure 8A, upper left sample The motion vectors of adjacent encoded blocks A2, B2, and B3 are in the upper left corner of the current block. sample Used as a motion vector candidate for the motion vector, upper right sampleThe motion vectors of adjacent encoded blocks B1 and B0 are in the upper right corner of the current block. sample Used as a candidate motion vector for the motion vector. Top left sample and upper right sample The motion vector candidates are combined to form multiple 2-tuples. The motion vectors of the two encoded blocks contained in the 2-tuple can be used as control point motion vector candidates for the current block, as shown in equation (11A) below. {vA2,vB1},{vA2,vB0},{vB2,vB1},{vB2,vB0},{vB3,vB1},{vB3,vB0} (11A) Here, vA2 represents the motion vector of A2, vB1 represents the motion vector of B1, vB0 represents the motion vector of B0, vB2 represents the motion vector of B2, and vB3 represents the motion vector of B3.

[0200] As shown in Figure 8A, upper left sample The motion vectors of adjacent encoded blocks A2, B2, and B3 are in the upper left corner of the current block. sample Used as a motion vector candidate for the motion vector, upper right sample The motion vectors of adjacent encoded blocks B1 and B0 are in the upper right corner of the current block. sample Used as a motion vector candidate for the motion vector, bottom left sample The motion vectors of adjacent encoded blocks A0 and A1 are located in the lower left of the current block. sample Used as a candidate motion vector for the motion vector. Top left sample , top right sample , and bottom left sample The motion vector candidates are combined to form a 3-tuple. The motion vectors of the three encoded blocks contained in the 3-tuple can be used as control point motion vector candidates for the current block, as shown in equations (11B) and (11C) below. {vA2,vB1,vA0},{vA2,vB0,vA0},{vB2,vB1,vA0},{vB2,vB0,vA0},{vB3,vB1,vA0},{vB3,vB0,vA0} (11B) {vA2,vB1,vA1},{vA2,vB0,vA1},{vB2,vB1,vA1},{vB2,vB0,vA1},{vB3,vB1,vA1},{vB3,vB0,vA1} (11C) Here, vA2 represents the motion vector of A2, vB1 represents the motion vector of B1, vB0 represents the motion vector of B0, vB2 represents the motion vector of B2, vB3 represents the motion vector of B3, vA0 represents the motion vector of A0, and vA1 represents the motion vector of A1.

[0201] It should be noted that other methods for combining control point motion vectors are also applicable to this application. Further details are not described herein.

[0202] It should be noted that methods used to represent the motion models of adjacent coding blocks and the current coding block for other control points are also applicable to this application. Further details are not described herein.

[0203] 5. Control point motion vector prediction method 2 constructed as shown in Figure 8B: Step 501: Obtain motion information for the control points of the current block.

[0204] For example, in Figure 8A, CP k (k=1,2,3,4) represents the k-th control point. A0, A1, A2, B0, B1, B2, and B3 are spatially adjacent positions of the current block and are used to predict CP1, CP2, or CP3, while T is a temporally adjacent position of the current block and is used to predict CP4.

[0205] The coordinates of CP1, CP2, CP3, and CP4 are assumed to be (0,0), (W,0), (H,0), and (W,H) respectively, where W and H represent the width and height of the current block.

[0206] For each control point, its motion information is obtained in the following order.

[0207] (1) For CP1, the checking order is B2 → A2 → B3. If B2 is available, the motion information of B2 is used for CP1. Otherwise, A2 and B3 are checked in order. If the motion information of all three positions is unavailable, the motion information of CP1 cannot be obtained.

[0208] (2) For CP2, the checking order is B0 → B1. If B0 is available, the motion information of B0 is used for CP2. Otherwise, B1 is checked. If the motion information of both positions is unavailable, the motion information of CP2 cannot be obtained.

[0209] (3) For CP3, the checking order is A0 → A1. If A0 is available, the motion information of A0 is used for CP3. Otherwise, A1 is checked. If the motion information of both positions is unavailable, the motion information of CP3 cannot be obtained.

[0210] (4) For CP4, the motion information of T is used.

[0211] In this specification, X being available means that block X (e.g., A0, A1, A2, B0, B1, B2, B3, or T) has already been encoded and the inter-prediction mode is used. Otherwise, X is unavailable.

[0212] Note that other methods for obtaining the motion information of control points are also applicable to this application. Details are not described in this specification.

[0213] Step 502: Combine the movement information of the control points to obtain the constructed control point movement information.

[0214] To construct a 4-parameter affine motion model, the movement information of two control points is combined to form a 2-tuple. The combinations of the movement information of two control points can be {CP1, CP4}, {CP2, CP3}, {CP1, CP2}, {CP2, CP4}, {CP1, CP3}, and {CP3, CP4}. For example, the 4-parameter affine motion model constructed by using the 2-tuple containing the movement information of control points CP1 and CP2 can be denoted as Affine(CP1, CP2).

[0215] To construct a 6-parameter affine motion model, the movement information of three control points is combined to form a 3-tuple. The combinations of the movement information of three control points can be {CP1, CP2, CP4}, {CP1, CP2, CP3}, {CP2, CP3, CP4}, and {CP1, CP3, CP4}. For example, the 6-parameter affine motion model constructed by using the 3-tuple containing the movement information of control points CP1, CP2, and CP3 can be denoted as Affine(CP1, CP2, CP3).

[0216] To construct an 8-parameter bilinear motion model, the movement information of four control points is combined to form a 4-tuple. The 8-parameter bilinear motion model constructed by using the 4-tuple containing the movement information of control points CP1, CP2, CP3, and CP4 can be denoted as Bilinear(CP, CP, CP, CP).

[0217] In this embodiment of the present application, for the sake of simplicity of description, the combination of the movement information of two control points (or two encoded blocks) is simply called a 2-tuple, the combination of the movement information of three control points (or three encoded blocks) is simply called a 3-tuple, and the combination of the movement information of four control points (or four encoded blocks) is simply called a 4-tuple.

[0218] These models are scanned in a predetermined order. If motion information for the control points corresponding to a combination model is unavailable, the model is considered unavailable. Otherwise, the model's reference frame index is determined, and the control point motion vectors are scaled. If all the motion information for all control points after scaling is consistent, the model is invalid. If all the motion information for the control points controlling the model is available and the model is valid, the motion information for the control points that make up the model is added to the motion information candidate list.

[0219] The control point motion vector scaling method is shown in equation (12).

[0220]

number

[0221] Here, CurPoc represents the POC number of the current frame, DesPoc represents the POC number of the current block's reference frame, SrcPoc represents the POC number of the control point's reference frame, and MV s represents the motion vector obtained after scaling, and MV represents the motion vector of the control point.

[0222] Note that different combinations of control points can be converted into control points at the same location.

[0223] For example, a four-parameter affine motion model obtained through the combinations {CP1,CP4}, {CP2,CP3}, {CP2,CP4}, {CP1,CP3}, or {CP3,CP4} is converted to a representation using {CP1,CP2} or {CP1,CP2,CP3}. The conversion method includes the steps of obtaining the model parameters by substituting the motion vectors and coordinate information of the control points {CP1,CP4}, {CP2,CP3}, {CP2,CP4}, {CP1,CP3}, or {CP3,CP4} into equation (2), and obtaining the motion vector of the control point {CP1,CP2} by substituting the coordinate information of {CP1,CP2} into equation (3).

[0224] More directly, the transformation may be performed according to equations (13) through (21) below, where W represents the width of the current block and H represents the height of the current block. In equations (13) through (21), (vx0, vy0) represents the motion vector of CP1, (vx1, vy1) represents the motion vector of CP2, (vx2, vy2) represents the motion vector of CP3, and (vx3, vy3) represents the motion vector of CP4.

[0225] {CP1,CP2} can be transformed into {CP1,CP2,CP3} by using equation (13) below. In other words, the motion vector of CP3 in {CP1,CP2,CP3} can be determined by using equation (13).

[0226]

number

[0227] {CP1,CP3} can be converted to {CP1,CP2} or {CP1,CP2,CP3} by using the following formula (14):

[0228]

number

[0229] {CP2, CP3} can be converted to {CP1, CP2} or {CP1, CP2, CP3} by using the following formula (15):

[0230]

Number

[0231] {CP1, CP4} can be converted to {CP1, CP2} or {CP1, CP2, CP3} by using the following formula (16) or (17):

[0232]

Number

[0233] {CP2, CP4} may be converted to {CP1, CP2} by using the following formula (18), and {CP2, CP4} may also be converted to {CP1, CP2, CP3} by using the following formulas (18) and (19).

[0234]

Number

[0235] {CP3, CP4} may be converted to {CP1, CP2} by using the following formula (20), and {CP3, CP4} may also be converted to {CP1, CP2, CP3} by using the following formulas (20) and (21).

[0236]

Number

[0237] For example, a six-parameter affine motion model obtained through the combination {CP1,CP2,CP4}, {CP2,CP3,CP4}, or {CP1,CP3,CP4} can be converted to a representation using {CP1,CP2,CP3}. The conversion method includes the steps of obtaining the model parameters by substituting the motion vectors and coordinate information of the control points {CP1,CP2,CP4}, {CP2,CP3,CP4}, or {CP1,CP3,CP4} into equation (4), and obtaining the motion vectors of {CP1,CP2,CP3} by substituting the coordinate information of {CP1,CP2,CP3} into equation (5).

[0238] More directly, the transformation may be performed according to equations (22) to (24) below, where W represents the width of the current block and H represents the height of the current block. In equations (13) to (21), (vx0, vy0) represents the motion vector of CP1, (vx1, vy1) represents the motion vector of CP2, (vx2, vy2) represents the motion vector of CP3, and (vx3, vy3) represents the motion vector of CP4.

[0239] {CP1,CP2,CP4} can be converted to {CP1,CP2,CP3} by using the following equation (22):

[0240]

number

[0241] {CP2,CP3,CP4} can be converted to {CP1,CP2,CP3} by using the following equation (23):

[0242]

number

[0243] {CP1,CP3,CP4} can be converted to {CP1,CP2,CP3} by using the following equation (24):

[0244]

number

[0245] 6. Affine motion model-based advanced motion vector prediction mode (affine AMVP mode) (1) Construction of a list of motion vector candidates The motion vector candidate list for the affine motion model-based AMVP mode is constructed by using the inherited control point motion vector prediction method and / or constructed control point motion vector prediction method described above. In this embodiment of the present application, the motion vector candidate list for the affine motion model-based AMVP mode may be referred to as the control point motion vector predictor candidate list. Each control point motion vector predictor includes motion vectors for two (four-parameter affine motion model) control points or motion vectors for three (six-parameter affine motion model) control points.

[0246] Optionally, the list of control point motion vector predictor candidates may be pruned and classified according to certain rules, and truncated or padded to include a certain number of control point motion vector predictor candidates.

[0247] (2) Determining the optimal control point motion vector predictor candidate On the encoder side, the motion vector of each sub-motion compensation unit in the current coding block is obtained based on each control point motion vector predictor candidate (e.g., X-tuple candidate) in the control point motion vector predictor candidate list by using equation (3) or (5). The obtained motion vector can be used to obtain a sample value at the corresponding position in the reference frame pointed to by the motion vector of the sub-motion compensation unit. This sample value is used as a predictor for motion compensation using an affine motion model. The mean difference between the original value and the predicted value for each sample in the current coding block is calculated. The control point motion vector predictor candidate corresponding to the smallest mean difference is selected as the optimal control point motion vector predictor candidate and is used as the motion vector predictor for two or three control points in the current coding block. The index number representing the position of the optimal control point motion vector predictor candidate (e.g., X-tuple candidate) in the control point motion vector predictor candidate list is encoded into a bitstream and sent to the decoder.

[0248] On the decoder side, the index number is analyzed, and a control point motion vector predictor (CPMVP) (for example, an X-tuple candidate) is determined from a list of control point motion vector predictor candidates based on the index number.

[0249] (3) Determination of control point motion vectors On the encoder side, the control point motion vector predictor is used as a starting point for motion searching within a specific search range in order to obtain the control point motion vector (CPMV). The difference between each control point motion vector and the control point motion vector predictor (control point motion vector difference, CPMVD) is transmitted to the decoder side.

[0250] On the decoder side, the control point motion vector difference is obtained by analyzing the bitstream and added to the respective control point motion vector predictor in order to obtain the motion vector for each control point.

[0251] 7. Affine Merge Mode The list of control point motion vector merge candidates is constructed by using the inherited control point motion vector prediction method and / or the constructed control point motion vector prediction method described above.

[0252] Optionally, the list of control point motion vector merge candidates may be pruned and categorized according to specific rules, truncated to a specific amount, or padded.

[0253] On the encoder side, the motion vector of each sub-motion compensation unit (a sample or N1×N2 sample block obtained based on a specific segmentation method) in the current coding block is obtained based on each control point motion vector candidate (e.g., an X-tuple candidate) in the merge candidate list by using equation (3) or (5). The obtained motion vector can be used to obtain the sample value at the position in the reference frame pointed to by the motion vector of each sub-motion compensation unit. These sample values ​​are used as predicted sample values ​​for affine motion compensation. The mean difference between the original value and the predicted value of each sample in the current coding block is calculated. The control point motion vector (CPMV) candidate (e.g., a 2-tuple candidate or a 3-tuple candidate) corresponding to the smallest mean difference is selected as the motion vector for two or three control points in the current coding block. The index numbers representing the positions of the control point motion vectors in the candidate list are encoded into the video bitstream and sent to the decoder.

[0254] On the decoder side, the index number is analyzed, and the control point motion vector (CPMV) is determined from a list of control point motion vector merge candidates based on the index number.

[0255] In addition, please note that in this application, “at least one” means one or more, and “multiple” means at least two. The term “and / or” describes a correlation for describing related objects and indicates that three relationships may exist. For example, A and / or B may represent the following cases: only A exists, both A and B exist, and only B exists. Here, A and B may be singular or plural. The letter “ / ” generally indicates an “or” relationship between related objects. “At least one of the following items” or similar expressions refer to any combination of these items, including any single item (fragment) or any combination of multiple items (fragments). For example, at least one of a, b, or c may represent a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c may be singular or plural.

[0256] In this application, when an interpredictive mode is used to decode the current block, a syntax element may be used to signal the interpredictive mode.

[0257] For some of the currently used syntax structures for the interprediction mode used to parse the current block, see Table 1, which lists some of the syntax for the interprediction mode. Alternatively, syntax elements within the syntax structure may be represented by other identifiers.

[0258] [Table 1A]

[0259] [Table 1B]

[0260] In Table 1, an inter_affine_flag[x0][y0] equal to 1 indicates that when decoding a P or B tile group, affine model-based motion compensation is used for the current coding unit to generate a predictive sample for that unit. An inter_affine_flag[x0][y0] equal to 0 indicates that the coding unit is not predicted by affine model-based motion compensation. If inter_affine_flag[x0][y0] is not present, it is presumed to be equal to 0.

[0261] inter_pred_idc[x0][y0] specifies whether list0, list1, or biprediction is used for the current coding unit, according to Table 2. Array indices x0, y0 specify the position (x0, y0) of the top-left luma sample of the coding block being considered, relative to the top-left luma sample of the picture.

[0262] If inter_pred_idc[x0][y0] does not exist, it is presumed to be equal to PRED_L0.

[0263] [Table 2]

[0264] The sps_affine_enabled_flag specifies whether affine model-based motion compensation can be used for inter-prediction. If sps_affine_enabled_flag is equal to 0, the syntax is constrained so that affine model-based motion compensation is not used in CVS and inter_affine_flag and cu_affine_type_flag do not exist in the CVS coding unit syntax. Otherwise (sps_affine_enabled_flag is equal to 1), affine model-based motion compensation can be used in CVS.

[0265] The syntax element inter_affine_flag[x0][y0] (or affine_inter_flag[x0][y0]) may be used to indicate whether an affine motion model-based AMVP mode is used for the current block when the slice in which the current block is located is a P-type or B-type slice. When this syntax element does not appear in the bitstream, it defaults to 0. For example, inter_affine_flag[x0][y0]=1 indicates that an affine motion model-based AMVP mode is used for the current block, and inter_affine_flag[x0][y0]=0 indicates that an affine motion model-based AMVP mode is not used for the current block and a translational motion model-based AMVP mode may be used. That is, inter_affine_flag[x0][y0] equal to 1 specifies that when decoding a P or B tile group, an affine model-based motion compensation is used for the current coding unit to generate the predicted sample for the current coding unit. An inter_affine_flag[x0][y0] equal to 0 indicates that the coding unit is not predicted by affine model-based motion compensation. When inter_affine_flag[x0][y0] does not exist, it is inferred to be equal to 0.

[0266] The syntax element cu_affine_type_flag[x0][y0] can be used to indicate whether a 6-parameter affine motion model is used to perform motion compensation for the current block when the slice in which the current block is located is a P-type or B-type slice. cu_affine_type_flag[x0][y0]=0 indicates that a 6-parameter affine motion model is not used to perform motion compensation for the current block, and only a 4-parameter affine motion model may be used for motion compensation, while cu_affine_type_flag[x0][y0]=1 indicates that a 6-parameter affine motion model is used to perform motion compensation for the current block. That is, cu_affine_type_flag[x0][y0] equal to 1 specifies that when decoding a P or B-tile group, a 6-parameter affine model-based motion compensation is used for the current coding unit to generate predictive samples for the current coding unit. A value of 0 for cu_affine_type_flag[x0][y0] specifies that a 4-parameter affine model-based motion compensation is used to generate the prediction samples for the current coding unit.

[0267] As shown in Table 3, MotionModelIdc[x0][y0]=1 indicates that a 4-parameter affine motion model is used, MotionModelIdc[x0][y0]=2 indicates that a 6-parameter affine motion model is used, and MotionModelIdc[x0][y0]=0 indicates that a translational motion model is used.

[0268] [Table 3]

[0269] The variables MaxNumMergeCand and MaxAffineNumMrgCand are used to represent the maximum list length and indicate the maximum length of the constructed motion vector candidate list. inter_pred_idc[x0][y0] is used to indicate the prediction direction. PRED_L1 is used to indicate backward prediction. num_ref_idx_l0_active_minus1 indicates the number of reference frames in the forward reference frame list, and ref_idx_l0[x0][y0] indicates the forward reference frame index value of the current block. mvd_coding(x0, y0, 0, 0) indicates the first motion vector difference. mvp_l0_flag[x0][y0] indicates the forward MVP candidate list index value. PRED_L0 indicates forward prediction. num_ref_idx_l1_active_minus1 indicates the number of reference frames in the backward reference frame list. ref_idx_l1[x0][y0] indicates the backreference frame index value of the current block, and mvp_l1_flag[x0][y0] indicates the back MVP candidate list index value.

[0270] In Table 1, ae(v) represents a syntax element encoded using context-based adaptive binary arithmetic coding (cabac).

[0271] Figure 9A is a flowchart showing the process of a decoding method according to one embodiment of the present application. The process may be performed by the interprediction unit 344 of the video decoder 30. The process is described as a series of steps or operations. It should be understood that the process may be performed in various orders and / or simultaneously, and is not limited to the execution order shown in Figure 9A. The video decoder is intended to be used to decode a video data stream having multiple video frames by using a process including the interprediction process shown in Figure 9A.

[0272] Step 601: To determine the interprediction mode of the current block, analyze the bitstream based on the syntax structure shown in Table 1.

[0273] If it is determined that the current block's interpretation mode is an affine motion model-based AMVP mode, then step 602a is performed.

[0274] For example, the syntax elements merge_flag=0 and inter_affine_flag=1 indicate that the inter-prediction mode for the current block is an affine motion model-based AMVP mode.

[0275] If it is determined that the current block's interpretation mode is an affine motion model-based merge mode, then step 602b is performed.

[0276] For example, the syntax elements merge_flag=1 and inter_affine_flag=1 indicate that the inter-prediction mode for the current block is an affine motion model-based merge mode.

[0277] Step 602a: Construct a list of motion vector candidates corresponding to the affine motion model-based AMVP mode.

[0278] One or more control point motion vector candidates for the current block (e.g., one or more X-tuple candidates) can be derived by using inherited control point motion vector prediction methods and / or constructed control point motion vector prediction methods. These control point motion vector candidates can be added to the motion vector candidate list.

[0279] The motion vector candidate list may include a 2-tuple list (where a 4-parameter affine motion model is used for the current coding block) or a 3-tuple list. A 2-tuple list contains one or more 2-tuples used to construct a 4-parameter affine motion model. A 3-tuple list contains one or more 3-tuples used to construct a 6-parameter affine motion model. Each 2-tuple candidate may contain two control point motion vector candidates for the current block.

[0280] Optionally, the list of 2-tuple / 3-tuple motion vector candidates may be pruned and categorized according to certain rules, and truncated or padded to contain a specific number of 2-tuples or 3-tuples.

[0281] A1: The process of constructing a list of motion vector candidates by using an inherited control point motion vector prediction method is described.

[0282] Figure 7 is used as an example. In this example, an affine-coded block containing a block adjacent to the current block is found, and the control of the affine-coded block is performed. point To acquire motion information, adjacent blocks around the current block are scanned in the order A1→B1→B0→A0→B2. The affine-coded block's control point motion information can be used to construct a motion model and derive candidate control point motion information for the current block. Details of this process are given above in the description of the inherited control point motion vector prediction method in section 3.

[0283] In one example, the affine motion model used for the current block is a four-parameter affine motion model (i.e., MotionModelIdc=1). In this example, when the four-parameter affine motion model is used for an adjacent affine decoded block, the motion vectors of the two control points of that affine decoded block are obtained: the motion vector (vx4,vy4) of the top-left control point (x4,y4) and the motion vector (vx5,vy5) of the top-right control point (x5,y5). The affine decoded block is the affine-coded block that is predicted at the coding stage by using the affine motion model.

[0284] The motion vectors of the two control points of the current block, namely the upper-left and upper-right control points, are derived according to equations (6) and (7) of the four-parameter affine motion model, respectively, by using a four-parameter affine motion model that includes the two control points of the adjacent affine decoding block.

[0285] When a 6-parameter affine motion model is used for adjacent affine decoded blocks, the motion vectors of the three control points of the adjacent affine decoded block are obtained, for example, the motion vector (vx4,vy4) of the top-left control point (x4,y4) in Figure 7, the motion vector (vx5,vy5) of the top-right control point (x5,y5), and the motion vector (vx6,vy6) of the bottom-left control point (x6,y6).

[0286] The motion vectors of the two control points of the current block, namely the upper-left and upper-right control points, are derived according to equations (8) and (9) of the six-parameter affine motion model, respectively, by using a six-parameter affine motion model that includes the three control points of the adjacent affine decoding block.

[0287] In another example, the affine motion model used for the current decryption block is a 6-parameter affine motion model (i.e., MotionModelIdc=2).

[0288] If the affine motion model used for the adjacent affine decoded block is a 6-parameter affine motion model, then the motion vectors of the three control points of the adjacent affine decoded block are obtained, for example, the motion vector (vx4,vy4) of the top-left control point (x4,y4) in Figure 7, the motion vector (vx5,vy5) of the top-right control point (x5,y6), and the motion vector (vx6,vy6) of the bottom-left control point (x6,y6).

[0289] The motion vectors of the three control points of the current block, namely the upper-left, upper-right, and lower-left control points, are derived using a six-parameter affine motion model that includes the three control points of the adjacent affine decoding block, according to equations (8), (9), and (10), respectively, which correspond to the six-parameter affine motion model.

[0290] If the affine motion model used for adjacent affine decoded blocks is a four-parameter affine motion model, then motion vectors for two control points of the adjacent affine decoded block can be obtained. These motion vectors could be, for example, the motion vector (vx4,vy4) of the top-left control point (x4,y4) and the motion vector (vx5,vy5) of the top-right control point (x5,y5).

[0291] Motion vectors can be derived for three control points, such as the top-left, top-right, and bottom-left control points of the current block. For example, these motion vectors can be derived according to equations (6) and (7) of the four-parameter affine motion model, by using a four-parameter affine motion model expressed based on two control points of an adjacent affine decoding block.

[0292] It should be noted that other motion models, position candidates, and search sequences may also be used herein. Furthermore, methods for representing the motion models of adjacent coding blocks and the current coding block based on other control points may also be used.

[0293] A2: The process of constructing a list of motion vector candidates by using the constructed control point motion vector prediction method is described.

[0294] In one example, the affine motion model used for the current decoding block is a 4-parameter affine motion model (i.e., MotionModelIdc=1). In this example, the top left of the current coding block is... sample and upper right sample The motion vector is determined based on the motion information of the adjacent encoded blocks of the current coding block. Specifically, the motion vector candidate list can be constructed using the constructed control point motion vector prediction method 1 described above with respect to item 4, or the constructed control point motion vector prediction method 2 described above with respect to item 5.

[0295] In another example, the affine motion model used for the current decoding block is a 6-parameter affine motion model (i.e., MotionModelIdc=2). In this example, the top left of the current coding block is... sample , top right sample , and bottom left sample The motion vector is determined by using the motion information of the adjacent encoded blocks of the current coding block. Specifically, the motion vector candidate list is determined by the constructed control point motion vector prediction method 1, as described above with respect to item 4, or item 4. 5 This can be constructed by using the constructed control point motion vector prediction method 2 described above.

[0296] Note that other combinations of control point motion information may also be available.

[0297] Step 603a: Analyze the bitstream and determine the best control point motion vector predictor (i.e., the best candidate for multiple tuples).

[0298] B1: If the affine motion model used for the current decoding block is a 4-parameter affine motion model (MotionModelIdc=1), the index number is analyzed from the bitstream, and the best motion vector predictor for the two control points is determined from a list of motion vector candidates based on the index number.

[0299] For example, the index number is mvp_l0_flag or mvp_l1_flag.

[0300] B2: If the affine motion model used for the current decoding block is a 6-parameter affine motion model (MotionModelIdc=2), the index numbers are analyzed from the bitstream, and the best motion vector predictors for the three control points are determined from a list of motion vector candidates based on the index numbers.

[0301] Step 604a: Analyze the bitstream to determine the control point motion vectors.

[0302] C1: If the affine motion model used for the current decoded block is a 4-parameter affine motion model (MotionModelIdc=1), the motion vector differences of the two control points of the current block are obtained by decoding the bitstream. The motion vector values ​​of the two control points are then obtained based on the motion vector differences of the control points and the corresponding motion vector predictors. Using forward prediction as an example, the motion vector differences of the two control points are mvd_coding(x0,y0,0,0) and mvd_coding(x0,y0,0,1), respectively.

[0303] For example, the motion vector difference between the top-left and top-right control points is obtained by decoding the bitstream and added to the respective motion vector predictors to obtain the motion vectors of the top-left and top-right control points of the current block.

[0304] C2: The affine motion model used for the current decryption block is a 6-parameter affine motion model (MotionModelIdc=2).

[0305] The motion vector differences of the three control points in the current block are obtained by decoding the bitstream. The motion vector values ​​of these control points are obtained based on the motion vector differences of the control points and their respective motion vector predictors. Using forward prediction as an example (i.e., List 0), the motion vector differences of the three control points are mvd_coding(x0,y0,0,0), mvd_coding(x0,y0,0,1), and mvd_coding(x0,y0,0,2), respectively.

[0306] For example, the motion vector differences for the top-left, top-right, and bottom-left control points are obtained by decoding the bitstream. These motion vector differences are added to the respective motion vector predictors to obtain the motion vectors for the top-left, top-right, and bottom-left control points of the current block.

[0307] Step 605a: Based on the control point motion information and affine motion model used for the current decoded block, obtain the motion vectors for each subblock within the current block.

[0308] A subblock within the current affine decoding block may be equivalent to a single motion compensation unit, where the width and height of the subblock are less than the width and height of the current block. Motion information of a sample at a predetermined position within the subblock or motion compensation unit can be used to represent the motion information of all samples within the subblock or motion compensation unit. Assuming the size of the motion compensation unit is M × N, the sample at the predetermined position could be the central sample (M / 2, N / 2), the top-left sample (0, 0), the top-right sample (M-1, 0), or a sample at another position within the motion compensation unit. The following explanation uses the central sample of the motion compensation unit as an example for illustrative purposes. Referring to Figure 9C, V0 represents the motion vector of the top-left control point, and V1 represents the motion vector of the top-right control point. Each small box represents a single motion compensation unit.

[0309] The left side of the current affine decryption block Upper Sa The coordinates of the center sample of the relative motion compensation unit with respect to the sample are calculated according to equation (25). In equation (25), i represents the i-th motion compensation unit in the horizontal direction (left to right), j represents the j-th motion compensation unit in the vertical direction (top to bottom), and (x(i,j), y(i,j)) represent the coordinates of the center sample of the (i,j)-th motion compensation unit relative to the top-left control point sample in the current affine decoding block.

[0310] If the affine motion model used for the current affine decoding block is a 6-parameter affine motion model, then each motion compensation unit (vx (i,j) ,vy (i,j) To obtain the motion vector of the sample at the center of (x (i,j) ,y (i,j) This is replaced by equation (26) of the 6-parameter affine motion model. As discussed above, the motion vector of the pixel at the center of the motion compensation unit is used as the motion vector for all samples within the motion compensation unit.

[0311] When the affine motion model used for the current affine decoding block is a 4 - affine motion model, for each motion compensation unit (vx (i,j) , vy (i,j) ) used as the motion vector for all samples in the motion compensation unit, in order to obtain the motion vector of the central sample of (x (i,j) , y (i,j) ), (x (i,j) , y (i,j) ) is substituted into the formula (27) of the 4 - parameter affine motion model.

[0312]

Number

[0313] Step 606a: To obtain the predicted sample value of the sub - block, perform motion compensation for each sub - block based on the determined motion vector of the sub - block.

[0314] As discussed above, in step 601, if it is determined that the inter - prediction mode of the current block is an affine motion model - based merge mode, step 602b is executed.

[0315] Step 602b: Construct a motion information candidate list corresponding to the affine motion model - based merge mode.

[0316] Specifically, the motion information candidate list corresponding to the affine motion model - based merge mode can be constructed by using the inherited control - point motion vector prediction method and / or the constructed control - point motion vector prediction method.

[0317] Optionally, the motion information candidate list can be pruned, classified, trimmed, or padded according to specific rules so as to contain a specific amount of motion information.

[0318] D1: The process of constructing a list of motion vector candidates by using an inherited control point motion vector prediction method is described.

[0319] The current block's control point motion information candidates are derived using the inherited control point motion vector prediction method and added to the motion information candidate list.

[0320] In the example shown in Figure 8A, blocks at adjacent positions around the current block are scanned in the order A1→B1→B0→A0→B2 in order to find the affine-coded block where the adjacent block is located and to obtain the control point motion information of the affine-coded block. Furthermore, candidate control point motion information for the current block is derived by using the motion model of the current block.

[0321] If the motion vector candidate list is empty, the control point motion information candidate obtained above is added to the candidate list. Otherwise, the motion information in the motion vector candidate list is scanned sequentially to check if the same motion information as the control point motion information candidate exists in the motion vector candidate list. If the same motion information as the control point motion information candidate does not exist in the motion vector candidate list, the control point motion information candidate is added to the motion vector candidate list.

[0322] Determining whether two motion information candidates are the same can be done by determining whether the forward (List 0) and backward (List 1) reference frames of the motion information candidates, as well as the horizontal and vertical components of their respective forward and backward motion vectors, are the same. Two motion information candidates are considered different only when all of these elements are different.

[0323] The construction of the candidate list is complete when the amount of motion information in the motion vector candidate list reaches the maximum list length MaxAffineNumMrgCand (MaxAffineNumMrgCand is a positive integer such as 1, 2, 3, 4, or 5, with 5 being used as an example in the following explanation). Otherwise, the next adjacent block is scanned.

[0324] D2: Candidate control point motion information for the current block is derived using the constructed control point motion vector prediction method and added to the motion information candidate list. Figure 9B shows an example flowchart of the constructed control point motion vector prediction method.

[0325] Step 601c: Obtain motion information for the control points of the current block. This step is the same as step 501 in "5. Method for Predicting the Motion Vector of Constructed Control Points 2". Details are not repeated here.

[0326] Step 602c: Combine the control point motion information to obtain the constructed control point motion information. This step is similar to step 501 in Figure 8B, and the details of this step will not be explained again here.

[0327] Step 603c: Add the constructed control point motion information to the motion vector candidate list.

[0328] If the length of the candidate list is less than the maximum list length MaxAffineNumMrgCand, the combinations of control point motion information are scanned in a predetermined order, and any valid combinations obtained are used as control point motion information candidates. In this case, if the motion vector candidate list is empty, the control point motion information candidate is added to the motion vector candidate list. Otherwise, the motion information in the motion vector candidate list is scanned sequentially to check if the same motion information as the control point motion information candidate exists in the motion vector candidate list. If the same motion information as the control point motion information candidate does not exist in the motion vector candidate list, the control point motion information candidate is added to the motion vector candidate list.

[0329] For example, the pre-set order is as follows: Affine(CP1,CP2,CP3)→Affine(CP1,CP2,CP4)→Affine(CP1,CP3,CP4)→Affine(CP2,CP3,CP4)→Affine(CP1,CP2)→Affine(CP1,CP3)→Affine(CP2,CP3)→Affine(CP1,CP4)→Affine(CP2,CP4)→Affine(CP3,CP4). There are a total of 10 combinations.

[0330] If the control point motion information corresponding to a given combination is unavailable, the combination is considered unavailable. If a combination is available, the reference frame index for that combination is determined. For two control points, the minimum reference frame index is selected as the reference frame index for the combination. For more than two control points, the reference frame index with the highest frequency of occurrence is selected as the reference frame index for the combination. If multiple reference frame indices have the same frequency of occurrence, the minimum reference frame index is selected as the reference frame index. The control point motion vector is further scaled. If the motion information for all control points remains consistent after scaling, the combination is invalid.

[0331] Optionally, in this embodiment of the present application, the motion vector candidate list may be padded. For example, if, after the scanning process described above, the length of the motion vector candidate list is less than the maximum list length MaxAffineNumMrgCand, the motion vector candidate list may be padded until its list length is equal to MaxAffineNumMrgCand.

[0332] Padding can be performed by using a zero motion vector padding method, or by combining existing motion information candidates from an existing list (e.g., by weighted averaging). Note that other methods for padding motion vector candidate lists are also applicable to this application.

[0333] Step 603b: Analyze the bitstream to determine the optimal control point motion information.

[0334] The index number is analyzed, and the optimal control point motion information is determined from a list of motion vector candidates based on the index number.

[0335] Step 604b: Obtain the motion vectors for each subblock within the current block, based on the optimal control point motion information and affine motion model used for the current decoded block.

[0336] This step is the same as step 605a.

[0337] Step 605b: To obtain predicted sample values ​​for the subblocks, motion compensation is performed for each subblock based on the determined motion vector of the subblock.

[0338] As previously described, after the motion vectors for each subblock are obtained in steps 605a and 604b, motion compensation for the subblocks is performed in steps 606a and 605b, respectively. That is, the details of performing subblock-based affine motion compensation for the current subblock of an affine-coded block in order to obtain the predicted sample value of the current subblock of the affine-coded block are described above. In conventional designs, the subblock size is set to 4x4, i.e., motion compensation is performed for each 4x4 unit by using each / different motion vector. Generally, smaller subblock sizes lead to greater complexity of motion compensation calculations and better predictive effects. To account for both the complexity of motion compensation calculations and the accuracy of predictions, a process for predictive signal refinement (PROF) using optical flow is provided after subblock-level motion compensation. Exemplary steps of the process are as follows:

[0339] (1) After the motion vectors of each subblock are obtained using steps 605a and 604b, motion compensation for the subblocks is performed using steps 606a and 605b to obtain the predicted signal I(i,j) for the subblocks. It should be noted that step (1) is not included in the PROF process.

[0340] (2) Horizontal gradient value g of the predicted signal of the subblock x (i,j) and vertical gradient value g y The calculation method for (i,j) is as follows: g x (i,j)=I(i+1,j)-I(i-1,j) g y (i,j)=I(i,j+1)-I(i,j-1)

[0341] As shown in Figure 9D, the formula reveals that a 6x6 prediction signal window 900 is required to obtain the gradient value (4x4 gradient value) for a 4x4 block.

[0342] This can be done by using the following different methods. a) After the prediction matrix of a subblock is obtained based on the motion information (e.g., motion vector) of the subblock, the horizontal gradient matrix and vertical gradient matrix of the subblock are obtained. In other words, (M+2)*(N+2) prediction blocks are obtained through interpolation based on the motion vectors of M×N subblocks. For example, to obtain a 6×6 prediction signal and compute a 4×4 gradient value (i.e., a 4×4 gradient matrix), interpolation is performed directly based on the motion vectors of the subblocks. b) Interpolation is performed based on the motion vectors of the subblocks to obtain a 4x4 prediction signal (i.e., the first prediction matrix), and then edge extension is performed on the prediction signal to obtain a 6x6 prediction signal (i.e., the second prediction matrix) and compute a 4x4 gradient value (i.e., a 4x4 gradient matrix). c) Obtain each 4x4 prediction signal (i.e., the first prediction matrix) and perform interpolation based on the motion vector of each subblock to obtain the w*h prediction signal through combinations. Then, perform edge extension on the w*h prediction signal to obtain the (w+2)*(h+2) prediction signal and calculate the w*h gradient value (i.e., the w*h gradient matrix) to obtain each 4x4 gradient value (i.e., the 4x4 gradient matrix).

[0343] Please note that directly obtaining (M+2)*(N+2) predicted blocks through interpolation based on the motion vectors of M×N subblocks includes the following implementation forms.

[0344] a1) The upper left of the position indicated by the motion vector relative to the surrounding area (white sample in Figure 13). sample Integer samples are obtained. For the inner region (gray samples in Figure 13), samples are obtained at the positions indicated by the motion vectors. If the samples are fractional samples, the samples are obtained through interpolation by using an interpolation filter.

[0345] As shown in Figure 14, A, B, C, and D are integer samples, the motion vector of the M×N subblock has a 1 / 16 sample precision, and dx / 16 is the top left. sample is the horizontal distance between the fractional sample and the integer sample, and dy / 16 is the vertical distance between the fractional sample and the integer sample of the top-left sample. For the surrounding region, the sample value of A is used as the predicted sample value for the sample position. For the inner region, the predicted sample value for the sample position is obtained through interpolation by using an interpolation filter.

[0346] a2) For the surrounding area (white samples in Figure 13), the integer sample closest to the position indicated by the motion vector is obtained. For the inner area (gray samples in Figure 13), the sample at the position indicated by the motion vector is obtained. If the sample is a fractional sample, the sample is obtained through interpolation by using an interpolation filter.

[0347] As shown in Figure 14, the integer sample closest to the position indicated by the motion vector relative to the surrounding region is selected based on dx and dy.

[0348] a3) A sample is obtained for the position indicated by the motion vector, both for the surrounding and inner regions. If the sample is a fractional sample, the sample is obtained through interpolation by using an interpolation filter.

[0349] It should be understood that a), b), and c) represent three different implementation forms.

[0350] (3) Calculate the delta prediction value, and the calculation method is as follows: ΔI(i,j)=g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j)

[0351] (i,j) represents the current sample in the subblock, and Δv(i,j) is the difference between the motion vector of the current sample in the current subblock and the motion vector of the sample at the center of the subblock (as shown in Figure 10), which may be calculated according to the formula above. x (i,j) and Δv y(i,j) are the horizontal and vertical offset values ​​of the difference between the motion vector of the current sample in the current subblock and the motion vector of the sample at the center of the subblock. Alternatively, in a simplified method, the motion vector difference between the motion vector of each 2x2 sample block to which the current sample belongs and the motion vector of the sample at the center of the subblock can be calculated. In comparison, Δv(i,j): the motion vector difference needs to be calculated for each pixel or sample (for example, in a 4x4 subblock, the calculation needs to be performed 16 times). However, in the simplified method, the motion vector difference is calculated for each 2x2 subblock (for example, in a 4x4 subblock, the calculation is performed 4 times). Note that the subblock here can be a 4x4 subblock or an MxN subblock, for example, where m is 4 or greater, or where n is 4 or greater.

[0352]

number

[0353] For a 4-parameter affine model:

[0354]

number

[0355] For a 6-parameter affine model:

[0356]

number

[0357] Here, (v0x,v0y), (v1x,v1y), and (v2x,v2y) are the motion vectors of the top-left, top-right, and bottom-left control points, and w and h are the width and height of the affine-coded block (CU).

[0358] (4) Perform predictive refinement: I'(i,j)=I(i,j)+ΔI(i,j) Here, I(i,j) is the predicted value of the sample (i,j) in the subblock (i.e., the predicted sample value at position (i,j) within the subblock), ΔI(i,j) is the delta predicted value of the sample (i,j) in the subblock, and I'(ij) is the refined predicted sample value of the sample (i,j) in the subblock. The conditional execution of a Prediction Refinement (PROF) process using optical flow to refine subblock-based affine motion-compensated predictions using optical flow, according to embodiments of the present disclosure, is described as follows:

[0359] Embodiment 1 A method for obtaining delta predictions of subblocks (specifically, delta predictions for each sample in a subblock) using optical flow may be applied to unidirectional affine-coded blocks or to bidirectional affine-coded blocks. When the method is applied to bidirectional affine prediction blocks, steps (1) through (4) described above must be performed twice, which increases the computational complexity. To reduce the complexity of the method, the present invention provides constraints for applying PROF. Specifically, the predicted sample values ​​are refined by using this method only when the affine-coded block is a unidirectional affine-coded block.

[0360] On the decoder side, the syntax element obtained by parsing the bitstream indicates whether it is single-prediction or bi-prediction. This syntax element can be used to determine whether an affine-coded block is a unidirectional affine-coded block.

[0361] On the encoder side, the structure of the B-frame and P-frame is determined by different use cases, and whether single or bi-prediction is used in the B-frame is determined by RDO. In other words, for the B-frame, the encoder may decide whether single or bi-prediction is used for the current affine picture block based on the RDO cost. For example, the encoder may attempt to select the mechanism that minimizes RDO from forward, backward, and bidirectional prediction.

[0362] Embodiment 2 To reduce the complexity of predictive signal refinement using optical flow, the method of obtaining predictive offset values ​​for subblocks using optical flow may only be used when the subblock size is relatively large. For example, the subblock size of a unidirectional affine-coded block may be set to 4x4, and the subblock size of a bidirectional affine-coded block may be set to 8x8, 8x4, or 4x8. In this example, this method is used only when the subblock size is greater than 4x4. In another example, the subblock size may be adaptively selected based on information such as the motion vectors of the control points of the affine-coded block, and the width and height of the affine-coded block. This method is used only when the subblock size is greater than 4x4.

[0363] In addition, in step (2), both methods a) and b) can be ensured that the predictions for each 4x4 subblock of the affine-coded block are independent and can be performed simultaneously. However, method a) increases the complexity of the interpolation calculation. Method b) does not increase complexity, but the boundary gradient values ​​are obtained through calculations using extended samples and are not highly accurate. Method c) can improve the accuracy of the gradient calculation, but there are dependencies between each 4x4 subblock, meaning that the optical flow-based refinement can only be performed when the interpolation of the entire CU is complete.

[0364] As shown in Figure 9E, to consider both the simultaneity and accuracy of gradient calculation, this disclosure proposes gradient value calculation based on a 16 × 16 granularity. Assuming size_w = min(w, 16) and size_h = min(h, 16), for each size_w * size_h in the affine-coded block, predictors (predicted sample values) for each 4 × 4 subblock in the affine-coded block are calculated, and the size_w * size_h predicted signal is obtained through combination. Next, edge extension is performed on the size_w * size_h predicted signal to obtain the (size_w + 2) * (size_h + 2) predicted signal (for example, extended outward by 2 samples by padding). The size_w * size_h gradient value is calculated to obtain each 4 × 4 gradient value. It should be understood that the amount of samples extended outward in this application is not limited to 2 samples and is relevant to the gradient calculation. If the gradient resolution is 3 taps, it is extended outward by 2 samples. In other words, this relates to the filter for gradient calculation. Assuming the number of taps in the filter is T, the added or supported area around it is T / 2 (divisible) * 2.

[0365] Figure 11A shows a method for predictive refinement (PROF) using optical flow on an affine-coded block according to one embodiment, which can be performed by a coding device (such as a decoder or decoder). This method includes the following steps:

[0366] S1101. It is determined that multiple optical flow determination conditions are met.

[0367] Here, the optical flow determination conditions may also be called the conditions for allowing the application of PROF. If all of the optical flow determination conditions are met, PROF is applied for the current subblock of the affine-coded block. Examples of optical flow determination conditions are described below. In some examples, the optical flow determination conditions may be replaced or rephrased as constraint conditions for applying PROF. If the constraint conditions for applying PROF are met, PROF is not applied to the current subblock of the affine-coded block. In those examples, step S1101 is modified to determine that none of the constraint conditions for applying PROF are met.

[0368] S1102. To obtain the refined predicted sample value of the current subblock of the affine-coded block, a PROF process is performed on the current subblock of the affine-coded block, and all multiple optical flow determination conditions are met for the affine-coded block. Here, the refined predicted sample value of the current subblock can be understood as the final predicted sample value of the current subblock after predictive refinement has been applied.

[0369] In step S1102, an optical flow (predictive refinement with optical flow, PROF) process is performed on one or more subblocks in the current affine picture block (e.g., the current subblock or each subblock) to obtain the delta prediction value (e.g., ΔI(i,j)) of one or more subblocks in the current affine picture block (e.g., the current subblock or each subblock).

[0370] Step S1102 involves obtaining a refined predicted sample value of the subblock (e.g., predicted signal I'(i,j)) based on the predicted delta value of the subblock (e.g., ΔI(i,j)) and the predicted sample value of the subblock (e.g., predicted signal ΔI(i,j)).

[0371] Specifically, step S1102 involves obtaining a refined predictor of the current sample in the subblock (e.g., predicted signal I'(i,j)) based on the delta predicted value of the current sample in the subblock (e.g., ΔI(i,j)) and the predictor of the current sample in the subblock (e.g., predicted signal I(i,j)).

[0372] In one possible design, the multiple optical flow determination conditions include one or more of the following:

[0373] (a) Indicator information obtained through analysis or derivation (e.g., sps_prof_enabled_flag or sps_bdof_enabled_flag) indicates that PROF is enabled for the current sequence, picture, slice, or tile group. For example, sps_prof_enabled_flag or sps_bdof_enabled_flag=1. If a constraint for applying PROF is used in S1101 instead of an optical flow determination condition, it can be understood that this condition may be translated into a constraint for applying PROF as follows: (a) Indicator information indicates that PROF is disabled for the current sequence, picture, slice, or tile group. For example, sps_prof_disabled_flag or sps_bdof_disabled_flag=1.

[0374] The instruction information obtained by analyzing parameter sets such as SPS, PPS, slice headers, or tile group headers indicates whether PROF is enabled for the current sequence, picture, slice, or tile group.

[0375] Specifically, sps_prof_enabled_flag may be used for control, and the syntax and semantics of sps_prof_enabled_flag are as follows:

[0376] [Table 4]

[0377] A sps_prof_enabled_flag equal to 0 indicates that predictive refined optical flow for affine-based motion compensation is disabled. A sps_prof_enabled_flag equal to 1 indicates that predictive refined optical flow for affine-based motion compensation is enabled.

[0378] Alternatively, sps_bdof_enabled_flag is reused for control.

[0379] In this embodiment, it should be understood that, assuming the aforementioned conditions are met (for example, the main switch decides to enable PROF), additional conditions are derived. In other words, if PROF is enabled for the current sequence, picture, slice, or tile group, it is further determined whether the current affine picture block satisfies the additional optical flow determination conditions described below. If PROF is not enabled for the current sequence, picture, slice, or tile group, it is not necessary to determine whether the current affine picture block satisfies the additional optical flow determination conditions provided below.

[0380] (b) Derived directive information (e.g., the variable fallbackModeTriggered) indicates that the current affine-coded block should be partitioned. For example, fallbackModeTriggered=0. If the constraint for applying PROF is used in S1101 instead of the optical flow determination condition, it can be understood that condition (b) may be translated into a constraint for applying PROF as follows: (b) Derived directive information indicates that the current affine-coded block should not be partitioned. For example, fallbackModeTriggered=1.

[0381] The variable `fallbackModeTriggered` is derived based on the affine parameter, and whether PROF is used depends on `fallbackModeTriggered`. When `fallbackModeTriggered` is 1, it indicates that the current affine-coded block should not be partitioned. When `fallbackModeTriggered` is 0, it indicates that the affine-coded block should be partitioned (for example, the affine-coded block should be partitioned into subblocks, e.g., 4x4 subblocks). PROF will be used when the current affine-coded block should be partitioned.

[0382] Specifically, the variable `fallbackModeTriggered` can be derived by using the following process.

[0383] The variable fallbackModeTriggered is initially set to 1 and is further derived as follows: - Variables bxWX4, bxHX4, bxWX h bxHX h bxWX v and bxHX v This is derived as follows: maxW4=Max(0,Max(4*(2048+dHorX),Max(4*dHorY,4*(2048+dHorX)+4*dHorY))) (8-775) minW4=Min(0,Min(4*(2048+dHorX),Min(4*dHorY,4*(2048+dHorX)+4*dHorY))) (8-775) maxH4=Max(0,Max(4*dVerX,Max(4*(2048+dVerY),4*dVerX+4*(2048+dVerY)))) (8-775) minH4=Min(0,Min(4*dVerX,Min(4*(2048+dVerY),4*dVerX+4*(2048+dVerY)))) (8-775) bxWX4=((maxW4-minW4)>>11)+9 (8-775) bxHX4=((maxH4-minH4)>>11)+9 (8-775) bxWX h =((Max(0,4*(2048+dHorX))-Min(0,4*(2048+dHorX)))>>11)+9 (8-775) bxHX h =((Max(0,4*dVerX)-Min(0,4*dVerX))>>11)+9 (8-775) bxWX v =((Max(0,4*dVerY)-Min(0,4*dVerY))>>11)+9 (8-775) bxHX v =((Max(0,4*(2048+dHorY))-Min(0,4*(2048+dHorY)))>>11)+9 (8-775) - If inter_pred_idc[xCb][yCb] is equal to PRED_BI and bxWX4*bxHX4 is less than or equal to 225, then fallbackModeTriggered is set to equal to 0. - No, bxWX h *bxHX h is 165 or less, and bxWX v *bxHX vIf the value is 165 or less, fallbackModeTriggered is set to equal to 0.

[0384] (c) The current affine picture block is a single predictive affine picture block.

[0385] (d) The size of the subblocks within the affine picture block is greater than N × N, so N = 4.

[0386] (e) The current affine picture block is a single-prediction affine picture block, and the size of the subblocks within the affine picture block is equal to N × N, where N = 4.

[0387] (f) The current affine picture block is a bipredictive affine picture block, and the size of the subblocks within the affine picture block is greater than N × N, so N = 4.

[0388] The current affine picture block is the current affine encoding The fact that a block has been identified and that the current affine picture block is a single predictive affine picture block is determined by using the following method. On the encoder side, based on the rate-distortion-based RDO, it is determined that a single prediction will be used for the current affine picture block.

[0389] The current affine picture block is the current affine decoded block, and the fact that the current affine picture block is a single-prediction affine picture block is determined by the following method. On the decoder side, in AMVP mode, prediction direction indicator information is used to indicate a single prediction direction (e.g., forward prediction only or backward prediction only), and the prediction direction indicator information is obtained by analyzing the bitstream or through derivation, or On the decoder side, in merge mode, the motion information candidate corresponding to the candidate index in the candidate list contains the first motion information corresponding to the first reference frame list, or the motion information candidate corresponding to the candidate index in the candidate list contains the second motion information corresponding to the second reference frame list.

[0390] In one possible design, the predictive direction information includes the syntax element inter_pred_idc[x0][y0], inter_pred_idc[x0][y0]=PRED_L0, which is used to indicate forward prediction. inter_pred_idc[x0][y0]=PRED_L1, which is used to indicate backward prediction, or, Predictive direction information includes predFlagL0 and / or predFlagL1. predFlagL0=1 and predFlagL1=0, which are used to indicate forward prediction. predFlagL1=1 and predFlagL0=0, which are used to indicate backward prediction.

[0391] Note that the optical flow determination conditions (or constraints for applying PROF) are not limited to the examples above, and additional or different optical flow determination conditions (or constraints for applying PROF) may be set based on different application scenarios. For example, conditions (c) through (f) above may be replaced with other conditions, such as optical flow determination conditions. PROF may be applied to an affine-coded block if all control point MVs of the affine-coded block are different from each other or differ from the constraints for applying PROF. PROF does not apply to an affine-coded block if all control point MVs of the affine-coded block are the same. Optical flow determination conditions, etc.: PROF may be applied to an affine-coded block if the resolution of the current picture and the resolution of the reference picture of the affine-coded block are the same, for example, if RprConstraintsActive[X][refIdxLX] is 0 or equal to the constraint for applying PROF. PROF is not applied to affine-coded blocks if the resolution of the current picture and the resolution of the reference picture of the affine-coded block are different from each other, for example, if RprConstraintsActive[X][refIdxLX] is equal to 1.

[0392] In one possible design, step S1102, performing an optical flow (predictive refinement with optical flow, PROF) process on one or more subblocks in the current affine picture block (e.g., each subblock or the current subblock) to obtain a delta prediction value (e.g., ΔI(i,j)) for one or more subblocks in the current affine picture block (e.g., each subblock or the current subblock), may include the following steps:

[0393] Step 1. Obtain a second prediction matrix based on the motion information (e.g., motion vector) of the current subblock within the current affine picture block.

[0394] For example, (M+2)*(N+2) prediction blocks (i.e., the second prediction matrix) are obtained through interpolation based on the motion vectors of M×N subblocks. Various implementation forms are given above.

[0395] Step 2. Based on the second prediction matrix, calculate the horizontal prediction gradient matrix and the vertical prediction gradient matrix, ensuring that the size of the second prediction matrix is ​​greater than or equal to the size of the horizontal prediction gradient matrix and the vertical prediction gradient matrix.

[0396] Step 3. Calculate the predicted delta value (ΔI(i,j)) for the current sample in the subblock based on the predicted horizontal gradient value of the current sample in the subblock in the horizontal prediction gradient matrix, the predicted vertical gradient value of the current sample in the vertical prediction gradient matrix, and the difference between the motion vector of the current sample in the current subblock and the motion vector of the sample at the center of the subblock.

[0397] Accordingly, in step S1102, based on the subblock's delta prediction value (e.g., ΔI(i,j)) and the subblock's prediction sample value (e.g., prediction signal I(i,j)), the refined prediction sample value of the subblock (e.g., prediction signal I(i,j)) is obtained, The process may include a step of obtaining a refined predicted sample value (e.g., predicted signal I'(i,j)) for the current sample, based on the predicted delta value (e.g., ΔI(i,j)) and the predicted sample value (e.g., predicted signal I(i,j)) for the current sample within a subblock.

[0398] It should be understood that the predicted sample values ​​of a subblock (for example, the predicted signal I(i,j)) can be M×N predicted blocks within (M+2)*(N+2) predicted blocks.

[0399] Regarding step 3, in one implementation, the motion vector difference between the motion vectors of different samples in the current subblock and the motion vector of the sample at the center of the subblock is different. In another implementation, the motion vector difference between the motion vector of the current sample unit containing the current sample (e.g., a 2x2 sample block) and the motion vector of the sample at the center of the subblock is used as the motion vector difference between the motion vector of the current sample in the current subblock and the motion vector of the sample at the center of the subblock. In other words, to balance processing overhead and prediction accuracy, assuming that both samples A and B are included in the current sample unit, the motion vector difference between the motion vector of the current sample unit (e.g., a 2x2 sample block) and the motion vector of the sample at the center of the subblock can be used as the motion vector difference between the motion vector of sample A in the subblock and the motion vector of the sample at the center of the subblock. Also, the motion vector difference between the motion vector of the current sample unit and the motion vector of the sample at the center of the subblock can be used as the motion vector difference between the motion vector of sample B in the subblock and the motion vector of the sample at the center of the subblock.

[0400] In one implementation, the second prediction matrix in step 1 above is represented by I1(p,q), where the range of p is [-1,sbW] and the range of q is [-1,sbH]. The horizontal gradient prediction matrix is ​​represented by X(i,j), where the range of i is [0,sbW-1] and the range of j is [0,sbH-1]. The vertical gradient prediction matrix is ​​represented by Y(i,j), where the range of i is [0,sbW-1] and the range of j is [0,sbH-1]. sbW represents the width of the current subblock within the current affine picture block, sbH represents the height of the current subblock within the current affine picture block, (x,y) represents the position coordinates of each sample (also called a sample) within the current subblock within the current affine picture block, and the element located at (x,y) may correspond to the element located at (i,j).

[0401] In another possible design, step 1102 In order to obtain the delta prediction value (also called the predictor offset value, e.g., ΔI(i,j)) for one or more subblocks (e.g., each subblock) in the current affine picture block, performing an optical flow (predictive refinement with optical flow, PROF) process on one or more subblocks (e.g., each subblock) in the current affine picture block includes the following, as shown in Figure 12: S1202. A second prediction matrix is ​​obtained or generated based on the first prediction matrix, and the first prediction matrix for each subblock (e.g., the first prediction signal I(i,j) or 4x4 prediction) corresponds to the predicted sample value of the current subblock. As shown in Figure 9A, subblock-based affine motion compensation for the current subblock of the affine-coded block is performed to obtain the predicted sample value of the current subblock of the affine-coded block. S1203. Based on the second prediction matrix, the horizontal prediction gradient matrix and the vertical prediction gradient matrix are calculated, and the size of the second prediction matrix is ​​greater than or equal to the size of the first prediction matrix. ri, dai The size of the prediction matrix 2 is greater than or equal to the size of the horizontal prediction gradient matrix and the vertical prediction gradient matrix.

[0402] S1204. Based on the horizontal prediction gradient matrix, the vertical prediction gradient matrix, and the motion vector difference between the motion vector of the current sample unit of the subblock (e.g., the current sample or current sample block, such as a 2x2 sample block) and the motion vector of the sample at the center of the subblock, the delta prediction matrix of the subblock (e.g., ΔI(i,j) of the predicted signal) is calculated. The steps to obtain the refined predicted sample value of a subblock (e.g., predicted signal I'(i,j)) based on the subblock's predicted delta value (e.g., ΔI(i,j)) and the subblock's predicted sample value (e.g., predicted signal I(i,j)) include: S1205. A refined third prediction matrix (e.g., prediction signal I'(i,j)) of the subblock is obtained based on the delta prediction matrix (e.g., ΔI(i,j)) and the first prediction matrix (e.g., prediction signal I(i,j)).

[0403] In this specification, I(i,j) represents the predicted sample value of the current sample in the current subblock (e.g., the original prediction obtained through motion compensation), ΔI(i,j) represents the delta predicted value of the current sample in the current subblock, and I'(i,j) represents the refined predicted sample value of the current sample in the current subblock. For example, original predicted sample value + delta predicted value = refined predicted sample value. Obtaining refined predicted sample values ​​for multiple samples in the current subblock (e.g., all samples) is equivalent to obtaining the refined predicted sample value for the current subblock.

[0404] In different possible implementations, the gradient values ​​may be calculated per sample, and the delta prediction values ​​may be calculated per sample. Alternatively, a gradient matrix may be obtained, and then the delta prediction values ​​may be calculated. This is not limited to this application. In one alternative implementation, the first prediction matrix and the second prediction matrix represent the same prediction matrix.

[0405] If the size of the second prediction matrix is ​​equal to the size of the first prediction matrix, and the size of the second prediction matrix is ​​equal to the sizes of the horizontal and vertical prediction gradient matrices, then in one possible implementation, the (w-2)*(h-2) gradient matrix is ​​calculated by using the w*h prediction matrix, which is padded to obtain a size of w*h, where w*h represents the size of the current subblock. For example, both the first and second prediction matrices are prediction matrices of size w*h, or the first and second prediction matrices represent the same prediction matrix.

[0406] As shown in Figure 11B, another embodiment of the present application provides another method for predictive refinement (PROF) using optical flow on an affine-coded block, comprising the following steps:

[0407] S1110. It is determined whether multiple optical flow determination conditions are met or satisfied. Here, optical flow determination conditions refer to conditions that allow the application of PROF.

[0408] S1111. If multiple optical flow determination conditions are met, the first indicator (e.g., applyProfFlag) is set to true, and a Predictive Refinement with Optical Flow (PROF) process is performed on the current subblock of the affine-coded block to obtain refined predicted sample values ​​for the current subblock of the affine-coded block. In step S1111, the optical flow (Predictive Refinement with Optical Flow, PROF) process is performed on one or more subblocks (e.g., each subblock) in the current affine picture block to obtain delta predicted values ​​(also called predictor offset values, e.g., ΔI(i,j)) for one or more subblocks (e.g., each subblock) in the current affine picture block.

[0409] In step S1111, a refined predicted sample value (e.g., predicted signal I'(i,j)) of the subblock is obtained based on the predicted delta value (e.g., ΔI(i,j)) and the predicted sample value (e.g., predicted signal I(i,j)) of the subblock.

[0410] In this specification, I(i,j) represents the predicted sample value of the current sample in the current subblock (e.g., the original predicted sample value obtained through motion compensation), ΔI(i,j) represents the delta predicted value of the current sample in the current subblock, and I'(i,j) represents the refined predicted sample value of the current sample in the current subblock. For example, original predicted sample value + delta predicted value = refined predicted sample value. Obtaining refined predicted sample values ​​for multiple samples in the current subblock (e.g., all samples) is equivalent to obtaining the refined predicted sample value for the current subblock.

[0411] When refined predicted sample values ​​are generated for each subblock of an affine-coded block, it can be understood that the refined predicted sample values ​​for the affine-coded block are generated naturally. S1113, when at least one of the multiple optical flow determination conditions is not met or satisfied, the first indicator (e.g., applyProfFlag) is set to false and the PROF process is skipped.

[0412] If the constraints for applying PROF are used to determine whether or not to apply PROF, it can be understood that step S1110 may be modified to determine whether any of the constraints for applying PROF are not met. In this case, step S1111 is modified so that if any of the constraints for applying PROF are not met or satisfied, the first indicator (e.g., applyProfFlag) is set to equal to true, and a Predictive Refinement (PROF) process with optical flow is performed on the current subblock of the affine-coded block to obtain refined predicted sample values ​​for the current subblock of the affine-coded block. Thus, step S1113 is modified so that if at least one of the constraints for applying PROF is met, the first indicator (e.g., applyProfFlag) is set to equal to false, and the PROF process is skipped.

[0413] In one implementation, if a first indicator (e.g., applyProfFlag) is a first value (e.g., 1), then an optical flow (e.g., PROF) operation is performed on one or more subblocks (e.g., each subblock) within the current affine picture block, or Instead, if the first indicator (e.g., applyProfFlag) is a second value (e.g., 0), the execution of optical flow (e.g., PROF) processing for one or more subblocks (e.g., each subblock) within the current affine picture block is skipped.

[0414] In one implementation, the value of the first indicator depends on whether the optical flow determination condition is met, and the optical flow determination condition includes one or more of the following: A first indicator (e.g., sps_prof_enabled_flag or sps_bdof_enabled_flag) is used to indicate that PROF is enabled for the current picture unit. Note that in this specification, the current picture unit may be, for example, the current sequence, the current picture, the current slice, or the current tile group. These examples of the current picture unit are not limited to these. A second directive (e.g., fallbackModeTriggered) is used to indicate that the current affine picture block is being separated. The current affine picture block is a single predictive affine picture block. The size of the subblocks within the affine picture block is greater than N×N, and N=4. The current affine picture block is a single-prediction affine picture block, and the size of the subblocks within the affine picture block is equal to N × N, where N = 4. Alternatively, The current affine picture block is a bipredictive affine picture block, and the size of the subblocks within the affine picture block is greater than N×N, so N=4.

[0415] This application includes, but is not limited to, the aforementioned optical flow determination conditions, and it should be noted that additional or different optical flow determination conditions may be set based on different application scenarios.

[0416] In this embodiment of the present application, for example, applyProfFlag is set to 1 when all of the following conditions are met. - sps_prof_enabled_flag==1 - fallbackModeTriggered==0 - inter_pred_idc[x0][y0]=PRED_L0 or PRED_L1 (or predFlagL0=1, predFlagL1=0; or predFlagL1=1, predFlagL0=0) - Other conditions

[0417] In another embodiment, for example, applyProfFlag is set to 1 when all of the following conditions are met. - sps_prof_enabled_flag==1 - fallbackModeTriggered==0 - Other conditions

[0418] If the constraints for applying PROF are used in S1110 instead of the optical flow determination conditions, it can be understood that applyProfFlag is set to 1 when none of the following constraints are met. - sps_prof_disabled_flag==1 - fallbackModeTriggered==1 - Other conditions

[0419] For details regarding the execution entities of the steps in the prediction method provided in this embodiment of the present application, as well as extensions and variations of these steps, please refer to the preceding description of the corresponding method. For brevity, further details are not described herein.

[0420] Another embodiment of this application further states A step of obtaining a first prediction matrix for an M*N block based on motion information (e.g., motion vectors) of multiple subblocks within the current affine picture block, where, for example, the M*N block is 16*16 as shown in Figure 9E, and for example, the 16*16 block (or 16*16 window) contains 16 4*4 subblocks, and a step of A step of calculating a horizontal prediction gradient matrix and a vertical prediction gradient matrix based on a second prediction matrix, wherein the size of the second prediction matrix is ​​greater than or equal to the size of the first prediction matrix, and the size of the second prediction matrix is ​​greater than or equal to the size of the horizontal prediction gradient matrix and the vertical prediction gradient matrix. The steps include: calculating the delta prediction matrix of the M*N block (e.g., ΔI(i,j) of the predicted signal) based on the horizontal prediction gradient matrix, the vertical prediction gradient matrix, and the motion vector difference between the motion vector of the current pixel unit in the M*N block (e.g., the current pixel or current pixel block, such as a 2x2 pixel block) and the motion vector of the pixel at the center of the M*N block; Another PROF process is provided, which includes the step of obtaining a refined third prediction matrix (e.g., prediction signal I'(i,j)) of an M*N block based on a delta prediction matrix (e.g., ΔI(i,j)) and a first prediction matrix (e.g., prediction signal I(i,j)).

[0421] In another possible design, the first prediction matrix is ​​represented by I1(i,j), where the range of values ​​for i is [0,size_w-1] and the range of values ​​for j is [0,size_h-1]. The second prediction matrix is ​​represented by I2(i,j), where the range of i is [-1,size_w] and the range of j is [-1,size_h], with size_w=min(W,m), size_h=min(H,m), and m=16. The horizontal gradient prediction matrix is ​​represented by X(i,j), where the range of i is [0,size_w-1] and the range of j is [0,size_h-1]. The vertical gradient prediction matrix is ​​represented by Y(i,j), where the range of i is [0,size_w-1] and the range of j is [0,size_h-1]. W represents the width of the current affine picture block, H represents the height of the current affine picture block, and (x,y) represents the position coordinates of each sample within the current affine picture block.

[0422] As shown in Figure 9E, in some examples, the affine picture block is implicitly divided into 16x16 blocks, and the gradient matrix is ​​calculated for each 16x16 block. Correspondingly, the second prediction matrix is ​​represented by I2(i,j), where the range of i is [-1,size_w] and the range of j is [-1,size_h], with size_w=min(w,m), size_h=min(h,m), and m=16. The horizontal prediction gradient matrix is ​​represented by X(i,j), where the range of i is [0,size_w-1] and the range of j is [0,size_h-1]. The vertical prediction gradient matrix is ​​represented by Y(i,j), where the range of i is [0,size_w-1] and the range of j is [0,size_h-1].

[0423] In another possible design, the horizontal and vertical predicted gradient matrices (e.g., size_w*size_h gradient values) are calculated based on a second prediction matrix (e.g., (size_w+2)*(size_h+2) predicted signal). This includes calculating a horizontal prediction gradient matrix and a vertical prediction gradient matrix based on a second prediction matrix, wherein the horizontal prediction gradient matrix and the vertical prediction gradient matrix each include the horizontal prediction gradient matrix and the vertical prediction gradient matrix of a subblock, The second prediction matrix is ​​represented by I2(i,j), where the range of i is [-1,size_w] and the range of j is [-1,size_h], with size_w=min(W,m), size_h=min(H,m), and m=16. The horizontal gradient prediction matrix is ​​represented by X(i,j), where the range of i is [0,size_w-1] and the range of j is [0,size_h-1]. The vertical gradient prediction matrix is ​​represented by Y(i,j), where the range of i is [0,size_w-1] and the range of j is [0,size_h-1]. W represents the width of the current affine picture block, H represents the height of the current affine picture block, and (i,j) represents the position coordinates of each sample within the current affine picture block.

[0424] From the above explanation, it is clear that the current affine picture block is implicitly divided into 16x16 blocks, and the gradient matrix is ​​calculated for each 16x16 block. It should be understood that m=16 is used only as an example in this specification and should not be interpreted as limiting. Various other values ​​of m, such as m=32, can be used.

[0425] In one possible design, the method is used for single prediction, and the motion information includes first motion information corresponding to a first reference frame list, or second motion information corresponding to a second reference frame list. The first prediction matrix includes either the first initial prediction matrix or the second initial prediction matrix, where the first initial prediction matrix is ​​obtained based on first motion information and the second initial prediction matrix is ​​obtained based on second motion information. The horizontal prediction gradient matrix includes (and is) a first horizontal prediction gradient matrix or a second horizontal prediction gradient matrix, the first of which is obtained through calculations based on an extended first initial prediction matrix, and the second of which is obtained through calculations based on an extended second initial prediction matrix. The vertical prediction gradient matrix includes (and is) a first vertical prediction gradient matrix or a second vertical prediction gradient matrix, the first of which is obtained through calculations based on an extended first initial prediction matrix, and the second of which is obtained through calculations based on an extended second initial prediction matrix. The delta prediction matrix includes a first delta prediction matrix corresponding to a first reference frame list or a second delta prediction matrix corresponding to a second reference frame list, the first delta prediction matrix being obtained through calculations based on a first horizontal prediction gradient matrix, a first vertical prediction gradient matrix, and a first motion vector difference (e.g., forward motion vector difference) for each sample unit in the subblock relative to the sample at the center of the subblock, and the second delta prediction matrix being obtained through calculations based on a second horizontal prediction gradient matrix, a second vertical prediction gradient matrix, and a second motion vector difference (e.g., backward motion vector difference) for each sample unit in the subblock relative to the sample at the center of the subblock.

[0426] In one possible design, the method is used for biprediction, and the motion information includes first motion information corresponding to a first reference frame list and second motion information corresponding to a second reference frame list. The first prediction matrix includes a first initial prediction matrix and a second initial prediction matrix, the first initial prediction matrix being obtained based on first motion information, and the second initial prediction matrix being obtained based on second motion information. The horizontal prediction gradient matrix includes a first horizontal prediction gradient matrix and a second horizontal prediction gradient matrix, the first of which is obtained through calculations based on an extended first initial prediction matrix, and the second of which is obtained through calculations based on an extended second initial prediction matrix. The vertical prediction gradient matrix includes a first vertical prediction gradient matrix and a second vertical prediction gradient matrix, the first of which is obtained through calculations based on an extended first initial prediction matrix, and the second of which is obtained through calculations based on an extended second initial prediction matrix. The delta prediction matrix includes a first delta prediction matrix corresponding to a first reference frame list and a second delta prediction matrix corresponding to a second reference frame list, the first delta prediction matrix being obtained through calculations based on a first horizontal prediction gradient matrix, a first vertical prediction gradient matrix, and a first motion vector difference (e.g., forward motion vector difference) for each sample unit in the subblock relative to the sample at the center of the subblock, and the second delta prediction matrix being obtained through calculations based on a second horizontal prediction gradient matrix, a second vertical prediction gradient matrix, and a second motion vector difference (e.g., backward motion vector difference) for each sample unit in the subblock relative to the sample at the center of the subblock.

[0427] In one possible design, the method is used for simple prediction. The motion information includes first motion information corresponding to the first reference frame list, or second motion information corresponding to the second reference frame list. The first prediction matrix includes either the first initial prediction matrix or the second initial prediction matrix, the first initial prediction matrix being obtained based on first motion information, and the second initial prediction matrix being obtained based on second motion information.

[0428] In one possible design, the method is used for biprediction. The motion information includes first motion information corresponding to the first reference frame list and second motion information corresponding to the second reference frame list. The first prediction matrix includes a first initial prediction matrix and a second initial prediction matrix, the first initial prediction matrix being obtained based on first motion information, and the second initial prediction matrix being obtained based on second motion information. The step of obtaining the predicted matrix of a subblock based on the subblock's motion information is: The process includes a step of performing a weighted sum on the sample values ​​at the same position in the first and second initial prediction matrices to obtain the prediction matrix for the subblock. It should be understood that the sample values ​​in the first and second initial prediction matrices may be refined separately before the weighted sum is performed here.

[0429] In another possible design, the PROF process is described as consisting of the following four steps:

[0430] Step 1) Subblock-based affine motion compensation is performed to generate a subblock prediction I(i,j). For example, i has a value from [0,subW+1] or [-1,subW] and j has a value from [0,subH+1] or [-1,subH]. Since i has a value from [0,subW+1] and j has a value from [0,subH+1], it can be understood that the top-left sample (or origin of coordinates) is located at (1,1), while since i has a value from [-1,subW] and j has a value from [-1,subH], the top-left sample is located at (0,0).

[0431] Step 2) Spatial gradient g of subblock prediction x (i,j) and g y (i,j) is calculated at each sample position using a 3-tap filter [-1,0,1]. g x (i,j)=I(i+1,j)-I(i-1,j) g y (i,j)=I(i,j+1)-I(i,j-1)

[0432] The subblock prediction is extended by only one sample on each side for gradient calculation. To reduce memory bandwidth and complexity, the samples on the extended boundary are copied from the nearest integer sample position in the reference picture. Thus, additional interpolation for padding regions is avoided.

[0433] Step 3) The Luma prediction refinement is calculated using the optical flow formula. ΔI(i,j)=g x (i,j)*Δv x (i,j)+g y (i,j)*Δv y (i,j) Here, as shown in Figure 10, Δv(i,j) is the difference between the sample MV calculated for the sample position (i,j) represented by v(i,j) and the subblock MV of the subblock to which the sample (i,j) belongs.

[0434] In other words, the MV of the central sample in each 4x4 grid is calculated, and then the MV of each sample in the sub-block is calculated. The difference Δv(i,j) between the MV of each sample and the MV of the central sample can be obtained.

[0435] Since the affine model parameters and the sample position relative to the subblock center do not change from subblock to subblock, Δv(i,j) can be calculated for the first subblock and reused for other subblocks in the same CU. Let x and y be the horizontal and vertical offsets from the sample position relative to the subblock center, and Δv(x,y) can be derived by the following equation.

[0436]

number

[0437] For a 4-parameter affine model,

[0438]

number

[0439] For a 6-parameter affine model,

[0440]

number

[0441] Here, (v0x,v0y), (v1x,v1y), and (v2x,v2y) are the motion vectors of the top-left, top-right, and bottom-left control points, and w and h are the width and height of the control unit (CU).

[0442] Step 4) Finally, the Luma prediction refinement is added to the subblock prediction I(i,j). The final prediction I' is generated as shown in the following formula. I'(i,j)=I(i,j)+ΔI(i,j)

[0443] Figure 15 shows an apparatus 1500 for predictive refinement (PROF) using optical flow on an affine-coded block, according to another aspect of the present disclosure. In one example, Device 1500 is, A decision unit 1501 configured to determine that none of the multiple constraints for applying PROF are met, The system includes a prediction processing unit 1503 configured to perform prediction refinement using optical flow and a PROF process on the current subblock of the affine-coded block in order to obtain refined predicted sample values ​​for the current subblock of the affine-coded block. When refined predicted sample values ​​are generated for each subblock of the affine-coded block, it can be understood that refined predicted sample values ​​for the affine-coded block are generated naturally.

[0444] In another example, Device 1500 is, A decision unit 1501 configured to determine that multiple optical flow determination conditions are met, wherein the multiple optical flow determination conditions refer to conditions that allow the application of PROF, and The system includes a prediction processing unit 1503 configured to perform a PROF process on the current subblock of an affine-coded block in order to obtain refined predicted sample values ​​for the current subblock of the affine-coded block. When refined predicted sample values ​​are generated for each subblock of the affine-coded block, it can be understood that refined predicted sample values ​​for the affine-coded block are generated naturally.

[0445] Accordingly, in one example, the exemplary structure of device 1500 may correspond to the encoder 20 in Figure 2. In another example, the exemplary structure of device 1500 may correspond to the decoder 30 in Figure 3.

[0446] In another example, the exemplary structure of the device 1500 may correspond to the interpretation unit 244 in Figure 2. In yet another example, the exemplary structure of the device 1500 may correspond to the interpretation unit 344 in Figure 3.

[0447] The decision unit and prediction processing unit (corresponding to the interprediction module) in the encoder 20 or decoder 30 provided in this embodiment of the present application are functional entities for carrying out various execution steps included in the corresponding method described above, that is, they may be understood to have functional entities for fully carrying out the steps of the method of the present application and extensions and variations thereof. For further details, please refer to the above description of the corresponding method. For brevity, further details will not be described herein again.

[0448] The following is a description of the encoding and decoding methods, as shown in the embodiments mentioned above, and the application of systems using them.

[0449] Figure 16 is a block diagram showing a content supply system 3100 for realizing a content distribution service. This content supply system 3100 includes a capture device 3102, a terminal device 3106, and optionally a display 3126. The capture device 3102 communicates with the terminal device 3106 via a communication link 3104. The communication link may include the communication channel 13 described above. The communication link 3104 may include, but is not limited to, Wi-Fi, Ethernet, cable, wireless (3G / 4G / 5G), USB, or any combination of these.

[0450] The capture device 3102 can generate data and encode it using an encoding method as shown in the above embodiment. Alternatively, the capture device 3102 may deliver the data to a streaming server (not shown in the figure), which encodes the data and transmits the encoded data to the terminal device 3106. The capture device 3102 includes, but is not limited to, a camera, a smartphone or tablet, a computer or laptop, a video conferencing system, a PDA, an in-vehicle device, or any combination thereof. For example, the capture device 3102 may include a source device 12 as described above. When the data includes video, the video encoder 20 included in the capture device 3102 may actually perform video encoding. When the data includes audio (i.e., speech), the audio encoder included in the capture device 3102 may actually perform audio encoding. In some realistic scenarios, the capture device 3102 delivers the encoded video and audio data by multiplexing them together. In other realistic scenarios, for example in a video conferencing system, the encoded audio and video data are not multiplexed. The capture device 3102 distributes the encoded audio data and encoded video data separately to the terminal device 3106.

[0451] In the content supply system 3100, the terminal device 310 receives and plays back encoded data. The terminal device 3106 may be a data receiving and recovery capable device such as a smartphone or tablet 3108, a computer or laptop 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a TV 3114, a set-top box (STB) 3116, a video conferencing system 3118, a video surveillance system 3120, a personal digital assistant (PDA) 3122, an in-vehicle device 3124, or any combination thereof capable of decoding the encoded data mentioned above. For example, the terminal device 3106 may include the destination device 14 as described above. When the encoded data includes video, the video decoder 30 included in the terminal device preferentially performs video decoding. When the encoded data includes audio, the audio decoder included in the terminal device preferentially performs audio decoding.

[0452] In terminal devices with a display, such as a smartphone or tablet 3108, a computer or laptop 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a TV 3114, a personal digital assistant (PDA) 3122, or an in-vehicle device 3124, the terminal device can supply the decoded data to its display. In terminal devices without a display, such as an STB 3116, a video conferencing system 3118, or a video surveillance system 3120, an external display 3126 is connected to receive and display the decoded data.

[0453] When each device in this system performs encoding or decoding, a picture encoding device or a picture decoding device, such as those shown in the embodiments described above, may be used.

[0454] Figure 17 shows the structure of an example terminal device 3106. After the terminal device 3106 receives a stream from the capture device 3102, the protocol progression unit 3202 analyzes the transmission protocol of the stream. The protocol may include, but is not limited to, Real-Time Streaming Protocol (RTSP), Hypertext Transfer Protocol (HTTP), HTTP Live Streaming Protocol (HLS), MPEG-DASH, Real-Time Transport Protocol (RTP), Real-Time Messaging Protocol (RTMP), or any combination of these.

[0455] After the protocol processing unit 3202 processes the stream, a stream file is generated. This file is output to the demultiplexing unit 3204. The demultiplexing unit 3204 can separate the multiplexed data into encoded audio data and encoded video data. As described above, in some realistic scenarios, such as in a video conferencing system, the encoded audio data and encoded video data are not multiplexed. In this situation, the encoded data is sent to the video decoder 3206 and audio decoder 3208 without going through the demultiplexing unit 3204.

[0456] Through demultiplexing, a video elementary stream (ES), an audio ES, and optionally subtitles are generated. A video decoder 3206, including a video decoder 30 as described in the embodiments mentioned above, decodes the video ES by the decoding method shown in the embodiments mentioned above to generate video frames and supplies this data to the synchronization unit 3212. An audio decoder 3208 decodes the audio ES to generate audio frames and supplies this data to the synchronization unit 3212. Alternatively, video frames may be stored in a buffer (not shown in Figure Y) before supplying them to the synchronization unit 3212. Similarly, audio frames may be stored in a buffer (not shown in Figure Y) before supplying them to the synchronization unit 3212.

[0457] The synchronization unit 3212 synchronizes video frames and audio frames and supplies video / audio to the video / audio display 3214. For example, the synchronization unit 3212 synchronizes the presentation of video and audio information. The information may be coded in syntax using timestamps for the presentation of the coded audio and visual data, and timestamps for the delivery of the data stream itself.

[0458] If subtitles are included in the stream, the subtitle decoder 3210 decodes the subtitles, synchronizes them with the video and audio frames, and supplies the video / audio / subtitles to the video / audio / subtitle display 3216.

[0459] The present invention is not limited to the systems mentioned above, and either the picture encoding device or the picture decoding device in the embodiments mentioned above may be incorporated into other systems, such as car systems.

[0460] The mathematical operators used in this application are similar to those used in the C programming language. However, the results of integer division and arithmetic shift operations are more precisely defined, and additional operations such as exponential and real-valued division are defined.

[0461] For a description of the relevant content, implementation of the relevant steps, and beneficial effects in this embodiment, please refer to the corresponding sections above, or make minor modifications based on those sections. Further details will not be described here again.

[0462] It should be noted that, where no inconsistencies arise, some features from any two or more of the embodiments described above may be combined to form a new embodiment. In addition, some features from any one of the embodiments described above may be used independently as a separate embodiment.

[0463] The above primarily describes the solutions provided in the embodiments of this application from a methodological standpoint. To implement the aforementioned functions, corresponding hardware structures and / or software modules for performing the functions are included. In combination with the examples described in the embodiments disclosed herein, those skilled in the art should readily recognize that the units and algorithmic steps may be implemented in this application by hardware or by a combination of hardware and computer software. Whether the functions are performed by hardware or by hardware driven by computer software depends on the specific application and design constraints of the technical solution. For each particular application, those skilled in the art may implement the functions described using different methods, but such implementations should not be considered outside the scope of this application.

[0464] The partitioning of an encoder / decoder into functional modules in the embodiments of this application may be carried out based on the examples of the methods described above. For example, each functional module may be obtained through partitioning corresponding to each function, or at least two functions may be integrated into one processing module. The integrated module may be implemented in hardware form or in the form of a software functional module. Note that in the embodiments of this application, the partitioning of modules is illustrative and merely represents a logical partitioning of functions. In actual implementations, other partitioning methods may be used.

[0465] While embodiments of the present invention have been primarily described in relation to video coding, it should be noted that the coding system 10, encoder 20, and decoder 30 (and correspondingly system 10), as well as other embodiments described herein, may also be configured for processing or coding still pictures, i.e., for processing or coding individual pictures independent of preceding or consecutive pictures, as in video coding. Generally, when picture processing coding is limited to a single picture 17, only the interpretation units 244 (encoder) and 344 (decoder) may not be available. All other functions (also called tools or techniques) of the video encoder 20 and video decoder 30 may be equally used for still picture processing, such as residual calculation 204 / 304, transformation 206, quantization 208, inverse quantization 210 / 310, (inverse) transformation 212 / 312, segmentation 262 / 362, intra prediction 254 / 354, and / or loop filtering 220, 320, as well as entropy coding 270 and entropy decoding 304.

[0466] For example, embodiments of encoder 20 and decoder 30, and functions described herein with reference to encoder 20 and decoder 30, may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes over a communication medium and executed by a hardware-based processing unit. Computer-readable medium may include computer-readable storage mediums corresponding to tangible media such as data storage mediums, or communication mediums including any medium that facilitates the transfer of computer programs from one location to another, for example, according to a communication protocol. Thus, computer-readable mediums may generally correspond to (1) non-transient tangible computer-readable storage mediums, or (2) communication mediums such as signal waves or carrier waves. Data storage mediums may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for the implementation of the techniques described herein. Computer program products may include computer-readable mediums.

[0467] As an example, and not an limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media used to store desired program code in the form of instructions or data structures that can be accessed by a computer. Also, any connection is appropriately called computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but instead refer to non-temporary tangible storage media. As used herein, the terms "disk" and "disc" include Compact Disc (CD), LaserDisc®, Optical Disc, Digital Multipurpose Disc (DVD), Floppy Disc, and Blu-ray® Disc, where a "disk" typically reproduces data magnetically, and a "disc" reproduces data optically using a laser. Any combination of the above should also be included within the scope of computer-readable media.

[0468] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term “processor” as used herein may refer to any of the aforementioned structures or any other structures suitable for implementing the techniques described herein. In addition, in some embodiments, the functions described herein may be provided within dedicated hardware and / or software modules configured for or incorporated into synthesized codecs. Furthermore, the techniques may be fully implemented in one or more circuits or logic elements.

[0469] The techniques of this disclosure can be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to highlight the functional aspects of devices configured to perform the techniques disclosed, but they do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, or they may be provided by a collection of interoperable hardware units, including one or more processors as described above, in conjunction with appropriate software and / or firmware. [Explanation of symbols]

[0470] 10 Video Coding Systems 12 Source Devices 13 Communication Channels 14 Destination device 16 Picture Sources 17 Picture Data 18 preprocessors 19 Preprocessed picture data 20 encoders 21 Encoded picture data 22 Communication Interfaces 28 Communication Interfaces 30 Decoders 31 Decrypted picture data 32 Post-Processors 33 Post-processed picture data 34 Display Devices 40 Video Coding Systems 41 Imaging devices 42 Antennas 43 processors 44 Memory Store 45 Display devices 46 Processing Circuit 201 Input 203 Picture Block 204 Residual Calculation Unit 205 Residual Block 206 Conversion Processing Unit 207 Conversion coefficient 208 Quantization Units 209 Quantized coefficients 210 Inverse Quantization Unit 211 Inversely quantized coefficients 212 Inverse Transform Processing Unit 213 Reconstructed residual block 214 Reconstruction Unit 215 Reconstructed Blocks 220 Loop Filter Unit 221 filtered blocks 230 Decoded picture buffer 231 Decrypted picture 244 Interpretation Units 254 Intra Prediction Units 260 Mode Selection Unit 262 division units 265 Prediction Blocks 266 Syntax Elements 270 Entropy Coding Units 272 Output 304 Entropy Decoding Unit 309 Quantized coefficients 310 Inverse Quantization Unit 311 Inversely quantized coefficients 312 Inverse Transform Processing Unit 313 Reconstructed residual block 314 Reconstruction Unit 315 Reconstructed Blocks 320 Loop Filter 321 filtered blocks 330 Decode picture buffer 331 Decrypted picture 332 output 344 Interpretation Units 354 Intra Prediction Units 360 Mode Applicable Unit 365 Prediction Block 366 Syntax Elements 400 video coding devices 410 Input Ports 420 Receiver Unit 430 processors 440 Transmitter Unit 450 output ports 460 memory 470 coding modules 500 devices 502 Processors 504 memory 506 data 508 Operating Systems 510 Application Programs 512 Bus 518 displays 900 Prediction signal window 1500 equipment 1501 Decision Unit 1503 Prediction Processing Unit 3100 Content Supply System 3102 Capture Device 3104 Communication Link 3106 Terminal device 3108 Smartphone / Pad 3110 Computer / Laptop 3112 Network Video Recorder (NVR) / Digital Video Recorder (DVR) 3114 TV 3116 Set-top box (STB) 3118 Video conferencing system 3120 Video Surveillance System 3122 Personal Digital Assistant (PDA) 3124 In-vehicle devices 3126 Display 3202 Protocol Progress Unit 3204 Reverse Multiplexing Unit 3206 Video Decoder 3208 Audio Decoder 3210 Subtitle Decoder 3212 Synchronization Unit 3214 Video / Audio Display 3216 Video / Audio / Subtitle Display

Claims

1. A method for predictive refinement (PROF) using optical flow on affine-coded blocks in video encoding or video decoding, When the first indicator is set to true equality, the process includes the step of performing a PROF process on the current subblock of the affine-coded block to obtain a refined predicted sample value for the current subblock of the affine-coded block. If none of the constraints for applying PROF are met for the affine-coded block, the first indicator is set to equal to true; if one or more of the constraints for applying PROF are met for the affine-coded block, the first indicator is set to equal to false. The aforementioned constraints for applying PROF are: The first instruction indicates that PROF is invalid for the picture containing the affine-coded block, and The second instruction information indicates that the affine-coded block is not divided, The step of performing the PROF process on the current subblock of the affine-coded block is: A step of obtaining a prediction matrix, wherein the prediction matrix is ​​generated based on the motion information of the current subblock, A step of generating a horizontal prediction gradient matrix and a vertical prediction gradient matrix based on the prediction matrix, wherein the horizontal prediction gradient matrix and the vertical prediction gradient matrix have the same size, and the size of the prediction matrix is ​​greater than or equal to the size of the horizontal prediction gradient matrix and the vertical prediction gradient matrix, The steps include calculating the delta prediction value of the current sample in the current subblock based on the horizontal prediction gradient value of the current sample in the horizontal prediction gradient matrix, the vertical prediction gradient value of the current sample in the vertical prediction gradient matrix, and the difference between the motion vector of the current sample in the current subblock and the motion vector of the sample at the center of the current subblock, A method comprising the step of obtaining a refined predicted sample value of the current sample in the current subblock based on the predicted delta value of the current sample and the predicted sample value of the current sample in the current subblock.

2. The method according to claim 1, wherein one of the plurality of constraints for applying PROF is that the variable fallbackModeTriggered is set to 1.

3. A step of determining whether the plurality of constraints for applying PROF are satisfied for the affine-coded block, If all of the constraints for applying PROF are not met for the affine-coded block, the first indicator is set to true equality, and the PROF process is performed on the current subblock of the affine-coded block to obtain the refined predicted sample value of the current subblock of the affine-coded block. The method according to claim 1 or 2, further comprising the step that if one or more of the plurality of constraints for applying PROF are met for the affine-coded block, the first indicator is set to equal to false and the PROF process for the current subblock of the affine-coded block is skipped.

4. The step of obtaining a prediction matrix is: A step of generating an initial prediction matrix, wherein the elements of the initial prediction matrix correspond to the predicted sample values ​​of the current subblock, and the predicted sample values ​​of the current subblock are obtained by motion compensation based on the motion information of the current subblock; and a step of generating a prediction matrix based on the initial prediction matrix, wherein the size of the prediction matrix is ​​greater than the size of the current subblock, or The method according to claim 1, comprising the step of generating the prediction matrix, wherein the elements of the prediction matrix correspond to predicted sample values ​​of the current subblock, the predicted sample values ​​of the current subblock are obtained by motion compensation based on the motion information of the current subblock, and the size of the prediction matrix is ​​equal to the size of the current subblock.

5. The elements of the aforementioned prediction matrix are I 1 It is represented by (p,q), where the range of p is [-1, sbW] and the range of q is [-1, sbH]. The elements of the horizontal prediction gradient matrix are represented by X(i,j), which corresponds to the sample(i,j) of the current subblock in the affine-coded block, where the range of the value of i is [0,sbW-1] and the range of the value of j is [0,sbH-1]. The elements of the vertical prediction gradient matrix are represented by Y(i,j), which corresponds to the sample(i,j) of the current subblock in the affine-coded block, where the range of the value of i is [0,sbW-1] and the range of the value of j is [0,sbH-1]. The method according to any one of claims 1 to 4, wherein sbW represents the width of the current subblock in the affine-coated block, and sbH represents the height of the current subblock in the affine-coated block.

6. An encoder (20) comprising a processing circuit for performing the method described in any one of claims 1 to 5.

7. A decoder (30) comprising a processing circuit for performing the method described in any one of claims 1 to 5.

8. A computer program comprising program code for performing the method described in any one of claims 1 to 5.

9. A computer-readable medium carrying program code that, when executed by a computer device, causes the computer device to perform the method described in any one of claims 1 to 5.

Images