Inter-frame prediction method, encoder, decoder and computer storage medium

Abstract

The embodiment of the present application discloses an inter-frame prediction method, an encoder, a decoder and a computer storage medium. The decoder parses a code stream to obtain a prediction mode parameter of a current block; when the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using the inter-frame prediction mode, a first motion vector of a sub-block of the current block is determined; wherein the current block includes multiple sub-blocks; based on the first motion vector, a first prediction value of the sub-block and a first motion vector deviation between each pixel position in the sub-block and the sub-block are determined; according to the first motion vector deviation, a center position and a second motion vector deviation corresponding to each pixel position are determined; based on the first prediction value, secondary prediction or PROF processing is performed according to the center position and the second motion vector deviation to determine a second prediction value of the sub-block, and the second prediction value is determined as the inter-frame prediction value of the sub-block.

Description

Technical Field

[0001] The present application relates to the technical field of video coding and decoding, and in particular to an inter-frame prediction method, an encoder, a decoder and a computer storage medium. Background Art

[0002] In the field of video coding and decoding, in order to balance performance and cost, in general, affine prediction in Versatile Video Coding (VVC) and Audio Videocoding Standard Workgroup of China (AVS) is implemented based on sub-blocks. At present, on the one hand, prediction refinement with optical flow (PROF) is proposed to use the optical flow principle to correct the affine prediction based on sub-blocks; on the other hand, it is proposed to use secondary prediction after sub-block-based prediction to obtain more accurate prediction values, thereby achieving improvement in affine prediction. Specifically, PROF relies on the horizontal and vertical gradients of the reference position to correct the affine prediction results, while the secondary prediction uses a filter to predict a pixel position in the sub-block again.

[0003] However, when there is a large motion vector deviation in the center position of the filter required for secondary prediction or PROF, the prediction accuracy may be greatly reduced. In other words, the existing PROF corrected prediction value method and secondary prediction method are not rigorous. When improving affine prediction, they cannot be well applied to all scenarios, and the encoding performance needs to be improved. Summary of the invention

[0004] The present application proposes an inter-frame prediction method, an encoder, a decoder and a computer storage medium, which can reduce the prediction error, greatly improve the coding performance, and thus improve the coding and decoding efficiency.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an inter-frame prediction method, which is applied to a decoder. The method includes:

[0007] Parse the bitstream and obtain the prediction mode parameters of the current block;

[0008] When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a sub-block of the current block; wherein the current block includes a plurality of sub-blocks;

[0009] determining a first prediction value of the sub-block based on the first motion vector and a first motion vector deviation between each pixel position in the sub-block and the sub-block;

[0010] Determine a center position and a second motion vector deviation corresponding to each pixel position according to the first motion vector deviation;

[0011] Based on the first prediction value, secondary prediction or PROF processing is performed according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and the second prediction value is determined as the inter-frame prediction value of the sub-block.

[0012] In a second aspect, an embodiment of the present application provides an inter-frame prediction method, which is applied to an encoder, and the method includes:

[0013] Determining prediction mode parameters for the current block;

[0014] When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a sub-block of the current block; wherein the current block includes a plurality of sub-blocks;

[0015] determining a first prediction value of the sub-block based on the first motion vector and a first motion vector deviation between each pixel position in the sub-block and the sub-block;

[0016] Determine a center position and a second motion vector deviation corresponding to each pixel position according to the first motion vector deviation;

[0017] Based on the first prediction value, secondary prediction or PROF processing is performed according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and the second prediction value is determined as the inter-frame prediction value of the sub-block.

[0018] In a third aspect, an embodiment of the present application provides a decoder, the decoder comprising a parsing unit, a first determining unit;

[0019] The parsing unit is used to parse the bitstream and obtain the prediction mode parameters of the current block;

[0020] The first determination unit is used to determine the first motion vector of a sub-block of the current block when the prediction mode parameter indicates that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode; wherein the current block includes multiple sub-blocks; and determine the first prediction value of the sub-block based on the first motion vector, and the first motion vector deviation between each pixel position in the sub-block and the sub-block; and determine the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation; and based on the first prediction value, perform secondary prediction or PROF processing according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and determine the second prediction value as the inter-frame prediction value of the sub-block.

[0021] In a fourth aspect, an embodiment of the present application provides a decoder, comprising a first processor and a first memory storing instructions executable by the first processor, wherein when the instructions are executed, the first processor implements the inter-frame prediction method as described above.

[0022] In a fifth aspect, an embodiment of the present application provides an encoder, the encoder comprising a second determining unit;

[0023] The second determination unit is used to determine the prediction mode parameters of the current block; and when the prediction mode parameters indicate that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode, determine the first motion vector of the sub-block of the current block; wherein the current block includes multiple sub-blocks; and determine the first prediction value of the sub-block based on the first motion vector, and the first motion vector deviation between each pixel position in the sub-block and the sub-block; and determine the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation; and based on the first prediction value, perform secondary prediction or PROF processing according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and determine the second prediction value as the inter-frame prediction value of the sub-block.

[0024] In a sixth aspect, an embodiment of the present application provides an encoder, comprising a second processor and a second memory storing instructions executable by the second processor, wherein when the instructions are executed, the second processor implements the inter-frame prediction method as described above.

[0025] In a seventh aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a first processor and a second processor, the inter-frame prediction method as described above is implemented.

[0026] The embodiments of the present application provide an inter-frame prediction method, an encoder, a decoder and a computer storage medium. The decoder parses a bit stream to obtain prediction mode parameters of a current block; when the prediction mode parameters indicate that an inter-frame prediction value of the current block is determined using the inter-frame prediction mode, a first motion vector of a sub-block of the current block is determined; wherein the current block includes multiple sub-blocks; based on the first motion vector, a first prediction value of the sub-block and a first motion vector deviation between each pixel position in the sub-block and the sub-block are determined; according to the first motion vector deviation, a center position and a second motion vector deviation corresponding to each pixel position are determined; based on the first prediction value, secondary prediction or PROF processing is performed according to the center position and the second motion vector deviation to determine a second prediction value of the sub-block, and the second prediction value is determined as the inter-frame prediction value of the sub-block. The encoder determines the prediction mode parameters of the current block; when the prediction mode parameters indicate that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode, the first motion vector of the sub-block of the current block is determined; wherein the current block includes multiple sub-blocks; based on the first motion vector, the first prediction value of the sub-block is determined, and the first motion vector deviation between each pixel position in the sub-block and the sub-block is determined; according to the first motion vector deviation, the center position and the second motion vector deviation corresponding to each pixel position are determined; based on the first prediction value, secondary prediction or PROF processing is performed according to the center position and the second motion vector deviation, and the second prediction value of the sub-block is determined, and the second prediction value is determined as the inter-frame prediction value of the sub-block. That is, the inter-frame prediction method proposed in the present application can, after the prediction based on the sub-block, for the pixel position where the first motion vector deviation between the motion vector and the motion vector of the sub-block is large, the center position and the second motion vector deviation used for secondary prediction or PROF processing can be re-determined based on the first motion vector deviation, so that the second prediction value can be obtained by performing point-based secondary prediction based on the center position and the second motion vector deviation on the basis of the first prediction value of the sub-block. It can be seen that the inter-frame prediction method proposed in this application can be well applied to all scenarios, can reduce the prediction error, greatly improve the coding performance, and thus improve the coding and decoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is an illustration of an affine model. Figure 1 ;

[0028] Figure 2 This is an illustration of an affine model. Figure 2 ;

[0029] Figure 3 Schematic diagram of pixel interpolation;

[0030] Figure 4 Interpolation for sub-blocks Figure 1 ;

[0031] Figure 5Interpolation for sub-blocks Figure 2 ;

[0032] Figure 6 A schematic diagram of the motion vector for each sub-block;

[0033] Figure 7 Schematic diagram of sample location;

[0034] Figure 8 Indication of the center position Figure 1 ;

[0035] Fig. 9 Indication of the center position Figure 2 ;

[0036] Fig.10 A schematic block diagram of a video encoding system provided in an embodiment of the present application;

[0037] Fig.11 A schematic block diagram of a video decoding system provided in an embodiment of the present application;

[0038] Fig.12 Schematic diagram of the implementation process of the inter-frame prediction method Figure 1 ;

[0039] Fig.13 Indication of the center position Figure 3 ;

[0040] Fig.14 Schematic diagram of the implementation process of the inter-frame prediction method Figure 2 ;

[0041] Fig.15 Schematic diagram of the implementation process of the inter-frame prediction method Figure 3 ;

[0042] Fig.16 Schematic diagram of the implementation process of the inter-frame prediction method Figure 4 ;

[0043] Fig.17 Schematic diagram of a two-dimensional filter Figure 2 ;

[0044] Fig.18 Schematic diagram of the implementation process of the inter-frame prediction method Figure 3 ;

[0045] Fig.19 Schematic diagram of the implementation process of the inter-frame prediction method Figure 4 ;

[0046] Fig. 20 The structure of the decoder is shown in Figure 2. Figure 1 ;

[0047] Fig.21 The structure of the decoder is shown in Figure 2. Figure 2 ;

[0048] Fig. 22 The structure of the encoder is shown in Figure 2. Figure 1 ;

[0049] Fig.23 The structure of the encoder is shown in Figure 2. Figure 2 . DETAILED DESCRIPTION

[0050] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for the convenience of description, only the parts related to the related applications are shown in the accompanying drawings.

[0051] At present, the common video coding standards are based on a block-based hybrid coding framework. Each frame in the video image is divided into square largest coding units (LCU) of the same size (such as 128×128, 64×64, etc.), and each largest coding unit can be divided into rectangular coding units (CU) according to rules; and the coding unit may be divided into smaller prediction units (PU). Specifically, the hybrid coding framework may include prediction, transform, quantization, entropy coding, in loop filter and other modules; among them, the prediction module may include intra-frame prediction and inter-frame prediction, and inter-frame prediction may include motion estimation and motion compensation. Since there is a strong correlation between adjacent pixels in a frame of a video image, the intra-frame prediction method used in video coding and decoding technology can eliminate the spatial redundancy between adjacent pixels; however, since there is also a strong similarity between adjacent frames in a video image, the inter-frame prediction method is used in video coding and decoding technology to eliminate the temporal redundancy between adjacent frames, thereby improving the coding efficiency. The following application will be described in detail with inter-frame prediction.

[0052] Inter-frame prediction is to use the frames that have been encoded / decoded to predict the part of the current frame that needs to be encoded / decoded. In the block-based coding and decoding framework, the part that needs to be encoded / decoded is usually a coding unit or a prediction unit. Here, the coding unit or prediction unit that needs to be encoded / decoded is collectively referred to as the current block. Translational motion is a common and simple motion mode in video, so the prediction of translation is also a traditional prediction method in video coding and decoding. Translational motion in video can be understood as a part of the content moving from a certain position on one frame to a certain position on another frame over time. A simple unidirectional prediction of translation can be represented by a motion vector (motion vector, MV) between a certain frame and the current frame. The certain frame mentioned here is a reference frame of the current frame. The current block can find a reference block on the reference frame with the same size as the current block through the motion information containing the reference frame and the motion vector, and use this reference block as the prediction block of the current block. In an ideal translational motion, the content of the current block does not have deformation, rotation, brightness, color, etc. between different frames. However, the content in the video does not always meet this ideal situation. Bidirectional prediction can solve the above problems to a certain extent. Usually, bidirectional prediction refers to bidirectional translation prediction. Bidirectional prediction is to use the motion information of two reference frames and motion vectors to find two reference blocks of the same size as the current block from two reference frames (the two reference frames may be the same reference frame), and use these two reference blocks to generate a prediction block of the current block. The generation methods include averaging, weighted averaging, and some other calculations.

[0053] In the present application, prediction can be considered as a part of motion compensation. Some documents refer to the prediction in the present application as motion compensation. For example, the affine prediction mentioned in the present application may be referred to as affine motion compensation in some documents.

[0054] Rotation, enlargement, reduction, distortion, deformation, etc. are also common changes in videos. However, ordinary translation prediction cannot handle such changes well, so the affine prediction model is applied to video encoding and decoding, such as the affine in VVC and AVS, where the affine prediction models of VVC and AVS3 are similar. In changes such as rotation, enlargement, reduction, distortion, and deformation, it can be considered that not all points in the current block use the same MV, so it is necessary to derive the MV of each point. The affine prediction model uses a small number of parameters to derive the MV of each point through calculation. The affine prediction models of VVC and AVS3 both use 2 control point (4 parameter) and 3 control point (6 parameter) models. 2 control points are the 2 control points at the upper left and upper right corners of the current block, and 3 control points are the 3 control points at the upper left, upper right, and lower left corners of the current block. For example, Figure 1 This is an illustration of an affine model. Figure 1 , Figure 2 This is an illustration of an affine model. Figure 2 ,like Figure 1 and 2 As shown in Figure 2, each MV consists of an x ​​component and a y component, so there are 4 parameters for 2 control points and 6 parameters for 3 control points.

[0055] According to the affine prediction model, an MV can be derived for each pixel position. Each pixel position can find its corresponding position in the reference frame. If this position is not an integer pixel position, then the value of this sub-pixel position needs to be obtained by interpolation. The interpolation method used in the current video coding standard is usually implemented by a finite impulse response (FIR) filter, and the complexity (cost) of implementing this method is very high. For example, in AVS3, an 8-tap interpolation filter is used for the brightness component, and the sub-pixel accuracy of the normal mode is 1 / 4 pixel, and the sub-pixel accuracy of the affine mode is 1 / 16 pixel. For each sub-pixel point that meets the 1 / 16 pixel accuracy, 8 integer pixels in the horizontal direction and 8 integer pixels in the vertical direction, that is, 64 integer pixels, need to be interpolated. Figure 3 is a schematic diagram of pixel interpolation, such as Figure 3 As shown, the circular pixel is the desired sub-pixel point, the dark square pixel is the position of the whole pixel corresponding to the sub-pixel, the vector between the two is the motion vector of the sub-pixel, and the light square pixel is the pixel needed to interpolate the circular sub-pixel position. To obtain the value of the sub-pixel position, the pixel values ​​of these 8x8 light square pixel areas need to be interpolated, including the dark pixel position.

[0056] In the traditional translation prediction, the MV of each pixel position of the current block is the same. If the concept of sub-block is further introduced, the size of the sub-block is 4x4, 8x8, etc. Figure 4 Interpolation for sub-blocks Figure 1 , the pixel area required for 4x4 block interpolation is as follows Figure 4 shown. Figure 5 Interpolation for sub-blocks Figure 2 , the pixel area required for 8x8 block interpolation is as follows Figure 5 shown.

[0057] If the MV of each pixel position in a sub-block is the same, then the pixel positions in a sub-block can be interpolated together, thereby sharing the bandwidth, using filters with the same phase, and sharing the intermediate values ​​of the interpolation process. However, if each pixel uses an MV, the bandwidth will increase, and filters with different phases may be used and the intermediate values ​​of the interpolation process cannot be shared.

[0058] Point-based affine prediction is very expensive. Therefore, in order to balance performance and cost, affine prediction in VVC and AVS3 is implemented based on sub-blocks. The sub-block size in AVS3 is 4x4 and 8x8, and the sub-block size of 4x4 is used in VVC. Each sub-block has an MV, and the pixel positions inside the sub-block share the same MV. Therefore, all pixel positions inside the sub-block are uniformly interpolated. Through the above method, the motion compensation complexity of sub-block-based affine prediction is similar to that of other sub-block-based prediction methods.

[0059] It can be seen that in the affine prediction method based on sub-blocks, the pixel positions inside the sub-block share the same MV, and the method for determining this shared MV is to take the MV of the center of the current sub-block. For sub-blocks such as 4x4 and 8x8 that have at least one even number of pixels in the horizontal and vertical directions, their centers actually fall on non-integer pixel positions. The current standards all take an integer pixel position. For example, for a 4x4 sub-block, take the pixel position that is (2, 2) away from the upper left corner. For an 8x8 sub-block, take the pixel position that is (4, 4) away from the upper left corner.

[0060] The affine prediction model can derive the MV of each pixel position based on the control points (2 control points or 3 control points) used by the current block. In the sub-block-based affine prediction, the MV of this position is calculated based on the pixel position in the previous paragraph as the MV of the sub-block. Figure 6 A schematic diagram of the motion vector for each sub-block, such as Figure 6 As shown, in order to derive the motion vector of each sub-block, the motion vector of the center sample of each sub-block is rounded to 1 / 16 accuracy as shown in the figure, and then motion compensation is performed.

[0061] With the development of technology, a method called PROF, an improved prediction technique using optical flow, has been proposed. This technology can improve the prediction value of block-based affine prediction without increasing bandwidth. After the affine prediction based on the sub-block is completed, the horizontal and vertical gradients are calculated for each pixel point that has completed the affine prediction based on the sub-block. In VVC, PROF uses a 3-tap filter [-1, 0, 1] to calculate the gradient, and its calculation method is the same as that of bidirectional optical flow (BDOF). After that, for each pixel position, its motion vector deviation is calculated. The motion vector deviation is the difference between the motion vector of the current pixel position and the MV used by the entire sub-block. These motion vector deviations can be calculated according to the formula of the affine prediction model. Due to the characteristics of the formula, the motion vector deviations of the same positions of some sub-blocks are the same. For these sub-blocks, only one set of motion vector deviations needs to be calculated, and other sub-blocks can directly reuse these values. For each pixel position, the horizontal pixel vertical gradient and motion vector deviation (including horizontal deviation and vertical deviation) of the point are used to calculate the correction value of the predicted value of the pixel position, and then the original prediction value, that is, the prediction value based on the affine prediction of the sub-block, is added to the correction value of the prediction value to obtain the corrected prediction value.

[0062] When calculating the gradients in the horizontal and vertical directions, a [-1, 0, 1] filter is used, that is, for the current pixel position, the predicted values ​​of the pixel position with a distance of 1 to the left and the pixel position with a distance of 1 to the right will be used in the horizontal direction, and the predicted values ​​of the pixel position with a distance of 1 to the top and the pixel position with a distance of 1 to the bottom will be used in the vertical direction. If the current pixel position is the boundary position of the current block, then some of the above-mentioned pixel positions will exceed the boundary of the current block by one pixel. The predicted value of the boundary of the current block is used to fill the position of one pixel distance outward to satisfy the gradient calculation, so there is no need to add the predicted value of one pixel distance beyond the boundary of the current block. Since the gradient calculation only needs to use the predicted value based on the affine prediction of the sub-block, there is no need to add additional bandwidth.

[0063] In the VVC standard text, the MV of each sub-block of the current block and the motion vector deviation of each pixel position in the sub-block are derived based on the MV of the control point. The pixel positions used as the sub-block MV of each sub-block in VVC are the same, so only one set of motion vector deviations of sub-blocks need to be derived, and other sub-blocks can reuse the sub-blocks. Furthermore, in the description of the PROF process in the VVC standard text, the calculation of the motion vector deviation by PROF is included in the above process.

[0064] AVS3's affine prediction has the same basic principle as VVC when calculating the MV of sub-blocks, but AVS3 has special processing for the upper left sub-block A, upper right sub-block B, and lower left sub-block C of the current block.

[0065] The following is the description of the derivation of the affine motion unit sub-block motion vector array in the AVS3 standard text:

[0066] If there are 3 motion vectors in the affine control point motion vector group, the motion vector group can be expressed as mvsAffine (mv0, mv1, mv2); if there are 2 motion vectors in the affine control point motion vector group, the motion vector group can be expressed as mvsAffine (mv0, mv1). Then, the affine motion unit sub-block motion vector array can be derived according to the following steps:

[0067] 1. Calculate the variables dHorX, dVerX, dHorY and dVerY:

[0068] dHorX=(mv1_x-mv0_x)<<(7-Log(width));

[0069] dHorY=(mv1_y-mv0_y)<<(7-Log(width));

[0070] If the motion vector group is mvsAffine(mv0, mv1, mv2), then:

[0071] dVerX=(mv2_x-mv0_x)<<(7-Log(height));

[0072] dVerY=(mv2_y-mv0_y)<<(7-Log(height));

[0073] If the motion vector group is mvsAffine(mv0, mv1), then:

[0074] dVerX=-dHorY;

[0075] dVerY = dHorX;

[0076] It should be noted that Figure 7 is a schematic diagram of the sample location, such as Figure 7As shown, (xE, yE) is the position of the upper left corner sample of the brightness prediction block of the current prediction unit in the brightness sample matrix of the current image, the width and height of the current prediction unit are width and height respectively, the width and height of each sub-block are subwidth and subheight respectively, the sub-block where the upper left corner sample of the brightness prediction block of the current prediction unit is located is A, the sub-block where the upper right corner sample is located is B, and the sub-block where the lower left corner sample is located is C.

[0077] 2.1. If the prediction reference mode of the current prediction unit is 'Pred_List01' or AffineSubblockSizeFlag is equal to 1 (AffineSubblockSizeFlag is used to indicate the size of the subblock), subwidth and subheight are both equal to 8, (x, y) is the coordinate of the upper left corner of the subblock of size 8x8, then the motion vector mvE (mvE_x, mvE_y) of each 8x8 luminance subblock can be calculated:

[0078] If the sub-block is A, both xPos and yPos are equal to 0;

[0079] If the sub-block is B, then xPos is equal to width and yPos is equal to 0;

[0080] If the sub-block is C and there are 3 motion vectors in mvsAffine, then xPos is equal to 0 and yPos is equal to height;

[0081] Otherwise, xPos is equal to (x-xE)+4, and yPos is equal to (y-yE)+4;

[0082] Therefore, the motion vector mvE of the current 8x8 sub-block is:

[0083] mvE_x=Clip3(-131072, 131071, Rounding((mv0_x<<7)+dHorX×xPos+dVerX×yPos, 7));

[0084] mvE_y=Clip3(-131072, 131071, Rounding((mv0_y<<7)+dHorY×xPos+dVerY×yPos, 7));

[0085] 2.2. If the prediction reference mode of the current prediction unit is 'Pred_List0' or 'Pred_List1', and AffineSubblockSizeFlag is equal to 0, then subwidth and subheight are both equal to 4, (x, y) is the coordinate of the upper left corner of the sub-block of size 4x4, and the motion vector mvE (mvE_x, mvE_y) of each 4x4 luminance sub-block is calculated:

[0086] If the sub-block is A, both xPos and yPos are equal to 0;

[0087] If the sub-block is B, then xPos is equal to width and yPos is equal to 0;

[0088] If the sub-block is C and there are 3 motion vectors in mvAffine, then xPos is equal to 0 and yPos is equal to height;

[0089] Otherwise, xPos is equal to (x-xE)+2, and yPos is equal to (y-yE)+2;

[0090] Therefore, the motion vector mvE of the current 4x4 sub-block is:

[0091] mvE_x=Clip3(-131072, 131071, Rounding((mv0_x<<7)+dHorX×xPos+dVerX×yPos, 7));

[0092] mvE_y=Clip3(-131072, 131071, Rounding((mv0_y<<7)+dHorY×xPos+dVerY×yPos, 7)).

[0093] The following is the description of AVS3 text for affine prediction sample derivation and brightness and chrominance sample interpolation:

[0094] If the position of the upper left corner sample of the brightness prediction block of the current prediction unit in the brightness sample matrix of the current image is (xE, yE).

[0095] If the prediction reference mode of the current prediction unit is 'PRED_List0' and the value of AffineSubblockSizeFlag is 0, mv0E0 is the LO motion vector of the 4x4 unit at the position (xE+x, yE+y) of the MvArrayL0 motion vector set. The value of the element predMatrixL0[x][y] in the luma prediction sample matrix predMatrixL0 is the sample value at position (((xE+x)<<4)+mv0E0_x, (yE+y)<<4)+mv0E0_y))) in the 1 / 16 precision luma sample matrix with reference index RefIdxL0 in reference picture queue 0. The value of the element predMatrixL0[x][y] in the chroma prediction sample matrix predMatrixL0 is the sample value at position (((xE+2x)<<4)+MvC_x, (yE+2y)<<4)+MvC_y))) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL0 in reference picture queue 0. Among them, x1=((xE+2x)>>3)<<3, y1=((yE+2y)>>3)<<3, mv1E0 is the LO motion vector of the 4x4 unit of the MvArrayL0 motion vector set at the position (x1, y1), mv2E0 is the LO motion vector of the 4x4 unit of the MvArrayL0 motion vector set at the position (x1+4, y1), mv3E0 is the LO motion vector of the 4x4 unit of the MvArrayL0 motion vector set at the position (x1, y1+4), and mv4E0 is the LO motion vector of the 4x4 unit of the MvArrayL0 motion vector set at the position (x1+4, y1+4).

[0096] MvC_x=(mv1E0_x+mv2E0_x+mv3E0_x+mv4E0_x+2)>>2

[0097] MvC_y=(mv1E0_y+mv2E0_y+mv3E0_y+mv4E0_y+2)>>2

[0098] If the prediction reference mode of the current prediction unit is 'PRED_List0' and the value of AffineSubblockSizeFlag is 1, mv0E0 is the LO motion vector of the 8x8 unit at the position (xE+x, yE+y) of the MvArrayL0 motion vector set. The value of the element predMatrixL0[x][y] in the luma prediction sample matrix predMatrixL0 is the sample value at position (((xE+x)<<4)+mv0E0_x, (yE+y)<<4)+mv0E0_y))) in the 1 / 16 precision luma sample matrix with reference index RefIdxL0 in reference picture queue 0, and the value of the element predMatrixL0[x][y] in the chroma prediction sample matrix predMatrixL0 is the sample value at position (((xE+2x)<<4)+MvC_x, (yE+2y)<<4)+MvC_y))) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL0 in reference picture queue 0. Among them, MvC_x is equal to mv0E0_x, and MvC_y is equal to mv0E0.

[0099] If the prediction reference mode of the current prediction unit is 'PRED_List1' and the value of AffineSubblockSizeFlag is 0, mv0E1 is the L1 motion vector of the 4x4 unit at the position (xE+x, yE+y) of the MvArrayL1 motion vector set. The value of the element predMatrixL1[x][y] in the luma prediction sample matrix predMatrixL1 is the sample value at position (((xE+x)<<4)+mv0E1_x, (yE+y)<<4)+mv0E1_y))) in the 1 / 16 precision luma sample matrix with reference index RefIdxL1 in reference picture queue 1. The value of the element predMatrixL1[x][y] in the chroma prediction sample matrix predMatrixL1 is the sample value at position (((xE+2x)<<4)+MvC_x, (yE+2y)<<4)+MvC_y))) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL1 in reference picture queue 1. Among them, x1=((xE+2x)>>3)<<3, y1=((yE+2y)>>3)<<3, mv1E1 is the L1 motion vector of the 4x4 unit of the MvArrayL1 motion vector set at the position (x1, y1), mv2E1 is the L1 motion vector of the 4x4 unit of the MvArrayL1 motion vector set at the position (x1+4, y1), mv3E1 is the L1 motion vector of the 4x4 unit of the MvArrayL1 motion vector set at the position (x1, y1+4), and mv4E1 is the L1 motion vector of the 4x4 unit of the MvArrayL1 motion vector set at the position (x1+4, y1+4).

[0100] MvC_x=(mv1E1_x+mv2E1_x+mv3E1_x+mv4E1_x+2)>>2

[0101] MvC_y=(mv1E1_y+mv2E1_y+mv3E1_y+mv4E1_y+2)>>2

[0102] If the prediction reference mode of the current prediction unit is 'PRED_List1' and the value of AffineSubblockSizeFlag is 1, mv0E1 is the L1 motion vector of the 8x8 unit at the position (xE+x, yE+y) of the MvArrayL1 motion vector set. The value of the element predMatrixL1[x][y] in the luma prediction sample matrix predMatrixL1 is the sample value at position (((xE+x)<<4)+mv0E1_x, (yE+y)<<4)+mv0E1_y))) in the 1 / 16 precision luma sample matrix with reference index RefIdxL1 in reference picture queue 1, and the value of the element predMatrixL1[x][y] in the chroma prediction sample matrix predMatrixL1 is the sample value at position (((xE+2x)<<4)+MvC_x, (yE+2y)<<4)+MvC_y))) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL1 in reference picture queue 1. Where MvC_x is equal to mv0E1_x, and MvC_y is equal to mv0E1.

[0103] If the prediction mode of the current prediction unit is 'PRED_List01', mv0E0 is the L0 motion vector of the 8x8 unit at position (xE+x, yE+y) of the MvArrayL0 motion vector set, and mv0E1 is the L1 motion vector of the 8x8 unit at position (x, y) of the MvArrayL1 motion vector set. The value of the element predMatrixL0[x][y] in the luma prediction sample matrix predMatrixL0 is the sample value at position (((xE+x)<<4)+mv0E0_x, (yE+y)<<4)+mv0E0_y))) in the 1 / 16 precision luma sample matrix with reference index RefIdxL0 in reference picture queue 0. The value of the element predMatrixL0[x][y] in the chroma prediction sample matrix predMatrixL0 is the sample value at position (((xE+2x)<<4)+MvC0_x, (yE+2y)<<4)+MvC0_y))) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL0 in reference picture queue 0. The value of the element predMatrixL1[x][y] in the luminance prediction sample matrix predMatrixL1 is the sample value at position ((((xE+x)<<4)+mv0E1_x, (yE+y)<<4)+mv0E1_y)))) in the 1 / 16 precision luminance sample matrix with reference index RefIdxL1 in reference picture queue 1. The value of the element predMatrixL1[x][y] in the chrominance prediction sample matrix predMatrixL1 is the sample value at position (((xE+2x)<<4)+MvC1_x, (yE+2y)<<4)+MvC1_y))) in the 1 / 32 precision chrominance sample matrix with reference index RefIdxL1 in reference picture queue 1. Among them, MvC0_x is equal to mv0E0_x, MvC0_y is equal to mv0E0_y, MvC1_x is equal to mv0E1_x, and MvC1_y is equal to mv0E1_y.

[0104] The element values ​​of each position in the 1 / 16 precision sample matrix of luminance and 1 / 32 precision sample matrix of chrominance of the reference image are obtained by the interpolation method defined by the following affine luminance sample interpolation process and affine chrominance sample interpolation process. Integer samples outside the reference image should be replaced by the integer samples (edge ​​or corner samples) closest to the samples in the image, that is, the motion vector can point to samples outside the reference image.

[0105] Specifically, the affine brightness sample interpolation process is as follows:

[0106] A, B, C, D are adjacent integer pixel samples, dx and dy are the horizontal and vertical distances between the sub-pixel sample a (dx, dy) around the integer pixel sample A and A, dx is equal to fx & 15, dy is equal to fy & 15, where (fx, fy) is the coordinate of the sub-pixel sample in the luminance sample matrix with 1 / 16 accuracy. Integer pixel A x，y There are 255 sub-pixel samples a around x，y (dx, dy).

[0107] Specifically, the sample position a x，0 (x=1-15) is obtained by filtering the 8 integer values ​​closest to the interpolation point in the horizontal direction. The prediction value is obtained as follows:

[0108] a x，0 =Clip1((fL[x][0]×A -3，0 +fL[x][1]×A -2，0 +fL[x][2]×A -1，0 +fL[x][3]×A 0，0 +fL[x][4]×A 1，0 +fL[x][5]×A 2，0 +fL[x][6]×A 3，0 +fL[x][7]×A 4，0 +32)>>6).

[0109] Specifically, the sample position a 0，y (y=1-15) is obtained by filtering the 8 integer values ​​closest to the interpolation point in the vertical direction. The predicted value is obtained as follows:

[0110] a 0，y =Clip1((fL[y][0]×A 0，-3 +fL[y][1]×A -2，0 +fL[y][2]×A -1，0 +fL[y][3]×A 0，0 +fL[y][4]×A 1，0 +fL[y][5]×A 2，0 +fL[y][6]×A 3，0 +fL[y][7]×A -4，0 +32)>>6).

[0111] Specifically, the sample position a x，y The method for obtaining the predicted value of (x=1-15, y=1-15) is as follows:

[0112] a x，y = Clip1((fL[y][0]×a' x，y-3 +fL[y][1]×a'x，y-2 +fL[y][2]× a'x，y-1 +fL[y][3]× a'x，y +fL[y][4]× a'x，y+1 +fL[y][5]× a'x，y +2+fL[y][6]× a'x，y+3 +fL[y][7]× a'x，y+4 +(1<<(19-BitDepth)))>>(20-BitDepth)).

[0113] in:

[0114] a' x，y =(fL[x][0]×A -3，y +fL[x][1]×A -2，y +fL[x][2]×A -1，y+ fL[x][3]×A 0，y +fL[x][4]×A 1，y +fL[x][5]×A 2，y +fL[x][6]×A 3，y +fL[x][7]×A 4，y+ ((1<<(BitDepth-8))>>1))>>(BitDepth-8).

[0115] The brightness interpolation filter coefficients are shown in Table 1:

[0116] Table 1

[0117]

[0118] Specifically, the affine chromaticity sample interpolation process is as follows:

[0119] A, B, C, D are adjacent integer pixel samples, dx and dy are the horizontal and vertical distances between the sub-pixel sample a (dx, dy) around the integer pixel sample A and A, dx is equal to fx & 31, dy is equal to fy & 31, where (fx, fy) is the coordinate of the sub-pixel sample in the chroma sample matrix with 1 / 32 accuracy. Integer pixel A x，y There are 1023 sub-pixel samples around a x，y (dx, dy).

[0120] Specifically, for sub-pixel points where dx is equal to 0 or dy is equal to 0, the chrominance integer pixel interpolation can be directly used. For points where dx is not equal to 0 and dy is not equal to 0, the sub-pixels on the integer pixel row (dy is equal to 0) are used for calculation:

[0121] if(dx==0){

[0122] ax，y (0,dy)=Clip3(0,(1< <BitDepth)-1,(fC[dy][0]×A x，y -1+fC[dy][1]×A x，y +fC[dy][2]×A x，y +1+fC[dy][3]×A x，y +2+32)>>6)

[0123] }

[0124] else if(dy==0){

[0125] a x，y (dx, 0) = Clip3 (0, (1 < <BitDepth)-1,(fC[dx][0]×A x-1，y +fC[dx][1]×A x，y +fC[dx][2]×A x+1，y +fC[dx][3]×A x+2，y+32 )>>6)

[0126] }

[0127] else{

[0128] a x，y (dx, dy) = Clip3 (0, (1 < <BitDepth)-1,(C[dy][0]×a' x，y-1 (dx, 0) + C[dy][1] × a' x，y (dx, 0) + C[dy][2] × a' x，y +1

[0129] (dx, 0) + C[dy][3] × a' x，y +2(dx,0)+(1<<(19-BitDepth)))>>(20-BitDepth))

[0130] }

[0131] Among them, a' x，y (dx, 0) is the temporary value of the sub-pixel on the integer pixel row, defined as: a' x，y (dx, 0) = (fC[dx][0] × A x-1，y +fC[dx][1]×A x，y +fC[dx][2]×A x+1，y+ fC[dx][3]×A x+2，y +((1<<(BitDepth-8))>>1))>>(BitDepth-8).

[0132] The chroma interpolation filter coefficients are shown in Table 2:

[0133] Table 2

[0134]

[0135]

[0136] Common affine prediction methods can include the following steps:

[0137] Step 101: Determine the motion vector of the control point.

[0138] Step 102: Determine the motion vector of the sub-block according to the motion vector of the control point.

[0139] Step 103: Predict the sub-block according to the motion vector of the sub-block.

[0140] Currently, when improving the prediction value of block-based affine prediction through PROF, the following steps may be specifically included:

[0141] Step 101: Determine the motion vector of the control point.

[0142] Step 102: Determine the motion vector of the sub-block according to the motion vector of the control point.

[0143] Step 103: Predict the sub-block according to the motion vector of the sub-block.

[0144] Step 104: Determine the motion vector deviation between each position in the sub-block and the sub-block according to the motion vector of the control point and the motion vector of the sub-block.

[0145] Step 105: Determine the motion vector of the sub-block according to the motion vector of the control point.

[0146] Step 106: Use the sub-block-based prediction value to derive the gradients in the horizontal and vertical directions for each position.

[0147] Step 107: Using the optical flow principle, according to the motion vector deviation and horizontal and vertical gradients of each position, calculate the deviation value of the predicted value at each position.

[0148] Step 108: For each position, add the prediction value based on the sub-block to the deviation value of the prediction value to obtain a corrected prediction value.

[0149] Currently, when improving the prediction value of block-based affine prediction through PROF, the following steps may be specifically included:

[0150] Step 101: Determine the motion vector of the control point.

[0151] Step 109: Determine the motion vector of the sub-block and the deviation between each position in the sub-block and the motion vector of the sub-block according to the motion vector of the control point.

[0152] Step 103: Predict the sub-block according to the motion vector of the sub-block.

[0153] Step 106: Use the sub-block-based prediction value to derive the gradients in the horizontal and vertical directions for each position.

[0154] Step 107: Using the optical flow principle, according to the motion vector deviation and horizontal and vertical gradients of each position, calculate the deviation value of the predicted value at each position.

[0155] Step 108: For each position, add the prediction value based on the sub-block to the deviation value of the prediction value to obtain a corrected prediction value.

[0156] PROF can use the optical flow principle to correct the affine prediction based on sub-blocks, thereby improving the compression performance. However, the application of PROF is based on the situation that the deviation between the motion vector of the pixel position in the sub-block and the sub-block motion vector is very small. In other words, the optical flow calculation method using PROF is effective when the deviation between the motion vector of the pixel position in the sub-block and the sub-block motion vector is very small. However, since PROF relies on the horizontal and vertical gradients of the reference position, when the actual position is far from the reference position, the horizontal and vertical gradients of the reference position cannot truly reflect the horizontal and vertical gradients between the reference position and the actual position. Therefore, when the deviation between the motion vector of the pixel position in the sub-block and the sub-block motion vector is large, this method is not particularly effective.

[0157] At this time, a secondary prediction method is proposed to overcome the defects of PROF. The method for the decoder to perform secondary prediction may include the following steps:

[0158] Step 201: predict the sub-block according to the motion vector of the sub-block to obtain a prediction value.

[0159] Step 202: Determine the motion vector deviation between each position in the sub-block and the sub-block.

[0160] Step 203: According to the motion vector deviation of each position, the predicted value is filtered using a two-dimensional filter to obtain a predicted value of secondary prediction.

[0161] That is to say, after the sub-block-based prediction, the secondary prediction method can perform point-based secondary prediction on the basis of the sub-block-based prediction for the pixel position where the motion vector deviates from the motion vector of the sub-block, and finally complete the correction of the predicted value to obtain a new predicted value, that is, the predicted value of the secondary prediction.

[0162] Specifically, the point-based secondary prediction uses a two-dimensional filter. The two-dimensional filter is a filter composed of adjacent points that form a preset shape. The adjacent points that form the preset shape can be 9 points. For a pixel position, the result of the filter processing is the predicted value of the secondary prediction of the position. Among them, the filter coefficient of the two-dimensional filter is determined by the motion vector deviation of each position, the input of the two-dimensional filter is the predicted value, and the output is the predicted value of the secondary prediction.

[0163] Furthermore, in the present application, if the affine mode is used for the current block in the present application, the motion vector of the control point needs to be determined first. The method for the decoder to perform secondary prediction may include the following steps:

[0164] Step 204: Determine the motion vector of the control point.

[0165] Step 205: Determine the motion vector of the sub-block according to the motion vector of the control point.

[0166] Step 201: predict the sub-block according to the motion vector of the sub-block to obtain a prediction value.

[0167] Step 202: Determine the motion vector deviation between each position in the sub-block and the sub-block.

[0168] Step 203: According to the motion vector deviation of each position, the predicted value is filtered using a two-dimensional filter to obtain a predicted value of secondary prediction.

[0169] It can be seen that in the embodiments of the present application, after the motion vector of the control point is determined, the motion vector of the control point can be used to perform prediction processing on the sub-block, and after determining the deviation between the pixel position in the sub-block and the motion vector of the sub-block, for the pixel position where the motion vector deviates from the motion vector of the sub-block, a point-based secondary prediction is performed on the basis of the sub-block-based prediction, and finally the prediction value is corrected to obtain a new prediction value, i.e., the prediction value of the secondary prediction.

[0170] Furthermore, in the present application, the two steps of determining the motion vector of the sub-block and determining the deviation of the motion vector of each position in the sub-block from the sub-block can also be performed simultaneously. The method for the decoder to perform secondary prediction may include the following steps:

[0171] Step 204: Determine the motion vector of the control point.

[0172] Step 206: Determine the motion vector of the sub-block according to the motion vector of the control point, and the deviation of the motion vector of each position in the sub-block from the sub-block.

[0173] Step 201: predict the sub-block according to the motion vector of the sub-block to obtain a prediction value.

[0174] Step 203: According to the motion vector deviation of each position, the predicted value is filtered using a two-dimensional filter to obtain a predicted value of secondary prediction.

[0175] It can be seen that in the embodiments of the present application, after the motion vector of the control point is determined, the motion vector of the sub-block and the deviation of each position in the sub-block from the motion vector of the sub-block can be determined simultaneously, and then for the pixel positions where the motion vector deviates from the motion vector of the sub-block, a point-based secondary prediction is performed on the basis of the sub-block-based prediction, and finally the prediction value is corrected to obtain a new prediction value, i.e., the prediction value of the secondary prediction.

[0176] It should be noted that, because the affine model can clearly calculate the motion vector of each pixel position, or the deviation between each pixel position in the sub-block and the sub-block motion vector, the secondary prediction method can be used to improve the affine prediction. Of course, the secondary prediction can also be applied to improve other sub-block-based predictions. That is to say, the sub-block-based prediction proposed in this application includes but is not limited to the affine sub-block-based prediction.

[0177] Furthermore, the secondary prediction method may be based on the AVS3 standard, and may also be applied to the VVC standard, which is not specifically limited in this application.

[0178] For each pixel position in the sub-block, when performing secondary prediction or PROF processing, the default filter center point is the pixel position in the prediction block based on the prediction of the sub-block that is the same as the pixel position. Assuming that the motion vector deviation is 0, the prediction value of the pixel position is the value of the pixel position in the prediction block based on the prediction of the sub-block that is the same as the pixel position. In general, the above method can be applied to the condition that the motion vector deviation between the pixel position and the sub-block is very small. However, for the case where the motion vector deviation between the pixel position and the sub-block is not very small, for example, in the horizontal or vertical direction, when the deviation exceeds half a pixel or a certain ratio, the prediction effect obtained by secondary prediction or PROF processing may be greatly reduced.

[0179] Figure 8 Indication of the center position Figure 1 ,like Figure 8 As shown, the square represents the pixel position of the prediction block based on the prediction of the sub-block, and the circle represents the actual position of the pixel position to be predicted in the prediction block based on the prediction of the sub-block, wherein the square 1 represents the pixel position in the prediction block based on the prediction of the sub-block that is the same as the pixel position to be predicted, that is, the (1, 1) position of the sub-block. The square 1 is the default reference position for point-based prediction, that is, the center position of the filter, and its motion vector deviation is recorded as (dmv_x0, dmv_y0).

[0180] Fig. 9Indication of the center position Figure 2 ,like Fig. 9 As shown, if dmv_x0 and dmv_y0 are both greater than half a pixel, then in the sub-block, there is a large distance between the square 1 and the pixel position to be predicted. If the square 1 is used as the center position of the filter for secondary prediction or PROF processing, the final prediction result will have a large error.

[0181] It can be seen from this that when there is a large motion vector deviation in the center position of the filter required for secondary prediction or PROF, it may cause the prediction performance to deteriorate. That is, the existing PROF corrected prediction value method and secondary prediction method are not rigorous. When improving affine prediction, they cannot be well applied to all scenarios, and the encoding performance needs to be improved.

[0182] In order to solve the defects existing in the prior art, in the embodiment of the present application, after the prediction based on the sub-block, the pixel position with a large first motion vector deviation between the motion vector and the motion vector of the sub-block can be re-determined based on the first motion vector deviation for secondary prediction or PROF processing, so that the center position and the second motion vector deviation can be used to perform point-based secondary prediction on the basis of the first prediction value based on the sub-block to obtain the second prediction value. It can be seen that the inter-frame prediction method proposed in the present application can be well applied to all scenes, can reduce the prediction error, greatly improve the encoding performance, and thus improve the encoding and decoding efficiency.

[0183] It should be understood that the embodiment of the present application provides a video encoding system. Fig.10 A schematic block diagram of a video encoding system provided in an embodiment of the present application is shown in FIG. Fig.10As shown, the video coding system 11 may include: a transform unit 111, a quantization unit 112, a mode selection and coding control logic unit 113, an intra-frame prediction unit 114, an inter-frame prediction unit 115 (including: motion compensation and motion estimation), an inverse quantization unit 116, an inverse transform unit 117, a loop filtering unit 118, a coding unit 119 and a decoded image buffer unit 110; for the input original video signal, a coding tree block (Coding A video reconstruction block can be obtained by dividing the video reconstruction block into a plurality of blocks, and the coding mode is determined by the mode selection and coding control logic unit 113. Then, the residual pixel information obtained after intra-frame or inter-frame prediction is transformed by the transform unit 111 and the quantization unit 112, including transforming the residual information from the pixel domain to the transform domain, and quantizing the obtained transform coefficients to further reduce the bit rate; the intra-frame prediction unit 114 is used to perform intra-frame prediction on the video reconstruction block; wherein the intra-frame prediction unit 114 is used to determine the optimal intra-frame prediction mode (i.e., the target prediction mode) of the video reconstruction block; the inter-frame prediction unit 115 is used to perform inter-frame prediction coding of the received video reconstruction block relative to one or more blocks in one or more reference frames to provide temporal prediction information; wherein the motion estimation is The process of generating a motion vector, which can estimate the motion of the video reconstruction block, and then motion compensation is performed based on the motion vector determined by the motion estimation; after determining the inter-frame prediction mode, the inter-frame prediction unit 115 is also used to provide the selected inter-frame prediction data to the encoding unit 119, and the calculated and determined motion vector data is also sent to the encoding unit 119; in addition, the inverse quantization unit 116 and the inverse transformation unit 117 are used to reconstruct the video reconstruction block, reconstruct the residual block in the pixel domain, and the reconstructed residual block is removed by the loop filter unit 118. The reconstructed residual block is then added to a predictive block in the frame of the decoded image cache unit 110 to generate a reconstructed video reconstruction block; the encoding unit 119 is used to encode various encoding parameters and quantized transform coefficients. The decoded image cache unit 110 is used to store the reconstructed video reconstruction block for prediction reference. As the video image encoding proceeds, new reconstructed video reconstruction blocks will be continuously generated, and these reconstructed video reconstruction blocks will be stored in the decoded image cache unit 110.

[0184] The embodiment of the present application also provides a video decoding system, Fig.11 A schematic block diagram of a video decoding system provided in an embodiment of the present application is shown in FIG. Fig.11As shown, the video decoding system 12 may include: a decoding unit 121, an inverse transform unit 127, an inverse quantization unit 122, an intra-frame prediction unit 123, a motion compensation unit 124, a loop filter unit 125 and a decoded image cache unit 126; after the input video signal is encoded by the video encoding system 11, a code stream of the video signal is output; the code stream is input into the video decoding system 12, and first passes through the decoding unit 121 to obtain a decoded transform coefficient; the transform coefficient is processed by the inverse transform unit 127 and the inverse quantization unit 122 to generate a residual block in the pixel domain; the intra-frame prediction unit 123 can be used to generate a prediction number of the current video decoding block based on the determined intra-frame prediction direction and the data of the previously decoded block from the current frame or picture. The motion compensation unit 124 determines prediction information for the video decoding block by analyzing the motion vector and other associated syntax elements, and uses the prediction information to generate a predictive block of the video decoding block being decoded; a decoded video block is formed by summing the residual block from the inverse transform unit 127 and the inverse quantization unit 122 with the corresponding predictive block generated by the intra-frame prediction unit 123 or the motion compensation unit 124; the decoded video signal passes through the loop filtering unit 125 to remove the block effect artifacts, which can improve the video quality; the decoded video block is then stored in the decoded image cache unit 126, which stores the reference image used for subsequent intra-frame prediction or motion compensation, and is also used for the output of the video signal to obtain the restored original video signal.

[0185] An inter-frame prediction method provided in an embodiment of the present application mainly acts on the inter-frame prediction unit 215 of the video encoding system 11 and the inter-frame prediction unit of the video decoding system 12, namely, the motion compensation unit 124; that is, if the video encoding system 11 can obtain a better prediction effect through the inter-frame prediction method provided in an embodiment of the present application, then, correspondingly, in the video decoding system 12, the video decoding recovery quality can also be improved.

[0186] Based on this, the technical solution of the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. Before the detailed description, it should be noted that the "first", "second", "third", etc. mentioned throughout the specification are only for distinguishing different features and do not have the function of limiting priority, sequence, size relationship, etc.

[0187] It should be noted that this embodiment is exemplified based on the AVS3 standard. The inter-frame prediction method proposed in this application can also be applied to other coding standard technologies such as VVC, and the application does not make any specific limitation on this.

[0188] The embodiment of the present application provides an inter-frame prediction method, which is applied to a video decoding device, i.e., a decoder. The function implemented by the method can be implemented by calling a computer program by a first processor in the decoder, and of course the computer program can be stored in a first memory. It can be seen that the decoder includes at least a first processor and a first memory.

[0189] Furthermore, in the embodiments of the present application, Fig.12 Schematic diagram of the implementation process of the inter-frame prediction method Figure 1 ,like Fig.12 As shown, the method for the decoder to perform inter-frame prediction may include the following steps:

[0190] Step 301: parse the bitstream to obtain prediction mode parameters of the current block.

[0191] In an embodiment of the present application, the decoder may first parse the binary code stream to obtain prediction mode parameters of the current block, wherein the prediction mode parameters may be used to determine the prediction mode used by the current block.

[0192] It should be noted that the image to be decoded can be divided into multiple image blocks, and the current image block to be decoded can be called a current block (which can be represented by CU), and the image blocks adjacent to the current block can be called adjacent blocks; that is, in the image to be decoded, there is an adjacent relationship between the current block and the adjacent blocks. Here, each current block can include a first image component, a second image component, and a third image component, that is, the current block represents an image block in the image to be decoded that is currently to be predicted for the first image component, the second image component, or the third image component.

[0193] Among them, assuming that the current block predicts the first image component, and the first image component is a luminance component, that is, the image component to be predicted is a luminance component, then the current block can also be called a luminance block; or, assuming that the current block predicts the second image component, and the second image component is a chrominance component, that is, the image component to be predicted is a chrominance component, then the current block can also be called a chrominance block.

[0194] Further, in an embodiment of the present application, the prediction mode parameter may not only indicate the prediction mode adopted by the current block, but also indicate parameters related to the prediction mode.

[0195] It can be understood that, in the embodiments of the present application, the prediction mode may include an inter-frame prediction mode, a traditional intra-frame prediction mode, a non-traditional intra-frame prediction mode, and the like.

[0196] That is to say, on the encoding side, the encoder can select the optimal prediction mode to pre-encode the current block. In this process, the prediction mode of the current block can be determined, and then the prediction mode parameters used to indicate the prediction mode can be determined, so that the corresponding prediction mode parameters are written into the bitstream and transmitted from the encoder to the decoder.

[0197] Accordingly, on the decoder side, the decoder can directly obtain the prediction mode parameters of the current block by parsing the bitstream, and determine the prediction mode used by the current block and the relevant parameters corresponding to the prediction mode according to the prediction mode parameters obtained by parsing.

[0198] Further, in an embodiment of the present application, after parsing and obtaining the prediction mode parameters, the decoder may determine whether the current block uses the inter-frame prediction mode based on the prediction mode parameters.

[0199] Step 302: When the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, determine a first motion vector of a sub-block of the current block; wherein the current block includes a plurality of sub-blocks.

[0200] In an embodiment of the present application, after the decoder parses and obtains the prediction mode parameters, if the parse and obtain prediction mode parameters indicate that the current block uses the inter-frame prediction mode to determine the inter-frame prediction value of the current block, the decoder can first determine the first motion vector of each sub-block of the current block. One sub-block corresponds to one first motion vector.

[0201] It should be noted that in the embodiment of the present application, the current block is an image block to be decoded in the current frame, and the current frame is decoded in the form of image blocks in a certain order, and the current block is an image block to be decoded at the next moment in the current frame in the order. The current block can have a variety of specifications, such as 16×16, 32×32 or 32×16, where the numbers represent the number of rows and columns of pixels on the current block.

[0202] Furthermore, in an embodiment of the present application, the current block may be divided into a plurality of sub-blocks, wherein each sub-block has the same size, and the sub-block is a set of pixels of a smaller size. The size of the sub-block may be 8×8 or 4×4.

[0203] Exemplarily, in the present application, the size of the current block is 16×16, and can be divided into 4 sub-blocks of 8×8 size each.

[0204] It can be understood that in the embodiment of the present application, when the decoder parses the code stream and obtains the prediction mode parameter indicating the use of the inter-frame prediction mode to determine the inter-frame prediction value of the current block, the inter-frame prediction method provided in the embodiment of the present application can continue to be used.

[0205] In an embodiment of the present application, further, when the prediction mode parameter indicates that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode, the method for the decoder to determine the first motion vector of the sub-block of the current block may include the following steps:

[0206] Step 302a: parse the bitstream to obtain the affine mode parameters and prediction reference mode of the current block.

[0207] Step 302b: When the affine mode parameter indicates to use the affine mode, determine the control point mode and sub-block size parameters.

[0208] Step 302c: determine a first motion vector according to the prediction reference mode, the control point mode and the sub-block size parameter.

[0209] In an embodiment of the present application, after the decoder parses and obtains the prediction mode parameters, if the prediction mode parameters obtained by parsing indicate that the current block uses the inter-frame prediction mode to determine the inter-frame prediction value of the current block, then the decoder can obtain the affine mode parameters and the prediction reference mode by parsing the bitstream.

[0210] It should be noted that, in the embodiment of the present application, the affine mode parameter is used to indicate whether to use the affine mode. Specifically, the affine mode parameter may be an affine motion compensation enable flag affine_enable_flag, and the decoder may further determine whether to use the affine mode by determining the value of the affine mode parameter.

[0211] That is, in the present application, the affine mode parameter may be a binary variable. If the value of the affine mode parameter is 1, it indicates that the affine mode is used; if the value of the affine mode parameter is 0, it indicates that the affine mode is not used.

[0212] It can be understood that, in the present application, if the decoder parses the code stream and fails to obtain the affine mode parameters, it can be understood as an indication that the affine mode is not used.

[0213] Exemplarily, in the present application, the value of the affine mode parameter may be equal to the value of the affine motion compensation enable flag affine_enable_flag. If the value of affine_enable_flag is '1', it indicates that affine motion compensation can be used; if the value of affine_enable_flag is '0', it indicates that affine motion compensation should not be used.

[0214] Further, in an embodiment of the present application, if the affine mode parameters obtained by the decoder when parsing the code stream indicate the use of the affine mode, the decoder can obtain the control point mode and sub-block size parameters.

[0215] It should be noted that, in the embodiment of the present application, the control point mode is used to determine the number of control points. In the affine model, a sub-block may have 2 control points or 3 control points, and accordingly, the control point mode may be a control point mode corresponding to 2 control points, or a control point mode corresponding to 3 control points. That is, the control point mode may include a 4-parameter mode and a 6-parameter mode.

[0216] It can be understood that in the embodiments of the present application, for the AVS3 standard, if the current block uses the affine mode, the decoder also needs to determine the number of control points of the current block in the affine mode, so as to determine whether the 4-parameter (2 control point) mode or the 6-parameter (3 control point) mode is used.

[0217] Further, in an embodiment of the present application, if the affine mode parameter obtained by the decoder when parsing the bitstream indicates that the affine mode is used, the decoder can further obtain the sub-block size parameter by parsing the bitstream.

[0218] Specifically, the sub-block size parameter can be determined by the affine prediction sub-block size flag affine_subblock_size_flag. The decoder obtains the sub-block size flag by parsing the bitstream, and determines the size of the sub-block of the current block according to the value of the sub-block flag. The size of the sub-block can be 8×8 or 4×4. Specifically, in the present application, the sub-block size flag can be a binary variable. If the value of the sub-block size flag is 1, it indicates that the sub-block size parameter is 8×8; if the value of the sub-block size flag is 0, it indicates that the sub-block size parameter is 4×4.

[0219] Exemplarily, in the present application, the value of the subblock size flag may be equal to the value of the affine prediction subblock size flag affine_subblock_size_flag. If the value of affine_subblock_size_flag is '1', the current block is divided into subblocks of size 8×8; if the value of affine_subblock_size_flag is '0', the current block is divided into subblocks of size 4×4.

[0220] It is understood that in this application, if the decoder parses the bitstream and does not parse the subblock size flag, it can be understood that the current block is divided into 4×4 subblocks. In other words, if affine_subblock_size_flag does not exist in the bitstream, the value of the subblock size flag can be directly set to 0.

[0221] Furthermore, in an embodiment of the present application, after determining the control point mode and the sub-block size parameter, the decoder can further determine the first motion vector of the sub-block in the current block according to the prediction reference mode, the control point mode and the sub-block size parameter.

[0222] Specifically, in an embodiment of the present application, the decoder may first determine a control point motion vector group according to a prediction reference mode; and then determine a first motion vector of the sub-block based on the control point motion vector group, the control point mode and the sub-block size parameter.

[0223] It can be understood that, in the embodiment of the present application, the control point motion vector group can be used to determine the motion vector of the control point.

[0224] It should be noted that in an embodiment of the present application, the decoder can traverse each sub-block in the current block according to the above method, and use the control point motion vector group, control point mode and sub-block size parameters of each sub-block to determine the first motion vector of each sub-block, so that a motion vector set can be constructed based on the first motion vector of each sub-block.

[0225] It can be understood that, in the embodiment of the present application, the motion vector set of the current block may include the first motion vector of each sub-block of the current block.

[0226] Furthermore, in an embodiment of the present application, when the decoder determines the first motion vector based on the control point motion vector group, the control point mode and the sub-block size parameters, it can first determine the difference variable based on the control point motion vector group, the control point mode and the size parameters of the current block; then it can determine the sub-block position based on the prediction mode parameters and the sub-block size parameters; finally, it can use the difference variable and the sub-block position to determine the first motion vector of the sub-block, and then obtain a set of motion vectors of multiple sub-blocks of the current block.

[0227] Exemplarily, in the present application, the difference variable may include four variables, namely dHorX, dVerX, dHorY and dVerY. When calculating the difference variable, the decoder needs to first determine the control point motion vector group, where the control point motion vector group can characterize the motion vector of the control point.

[0228] Specifically, if the control point mode is a 6-parameter mode, that is, there are 3 control points, then the control point motion vector group can be a motion vector group including 3 motion vectors, expressed as mvsAffine(mv0, mv1, mv2); if the control point mode is a 4-parameter mode, that is, there are 2 control points, then the control point motion vector group can be a motion vector group including 2 motion vectors, expressed as mvsAffine(mv0, mv1).

[0229] The decoder can then use the control point motion vector group to calculate the difference variable:

[0230] dHorX=(mv1_x-mv0_x)<<(7-Log(width));

[0231] dHorY=(mv1_y-mv0_y)<<(7-Log(width));

[0232] If the motion vector group is mvsAffine(mv0, mv1, mv2), then:

[0233] dVerX=(mv2_x-mv0_x)<<(7-Log(height));

[0234] dVerY=(mv2_y-mv0_y)<<(7-Log(height));

[0235] If the motion vector group is mvsAffine(mv0, mv1), then:

[0236] dVerX=-dHorY;

[0237] dVerY=dHorX.

[0238] Wherein, width and height are respectively the width and height of the current block, that is, the size parameter of the current block. Specifically, the size parameter of the current block may be obtained by the decoder by parsing the bitstream.

[0239] Further, in an embodiment of the present application, after determining the difference value variable, the decoder can then determine the sub-block position based on the prediction mode parameter and the sub-block size parameter. Specifically, the decoder can determine the size of the sub-block through the word block size flag, and can determine which prediction mode to use through the prediction mode parameter, and then can determine the sub-block position according to the size of the sub-block and the prediction mode used.

[0240] Exemplarily, in the present application, if the prediction reference mode of the current block is 2, that is, the third reference mode 'Pred_List01', or the sub-block size flag is 1, that is, the width subwidth and height subheight of the sub-block are both equal to 8, then (x, y) is the coordinate of the upper left corner position of the 8×8 sub-block, then the coordinates xPos and yPos of the sub-block position can be determined by the following method:

[0241] If the sub-block is the control point of the upper left corner of the current block, xPos and yPos are both equal to 0;

[0242] If the sub-block is the control point of the upper right corner of the current block, xPos is equal to width and yPos is equal to 0;

[0243] If the sub-block is the control point of the lower left corner of the current block, the control point motion vector group may be a motion vector group including 3 motion vectors, then xPos is equal to 0, and yPos is equal to height;

[0244] Otherwise, xPos is equal to (x-xE)+4, and yPos is equal to (y-yE)+4.

[0245] Exemplarily, in the present application, if the prediction reference mode of the current block takes a value of 0 or 1, that is, the first reference mode 'Pred_List0' or the second reference mode 'Pred_List1', and the sub-block size flag takes a value of 0, that is, the width subwidth and height subheight of the sub-block are both equal to 4, (x, y) is the coordinate of the upper left corner position of the 4×4 sub-block, then the coordinates xPos and yPos of the sub-block position can be determined by the following method:

[0246] If the sub-block is the control point of the upper left corner of the current block, xPos and yPos are both equal to 0;

[0247] If the sub-block is the control point of the upper right corner of the current block, xPos is equal to width and yPos is equal to 0;

[0248] If the sub-block is the control point of the lower left corner of the current block, the control point motion vector group may be a motion vector group including 3 motion vectors, then xPos is equal to 0, and yPos is equal to height;

[0249] Otherwise, xPos is equal to (x-xE)+2, and yPos is equal to (y-yE)+2.

[0250] Furthermore, in an embodiment of the present application, after the decoder calculates the sub-block position, it can then determine the first motion vector of the sub-block based on the sub-block position and the difference variable, and finally, traverse each sub-block of the current block to obtain the first motion vector of each sub-block, so as to construct a motion vector set of multiple sub-blocks of the current block.

[0251] Exemplarily, in the present application, after determining the sub-block positions xPos and yPos, the decoder may determine the first motion vector mvE (mvE_x, mvE_y) of the sub-block in the following manner:

[0252] mvE_x=Clip3(-131072, 131071, Rounding((mv0_x<<7)+dHorX×xPos+dVerX×yPos, 7));

[0253] mvE_y=Clip3(-131072, 131071, Rounding((mv0_y<<7)+dHorY×xPos+dVerY×yPos, 7)).

[0254] It should be noted that in the present application, when determining the deviation of each position within a sub-block from the motion vector of the sub-block, if the current block uses an affine prediction model, the motion vector of each position within the sub-block can be calculated according to the formula of the affine prediction model, and their deviations can be obtained by subtracting the motion vector of the sub-block. If the motion vectors of the sub-blocks all select the motion vectors at the same position within the sub-block, such as a 4x4 block using the position (2, 2) from the upper left corner, and an 8x8 block using the position (4, 4) from the upper left corner, according to the affine model used in current standards including VVC and AVS3, the motion vector deviations at the same position of each sub-block are the same. However, AVS has the same deviations in the upper left corner, upper right corner, and lower left corner with three control points (such as in the above AVS3 text). Figure 7 The A, B, C positions shown) are different from the positions used by other blocks, and accordingly the motion vector deviations of the sub-blocks in the upper left corner, upper right corner, and lower left corner in the case of 3 control points are calculated differently from other blocks.

[0255] Step 303: determine a first prediction value of the sub-block based on the first motion vector, and a first motion vector deviation between each pixel position in the sub-block and the sub-block.

[0256] In an embodiment of the present application, after determining the first motion vector of each sub-block of the current block, the decoder can determine the first prediction value of the sub-block and the first motion vector deviation between each pixel position in the sub-block and the sub-block based on the first motion vector of the sub-block, wherein one pixel position corresponds to one first motion vector deviation.

[0257] It can be understood that, in the embodiment of the present application, step 303 may specifically include:

[0258] Step 303a: Determine a first prediction value of the sub-block based on the first motion vector.

[0259] Step 303b: determine a first motion vector deviation between each pixel position in the sub-block and the sub-block based on the first motion vector.

[0260] Among them, the inter-frame prediction method proposed in the embodiment of the present application does not limit the order in which the decoder executes step 303a and step 303b. That is to say, in the present application, after determining the first motion vector of each sub-block of the current block, the decoder can first execute step 303a and then execute step 303b, or first execute step 303b and then execute step 303a, or execute step 303a and step 303b at the same time.

[0261] Further, in an embodiment of the present application, when the decoder determines the first prediction value of the sub-block based on the first motion vector, it can first determine the sample matrix; wherein the sample matrix includes a luminance sample matrix and a chrominance sample matrix; and then the first prediction value can be determined based on the prediction reference mode, the sub-block size parameter, the sample matrix, and the motion vector set.

[0262] It should be noted that, in an embodiment of the present application, when the decoder determines the first prediction value based on the prediction reference mode, sub-block size parameters, sample matrix and motion vector set, it can first determine the target motion vector from the motion vector set based on the prediction reference mode and sub-block size parameters; then, the prediction sample matrix can be determined using the reference image queue and reference index sample matrix corresponding to the prediction reference mode and the target motion vector; wherein the prediction sample matrix includes the first prediction values ​​of multiple sub-blocks.

[0263] Specifically, in an embodiment of the present application, the sample matrix may include a luma sample matrix and a chroma sample matrix, and accordingly, the prediction sample matrix determined by the decoder may include a luma prediction sample matrix and a chroma prediction sample matrix, wherein the luma prediction sample matrix includes first luma prediction values ​​of multiple sub-blocks, and the chroma prediction sample matrix includes first chroma prediction values ​​of multiple sub-blocks, and the first luma prediction value and the first chroma prediction value constitute the first prediction value of the sub-block.

[0264] Exemplarily, in the present application, it is assumed that the position of the upper left corner sample of the current block in the brightness sample matrix of the current image is (xE, yE). If the prediction reference mode of the current block is 0, that is, the first reference mode 'PRED_List0' is used, and the sub-block size flag is 0, that is, the sub-block size parameter is 4×4, then the target motion vector mv0E0 is the first motion vector of the 4×4 sub-block at the position (xE+x, yE+y) of the motion vector set of the current block. The value of the element predMatrixL0[x][y] in the luma prediction sample matrix predMatrixL0 is the sample value at position (((xE+x)<<4)+mv0E0_x, (yE+y)<<4)+mv0E0_y))) in the 1 / 16 precision luma sample matrix with reference index RefIdxL0 in reference picture queue 0. The value of the element predMatrixL0[x][y] in the chroma prediction sample matrix predMatrixL0 is the sample value at position (((xE+2×x)<<4)+MvC_x, (yE+2×y)<<4)+MvC_y))) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL0 in reference picture queue 0. Among them, x1=((xE+2×x)>>3)<<3, y1=((yE+2×y)>>3)<<3, mv1E0 is the first motion vector of the 4×4 unit of the motion vector set of the current block at the position (x1, y1), mv2E0 is the first motion vector of the 4×4 unit of the motion vector set of the current block at the position (x1+4, y1), mv3E0 is the first motion vector of the 4×4 unit of the motion vector set of the current block at the position (x1, y1+4), and mv4E0 is the first motion vector of the 4×4 unit of the motion vector set of the current block at the position (x1+4, y1+4).

[0265] Specifically, MvC_x and MvC_y can be determined by:

[0266] MvC_x=(mv1E0_x+mv2E0_x+mv3E0_x+mv4E0_x+2)>>2

[0267] MvC_y=(mv1E0_y+mv2E0_y+mv3E0_y+mv4E0_y+2)>>2

[0268] Exemplarily, in the present application, it is assumed that the position of the upper left corner sample of the current block in the brightness sample matrix of the current image is (xE, yE). If the prediction reference mode of the current block is 0, that is, the first reference mode 'PRED_List0' is used, and the sub-block size flag is 1, that is, the sub-block size parameter is 8×8, then the target motion vector mv0E0 is the first motion vector of the 8×8 unit of the motion vector set of the current block at the position (xE+x, yE+y). The value of the element predMatrixL0[x][y] in the luma prediction sample matrix predMatrixL0 is the sample value at position (((xE+x)<<4)+mv0E0_x, (yE+y)<<4)+mv0E0_y))) in the 1 / 16 precision luma sample matrix with reference index RefIdxL0 in reference picture queue 0, and the value of the element predMatrixL0[x][y] in the chroma prediction sample matrix predMatrixL0 is the sample value at position (((xE+2×x)<<4)+MvC_x, (yE+2×y)<<4)+MvC_y))) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL0 in reference picture queue 0. Among them, MvC_x is equal to mv0E0_x, and MvC_y is equal to mv0E0.

[0269] Exemplarily, in the present application, it is assumed that the position of the upper left corner sample of the current block in the brightness sample matrix of the current image is (xE, yE). If the prediction reference mode of the current block is 1, that is, the second reference mode 'PRED_List1' is used, and the sub-block size flag is 0, that is, the sub-block size parameter is 4×4, then the target motion vector mv0E1 is the first motion vector of the 4×4 unit of the motion vector set of the current block at the position (xE+x, yE+y). The value of the element predMatrixL1[x][y] in the luma prediction sample matrix predMatrixL1 is the sample value at position (((xE+x)<<4)+mv0E1_x, (yE+y)<<4)+mv0E1_y))) in the 1 / 16 precision luma sample matrix with reference index RefIdxL1 in reference picture queue 1. The value of the element predMatrixL1[x][y] in the chroma prediction sample matrix predMatrixL1 is the sample value at position (((xE+2×x)<<4)+MvC_x, (yE+2×y)<<4)+MvC_y))) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL1 in reference picture queue 1. Among them, x1=((xE+2×x)>>3)<<3, y1=((yE+2×y)>>3)<<3, mv1E1 is the first motion vector of the 4×4 unit of the MvArray first motion vector set at the position (x1, y1), mv2E1 is the first motion vector of the 4×4 unit of the MvArray first motion vector set at the position (x1+4, y1), mv3E1 is the first motion vector of the 4×4 unit of the MvArray first motion vector set at the position (x1, y1+4), and mv4E1 is the first motion vector of the 4×4 unit of the MvArray first motion vector set at the position (x1+4, y1+4).

[0270] Specifically, MvC_x and MvC_y can be determined by:

[0271] MvC_x=(mv1E1_x+mv2E1_x+mv3E1_x+mv4E1_x+2)>>2

[0272] MvC_y=(mv1E1_y+mv2E1_y+mv3E1_y+mv4E1_y+2)>>2

[0273] Exemplarily, in the present application, it is assumed that the position of the upper left corner sample of the current block in the brightness sample matrix of the current image is (xE, yE). If the prediction reference mode of the current block is 1, that is, the second reference mode 'PRED_List1' is used, and the sub-block size flag is 1, that is, the sub-block size parameter is 8×8, then the target motion vector mv0E1 is the first motion vector of the 8×8 unit of the motion vector set of the current block at the position (xE+x, yE+y). The value of the element predMatrixL1[x][y] in the luma prediction sample matrix predMatrixL1 is the sample value at position (((xE+x)<<4)+mv0E1_x, (yE+y)<<4)+mv0E1_y))) in the 1 / 16 precision luma sample matrix with reference index RefIdxL1 in reference picture queue 1, and the value of the element predMatrixL1[x][y] in the chroma prediction sample matrix predMatrixL1 is the sample value at position (((xE+2×x)<<4)+MvC_x, (yE+2×y)<<4)+MvC_y))) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL1 in reference picture queue 1. Where MvC_x is equal to mv0E1_x, and MvC_y is equal to mv0E1.

[0274] Exemplarily, in the present application, it is assumed that the position of the upper left corner sample of the current block in the brightness sample matrix of the current image is (xE, yE). If the prediction reference mode of the current block takes a value of 2, that is, the third reference mode 'PRED_List01' is used, then the target motion vector mv0E0 is the first motion vector of the 8×8 unit of the motion vector set of the current block at the position (xE+x, yE+y), and the target motion vector mv0E1 is the first motion vector of the 8×8 unit of the motion vector set of the current block at the position (x, y). The value of the element predMatrixL0[x][y] in the luma prediction sample matrix predMatrixL0 is the sample value at position (((xE+x)<<4)+mv0E0_x, (yE+y)<<4)+mv0E0_y))) in the 1 / 16 precision luma sample matrix with reference index RefIdxL0 in reference picture queue 0. The value of the element predMatrixL0[x][y] in the chroma prediction sample matrix predMatrixL0 is the sample value at position (((xE+2×x)<<4)+MvC0_x, (yE+2×y)<<4)+MvC0_y))) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL0 in reference picture queue 0. The value of the element predMatrixL1[x][y] in the luminance prediction sample matrix predMatrixL1 is the sample value at position ((((xE+x)<<4)+mv0E1_x, (yE+y)<<4)+mv0E1_y)))) in the 1 / 16 precision luminance sample matrix with reference index RefIdxL1 in reference picture queue 1. The value of the element predMatrixL1[x][y] in the chrominance prediction sample matrix predMatrixL1 is the sample value at position (((xE+2×x)<<4)+MvC1_x, (yE+2×y)<<4)+MvC1_y))) in the 1 / 32 precision chrominance sample matrix with reference index RefIdxL1 in reference picture queue 1. Among them, MvC0_x is equal to mv0E0_x, MvC0_y is equal to mv0E0_y, MvC1_x is equal to mv0E1_x, and MvC1_y is equal to mv0E1_y.

[0275] It should be noted that, in the embodiment of the application, the brightness sample matrix in the sample matrix can be a brightness sample matrix with 1 / 16 precision, and the chrominance sample matrix in the sample matrix can be a chrominance sample matrix with 1 / 32 precision.

[0276] It can be understood that, in the embodiments of the present application, for different prediction reference modes, the reference picture queues and reference indexes obtained by the decoder through parsing the bitstream are different.

[0277] Furthermore, in an embodiment of the present application, when the decoder determines the sample matrix, it can first obtain the luminance interpolation filter coefficients and the chrominance interpolation filter coefficients; then the luminance sample matrix can be determined based on the luminance interpolation filter coefficients, and at the same time, the chrominance sample matrix can be determined based on the chrominance interpolation filter coefficients.

[0278] Exemplarily, in the present application, when the decoder determines the brightness sample matrix, the brightness interpolation filter coefficients obtained are as shown in Table 1 above, and then the brightness sample matrix is ​​calculated according to the pixel position and the sample position.

[0279] Specifically, the sample position a x，0 (x=1-15) is obtained by filtering the 8 integer values ​​closest to the interpolation point in the horizontal direction. The prediction value is obtained as follows:

[0280] a x，0 =Clip1((fL[x][0]×A -3，0 +fL[x][1]×A -2，0 +fL[x][2]×A -1，0 +fL[x][3]×A 0，0 +fL[x][4]×A 1，0 +fL[x][5]×A 2，0 +fL[x][6]×A 3，0 +fL[x][7]×A 4，0 +32)>>6).

[0281] Specifically, the sample position a 0，y (y=1-15) is obtained by filtering the 8 integer values ​​closest to the interpolation point in the vertical direction. The predicted value is obtained as follows:

[0282] a 0，y =Clip1((fL[y][0]×A 0，-3 +fL[y][1]×A -2，0 +fL[y][2]×A -1，0 +fL[y][3]×A 0，0 +fL[y][4]×A 1，0 +fL[y][5]×A 2，0 +fL[y][6]×A 3，0 +fL[y][7]×A -4，0 +32)>>6).

[0283] Specifically, the sample position a x，y The method for obtaining the predicted value of (x=1-15, y=1-15) is as follows:

[0284] a x，y＝Clip1((fL[y][0]×a' x，y-3 +fL[y][1]×a' x，y-2 +fL[y][2]× a'x，y-1 +fL[y][3]× a'x，y +fL[y][4]× a'x，y+1 +fL[y][5]× a'x，y +2+fL[y][6]× a'x，y+3 +fL[y][7]× a'x，y+4 +(1<<(19-BitDepth)))>>(20-BitDepth)).

[0285] in:

[0286] a' x，y =(fL[x][0]×A -3，y +fL[x][1]×A -2，y +fL[x][2]×A -1，y+ fL[x][3]×A 0，y +fL[x][4]×A 1，y +fL[x][5]×A 2，y +fL[x][6]×A 3，y +fL[x][7]×A 4，y+ ((1<<(BitDepth-8))>>1))>>(BitDepth-8).

[0287] Exemplarily, in the present application, when determining the chroma sample matrix, the decoder may first parse the bitstream to obtain the chroma interpolation filter coefficients as shown in Table 2 above, and then calculate the chroma sample matrix according to the pixel position and sample position.

[0288] Specifically, for sub-pixel points where dx is equal to 0 or dy is equal to 0, the chrominance integer pixel interpolation can be directly used. For points where dx is not equal to 0 and dy is not equal to 0, the sub-pixels on the integer pixel row (dy is equal to 0) are used for calculation:

[0289] if(dx==0){

[0290] a x，y (0,dy)=Clip3(0,(1< <BitDepth)-1,(fC[dy][0]×A x，y -1+fC[dy][1]×A x，y +fC[dy][2]×A x，y +1+fC[dy][3]×A x，y +2+32)>>6)}

[0291] else if(dy==0){

[0292] a x，y (dx, 0) = Clip3 (0, (1 < <BitDepth)-1,(fC[dx][0]×A x-1，y +fC[dx][1]×A x，y +fC[dx][2]×A x+1，y +fC[dx][3]×A x+2，y+32 )>>6)}

[0293] else{

[0294] a x，y (dx, dy) = Clip3 (0, (1 < <BitDepth)-1,(C[dy][0]×a' x，y-1 (dx, 0) + C[dy][1] × a' x，y (dx, 0) + C[dy][2] × a' x，y +1(dx,0)+C[dy][3]×a' x，y +2(dx,0)+(1<<(19-BitDepth)))>>(20-BitDepth))

[0295] }

[0296] Among them, a' x，y (dx, 0) is the temporary value of the sub-pixel on the integer pixel row, defined as: a' x，y (dx, 0) = (fC[dx][0] × A x-1，y +fC[dx][1]×A x，y +fC[dx][2]×A x+1，y+ fC[dx][3]×A x+2，y +((1<<(BitDepth-8))>>1))>>(BitDepth-8).

[0297] Further, in an embodiment of the present application, the decoder can first parse the code stream and obtain secondary prediction parameters based on the first motion vector deviation between the pixel position and the sub-block; if the secondary prediction parameters indicate the use of secondary prediction, the decoder can determine the first motion vector deviation between the sub-block and each pixel position based on the value variable.

[0298] Specifically, in an embodiment of the present application, when the decoder determines the first motion vector deviation between the sub-block and each pixel position based on the difference variable, it can determine the four difference variables dHorX, dVerX, dHorY and dVerY according to the control point motion vector group, the control point mode and the size parameters of the current block according to the method proposed in the above step 302, and then use the difference variables to further determine the first motion vector deviation corresponding to each pixel position in the sub-block.

[0299] Exemplarily, in this application, width and height are the width and height of the current block obtained by the decoder, respectively, and the width subwidth and height subheight of the subblock are determined by the subblock size parameter. Assuming (i, j) is the coordinate of any pixel point inside the subblock, where the value range of i is 0 to (subwidth-1), and the value range of j is 0 to (subheight-1), then the first motion vector deviation of each pixel (i, j) position inside the 4 different types of subblocks can be calculated by the following method:

[0300] If the sub-block is the control point A at the upper left corner of the current block, then the first motion vector deviation dMvA[i][j] of the (i, j) pixel is:

[0301] dMvA[i][j][0]=dHorX×i+dVerX×j

[0302] dMvA[i][j][1]=dHorY×i+dVerY×j;

[0303] If the sub-block is the control point B at the upper right corner of the current block, then the first motion vector deviation dMvB[i][j] of the (i, j) pixel is:

[0304] dMvB[i][j][0]=dHorX×(i-subwidth)+dVerX×j

[0305] dMvB[i][j][1]=dHorY×(i-subwidth)+dVerY×j;

[0306] If the sub-block is the control point C at the lower left corner of the current block, the control point motion vector group may be a motion vector group including three motion vectors, then the first motion vector deviation dMvC[i][j] of the (i, j) pixel is:

[0307] dMvC[i][j][0]=dHorX×i+dVerX×(j-subheight)

[0308] dMvC[i][j][1]=dHorY×i+dVerY×(j-subheight);

[0309] Otherwise, the first motion vector deviation dMvN[i][j] of the (i, j) pixel is:

[0310] dMvN[i][j][0]=dHorX×(i–(subwidth>>1))+dVerX×(j–(subheight>>1))

[0311] dMvN[i][j][1]=dHorY×(i–(subwidth>>1))+dVerY×(j–(subheight>>1)).

[0312] Wherein, dMvX[i][j][0] represents the deviation value of the first motion vector deviation in the horizontal component, and dMvX[i][j][1] represents the deviation value of the first motion vector deviation in the vertical component. X is A, B, C or N.

[0313] It can be understood that in the embodiment of the present application, after the decoder determines the first motion vector deviation between the sub-block and each pixel position based on the difference variable, it can use all the first motion vector deviations corresponding to all the pixel positions in the sub-block to construct a motion vector deviation matrix corresponding to the sub-block. It can be seen that the motion vector deviation matrix includes the motion vector deviation between the sub-block and any internal pixel point, that is, the first motion vector deviation.

[0314] Further, in an embodiment of the present application, if the secondary prediction parameters obtained by the decoder when parsing the bitstream indicate that secondary prediction is not to be used, then the decoder may choose to directly use the first prediction value of the sub-block of the current block obtained in the above step 303a as the second prediction value of the sub-block, without performing the following steps 304 and 305.

[0315] Specifically, in an embodiment of the present application, if the secondary prediction parameter indicates that secondary prediction is not used, the decoder can determine the second prediction value using the prediction sample matrix. Wherein the prediction sample matrix includes the first prediction values ​​of multiple sub-blocks, and the decoder can determine the first prediction value of the sub-block where the pixel position is located as its own second prediction value.

[0316] Exemplarily, in the present application, if the prediction reference mode of the current block takes a value of 0 or 1, that is, the first reference mode 'PRED_List0' is used, or the second reference mode 'PRED_List1' is used, then the first prediction value of the sub-block where the pixel position is located can be directly selected from the prediction sample matrix including a luminance prediction sample matrix and two chrominance prediction sample matrices, and the first prediction value is determined as the inter-frame prediction value of the pixel position, that is, the second prediction value.

[0317] Exemplarily, in the present application, if the prediction reference mode value of the current block is 2, that is, the third reference mode 'PRED_List01' is used, then the two luminance prediction sample matrices (2 groups of 4 chrominance prediction sample matrices) included in the prediction sample matrix can be averaged to obtain one averaged luminance prediction sample (2 averaged chrominance prediction samples), and finally, the first prediction value of the sub-block where the pixel position is located is selected from the averaged luminance prediction sample (2 averaged chrominance prediction samples), and the first prediction value is determined as the inter-frame prediction value of the pixel position, that is, the second prediction value.

[0318] Step 304: Determine the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation.

[0319] In an embodiment of the present application, after the decoder determines the first prediction value of the sub-block and the first motion vector deviation between each pixel position in the sub-block and the sub-block based on the first motion vector, the center position and the second motion vector deviation corresponding to each pixel position can be determined according to the first motion vector deviation.

[0320] It should be noted that, in an embodiment of the present application, for each pixel position in a sub-block, when performing secondary prediction or PROF processing, if the default center position used by the filter has a large deviation from the motion vector between the sub-block, then the prediction result obtained may have a large error. Therefore, after determining the first motion vector deviation between each pixel position and the sub-block, the decoder can first confirm whether each pixel position is suitable as the center position of the filter used for secondary prediction or PROF processing of a certain pixel. If not, the center position can be readjusted.

[0321] It can be understood that, in the present application, the center position and the second motion vector deviation corresponding to a pixel position may be the center position and the motion vector deviation of the filter used when performing secondary prediction or PROF processing on the pixel position. That is, the center position and the second motion vector deviation are used for secondary prediction or PROF processing on the pixel position.

[0322] Further, in an embodiment of the present application, a method for a decoder to determine a center position and a second motion vector deviation corresponding to each pixel position according to the first motion vector deviation may include the following steps:

[0323] Step 304a, determining a horizontal deviation and a vertical deviation of the first motion vector deviation;

[0324] Step 304b: determine the center position and the second motion vector deviation according to the first absolute value of the horizontal deviation, the second absolute value of the vertical deviation and a preset deviation threshold.

[0325] In an embodiment of the present application, the decoder may first determine the deviation values ​​of the first motion vector deviation in different directions, that is, determine the horizontal deviation and vertical deviation corresponding to the first motion vector deviation. Then, the decoder may further calculate the absolute value of the horizontal deviation, that is, the first absolute value, and the absolute value of the vertical deviation, that is, the second absolute value. Finally, the decoder may further determine the center position and the second motion vector deviation used in secondary prediction or PROF processing based on the first absolute value, the second absolute value, and the preset deviation threshold.

[0326] It should be noted that, in the embodiment of the application, the decoder may compare the first absolute value of the horizontal deviation and the second absolute value of the vertical deviation with the preset deviation thresholds, respectively, so as to determine the center position and the second motion vector deviation according to the comparison results. The preset deviation threshold may be a pre-set value used to determine whether to adjust the center position and the motion vector deviation.

[0327] Exemplarily, in the present application, the preset deviation threshold may be in pixels, specifically, the preset deviation threshold may be k pixels, wherein k is greater than 0.5 and less than or equal to 1. That is, the preset deviation threshold may be set in advance to half a pixel, three quarters of a pixel, or one pixel.

[0328] Further, in an embodiment of the present application, when the decoder determines the center position and the second motion vector deviation based on the horizontal deviation, the vertical deviation and the preset deviation threshold, if the first absolute value and the second absolute value are both smaller than the preset deviation threshold, then the first motion vector deviation can be determined as the second motion vector deviation, and each pixel position can be determined as the center position.

[0329] It can be understood that in the embodiment of the application, after the decoder compares the first absolute value of the horizontal deviation and the second absolute value of the vertical deviation with the preset deviation threshold values ​​respectively, if the first absolute value and the second absolute value are both smaller than the preset deviations, it can be considered that the vector deviation between the pixel position and the sub-block is small, and the prediction result obtained by using the pixel position as the center position of the filter for secondary prediction or PROF processing is relatively accurate. Therefore, the first motion vector deviation corresponding to the pixel position can be directly determined as the second motion vector deviation for secondary prediction or PROF processing, and at the same time, the pixel position can be directly determined as the center position used by the filter for secondary prediction or PROF processing.

[0330] Further, in an embodiment of the present application, when the decoder determines the center position and the second motion vector deviation based on the horizontal deviation, the vertical deviation and the preset deviation threshold, if the first absolute value is greater than or equal to the preset deviation threshold, and the second absolute value is less than the preset deviation threshold, then the first adjustment direction can be determined based on the horizontal deviation; and then the center position and the second motion vector deviation can be determined based on the first adjustment direction.

[0331] It should be noted that, in the embodiment of the present application, the first adjustment direction is used to adjust the first motion vector in the horizontal direction, and therefore, the first adjustment direction includes the left side and the right side.

[0332] It can be understood that in the embodiment of the application, after the decoder compares the first absolute value of the horizontal deviation and the second absolute value of the vertical deviation with the preset deviation threshold respectively, if the first absolute value is greater than or equal to the preset deviation threshold, and the second absolute value is less than the preset deviation threshold, it can be considered that the motion vector deviation between the pixel position and the sub-block in the horizontal direction is large. If the pixel is used as the center position of the filter for secondary prediction or PROF processing, the prediction result obtained may have errors. Therefore, it is necessary to further determine the first adjustment direction according to the horizontal deviation, and then adjust according to the first adjustment direction. Finally, the center position and the second motion vector deviation used for secondary prediction or PROF processing can be determined.

[0333] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (0.75, 0) and the preset deviation threshold is 1 / 2 pixel, the first absolute value is 0.75 and the second absolute value is 0. It can be seen that the first absolute value is greater than the preset deviation threshold and the second absolute value is less than the preset deviation threshold, that is, the motion vector deviation of the pixel position in the horizontal direction is larger, and based on the corresponding horizontal deviation of 0.75, it can be determined that it is closer to the right side. Therefore, it can be determined that the first adjustment direction is the right side.

[0334] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (-0.75, 0) and the preset deviation threshold is 1 / 2 pixel, the first absolute value is 0.75 and the second absolute value is 0. It can be seen that the first absolute value is greater than the preset deviation threshold and the second absolute value is less than the preset deviation threshold, that is, the motion vector deviation of the pixel position in the horizontal direction is larger, and based on the corresponding horizontal deviation of -0.75, it can be determined that it is closer to the left side. Therefore, it can be determined that the first adjustment direction is the left side.

[0335] Further, in an embodiment of the present application, when the decoder determines the center position and the second motion vector deviation based on the first adjustment direction, if the first adjustment direction is the left side, then the adjacent left pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be added with (1, 0) to obtain the corresponding second motion vector deviation.

[0336] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (-0.75, 0) and the preset deviation threshold is 1 / 2 pixel, the first adjustment direction is determined to be the left side. Then, the pixel position on the left side adjacent to the pixel position can be used as the center position of the filter. Then, the corresponding first motion vector deviation (-0.75, 0) can be added to (1, 0), and the final second motion vector deviation is (0.25, 0).

[0337] Further, in an embodiment of the present application, the decoder determines the center position and the second motion vector deviation based on the first adjustment direction. If the first adjustment direction is the right side, the adjacent right pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be subtracted from (1, 0) to obtain the second motion vector deviation.

[0338] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (0.75, 0) and the preset deviation threshold is 1 / 2 pixel, the first adjustment direction is determined to be the right side. Then, the pixel position on the right side adjacent to the pixel position can be used as the center position of the filter. Then, the corresponding first motion vector deviation (0.75, 0) can be subtracted from (1, 0), and the final second motion vector deviation is (-0.25, 0).

[0339] Further, in an embodiment of the present application, when the decoder determines the center position and the second motion vector deviation based on the horizontal deviation, the vertical deviation and the preset deviation threshold, if the second absolute value is greater than or equal to the preset deviation threshold, and the first absolute value is less than the preset deviation threshold, then the second adjustment direction can be determined based on the vertical deviation; then the center position and the second motion vector deviation can be determined based on the second adjustment direction.

[0340] It should be noted that, in the embodiment of the present application, the second adjustment direction is used to adjust the first motion vector in the vertical direction, and therefore, the second adjustment direction includes an upper side and a lower side.

[0341] It can be understood that in the embodiment of the application, after the decoder compares the first absolute value of the horizontal deviation and the second absolute value of the vertical deviation with the preset deviation threshold respectively, if the second absolute value is greater than or equal to the preset deviation threshold, and the first absolute value is less than the preset deviation threshold, it can be considered that the motion vector deviation between the pixel position and the sub-block in the vertical direction is large. If the pixel is used as the center position of the filter for secondary prediction or PROF processing, the prediction result obtained may have errors. Therefore, it is necessary to further determine the second adjustment direction according to the vertical deviation, and then adjust according to the second adjustment direction. Finally, the center position and the second motion vector deviation used for secondary prediction or PROF processing can be determined.

[0342] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (0, 0.75) and the preset deviation threshold is 1 / 2 pixel, the second absolute value is 0.75, and the first absolute value is 0. It can be seen that the second absolute value is greater than the preset deviation threshold, and the first absolute value is less than the preset deviation threshold, that is, the motion vector deviation of the pixel position in the vertical direction is larger, and based on the corresponding vertical deviation of 0.75, it can be determined that it is closer to the lower side. Therefore, it can be determined that the second adjustment direction is the lower side.

[0343] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (0, -0.75) and the preset deviation threshold is 1 / 2 pixel, the second absolute value is 0.75, and the first absolute value is 0. It can be seen that the second absolute value is greater than the preset deviation threshold, and the first absolute value is less than the preset deviation threshold, that is, the motion vector deviation of the pixel position in the vertical direction is larger, and based on the corresponding vertical deviation of -0.75, it can be determined that it is closer to the upper side. Therefore, it can be determined that the second adjustment direction is the upper side.

[0344] Further, in an embodiment of the present application, when the decoder determines the center position and the second motion vector deviation according to the second adjustment direction, if the second adjustment direction is upward, then the adjacent upper pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be added with (0, 1) to obtain the corresponding second motion vector deviation.

[0345] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (0, -0.75) and the preset deviation threshold is 1 / 2 pixel, the second adjustment direction is determined to be the upper side, then the pixel position adjacent to the pixel position and on the upper side can be used as the center position of the filter, and then the corresponding first motion vector deviation (0, -0.75) can be added to (0, 1), and the final second motion vector deviation is (0, 0.25).

[0346] Further, in an embodiment of the present application, the decoder determines the center position and the second motion vector deviation based on the second adjustment direction. If the second adjustment direction is the lower side, the adjacent lower pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be subtracted from (0, 1) to obtain the second motion vector deviation.

[0347] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (0, 0.75) and the preset deviation threshold is 1 / 2 pixel, the second adjustment direction is determined to be the lower side, then the pixel position adjacent to the pixel position and on the lower side can be used as the center position of the filter, and then the corresponding first motion vector deviation (0, 0.75) can be subtracted from (0, 1), and the final second motion vector deviation is (0, 0.25).

[0348] Further, in an embodiment of the present application, when the decoder determines the center position and the second motion vector deviation based on the horizontal deviation, the vertical deviation and the preset deviation threshold, if the first absolute value and the second absolute value are both greater than or equal to the preset deviation threshold, then the third adjustment direction can be determined based on the horizontal deviation and the vertical deviation; then the center position and the second motion vector deviation can be determined based on the third adjustment direction.

[0349] It should be noted that, in the embodiment of the present application, the third adjustment direction is used to adjust the first motion vector in the horizontal direction and the vertical direction at the same time. Therefore, the third adjustment direction includes the upper right side, the lower right side, the upper left side and the lower left side.

[0350] It can be understood that in the embodiment of the application, after the decoder compares the first absolute value of the horizontal deviation and the second absolute value of the vertical deviation with the preset deviation threshold respectively, if the first absolute value and the second absolute value are both greater than or equal to the preset deviation threshold, it can be considered that the motion vector deviations between the pixel position and the sub-block in the horizontal and vertical directions are large. If the pixel is used as the center position of the filter for secondary prediction or PROF processing, the prediction result obtained may have errors. Therefore, it is necessary to further determine the third adjustment direction according to the horizontal deviation and the vertical deviation, and then adjust according to the third adjustment direction. Finally, the center position and the second motion vector deviation used for secondary prediction or PROF processing can be determined.

[0351] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (0.75, 0.75) and the preset deviation threshold is 1 / 2 pixel, the first absolute value is 0.75 and the second absolute value is 0.75. It can be seen that both the first absolute value and the second absolute value are greater than the preset deviation threshold, that is, the motion vector deviations of the pixel position in the horizontal and vertical directions are large, and based on the corresponding horizontal deviation of 0.75, it can be determined that it is closer to the right side. At the same time, based on the corresponding vertical deviation of 0.75, it can be determined that it is closer to the bottom side. Therefore, it can be determined that the third adjustment direction is the lower right side.

[0352] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (-0.75, -0.75) and the preset deviation threshold is 1 / 2 pixel, the first absolute value is -0.75, and the second absolute value is -0.75. It can be seen that both the first absolute value and the second absolute value are greater than the preset deviation threshold, that is, the motion vector deviations of the pixel position in the horizontal and vertical directions are large, and based on the corresponding horizontal deviation of -0.75, it can be determined that it is closer to the left side. At the same time, based on the corresponding vertical deviation of -0.75, it can be determined that it is closer to the upper side. Therefore, it can be determined that the third adjustment direction is the upper left side.

[0353] Further, in an embodiment of the present application, when the decoder determines the center position and the second motion vector deviation according to the third adjustment direction, if the third adjustment direction is the upper left side, then the adjacent upper left pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be added with (1, 1) to obtain the corresponding second motion vector deviation.

[0354] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (-0.75, -0.75) and the preset deviation threshold is 1 / 2 pixel, the third adjustment direction is determined to be the upper left side. Then, the pixel position adjacent to the pixel position and on the upper left side can be used as the center position of the filter. Then, the corresponding first motion vector deviation (-0.75, -0.75) can be added to (1, 1), and the final second motion vector deviation is (0.25, 0.25).

[0355] Further, in an embodiment of the present application, the decoder determines the center position and the second motion vector deviation based on the third adjustment direction. If the third adjustment direction is the lower right side, then the adjacent lower right pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be subtracted from (1, 1) to obtain the second motion vector deviation.

[0356] Exemplarily, in the present application, when the first motion vector deviation of the pixel position is (0.75, 0.75) and the preset deviation threshold is 1 / 2 pixel, the third adjustment direction is determined to be the lower right side. Then, the pixel position adjacent to the pixel position and on the lower right side can be used as the center position of the filter. Then, the corresponding first motion vector deviation (0.75, 0.75) can be subtracted from (1, 1), and the final second motion vector deviation is (-0.25, -0.25).

[0357] It can be understood that, in the embodiment of the present application, the deviation values ​​of the second motion vector deviation in the horizontal direction and the vertical direction are both smaller than the preset deviation threshold.

[0358] That is to say, in the present application, after determining the deviation between the motion vector of each pixel position in the sub-block and the motion vector of the sub-block, and before filtering the predicted block based on the prediction of the sub-block, the decoder can first determine the center position and the second motion vector deviation, wherein the center position and the second motion vector are used for secondary prediction or PROF processing.

[0359] Specifically, in the present application, for each pixel position in the sub-block, the size of the first motion vector deviation between the pixel position and the sub-block can be determined first, including the horizontal deviation and vertical deviation of the first motion vector deviation in the horizontal direction and the vertical direction. If the absolute value of the motion vector deviation in the horizontal direction and / or the vertical direction of a certain pixel position is greater than a preset deviation threshold, then it can be determined that the center position of the filter corresponding to the pixel position is no longer the pixel position itself, and the center position and the second motion vector deviation need to be further determined.

[0360] For example, in the present application, it is assumed that the previous pixel position is CURRENT. Accordingly, the pixel position to the left of the current pixel position in the horizontal direction is LEFT, then LEFT is the coordinate of CURRENT plus (-1, 0); the pixel position to the right of the current pixel position in the horizontal direction is RIGHT, then RIGHT is the coordinate of CURRENT plus (1, 0); the pixel position to the upper side of the current pixel position in the vertical direction is UP, then UP is the coordinate of CURRENT plus (0, -1); the pixel position to the lower side of the current pixel position in the vertical direction is DOWN, then DOWN is the coordinate of CURRENT. Add (0, 1); if the pixel position to the upper left of the current pixel position is UPLEFT, then UPLEFT is the coordinate of CURRENT plus (-1, -1); if the pixel position to the upper right of the current pixel position is UPRIGHT, then UPRIGHT is the coordinate of CURRENT plus (1, -1); if the pixel position to the lower left of the current pixel position is DOWNLEFT, then DOWNLEFT is the coordinate of CURRENT plus (-1, 1); if the pixel position to the lower right of the current pixel position is DOWNRIGHT, then DOWNRIGHT is the coordinate of CURRENT plus (1, 1).

[0361] Further, in an embodiment of the present application, before performing secondary prediction or PROF processing on a pixel position in a sub-block, if it is determined that the absolute value of the deviation component of the first motion vector deviation between the pixel position and the sub-block in the horizontal direction and / or vertical direction is greater than or equal to a preset deviation threshold (such as 1 / 2 pixel, 3 / 4 pixel or 1 pixel), then it is necessary to adjust the starting point used for the secondary prediction or PROF processing, that is, the center position of the two-dimensional filter used for the secondary prediction or PROF processing is adjusted.

[0362] It can be understood that in the present application, when adjusting the center position and the motion vector deviation, another pixel position whose motion vector deviation from the pixel position in the horizontal direction and the vertical direction is less than or equal to the preset deviation threshold can be selected in the current block.

[0363] Exemplarily, in the present application, if the first motion vector deviation between the motion vector of the current pixel position in the sub-block and the motion vector of the sub-block is (0.75, 0), and the preset deviation threshold is 1 / 2 pixel, then the pixel position to the right of the current pixel position CURRENT, that is, RIGHT, can be used as the starting position of the secondary prediction of the current pixel position, that is, the center position of the two-dimensional filter. Accordingly, after adjusting the starting position of the secondary prediction, the first motion vector deviation needs to be adjusted accordingly. Specifically, the second motion vector deviation can be determined according to the deviation between the motion vector of the current pixel position and the motion vector of the center position. For example, since the starting position is adjusted from CURRENT to RIGHT, the first motion vector deviation needs to be subtracted by (1, 0), and the adjusted second motion vector deviation is (-0.25, 0).

[0364] Exemplarily, in the present application, if the first motion vector deviation between the motion vector of the current pixel position in the sub-block and the motion vector of the sub-block is (0.75, 0.75), and the preset deviation threshold is 1 / 2 pixel, then the pixel position on the lower right side of the current pixel position CURRENT, i.e., DOWNRIGHT, can be used as the starting position of the PROF processing of the current pixel position, i.e., the center position of the one-dimensional filter. Accordingly, after adjusting the starting position of the secondary prediction, the first motion vector deviation needs to be adjusted accordingly. Specifically, the second motion vector deviation can be determined according to the deviation between the motion vector of the current pixel position and the motion vector of the center position. For example, since the starting position is adjusted from CURRENT to DOWNRIGHT, the first motion vector deviation needs to be subtracted by (1, 1), and the adjusted second motion vector deviation is (-0.25, -0.25).

[0365] Based on the above Fig. 9 , Fig.13 Indication of the center position Figure 3 ,like Fig.13 As shown in the figure, dmv_x0 and dmv_y0 are both greater than half a pixel. It can be seen that the pixel position of square 2 with coordinates (2, 2) is closest to the circle. If the reference position, that is, the center position of the filter, is set at this position, it can be considered that the prediction result obtained is more accurate than the original one. The adjusted motion vector deviation is (dmv_x, dmv_y), the absolute value of dmv_x is smaller than dmv_x0, and the absolute value of dmv_y is smaller than dmv_y0. The dotted box in the figure indicates the pixel position used by the filter before adjustment, and the solid box in the figure indicates the pixel position used by the filter after adjustment.

[0366] In the embodiments of the present application, further, Fig.14 Schematic diagram of the implementation process of the inter-frame prediction method Figure 2 ,like Fig.14 As shown, before determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, that is, before step 304, the method for the decoder to perform inter-frame prediction may further include the following steps:

[0367] Step 306: Limit the first motion vector deviation according to a preset deviation range; wherein the preset deviation range includes a lower deviation limit value and an upper deviation limit value.

[0368] In an embodiment of the present application, before the decoder determines the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, the decoder may first limit the first motion vector. Specifically, the decoder may limit the first motion vector deviation according to a preset deviation range; wherein the preset deviation range includes a lower deviation limit value and an upper deviation limit value.

[0369] Further, in an embodiment of the present application, when the decoder performs limit processing on the first motion vector deviation according to a preset deviation range, if the horizontal deviation and / or vertical deviation is less than the deviation lower limit value, then the horizontal deviation and / or vertical deviation can be set to the deviation lower limit value; if the horizontal deviation and / or vertical deviation is greater than the deviation upper limit value, then the horizontal deviation and / or vertical deviation can be set to the deviation upper limit value.

[0370] It should be noted that, in the present application, the decoder may not restrict the first motion vector deviation, that is, may not perform step 306, but may directly adjust the center position to any suitable pixel position according to the first motion vector deviation. It may also first perform restriction processing on the first motion vector deviation according to step 306, or restrict the adjustable range and motion vector deviation.

[0371] Further, in an embodiment of the present application, the preset deviation range can be composed of a deviation lower limit value min and a deviation upper limit value max, that is, the preset deviation range can be expressed as (min, max). When the first motion vector deviation is restricted, if the deviation value of the first motion vector deviation in the horizontal direction and / or vertical direction is less than the deviation lower limit value min, then the deviation value can be directly set to min; if the deviation value of the first motion vector deviation in the horizontal direction and / or vertical direction is greater than max, then the deviation value can be directly set to max.

[0372] Exemplarily, in the present application, it is assumed that the upper limit value max is 1, that is, the first motion vector deviation is limited to 1 pixel. If the absolute value of the deviation value of the first motion vector deviation in the horizontal direction and / or vertical direction (the first absolute value and / or the second absolute value) is greater than 1, the absolute value of the deviation value of the first motion vector deviation in the horizontal direction and / or vertical direction is set to 1 pixel.

[0373] In the embodiments of the present application, further, Fig.15 Schematic diagram of the implementation process of the inter-frame prediction method Figure 3 ,like Fig.15 As shown, after determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, that is, after step 304, the method for the decoder to perform inter-frame prediction may further include the following steps:

[0374] Step 307: If the center position does not belong to the current block, redetermine the center position according to the pixel positions in the current block.

[0375] In an embodiment of the present application, after the decoder determines the center position and the second motion vector deviation corresponding to each pixel position based on the first motion vector deviation, it can first determine whether the center position exceeds the current block. If the center position exceeds the range of the current block, that is, the center position does not belong to the current block, then the decoder needs to redetermine the center position according to the pixel position in the current block.

[0376] In the embodiments of the present application, further, Fig.16 Schematic diagram of the implementation process of the inter-frame prediction method Figure 4 ,like Fig.16 As shown, after determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, that is, after step 304, the method for the decoder to perform inter-frame prediction may further include the following steps:

[0377] Step 308: If the center position does not belong to the current block, directly determine each pixel position as the center position corresponding to each pixel position.

[0378] In an embodiment of the present application, after the decoder determines the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, it can first determine whether the center position exceeds the current block. If the center position exceeds the range of the current block, that is, the center position does not belong to the current block, then the decoder can directly determine the corresponding, original pixel position as the center position. That is, if the center position determined based on the first motion vector deviation corresponding to each pixel position in the sub-block exceeds the range of the current block, then the decoder can choose to still use each pixel position as the center position.

[0379] That is to say, in an embodiment of the present application, for the starting position of the adjusted secondary prediction, that is, the center position used by the filter determined based on the first motion vector deviation, the decoder can limit the center position to not exceed the range of the current block, that is, it will not exceed the range of the starting position of the secondary prediction of all pixel positions in the current block before adjustment.

[0380] Step 305: Based on the first prediction value, perform secondary prediction or PROF processing according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and determine the second prediction value as the inter-frame prediction value of the sub-block.

[0381] In an embodiment of the present application, after the decoder determines the center position and the second motion vector deviation corresponding to each pixel position based on the first motion vector deviation, it can perform secondary prediction or PROF processing based on the first prediction value according to the center position and the second motion vector deviation, thereby obtaining the predicted value of the pixel at each pixel position.

[0382] Furthermore, in an embodiment of the present application, after the decoder traverses each pixel position in the sub-block and obtains the prediction value of the pixel at each pixel position, it can determine the second prediction value of the sub-block based on the prediction value of the pixel at each pixel position, thereby determining the second prediction value as the inter-frame prediction value of the sub-block.

[0383] It should be noted that in the embodiments of the present application, since the center position and the second motion vector deviation can be used to perform secondary prediction or PROF processing on the pixel points at each pixel position in the sub-block, after obtaining the center position and the second motion vector deviation, the decoder can perform secondary prediction or PROF processing on the pixel points at each pixel position based on the first prediction value using the center position and the second motion vector deviation, and finally obtain the second prediction value corresponding to the sub-block, so that the second prediction value can be determined as the inter-frame prediction value of the sub-block.

[0384] Further, in an embodiment of the present application, the decoder performs secondary prediction or PROF processing based on the first prediction value according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and the method for determining the second prediction value as the inter-frame prediction value of the sub-block may include the following steps:

[0385] Step 305a, parse the code stream and obtain PROF parameters;

[0386] Step 305b: when the PROF parameter indicates to perform PROF processing, determine the pixel horizontal gradient and the pixel vertical gradient corresponding to the center position based on the first prediction value;

[0387] Step 305c, calculating the deviation value corresponding to each pixel position according to the pixel horizontal gradient, the pixel vertical gradient and the second motion vector deviation;

[0388] Step 305d, obtaining a predicted value of a pixel at each pixel position based on the first predicted value and the deviation value;

[0389] Step 305e: Determine a second prediction value using the prediction value of the pixel at each pixel position.

[0390] In an embodiment of the present application, the decoder can first parse the bitstream to obtain PROF parameters. If the PROF parameters indicate PROF processing, the decoder can determine the pixel horizontal gradient and pixel vertical gradient corresponding to the center position based on the first prediction value; wherein the pixel horizontal gradient is the gradient value between the pixel value corresponding to the center position and the pixel value corresponding to the adjacent pixel position in the horizontal direction; the pixel vertical gradient is the gradient value between the pixel value corresponding to the center position and the pixel value corresponding to the adjacent pixel position in the vertical direction.

[0391] Further, in an embodiment of the present application, the decoder can calculate the deviation value corresponding to each pixel position according to the pixel horizontal gradient, pixel vertical gradient and second motion vector deviation of the center position corresponding to each pixel position, wherein the deviation value can be used to correct the predicted value of the pixel value of each pixel position.

[0392] It should be noted that, in an embodiment of the present application, the decoder can further obtain a corrected prediction value corresponding to any pixel position based on the first prediction value and the deviation value, and after traversing each pixel position in the current sub-block and obtaining the corrected prediction value corresponding to each pixel position, use the corrected prediction values ​​corresponding to all pixel positions to determine the second prediction value corresponding to the current sub-block, thereby determining the corresponding inter-frame prediction value. Specifically, in the present application, after completing the prediction based on the sub-block, the first prediction value of the current sub-block is used as the prediction value of each pixel position, and then, the first prediction value is added to the deviation value corresponding to each pixel position, that is, the correction processing of the prediction value of each pixel position can be completed to obtain the corrected prediction value, so that the second prediction value of the current sub-block can be further obtained, and the second prediction value is used as the inter-frame prediction value corresponding to the current sub-block.

[0393] Further, in an embodiment of the present application, the decoder performs secondary prediction or PROF processing based on the first prediction value according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and the method for determining the second prediction value as the inter-frame prediction value of the sub-block may include the following steps:

[0394] Step 305f, parsing the bitstream to obtain secondary prediction parameters;

[0395] Step 305g: when the secondary prediction parameter indicates to use secondary prediction, determine the filter coefficient of the two-dimensional filter according to the second motion vector deviation; wherein the two-dimensional filter is used to perform secondary prediction processing according to a preset shape;

[0396] Step 305h: determining a predicted value of a pixel at each pixel position based on the filter coefficient and the first predicted value;

[0397] Step 305i: Determine a second prediction value using the prediction value of the pixel at each pixel position.

[0398] In an embodiment of the present application, the decoder can first parse the bit stream to obtain secondary prediction parameters. If the secondary prediction parameters indicate the use of secondary prediction, the decoder can determine the filter coefficients of the two-dimensional filter based on the second motion vector deviation; wherein the two-dimensional filter is used to perform secondary prediction processing according to a preset shape.

[0399] It should be noted that, in the embodiment of the present application, the filter coefficient of the two-dimensional filter is related to the second motion vector deviation corresponding to the target pixel position. In other words, for different target pixel positions, if the corresponding second motion vector deviation is different, the filter coefficient of the two-dimensional filter used is also different.

[0400] It is understandable that in the embodiment of the present application, the two-dimensional filter is used to perform secondary prediction using a plurality of adjacent pixel positions forming a preset shape, wherein the preset shape is a rectangle, a diamond or any symmetrical shape.

[0401] That is to say, in the present application, the two-dimensional filter used for secondary prediction is a filter composed of adjacent points constituting a preset shape. The adjacent points constituting the preset shape may include multiple points, for example, 9 points. The preset shape may be a symmetrical shape, for example, the preset shape may include a rectangle, a rhombus, or any other symmetrical shape.

[0402] Exemplarily, in the present application, the two-dimensional filter is a rectangular filter, specifically, the two-dimensional filter is a filter composed of 9 adjacent pixel positions forming a rectangle. Among the 9 pixel positions, the pixel position at the center is the pixel position of the pixel that currently needs secondary prediction, that is, the current pixel position.

[0403] Furthermore, in an embodiment of the present application, when the decoder determines the filter coefficients of the two-dimensional filter based on the second motion vector deviation, it can first parse the bit stream to obtain the scale parameters, and then determine the filter coefficients corresponding to the pixel position based on the scale parameters and the second motion vector deviation.

[0404] It should be noted that, in an embodiment of the present application, the scale parameter may include at least one scale value, and the second motion vector deviation includes a horizontal deviation and a vertical deviation; wherein at least one scale value is a non-zero real number.

[0405] Specifically, in the present application, when the two-dimensional filter uses 9 adjacent pixel positions forming a rectangle for secondary prediction, the pixel position located at the center of the rectangle is the position to be predicted, that is, the current pixel position, and the other 8 target pixel positions are located in the 8 directions of upper left, upper, upper right, right, lower right, lower, lower left, and left of the current pixel position, respectively.

[0406] Accordingly, in the present application, the decoder can calculate 9 filter coefficients corresponding to 9 adjacent pixel positions according to a preset calculation rule based on at least one proportional value and the second motion vector deviation of the position to be predicted.

[0407] It should be noted that, in the present application, the preset calculation rules may include a variety of different calculation methods, such as addition, subtraction, multiplication, etc. Among them, for different pixel positions, different calculation methods may be used to calculate the filter coefficients.

[0408] It can be understood that in the present application, when the decoder calculates multiple filter coefficients corresponding to multiple pixel positions according to different calculation methods in the preset calculation rules, some filter coefficients can be a linear function of the second motion vector deviation, that is, the two are in a linear relationship, or can be a quadratic function or a higher-order function of the second motion vector deviation, that is, the two are in a non-linear relationship.

[0409] That is to say, in the present application, any one of the multiple filter coefficients corresponding to multiple adjacent pixel positions can be a linear function, a quadratic function, or a higher-order function of the second motion vector deviation.

[0410] Exemplarily, in the present application, it is assumed that the second motion vector deviation of the pixel position is (dmv_x, dmv_y), where if the coordinates of the target pixel position are (i, j), then dmv_x can be expressed as dMvX[i][j][0], which represents the deviation value of the second motion vector deviation in the horizontal component, and dmv_y can be expressed as dMvX[i][j][1], which represents the deviation value of the second motion vector deviation in the vertical component.

[0411] Correspondingly, Table 3 shows the filter coefficients obtained based on the second motion vector deviation (dmv_x, dmv_y). As shown in Table 3, for a two-dimensional filter, according to the second motion vector deviation of the pixel position (the horizontal deviation is dmv_x, the vertical deviation is dmv_y) and different scale parameters, such as m and n, 9 filter coefficients corresponding to 9 adjacent pixel positions can be obtained, wherein the decoder can directly set the filter coefficient of the current pixel position in the center to 1.

[0412] Table 3

[0413] Pixel location Filter coefficients Top left (-dmv_x-dmv_y)×m Left -dmv_x×n Lower left (-dmv_x+dmv_y)×m superior -dmv_y×n center 1 Down dmv_y×n Top right (dmv_x-dmv_y)×m right dmv_x×n Bottom right (dmv_x+dmv_y)×m

[0414] Among them, the scale parameters m and n are generally decimals or fractions. One possible case is that m and n are powers of 2, such as 1 / 2, 1 / 4, 1 / 8, etc. Here, dmv_x and dmv_y are their actual sizes, that is, 1 in dmv_x and dmv_y represents the distance of 1 pixel, and dmv_x and dmv_y are decimals or fractions.

[0415] It should be noted that, in the embodiment of the present application, compared with the existing 8-tap filter, the motion vectors of the integer pixel position and the sub-pixel position corresponding to the currently common 8-tap filter are non-negative in the horizontal and vertical directions, and the sizes are between 0 pixels and 1 pixel, that is, dmv_x and dmv_y cannot be negative. In the present application, the motion vectors of the integer pixel position and the sub-pixel position corresponding to the filter can be negative in the horizontal and vertical directions, that is, dmv_x and dmv_y can be negative.

[0416] For example, in the embodiment of the present application, if the ratio parameter m is 1 / 16 and n is 1 / 2, then the above Table 3 can be expressed as the following Table 4:

[0417] Table 4

[0418] Pixel location Filter coefficients Top left (-dmv_x-dmv_y) / 16 Left -dmv_x / 2 Lower left (-dmv_x+dmv_y) / 16 superior -dmv_y / 2 center 1 Down dmv_y / 2 Top right (dmv_x-dmv_y) / 16 right dmv_x / 2 Bottom right (dmv_x+dmv_y) / 16

[0419] It is understandable that in the embodiments of the present application, in video coding and decoding technology and standards, magnification is usually used to avoid decimals, floating point operations, and then the calculated result is reduced by an appropriate multiple to obtain the correct result. Left shift is usually used when magnifying, and right shift is usually used when reducing. Therefore, when performing secondary prediction through a two-dimensional filter, it will be written in the following form in actual application:

[0420] Assuming that the second motion vector deviation of the pixel position is (dmv_x, dmv_y), after left shifting by shift1, (dmv_x', dmv_y') is obtained. Based on the above Table 4, the coefficients of the two-dimensional filter can be expressed as the following Table 5:

[0421] Table 5

[0422] Pixel location Filter coefficients Top left -dmv_x' -dmv_y' Left -dmv_x'×8 Lower left -dmv_x'+dmv_y' superior -dmv_y'×8 center 16<<shift1 Down dmv_y'×8 Top right dmv_x'-dmv_y' right dmv_x×8 Bottom right dmv_x'+dmv_y'

[0423] Fig.17 Schematic diagram of a two-dimensional filter Figure 1 ,like Fig.17As shown in the figure, the result of the prediction based on the sub-block is used as the basis for the secondary prediction. The light-colored square is the integer pixel position of the filter, that is, the position obtained based on the prediction of the sub-block. The circle is the sub-pixel position that needs to be predicted twice, that is, the position of the pixel position. The dark square is the integer pixel position corresponding to the sub-pixel position. When interpolating to obtain this sub-pixel position, 9 integer pixel positions as shown in the figure are required.

[0424] Fig.18 Schematic diagram of a two-dimensional filter Figure 2 ,like Fig.18 As shown in the figure, the result of the prediction based on the sub-block is used as the basis for the secondary prediction. The light-colored square is the integer pixel position of the filter, that is, the position obtained based on the prediction of the sub-block. The circle is the sub-pixel position that needs to be predicted twice, that is, the position of the pixel position. The dark square is the integer pixel position corresponding to the sub-pixel position. When interpolating to obtain this sub-pixel position, 13 integer pixel positions are required as shown in the figure.

[0425] Furthermore, in an embodiment of the present application, after the decoder determines the filter coefficient of the two-dimensional filter according to the second motion vector deviation, it can determine the second prediction value of the current sub-block based on the filter coefficient and the first prediction value, thereby realizing secondary prediction of the current sub-block.

[0426] It can be understood that in the embodiment of the present application, the decoder determines the filter coefficient using the second motion vector deviation corresponding to the pixel position, so that the first prediction value can be corrected by a two-dimensional filter according to the filter coefficient to obtain a corrected second prediction value of the current sub-block. It can be seen that the second prediction value is a correction value based on the first prediction value.

[0427] Furthermore, in an embodiment of the present application, when the decoder determines the second prediction value of the current sub-block based on the filter coefficient and the first prediction value, it can first multiply the filter coefficient and the first prediction value to obtain a product result, and after traversing all pixel positions in the current sub-block, add the product results of all pixel positions in the current sub-block to obtain a sum result, and finally normalize the addition result to finally obtain the corrected second prediction value of the current sub-block.

[0428] It should be noted that in the embodiments of the present application, before performing secondary prediction, the first prediction value of the current sub-block where the pixel position is located is generally used as the prediction value of the pixel position before correction. Therefore, when filtering through a two-dimensional filter, the filter coefficient can be multiplied by the prediction value of the corresponding pixel position, that is, the first prediction value, and the product results corresponding to each pixel position are accumulated and then normalized.

[0429] It is understandable that in the present application, the decoder can perform normalization processing in a variety of ways, for example, the result of multiplying the filter coefficient with the prediction value of the corresponding pixel position and then accumulating it can be right-shifted by 4+shift1. Alternatively, the result of multiplying the filter coefficient with the prediction value of the corresponding pixel position and then accumulating it can be added (1<<(3+shift1)), and then right-shifted by 4+shift1.

[0430] It can be seen that in the present application, after obtaining the second motion vector deviation corresponding to the pixel position within the current sub-block, for each sub-block and each pixel position in each sub-block, a two-dimensional filter can be used for filtering based on the first prediction value of motion compensation of the current sub-block according to the second motion vector deviation to complete the secondary prediction of the current sub-block and obtain a new second prediction value.

[0431] Furthermore, in the embodiment of the present application, the two-dimensional filter can be understood as performing secondary prediction using a plurality of adjacent pixel positions forming a preset shape, wherein the preset shape can be a rectangle, a diamond or any symmetrical shape.

[0432] Specifically, in an embodiment of the present application, when a two-dimensional filter uses nine adjacent pixel positions forming a rectangle for secondary prediction, it can first determine a prediction sample matrix of the current block and a motion vector deviation matrix of the current sub-block of the current block; wherein the motion vector deviation matrix includes the second motion vector deviation corresponding to all pixel positions; and then based on the nine adjacent pixel positions forming a rectangle, the prediction sample matrix and the motion vector deviation matrix are used to determine the sample matrix of the current block after secondary prediction.

[0433] For example, in this application, if the width and height of the current block are width and height respectively, the width and height of each sub-block are subwidth and subheight respectively. Figure 7 As shown, the sub-block where the upper left corner sample of the brightness prediction sample matrix of the current block is located is A, the sub-block where the upper right corner sample is located is B, the sub-block where the lower left corner sample is located is C, and the sub-blocks where other positions are located are other sub-blocks.

[0434] For each sub-block in the current block, the motion vector deviation matrix of the sub-block can be recorded as dMv, then:

[0435] 1. If the sub-block is A, dMv is equal to dMvA;

[0436] 2. If the sub-block is B, dMv is equal to dMvB;

[0437] 3. If the sub-block is C, and there are 3 motion vectors in the control point motion vector group mvAffine of the current sub-block, dMv is equal to dMvC;

[0438] 4. If the sub-block is other than A, B, and C, dMv is equal to dMvN.

[0439] Further, assuming that (x, y) is the coordinate of the upper left corner of the current sub-block, (i, j) is the coordinate of the pixel inside the luminance sub-block, the value range of i is 0 to (subwidth-1), the value range of j is 0 to (subheight-1), the prediction sample matrix based on the sub-block is PredMatrixSb, the prediction sample matrix of the secondary prediction is PredMatrixS, dCenterX, dCenterY are the offset matrices of the horizontal and vertical directions of the secondary prediction starting position, the prediction sample PredMatrixS[x+i][y+j] of the secondary prediction of (x+i, y+j) can be calculated according to the following method:

[0440] dCenterX[i][j]=(dMv[i][j][0]<0?-1:1)*((abs(dMv[i][j][0])+(1<<10)) / (1<<11))

[0441] dCenterX[i][j]=CLIP3(-i,width-1-i)

[0442] dCenterY[i][j]=(dMv[i][j][1]<0?-1:1)*((abs(dMv[i][j][1])+(1<<10)) / (1<<11))

[0443] dCenterY[i][j]=CLIP3(-j,height-1-j)

[0444] dMv[i][j][0]=dMv[i][j][0]-dCenterX*(1<<11)

[0445] dMv[i][j][1]=dMv[i][j][1]-dCenterY*(1<<11)

[0446] PredMatrixS[x+i][y+j]=

[0447] (UPLEFT(x+i,y+j)×(-dMv[i][j][0]-dMv[i][j][1])+

[0448] UP(x+i,y+j)×((-dMv[i][j][1])<<3)+

[0449] UPRIGHT(x+i, y+j) × (dMv[i][j][0] - dMv[i][j][1]) +

[0450] LEFT(x+i, y+j) × ((-dMv[i][j][0]) << 3) +

[0451] CENTER(x+i, y+j) × (1 << 15) +

[0452] RIGHT(x+i, y+j) × (dMv[i][j][0] << 3) +

[0453] DOWNLEFT(x+i, y+j) × (-dMv[i][j][0] + dMv[i][j][1]) +

[0454] DOWN(x+i, y+j) × (dMv[i][j][1] << 3) +

[0455] DOWNRIGHT(x+i, y+j) × (dMv[i][j][0] + dMv[i][j][1]) +

[0456] (1 << 14)) >> 15

[0457] PredMatrixS[x+i][y+j] = Clip3(0, (1 << BitDepth) - 1, PredMatrixS[x+i][y+j]).

[0458] where, UPLEFT(x+i, y+j) = PredMatrixSb[max(0, x+i + dCenterX[i][j] - 1)][max(0, y+j + dCenterY[i][j] - 1)]

[0459] UP(x+i, y+j) = PredMatrixSb[x+i + dCenterX[i][j]][max(0, y+j + dCenterY[i][j] - 1)]

[0460] UPRIGHT(x+i, y+j) = PredMatrixSb[min(width - 1, x+i + dCenterX[i][j] + 1)][max(0, y+j + dCenterY[i][j] - 1)]

[0461] LEFT(x+i, y+j) = PredMatrixSb[max(0, x+i + dCenterX[i][j] - 1)][y+j + dCenterY[i][j]]

[0462] CENTER(x+i,y+j)=PredMatrixSb[x+i+dCenterX[i][j]][y+j+dCenterY[i][j]]

[0463] RIGHT(x+i,y+j)=PredMatrixSb[min(width-1,x+i+dCenterX[i][j]+1)][y+j+dCenterY[i][j]]

[0464] DOWNLEFT(x+i,y+j)=PredMatrixSb[max(0,x+i+dCenterX[i][j]-1)][min(height-1,y+j+dCenterY[i][j]+1)]

[0465] DOWN(x+i,y+j)=PredMatrixSb[x+i+dCenterX[i][j]][max(0,x+i+dCenterX[i][j]-1)][min(height-1,y+j+dCenterY[i][j]+1)]

[0466] DOWNRIGHT(x+i,y+j)=PredMatrixSb[min(width-1,x+i+dCenterX[i][j]+1)][max(0,x+i+dCenterX[i][j]-1)][min(height-1,y+j+dCenterY[i][j]+1)]

[0467] Among them, max(a, b) can be understood as taking the larger value of a and b, and min(a, b) can be understood as taking the smaller value of a and b.

[0468] Further, in the present application, if a method of expanding a prediction block based on sub-block prediction by one row and one column is adopted, that is, the rows of -1 and height and the columns of -1 and width of predMatrix are expanded, the above paragraph can be expressed as:

[0469] UPLEFT(x+i,y+j)=PredMatrixSb[x+i+dCenterX[i][j]-1][y+j+dCenterY[i][j]-1]

[0470] UP(x+i,y+j)=PredMatrixSb[x+i+dCenterX[i][j]][y+j+dCenterY[i][j]-1]

[0471] UPRIGHT(x+i,y+j)=PredMatrixSb[x+i+dCenterX[i][j]+1][y+j+dCenterY[i][j]-1]

[0472] LEFT(x+i,y+j)=PredMatrixSb[x+i+dCenterX[i][j]-1][y+j+dCenterY[i][j]]

[0473] CENTER(x+i,y+j)=PredMatrixSb[x+i+dCenterX[i][j]][y+j+dCenterY[i][j]]

[0474] RIGHT(x+i,y+j)=PredMatrixSb[x+i+dCenterX[i][j]+1][y+j+dCenterY[i][j]]

[0475] DOWNLEFT(x+i,y+j)=PredMatrixSb[x+i+dCenterX[i][j]-1][y+j+dCenterY[i][j]+1]

[0476] DOWN(x+i,y+j)=PredMatrixSb[x+i+dCenterX[i][j]][y+j+dCenterY[i][j]+1]

[0477] DOWNRIGHT(x+i,y+j)=PredMatrixSb[x+i+dCenterX[i][j]+1][y+j+dCenterY[i][j]+1].

[0478] Furthermore, in the present application, dCenterX and dCenterY may be written in the form of temporary variables instead of matrices.

[0479] It can be understood that in the embodiment of the present application, the CENTER (x+i, y+j) pixel position can be the center position of the above 9 adjacent pixel positions that form a rectangle, and then the secondary prediction process can be performed based on the (x+i, y+j) pixel position and the other 8 pixel positions adjacent to it. Specifically, the other 8 pixel positions are UP (up), UPRIGHT (upper right), LEFT (left), RIGHT (right), DOWNLEFT (lower left), DOWN (lower), DOWNRIGHT (lower right), UPLEFT (upper left).

[0480] It should be noted that in this application, the precision of the calculation formula of PredMatrixS[x+i][y+j] can use lower precision. For example, each term on the right side of the multiplication is shifted right, such as dMv[i][j][0] and dMv[i][j][1] are shifted right by shift3 positions. Accordingly, 1<<15 becomes 1<<(15-shift3), ...+(1<<10))>>11 becomes ...+(1<<(10-shift3)))>>(11-shift3).

[0481] Exemplarily, the size of the motion vector may be limited to a reasonable range, such as the positive and negative values ​​of the motion vector used above in the horizontal and vertical directions do not exceed 1 pixel, 1 / 2 pixel, 1 / 4 pixel, etc.

[0482] It is understandable that if the prediction reference mode of the current block is 'Pred_List01', the decoder averages multiple prediction sample matrices of each component to obtain the final prediction sample matrix of the component. For example, two brightness prediction sample matrices are averaged to obtain a new brightness prediction sample matrix.

[0483] Furthermore, in an embodiment of the present application, after obtaining the prediction sample matrix of the current block, if the current block has no transform coefficients, then the prediction matrix is ​​used as the decoding result of the current block. If the current block still has transform coefficients, then the transform coefficients can be decoded first, and the residual matrix can be obtained through inverse transformation and inverse quantization, and the residual matrix is ​​added to the prediction matrix to obtain the decoding result.

[0484] It can be understood that the inter-frame prediction method proposed in the present application can first adjust the initial position of the secondary prediction based on the first motion vector deviation corresponding to a pixel position in the sub-block, that is, adjust the center position used by the filter for secondary prediction or PROF processing, and then further adjust the first motion vector based on the center position determined after the adjustment to obtain the second motion vector deviation. Finally, the center position and the second motion vector deviation can be used to perform point-based prediction on the pixel position, thereby obtaining the second prediction value of the sub-block.

[0485] It should be noted that the inter-frame prediction method proposed in the present application can be applied to any image component. In the present embodiment, the secondary prediction scheme is used for the luminance component, but it can also be used for the chrominance component or any component in other formats. The inter-frame prediction method proposed in the present application can also be applied to any video format, including but not limited to the YUV format, including but not limited to the luminance component in the YUV format.

[0486] This embodiment provides an inter-frame prediction method, which can, after prediction based on sub-blocks, re-determine the center position and the second motion vector deviation used for secondary prediction or PROF processing for the pixel position where the first motion vector deviation between the motion vector and the motion vector of the sub-block is large based on the first motion vector deviation, so that the center position and the second motion vector deviation can be used to perform point-based secondary prediction on the basis of the first prediction value based on the sub-block to obtain the second prediction value. It can be seen that the inter-frame prediction method proposed in this application can be well applied to all scenarios, can reduce the prediction error, greatly improve the encoding performance, and thus improve the encoding and decoding efficiency.

[0487] The embodiment of the present application provides an inter-frame prediction method, which is applied to a video encoding device, i.e., an encoder. The function implemented by the method can be implemented by calling a computer program by a second processor in the encoder, and of course the computer program can be stored in a second memory. It can be seen that the encoder includes at least a second processor and a second memory.

[0488] Fig.19 Schematic diagram of the implementation process of the inter-frame prediction method Figure 5 ,like Fig.19 As shown, the method for the encoder to perform inter-frame prediction may include the following steps:

[0489] Step 401: Determine prediction mode parameters of the current block.

[0490] In an embodiment of the present application, the encoder may first determine the prediction mode parameters of the current block. Specifically, the encoder may first determine the prediction mode used by the current block, and then determine the corresponding prediction mode parameters based on the prediction mode. The prediction mode parameters may be used to determine the prediction mode used by the current block.

[0491] It should be noted that, in the embodiment of the present application, the image to be encoded can be divided into multiple image blocks, the current image block to be encoded can be called the current block, and the image blocks adjacent to the current block can be called adjacent blocks; that is, in the image to be encoded, there is an adjacent relationship between the current block and the adjacent blocks. Here, each current block may include a first image component, a second image component, and a third image component; that is, the current block is an image block in the image to be encoded that is currently to be predicted for the first image component, the second image component, or the third image component.

[0492] Among them, assuming that the current block predicts the first image component, and the first image component is a luminance component, that is, the image component to be predicted is a luminance component, then the current block can also be called a luminance block; or, assuming that the current block predicts the second image component, and the second image component is a chrominance component, that is, the image component to be predicted is a chrominance component, then the current block can also be called a chrominance block.

[0493] It should be noted that, in the embodiments of the present application, the prediction mode parameters indicate the prediction mode used by the current block and the parameters related to the prediction mode. Here, for determining the prediction mode parameters, a simple decision strategy can be adopted, such as determining according to the size of the distortion value; or a complex decision strategy can be adopted, such as determining according to the result of rate distortion optimization (RDO), which is not limited in the embodiments of the present application. Generally speaking, the RDO method can be used to determine the prediction mode parameters of the current block.

[0494] Specifically, in some embodiments, when determining the prediction mode parameters of the current block, the encoder may first perform pre-encoding processing on the current block using multiple prediction modes to obtain rate-distortion cost values ​​corresponding to each prediction mode; then select the minimum rate-distortion cost value from the multiple rate-distortion cost values ​​obtained, and determine the prediction mode parameters of the current block according to the prediction mode corresponding to the minimum rate-distortion cost value.

[0495] That is to say, on the encoder side, multiple prediction modes can be used for the current block to perform pre-coding processing on the current block respectively. Here, the multiple prediction modes generally include inter-frame prediction mode, traditional intra-frame prediction mode and non-traditional intra-frame prediction mode; among which, the traditional intra-frame prediction mode may include direct current (DC) mode, plane (PLANAR) mode and angle mode, etc., the non-traditional intra-frame prediction mode may include matrix-based intra-frame prediction (Matrix-based IntraPrediction, MIP) mode, cross-component linear model prediction (Cross-componentLinearmodelPrediction, CCLM) mode, intra-frame block copy (Intra Block Copy, IBC) mode and PLT (Palette) mode, etc., and the inter-frame prediction mode may include ordinary inter-frame prediction mode, GPM mode and AWP mode, etc.

[0496] In this way, after pre-encoding the current block using multiple prediction modes, the rate-distortion cost value corresponding to each prediction mode can be obtained; then the minimum rate-distortion cost value is selected from the multiple rate-distortion cost values ​​obtained, and the prediction mode corresponding to the minimum rate-distortion cost value is determined as the prediction mode parameter of the current block. In addition, after pre-encoding the current block using multiple prediction modes, the distortion value corresponding to each prediction mode can be obtained; then the minimum distortion value is selected from the multiple distortion values ​​obtained, and then the prediction mode corresponding to the minimum distortion value is determined as the prediction mode used by the current block, and the corresponding prediction mode parameters are set according to the prediction mode. In this way, the current block is finally encoded using the determined prediction mode parameters, and in this prediction mode, the prediction residual can be made smaller, which can improve the coding efficiency.

[0497] That is to say, on the encoding side, the encoder can select the optimal prediction mode to pre-encode the current block. In this process, the prediction mode of the current block can be determined, and then the prediction mode parameters used to indicate the prediction mode can be determined, so that the corresponding prediction mode parameters are written into the bitstream and transmitted from the encoder to the decoder.

[0498] Accordingly, on the decoder side, the decoder can directly obtain the prediction mode parameters of the current block by parsing the bitstream, and determine the prediction mode used by the current block and the relevant parameters corresponding to the prediction mode according to the prediction mode parameters obtained by parsing.

[0499] Step 402: When the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, determine a first motion vector of a sub-block of the current block; wherein the current block includes a plurality of sub-blocks.

[0500] In an embodiment of the present application, if the prediction mode parameter indicates that the current block uses the inter-frame prediction mode to determine the inter-frame prediction value of the current block, the encoder may first determine the first motion vector of each sub-block of the current block, wherein one sub-block corresponds to one first motion vector.

[0501] It should be noted that in the embodiment of the present application, the current block is an image block to be encoded in the current frame, the current frame is encoded in the form of image blocks in a certain order, and the current block is the image block to be encoded at the next moment in the current frame in the order. The current block can have a variety of specifications, such as 16×16, 32×32 or 32×16, where the numbers represent the number of rows and columns of pixels on the current block.

[0502] Furthermore, in an embodiment of the present application, the current block may be divided into a plurality of sub-blocks, wherein each sub-block has the same size, and the sub-block is a set of pixels of a smaller size. The size of the sub-block may be 8×8 or 4×4.

[0503] Exemplarily, in the present application, the size of the current block is 16×16, and can be divided into 4 sub-blocks of 8×8 size each.

[0504] It can be understood that in the embodiment of the present application, when the encoder determines that the prediction mode parameter indicates the use of the inter-frame prediction mode to determine the inter-frame prediction value of the current block, the inter-frame prediction method provided in the embodiment of the present application can continue to be used.

[0505] In an embodiment of the present application, further, when the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, when the encoder determines the first motion vector of the sub-block of the current block, the affine mode parameter and the prediction reference mode of the current block can be determined. When the affine mode parameter indicates that the affine mode is used, the control point mode and the sub-block size parameter are determined. Finally, the first motion vector can be determined according to the prediction reference mode, the control point mode and the sub-block size parameter.

[0506] In an embodiment of the present application, after determining the prediction mode parameters, if the prediction mode parameters indicate that the current block uses the inter-frame prediction mode to determine the inter-frame prediction value of the current block, the encoder can determine the affine mode parameters and the prediction reference mode.

[0507] It should be noted that, in the embodiment of the present application, the affine mode parameter is used to indicate whether to use the affine mode. Specifically, the affine mode parameter may be an affine motion compensation enable flag affine_enable_flag, and the encoder may further determine whether to use the affine mode by determining the value of the affine mode parameter.

[0508] That is, in the present application, the affine mode parameter may be a binary variable. If the value of the affine mode parameter is 1, it indicates that the affine mode is used; if the value of the affine mode parameter is 0, it indicates that the affine mode is not used.

[0509] Exemplarily, in the present application, the value of the affine mode parameter may be equal to the value of the affine motion compensation enable flag affine_enable_flag. If the value of affine_enable_flag is '1', it indicates that affine motion compensation can be used; if the value of affine_enable_flag is '0', it indicates that affine motion compensation should not be used.

[0510] Further, in an embodiment of the present application, if the affine mode parameters determined by the encoder indicate to use the affine mode, the encoder may obtain the control point mode and the sub-block size parameters.

[0511] It should be noted that, in the embodiment of the present application, the control point mode is used to determine the number of control points. In the affine model, a sub-block may have 2 control points or 3 control points, and accordingly, the control point mode may be a control point mode corresponding to 2 control points, or a control point mode corresponding to 3 control points. That is, the control point mode may include a 4-parameter mode and a 6-parameter mode.

[0512] It can be understood that in the embodiments of the present application, for the AVS3 standard, if the current block uses the affine mode, the encoder also needs to determine the number of control points of the current block in the affine mode, so as to determine whether the 4-parameter (2 control point) mode or the 6-parameter (3 control point) mode is used.

[0513] Further, in an embodiment of the present application, if the affine mode parameter determined by the encoder indicates to use the affine mode, the encoder may further determine a sub-block size parameter.

[0514] Specifically, the sub-block size parameter can be represented by the affine prediction sub-block size flag affine_subblock_size_flag, and the encoder can indicate the sub-block size parameter by setting the value of the sub-block size flag, that is, indicating the size of the sub-block of the current block. The size of the sub-block can be 8×8 or 4×4. Specifically, in the present application, the sub-block size flag can be a binary variable. If the value of the sub-block size flag is 1, it indicates that the sub-block size parameter is 8×8; if the value of the sub-block size flag is 0, it indicates that the sub-block size parameter is 4×4.

[0515] Exemplarily, in the present application, the value of the subblock size flag may be equal to the value of the affine prediction subblock size flag affine_subblock_size_flag. If the value of affine_subblock_size_flag is '1', the current block is divided into subblocks of size 8×8; if the value of affine_subblock_size_flag is '0', the current block is divided into subblocks of size 4×4.

[0516] Furthermore, in an embodiment of the present application, after determining the control point mode and the sub-block size parameter, the encoder can further determine the first motion vector of the sub-block in the current block according to the prediction reference mode, the control point mode and the sub-block size parameter.

[0517] Specifically, in an embodiment of the present application, the encoder may first determine a control point motion vector group according to a prediction reference mode; and then determine a first motion vector of the sub-block based on the control point motion vector group, the control point mode and the sub-block size parameter.

[0518] It can be understood that, in the embodiment of the present application, the control point motion vector group can be used to determine the motion vector of the control point.

[0519] It should be noted that in an embodiment of the present application, the encoder can traverse each sub-block in the current block according to the above method, and use the control point motion vector group, control point mode and sub-block size parameters of each sub-block to determine the first motion vector of each sub-block, so that a motion vector set can be constructed based on the first motion vector of each sub-block.

[0520] It can be understood that, in the embodiment of the present application, the motion vector set of the current block may include the first motion vector of each sub-block of the current block.

[0521] Furthermore, in an embodiment of the present application, when the encoder determines the first motion vector based on the control point motion vector group, the control point mode and the sub-block size parameters, it can first determine the difference variable based on the control point motion vector group, the control point mode and the size parameters of the current block; then it can determine the sub-block position based on the prediction mode parameters and the sub-block size parameters; finally, it can use the difference variable and the sub-block position to determine the first motion vector of the sub-block, and then obtain a set of motion vectors of multiple sub-blocks of the current block.

[0522] It should be noted that in the present application, when determining the deviation of each position within a sub-block from the motion vector of the sub-block, if the current block uses an affine prediction model, the motion vector of each position within the sub-block can be calculated according to the formula of the affine prediction model, and their deviations can be obtained by subtracting the motion vector of the sub-block from the motion vector. If the motion vectors of the sub-blocks all select the motion vectors at the same position within the sub-block, such as a 4x4 block using the position at (2, 2) from the upper left corner, and an 8x8 block using the position at (4, 4) from the upper left corner, according to the affine model used in current standards including VVC and AVS3, the motion vector deviations at the same position of each sub-block are the same. However, AVS has the same deviations in the upper left corner, upper right corner, and lower left corner with 3 control points (such as in the above AVS3 text). Figure 7 The positions A, B, and C shown in the figure are different from those used by other blocks, and accordingly, the motion vector deviations of the upper left corner, upper right corner, and lower left corner sub-blocks in the case of three control points are calculated differently from those of other blocks. Specific examples are shown in the embodiments.

[0523] Step 403: Determine a first prediction value of the sub-block based on the first motion vector, and a first motion vector deviation between each pixel position in the sub-block and the sub-block.

[0524] In an embodiment of the present application, after determining the first motion vector of each sub-block of the current block, the encoder can determine the first prediction value of the sub-block and the first motion vector deviation between each pixel position in the sub-block and the sub-block based on the first motion vector of the sub-block.

[0525] It can be understood that, in the embodiment of the present application, step 403 may specifically include:

[0526] Step 403a: Determine a first prediction value of the sub-block based on the first motion vector.

[0527] Step 403b: determine the first motion vector deviation between each pixel position in the sub-block and the sub-block based on the first motion vector.

[0528] Among them, the inter-frame prediction method proposed in the embodiment of the present application does not limit the order in which the encoder executes step 403a and step 403b. That is to say, in the present application, after determining the first motion vector of each sub-block of the current block, the encoder can first execute step 403a and then step 403b, or first execute step 403b and then step 403a, or execute step 403a and step 403b at the same time.

[0529] Further, in an embodiment of the present application, when the encoder determines the first prediction value of the sub-block based on the first motion vector, it can first determine the sample matrix; wherein the sample matrix includes a luminance sample matrix and a chrominance sample matrix; and then the first prediction value can be determined based on the prediction reference mode, the sub-block size parameter, the sample matrix, and the motion vector set.

[0530] It should be noted that, in an embodiment of the present application, when the encoder determines the first prediction value based on the prediction reference mode, sub-block size parameters, sample matrix and motion vector set, it can first determine the target motion vector from the motion vector set based on the prediction reference mode and sub-block size parameters; then, the prediction sample matrix can be determined using the reference image queue and reference index sample matrix corresponding to the prediction reference mode and the target motion vector; wherein the prediction sample matrix includes the first prediction values ​​of multiple sub-blocks.

[0531] Specifically, in an embodiment of the present application, the sample matrix may include a luma sample matrix and a chroma sample matrix, and accordingly, the prediction sample matrix determined by the encoder may include a luma prediction sample matrix and a chroma prediction sample matrix, wherein the luma prediction sample matrix includes first luma prediction values ​​of multiple sub-blocks, and the chroma prediction sample matrix includes first chroma prediction values ​​of multiple sub-blocks, and the first luma prediction value and the first chroma prediction value constitute the first prediction value of the sub-block.

[0532] It should be noted that, in the embodiment of the application, the brightness sample matrix in the sample matrix can be a brightness sample matrix with 1 / 16 precision, and the chrominance sample matrix in the sample matrix can be a chrominance sample matrix with 1 / 32 precision.

[0533] It can be understood that, in the embodiments of the present application, for different prediction reference modes, the reference image queues and reference indexes obtained by the encoder are different.

[0534] Furthermore, in an embodiment of the present application, when the encoder determines the sample matrix, it can first obtain the luminance interpolation filter coefficients and the chrominance interpolation filter coefficients; then the luminance sample matrix can be determined based on the luminance interpolation filter coefficients, and at the same time, the chrominance sample matrix can be determined based on the chrominance interpolation filter coefficients.

[0535] Further, in an embodiment of the present application, the encoder may determine a secondary prediction parameter when determining a first motion vector deviation between each pixel position in a sub-block and the sub-block; if the secondary prediction parameter indicates the use of secondary prediction, the encoder may determine the first motion vector deviation between the sub-block and each pixel position based on a difference variable.

[0536] It can be understood that in the embodiment of the present application, after the encoder determines the first motion vector deviation between the sub-block and each pixel position based on the difference variable, it can use all the first motion vector deviations corresponding to all the pixel positions in the sub-block to construct a motion vector deviation matrix corresponding to the sub-block. It can be seen that the motion vector deviation matrix includes the motion vector deviation between the sub-block and any pixel point inside, that is, the first motion vector deviation.

[0537] Further, in an embodiment of the present application, if the secondary prediction parameter determined by the encoder indicates that secondary prediction is not used, then the encoder may choose to directly use the first prediction value of the sub-block of the current block obtained in the above step 403a as the second prediction value of the sub-block, without performing the following steps 404 and 405.

[0538] Specifically, in an embodiment of the present application, if the secondary prediction parameter indicates that secondary prediction is not used, the encoder can determine the second prediction value using the prediction sample matrix. Wherein the prediction sample matrix includes the first prediction values ​​of multiple sub-blocks, and the encoder can determine the first prediction value of the sub-block where the pixel position is located as its own second prediction value.

[0539] Exemplarily, in the present application, if the prediction reference mode of the current block takes a value of 0 or 1, that is, the first reference mode 'PRED_List0' is used, or the second reference mode 'PRED_List1' is used, then the first prediction value of the sub-block where the pixel position is located can be directly selected from the prediction sample matrix including 1 luminance prediction sample matrix (2 chrominance prediction sample matrices), and the first prediction value is determined as the inter-frame prediction value of the pixel position, that is, the second prediction value.

[0540] Exemplarily, in the present application, if the prediction reference mode value of the current block is 2, that is, the third reference mode 'PRED_List01' is used, then the two luminance prediction sample matrices (2 groups of 4 chrominance prediction sample matrices) included in the prediction sample matrix can be averaged to obtain one averaged luminance prediction sample (2 averaged chrominance prediction samples), and finally, the first prediction value of the sub-block where the pixel position is located is selected from the averaged luminance prediction sample (2 averaged chrominance prediction samples), and the first prediction value is determined as the inter-frame prediction value of the pixel position, that is, the second prediction value.

[0541] Step 404: Determine the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation.

[0542] In an embodiment of the present application, after the encoder determines the first prediction value of the sub-block and the first motion vector deviation between each pixel position in the sub-block and the sub-block based on the first motion vector, the center position and the second motion vector deviation corresponding to each pixel position can be determined according to the first motion vector deviation.

[0543] It should be noted that, in an embodiment of the present application, for each pixel position in a sub-block, when performing secondary prediction or PROF processing, if the default center position used by the filter has a large deviation from the motion vector between the sub-block, then the prediction result obtained may have a large error. Therefore, after determining the first motion vector deviation between each pixel position and the sub-block, the encoder can first confirm whether each pixel position is suitable as the center position of the filter used for secondary prediction or PROF processing of a certain pixel. If not, the center position can be readjusted.

[0544] It can be understood that, in the present application, the center position and the second motion vector deviation corresponding to a pixel position may be the center position and the motion vector deviation of the filter used when performing secondary prediction or PROF processing on the pixel position. That is, the center position and the second motion vector deviation are used for secondary prediction or PROF processing on the pixel position.

[0545] Further, in an embodiment of the present application, a method for an encoder to determine a center position and a second motion vector deviation corresponding to each pixel position according to the first motion vector deviation may include the following steps:

[0546] Step 404a, determining a horizontal deviation and a vertical deviation of the first motion vector deviation;

[0547] Step 404b: determine the center position and the second motion vector deviation according to the first absolute value of the horizontal deviation, the second absolute value of the vertical deviation and a preset deviation threshold.

[0548] In an embodiment of the present application, the encoder may first determine the deviation values ​​of the first motion vector deviation in different directions, that is, determine the horizontal deviation and vertical deviation corresponding to the first motion vector deviation. Then, the encoder may further calculate the absolute value of the horizontal deviation, that is, the first absolute value, and the absolute value of the vertical deviation, that is, the second absolute value. Finally, the encoder may further determine the center position and the second motion vector deviation used in secondary prediction or PROF processing based on the first absolute value, the second absolute value, and the preset deviation threshold.

[0549] It should be noted that, in the embodiment of the application, the encoder may compare the first absolute value of the horizontal deviation and the second absolute value of the vertical deviation with the preset deviation thresholds, respectively, so as to determine the center position and the second motion vector deviation according to the comparison results. The preset deviation threshold may be a pre-set value used to determine whether to adjust the center position and the motion vector deviation.

[0550] Further, in an embodiment of the present application, when the encoder determines the center position and the second motion vector deviation based on the horizontal deviation, the vertical deviation and the preset deviation threshold, if the first absolute value and the second absolute value are both smaller than the preset deviation threshold, then the first motion vector deviation can be determined as the second motion vector deviation, and each pixel position can be determined as the center position.

[0551] It can be understood that in the embodiment of the application, after the encoder compares the first absolute value of the horizontal deviation and the second absolute value of the vertical deviation with the preset deviation threshold values ​​respectively, if the first absolute value and the second absolute value are both smaller than the preset deviations, it can be considered that the vector deviation between the pixel position and the sub-block is small, and the prediction result obtained by using the pixel position as the center position of the filter for secondary prediction or PROF processing is relatively accurate. Therefore, the first motion vector deviation corresponding to the pixel position can be directly determined as the second motion vector deviation for secondary prediction or PROF processing, and at the same time, the pixel position can be directly determined as the center position used by the filter for secondary prediction or PROF processing.

[0552] Further, in an embodiment of the present application, when the encoder determines the center position and the second motion vector deviation based on the horizontal deviation, the vertical deviation and the preset deviation threshold, if the first absolute value is greater than or equal to the preset deviation threshold, and the second absolute value is less than the preset deviation threshold, then the first adjustment direction can be determined based on the horizontal deviation; and then the center position and the second motion vector deviation can be determined based on the first adjustment direction.

[0553] It should be noted that, in the embodiment of the present application, the first adjustment direction is used to adjust the first motion vector in the horizontal direction, and therefore, the first adjustment direction includes the left side and the right side.

[0554] It can be understood that in the embodiment of the application, after the encoder compares the first absolute value of the horizontal deviation and the second absolute value of the vertical deviation with the preset deviation threshold respectively, if the first absolute value is greater than or equal to the preset deviation threshold, and the second absolute value is less than the preset deviation threshold, it can be considered that the motion vector deviation between the pixel position and the sub-block in the horizontal direction is large. If the pixel is used as the center position of the filter for secondary prediction or PROF processing, the prediction result obtained may have errors. Therefore, it is necessary to further determine the first adjustment direction according to the horizontal deviation, and then adjust according to the first adjustment direction. Finally, the center position and the second motion vector deviation used for secondary prediction or PROF processing can be determined.

[0555] Further, in an embodiment of the present application, when the encoder determines the center position and the second motion vector deviation based on the first adjustment direction, if the first adjustment direction is the left side, then the adjacent left pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be added with (1, 0) to obtain the corresponding second motion vector deviation.

[0556] Further, in an embodiment of the present application, the encoder determines the center position and the second motion vector deviation based on the first adjustment direction. If the first adjustment direction is the right side, the adjacent right pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be subtracted from (1, 0) to obtain the second motion vector deviation.

[0557] Further, in an embodiment of the present application, when the encoder determines the center position and the second motion vector deviation based on the horizontal deviation, the vertical deviation and the preset deviation threshold, if the second absolute value is greater than or equal to the preset deviation threshold, and the first absolute value is less than the preset deviation threshold, then the second adjustment direction can be determined based on the vertical deviation; and then the center position and the second motion vector deviation can be determined based on the second adjustment direction.

[0558] It should be noted that, in the embodiment of the present application, the second adjustment direction is used to adjust the first motion vector in the vertical direction, and therefore, the second adjustment direction includes an upper side and a lower side.

[0559] It can be understood that in the embodiment of the application, after the encoder compares the first absolute value of the horizontal deviation and the second absolute value of the vertical deviation with the preset deviation threshold respectively, if the second absolute value is greater than or equal to the preset deviation threshold, and the first absolute value is less than the preset deviation threshold, it can be considered that the motion vector deviation between the pixel position and the sub-block in the vertical direction is large. If the pixel is used as the center position of the filter for secondary prediction or PROF processing, the prediction result obtained may have errors. Therefore, it is necessary to further determine the second adjustment direction according to the vertical deviation, and then adjust according to the second adjustment direction. Finally, the center position and the second motion vector deviation used for secondary prediction or PROF processing can be determined.

[0560] Further, in an embodiment of the present application, when the encoder determines the center position and the second motion vector deviation according to the second adjustment direction, if the second adjustment direction is upward, then the adjacent upper pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be added with (0, 1) to obtain the corresponding second motion vector deviation.

[0561] Further, in an embodiment of the present application, the encoder determines the center position and the second motion vector deviation based on the second adjustment direction. If the second adjustment direction is the lower side, the adjacent lower pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be subtracted by (0, 1) to obtain the second motion vector deviation.

[0562] Further, in an embodiment of the present application, when the encoder determines the center position and the second motion vector deviation based on the horizontal deviation, the vertical deviation and the preset deviation threshold, if the first absolute value and the second absolute value are both greater than or equal to the preset deviation threshold, then the third adjustment direction can be determined based on the horizontal deviation and the vertical deviation; then the center position and the second motion vector deviation can be determined based on the third adjustment direction.

[0563] It should be noted that, in the embodiment of the present application, the third adjustment direction is used to adjust the first motion vector in the horizontal direction and the vertical direction at the same time. Therefore, the third adjustment direction includes the upper right side, the lower right side, the upper left side and the lower left side.

[0564] It can be understood that in the embodiment of the application, after the encoder compares the first absolute value of the horizontal deviation and the second absolute value of the vertical deviation with the preset deviation threshold respectively, if the first absolute value and the second absolute value are both greater than or equal to the preset deviation threshold, it can be considered that the motion vector deviations between the pixel position and the sub-block in the horizontal and vertical directions are large. If the pixel is used as the center position of the filter for secondary prediction or PROF processing, the prediction result obtained may have errors. Therefore, it is necessary to further determine the third adjustment direction according to the horizontal deviation and the vertical deviation, and then adjust according to the third adjustment direction. Finally, the center position and the second motion vector deviation used for secondary prediction or PROF processing can be determined.

[0565] Further, in an embodiment of the present application, when the encoder determines the center position and the second motion vector deviation according to the third adjustment direction, if the third adjustment direction is the upper left side, then the adjacent upper left pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be added with (1, 1) to obtain the corresponding second motion vector deviation.

[0566] Further, in an embodiment of the present application, the encoder determines the center position and the second motion vector deviation based on the third adjustment direction. If the third adjustment direction is the lower right side, then the adjacent lower right pixel position of any pixel position can be used as the center position, and then the first motion vector deviation can be subtracted from (1, 1) to obtain the second motion vector deviation.

[0567] It can be understood that, in the embodiment of the present application, the deviation values ​​of the second motion vector deviation in the horizontal direction and the vertical direction are both smaller than the preset deviation threshold.

[0568] That is to say, in the present application, after determining the deviation between the motion vector of each pixel position in the sub-block and the motion vector of the sub-block, and before filtering the predicted block based on the prediction of the sub-block, the encoder can first determine the center position and the second motion vector deviation, wherein the center position and the second motion vector are used for secondary prediction or PROF processing.

[0569] Specifically, in the present application, for each pixel position in the sub-block, the size of the first motion vector deviation between the pixel position and the sub-block can be determined first, including the horizontal deviation and vertical deviation of the first motion vector deviation in the horizontal direction and the vertical direction. If the absolute value of the motion vector deviation in the horizontal direction and / or the vertical direction of a certain pixel position is greater than a preset deviation threshold, then it can be determined that the center position of the filter corresponding to the pixel position is no longer the pixel position itself, and the center position and the second motion vector deviation need to be further determined.

[0570] Further, in an embodiment of the present application, before performing secondary prediction or PROF processing on a pixel position in a sub-block, if it is determined that the absolute value of the deviation component of the first motion vector deviation between the pixel position and the sub-block in the horizontal direction and / or vertical direction is greater than or equal to a preset deviation threshold (such as 1 / 2 pixel, 3 / 4 pixel or 1 pixel), then it is necessary to adjust the starting point used for the secondary prediction or PROF processing, that is, the center position of the two-dimensional filter used for the secondary prediction or PROF processing is adjusted.

[0571] It can be understood that in the present application, when adjusting the center position and the motion vector deviation, another pixel position whose motion vector deviation from the pixel position in the horizontal direction and the vertical direction is less than or equal to the preset deviation threshold can be selected in the current block.

[0572] In an embodiment of the present application, further, before determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, that is, before step 404, the method for the encoder to perform inter-frame prediction may further include the following steps:

[0573] Step 406: Limit the first motion vector deviation according to a preset deviation range; wherein the preset deviation range includes a lower deviation limit value and an upper deviation limit value.

[0574] In an embodiment of the present application, before the encoder determines the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, the encoder may first limit the first motion vector. Specifically, the encoder may limit the first motion vector deviation according to a preset deviation range; wherein the preset deviation range includes a lower deviation limit value and an upper deviation limit value.

[0575] Further, in an embodiment of the present application, when the encoder limits the first motion vector deviation according to a preset deviation range, if the horizontal deviation and / or vertical deviation is less than the lower limit value of the deviation, then the horizontal deviation and / or vertical deviation can be set to the lower limit value of the deviation; if the horizontal deviation and / or vertical deviation is greater than the upper limit value of the deviation, then the horizontal deviation and / or vertical deviation can be set to the upper limit value of the deviation.

[0576] It should be noted that, in the present application, the encoder may not limit the first motion vector deviation, that is, not perform step 407, but directly adjust the center position to any suitable pixel position according to the first motion vector deviation. It may also first perform the limiting process of the first motion vector deviation according to step 407, or limit the adjustable range and motion vector deviation.

[0577] Further, in an embodiment of the present application, the preset deviation range can be composed of a deviation lower limit value min and a deviation upper limit value max, that is, the preset deviation range can be expressed as (min, max). When the first motion vector deviation is restricted, if the deviation value of the first motion vector deviation in the horizontal direction and / or vertical direction is less than the deviation lower limit value min, then the deviation value can be directly set to min; if the deviation value of the first motion vector deviation in the horizontal direction and / or vertical direction is greater than max, then the deviation value can be directly set to max.

[0578] In an embodiment of the present application, further, after determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, that is, after step 404, the method for the encoder to perform inter-frame prediction may further include the following steps:

[0579] Step 407: If the center position does not belong to the current block, redetermine the center position according to the pixel positions in the current block.

[0580] In an embodiment of the present application, after the encoder determines the center position and the second motion vector deviation corresponding to each pixel position based on the first motion vector deviation, it can first determine whether the center position exceeds the current block. If the center position exceeds the range of the current block, that is, the center position does not belong to the current block, then the encoder needs to redetermine the center position according to the pixel position in the current block.

[0581] In an embodiment of the present application, further, after determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, that is, after step 404, the method for the encoder to perform inter-frame prediction may further include the following steps:

[0582] Step 408: If the center position does not belong to the current block, directly determine each pixel position as the center position corresponding to each pixel position.

[0583] In an embodiment of the present application, after the encoder determines the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, it can first determine whether the center position exceeds the current block. If the center position exceeds the range of the current block, that is, the center position does not belong to the current block, the encoder can directly determine the corresponding, original pixel position as the center position. That is, if the center position determined based on the first motion vector deviation corresponding to each pixel position in the sub-block exceeds the range of the current block, the encoder can choose to still use each pixel position as the center position.

[0584] That is to say, in an embodiment of the present application, for the starting position of the adjusted secondary prediction, that is, the center position used by the filter determined based on the first motion vector deviation, the encoder can limit the center position to not exceed the range of the current block, that is, it will not exceed the range of the starting position of the secondary prediction of all pixel positions in the current block before adjustment.

[0585] Step 405: Based on the first prediction value, perform secondary prediction or PROF processing according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and determine the second prediction value as the inter-frame prediction value of the sub-block.

[0586] In an embodiment of the present application, after the encoder determines the center position and the second motion vector deviation corresponding to each pixel position based on the first motion vector deviation, it can perform secondary prediction or PROF processing based on the first prediction value according to the center position and the second motion vector deviation, so as to obtain the predicted value of the pixel at each pixel position.

[0587] Furthermore, in an embodiment of the present application, after the encoder traverses each pixel position in the sub-block and obtains the prediction value of the pixel at each pixel position, it can determine the second prediction value of the sub-block based on the prediction value of the pixel at each pixel position, thereby determining the second prediction value as the inter-frame prediction value of the sub-block.

[0588] It should be noted that in the embodiments of the present application, since the center position and the second motion vector deviation can be used to perform secondary prediction or PROF processing on the pixel points at each pixel position in the sub-block, after obtaining the center position and the second motion vector deviation, the encoder can perform secondary prediction or PROF processing on the pixel points at each pixel position based on the first prediction value using the center position and the second motion vector deviation, and finally obtain the second prediction value corresponding to the sub-block, so that the second prediction value can be determined as the inter-frame prediction value of the sub-block.

[0589] Further, in an embodiment of the present application, the encoder performs secondary prediction or PROF processing based on the first prediction value according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and the method for determining the second prediction value as the inter-frame prediction value of the sub-block may include the following steps:

[0590] Step 405a, determining PROF parameters;

[0591] Step 405b: when the PROF parameter indicates to perform PROF processing, determine the pixel horizontal gradient and the pixel vertical gradient corresponding to the center position based on the first prediction value;

[0592] Step 405c, calculating the deviation value corresponding to each pixel position according to the pixel horizontal gradient, the pixel vertical gradient and the second motion vector deviation;

[0593] Step 405d, obtaining a predicted value of a pixel at each pixel position based on the first predicted value and the deviation value;

[0594] Step 405e: Determine a second prediction value using the prediction value of the pixel at each pixel position.

[0595] In an embodiment of the present application, the encoder may first determine the PROF parameters. If the PROF parameters indicate that PROF processing is to be performed, the encoder may determine the pixel horizontal gradient and the pixel vertical gradient corresponding to the center position based on the first prediction value; wherein the pixel horizontal gradient is the gradient value between the pixel value corresponding to the center position and the pixel value corresponding to the adjacent pixel position in the horizontal direction; the pixel vertical gradient is the gradient value between the pixel value corresponding to the center position and the pixel value corresponding to the adjacent pixel position in the vertical direction.

[0596] Further, in an embodiment of the present application, the encoder can calculate the deviation value corresponding to each pixel position according to the pixel horizontal gradient, pixel vertical gradient and second motion vector deviation of the center position corresponding to each pixel position, wherein the deviation value can be used to correct the predicted value of the pixel value of each pixel position.

[0597] It should be noted that, in an embodiment of the present application, the encoder can further obtain a corrected prediction value corresponding to any pixel position based on the first prediction value and the deviation value, and after traversing each pixel position in the current sub-block and obtaining the corrected prediction value corresponding to each pixel position, the corrected prediction value corresponding to all pixel positions is used to determine the second prediction value corresponding to the current sub-block, thereby determining the corresponding inter-frame prediction value. Specifically, in the present application, after completing the prediction based on the sub-block, the first prediction value of the current sub-block is used as the prediction value of each pixel position, and then, the first prediction value is added to the deviation value corresponding to each pixel position, that is, the correction processing of the prediction value of each pixel position can be completed to obtain the corrected prediction value, so that the second prediction value of the current sub-block can be further obtained, and the second prediction value is used as the inter-frame prediction value corresponding to the current sub-block.

[0598] Further, in an embodiment of the present application, the encoder performs secondary prediction or PROF processing based on the first prediction value according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and the method for determining the second prediction value as the inter-frame prediction value of the sub-block may include the following steps:

[0599] Step 405f, determining secondary prediction parameters;

[0600] Step 405g: when the secondary prediction parameter indicates to use secondary prediction, determine the filter coefficient of the two-dimensional filter according to the second motion vector deviation; wherein the two-dimensional filter is used to perform secondary prediction processing according to a preset shape;

[0601] Step 405h: determining a predicted value of a pixel at each pixel position based on the filter coefficient and the first predicted value;

[0602] Step 405i: Determine a second prediction value using the prediction value of the pixel at each pixel position.

[0603] In an embodiment of the present application, the encoder may first determine a secondary prediction parameter. If the secondary prediction parameter indicates the use of secondary prediction, the encoder may determine the filter coefficient of the two-dimensional filter based on the second motion vector deviation; wherein the two-dimensional filter is used to perform secondary prediction processing according to a preset shape.

[0604] It should be noted that, in the embodiment of the present application, the filter coefficient of the two-dimensional filter is related to the second motion vector deviation corresponding to the target pixel position. In other words, for different target pixel positions, if the corresponding second motion vector deviation is different, the filter coefficient of the two-dimensional filter used is also different.

[0605] It is understandable that in the embodiment of the present application, the two-dimensional filter is used to perform secondary prediction using a plurality of adjacent pixel positions forming a preset shape, wherein the preset shape is a rectangle, a diamond or any symmetrical shape.

[0606] That is to say, in the present application, the two-dimensional filter used for secondary prediction is a filter composed of adjacent points constituting a preset shape. The adjacent points constituting the preset shape may include multiple points, for example, 9 points. The preset shape may be a symmetrical shape, for example, the preset shape may include a rectangle, a rhombus, or any other symmetrical shape.

[0607] Exemplarily, in the present application, the two-dimensional filter is a rectangular filter, specifically, the two-dimensional filter is a filter composed of 9 adjacent pixel positions forming a rectangle. Among the 9 pixel positions, the pixel position at the center is the pixel position of the pixel that currently needs secondary prediction, that is, the current pixel position.

[0608] Furthermore, in an embodiment of the present application, when the encoder determines the filter coefficient of the two-dimensional filter based on the second motion vector deviation, it can first determine the scale parameter, and then determine the filter coefficient corresponding to the pixel position based on the scale parameter and the second motion vector deviation.

[0609] It should be noted that, in an embodiment of the present application, the scale parameter may include at least one scale value, and the second motion vector deviation includes a horizontal deviation and a vertical deviation; wherein at least one scale value is a non-zero real number.

[0610] Specifically, in the present application, when the two-dimensional filter uses 9 adjacent pixel positions forming a rectangle for secondary prediction, the pixel position located at the center of the rectangle is the position to be predicted, that is, the current pixel position, and the other 8 target pixel positions are located in the 8 directions of upper left, upper, upper right, right, lower right, lower, lower left, and left of the current pixel position, respectively.

[0611] Accordingly, in the present application, the encoder may calculate 9 filter coefficients corresponding to 9 adjacent pixel positions according to a preset calculation rule based on at least one proportional value and a second motion vector deviation of the position to be predicted.

[0612] It should be noted that, in the present application, the preset calculation rules may include a variety of different calculation methods, such as addition, subtraction, multiplication, etc. Among them, for different pixel positions, different calculation methods may be used to calculate the filter coefficients.

[0613] It can be understood that in the present application, when the encoder calculates multiple filter coefficients corresponding to multiple pixel positions according to different calculation methods in the preset calculation rules, some filter coefficients can be a linear function of the second motion vector deviation, that is, the two are in a linear relationship, or can be a quadratic function or a higher-order function of the second motion vector deviation, that is, the two are in a non-linear relationship.

[0614] That is to say, in the present application, any one of the multiple filter coefficients corresponding to multiple adjacent pixel positions can be a linear function, a quadratic function, or a higher-order function of the second motion vector deviation.

[0615] It is understandable that in the present application, the encoder may write prediction mode parameters, affine mode parameters, and prediction reference mode into the bitstream, and may also write PROF parameters and secondary prediction parameters into the bitstream.

[0616] This embodiment provides an inter-frame prediction method, which can, after prediction based on sub-blocks, re-determine the center position and the second motion vector deviation used for secondary prediction or PROF processing based on the first motion vector deviation for the pixel position where the first motion vector deviation between the motion vector and the motion vector of the sub-block is large, so that the center position and the second motion vector deviation can be used to perform point-based secondary prediction on the basis of the first prediction value based on the sub-block to obtain the second prediction value. It can be seen that the inter-frame prediction method proposed in this application can be well applied to all scenarios, can reduce the prediction error, greatly improve the encoding performance, and thus improve the encoding efficiency.

[0617] Based on the above embodiment, in yet another embodiment of the present application, Fig. 20 The structure of the decoder is shown in Figure 2. Figure 1 ,like Fig. 20 As shown, the decoder 300 proposed in the embodiment of the present application may include a parsing unit 301 and a first determining unit 302;

[0618] The parsing unit 301 is used to parse the bitstream and obtain the prediction mode parameters of the current block;

[0619] The first determination unit 302 is used to determine the first motion vector of a sub-block of the current block when the prediction mode parameter indicates that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode; wherein the current block includes multiple sub-blocks; determine the first prediction value of the sub-block based on the first motion vector, and the first motion vector deviation between each pixel position in the sub-block and the sub-block; determine the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation; based on the first prediction value, perform secondary prediction or PROF processing according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and determine the second prediction value as the inter-frame prediction value of the sub-block.

[0620] Fig.21 The structure of the decoder is shown in Figure 2. Figure 2 ,like Fig.21 As shown, the decoder 300 proposed in the embodiment of the present application may also include a first processor 303, a first memory 304 storing executable instructions of the first processor 303, a first communication interface 305, and a first bus 306 for connecting the first processor 303, the first memory 304 and the first communication interface 305.

[0621] Further, in an embodiment of the present application, the above-mentioned first processor 303 is used to parse the code stream and obtain prediction mode parameters of the current block; when the prediction mode parameters indicate the use of the inter-frame prediction mode to determine the inter-frame prediction value of the current block, determine the first motion vector of the sub-block of the current block; wherein the current block includes multiple sub-blocks; based on the first motion vector, determine the first prediction value of the sub-block, and the first motion vector deviation between each pixel position in the sub-block and the sub-block; determine the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation; based on the first prediction value, perform secondary prediction or PROF processing according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and determine the second prediction value as the inter-frame prediction value of the sub-block.

[0622] Fig. 22 The structure of the encoder is shown in Figure 2. Figure 1 ,like Fig. 22 As shown, the encoder 400 proposed in the embodiment of the present application may include a second determining unit 401;

[0623] The second determination unit 401 is used to determine the prediction mode parameters of the current block; when the prediction mode parameters indicate that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode, determine the first motion vector of the sub-block of the current block; wherein the current block includes multiple sub-blocks; determine the first prediction value of the sub-block based on the first motion vector, and the first motion vector deviation between each pixel position in the sub-block and the sub-block; determine the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation; based on the first prediction value, perform secondary prediction or PROF processing according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and determine the second prediction value as the inter-frame prediction value of the sub-block.

[0624] Fig.23 The structure of the encoder is shown in Figure 2. Figure 2 ,like Fig.23 As shown, the encoder 400 proposed in the embodiment of the present application may also include a second processor 402, a second memory 403 storing executable instructions of the second processor 402, a second communication interface 404, and a second bus 405 for connecting the second processor 402, the second memory 403 and the second communication interface 404.

[0625] Further, in an embodiment of the present application, the above-mentioned second processor 402 is used to determine a prediction mode parameter of the current block; when the prediction mode parameter indicates that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode, determine the first motion vector of the sub-block of the current block; wherein the current block includes multiple sub-blocks; based on the first motion vector, determine the first prediction value of the sub-block, and the first motion vector deviation between each pixel position in the sub-block and the sub-block; determine the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation; based on the first prediction value, perform secondary prediction or PROF processing according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and determine the second prediction value as the inter-frame prediction value of the sub-block.

[0626] The embodiment of the present application provides a decoder and an encoder, which can, after prediction based on sub-blocks, re-determine the center position and the second motion vector deviation used for secondary prediction or PROF processing for the pixel position where the first motion vector deviation between the motion vector and the motion vector of the sub-block is large, based on the first motion vector deviation, so that the center position and the second motion vector deviation can be used to perform point-based secondary prediction on the basis of the first prediction value based on the sub-block to obtain the second prediction value. It can be seen that the inter-frame prediction method proposed in the present application can be well applied to all scenarios, can reduce the prediction error, greatly improve the encoding performance, and thus improve the encoding efficiency.

[0627] The embodiments of the present application provide a computer-readable storage medium and a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the method described in the above embodiments is implemented.

[0628] Specifically, the program instructions corresponding to an inter-frame prediction method in this embodiment may be stored in a storage medium such as an optical disk, a hard disk, or a USB flash drive. When the program instructions corresponding to an inter-frame prediction method in the storage medium are read or executed by an electronic device, the following steps are included:

[0629] Parse the bitstream and obtain the prediction mode parameters of the current block;

[0630] When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a sub-block of the current block; wherein the current block includes a plurality of sub-blocks;

[0631] determining a first prediction value of the sub-block based on the first motion vector and a first motion vector deviation between each pixel position in the sub-block and the sub-block;

[0632] Determine a center position and a second motion vector deviation corresponding to each pixel position according to the first motion vector deviation;

[0633] Based on the first prediction value, secondary prediction or PROF processing is performed according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and the second prediction value is determined as the inter-frame prediction value of the sub-block.

[0634] Specifically, the program instructions corresponding to an inter-frame prediction method in this embodiment may be stored in a storage medium such as an optical disk, a hard disk, or a USB flash drive. When the program instructions corresponding to an inter-frame prediction method in the storage medium are read or executed by an electronic device, the following steps are included:

[0635] Determining prediction mode parameters for the current block;

[0636] When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a sub-block of the current block; wherein the current block includes a plurality of sub-blocks;

[0637] determining a first prediction value of the sub-block based on the first motion vector and a first motion vector deviation between each pixel position in the sub-block and the sub-block;

[0638] Determine a center position and a second motion vector deviation corresponding to each pixel position according to the first motion vector deviation;

[0639] Based on the first prediction value, secondary prediction or PROF processing is performed according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and the second prediction value is determined as the inter-frame prediction value of the sub-block.

[0640] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0641] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0642] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0643] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process described in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0644] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0645] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0646] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0647] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An inter-frame prediction method, characterized in that: Applied to a decoder, the method comprises: Parse the bitstream and obtain the prediction mode parameters of the current block; When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a sub-block of the current block; wherein the current block includes a plurality of sub-blocks; determining a first prediction value of the sub-block based on the first motion vector and a first motion vector deviation between each pixel position in the sub-block and the sub-block; Determine a horizontal deviation and a vertical deviation of the first motion vector deviation; and determine a center position corresponding to each pixel position and a second motion vector deviation according to a first absolute value of the horizontal deviation, a second absolute value of the vertical deviation, and a preset deviation threshold; Based on the first prediction value, secondary prediction is performed according to the center position and the second motion vector deviation or prediction correction PROF processing based on the optical flow principle is used to determine the second prediction value of the sub-block, and the second prediction value is determined as the inter-frame prediction value of the sub-block.

2. The method according to claim 1, characterized in that The determining a first motion vector of a sub-block of the current block comprises: Parsing the bitstream to obtain affine mode parameters and prediction reference mode of the current block; When the affine mode parameter indicates to use the affine mode, determining a control point mode and a sub-block size parameter; wherein the control point mode is used to determine the number of control points; The first motion vector is determined according to the prediction reference mode, the control point mode and the sub-block size parameter.

3. The method according to claim 2, characterized in that The determining the first motion vector according to the prediction reference mode, the control point mode and the sub-block size parameter includes: Determine a control point motion vector group according to the prediction reference mode; The first motion vector is determined according to the control point motion vector group, the control point mode and the sub-block size parameter.

4. The method according to claim 3, characterized in that The determining the first motion vector according to the control point motion vector group, the control point mode and the sub-block size parameter comprises: Based on the size parameter of the current block and the control point mode, performing subtraction on the motion vectors in the control point motion vector group to determine a difference variable; Determining a sub-block position based on the prediction mode parameter and the sub-block size parameter; The first motion vector of the current sub-block is determined by using the difference variable and the sub-block position.

5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: Each sub-block of the current block is traversed, and a motion vector set is constructed according to the first motion vector of each sub-block.

6. The method according to claim 5, characterized in that The determining the first motion vector according to the control point motion vector group, the control point mode and the sub-block size parameter comprises: Based on the size parameter of the current block and the control point mode, performing subtraction on the motion vectors in the control point motion vector group to determine a difference variable; Determining a sub-block position based on the prediction mode parameter and the sub-block size parameter; The first motion vector of the sub-block is determined using the difference variable and the sub-block position.

7. The method according to claim 5, characterized in that The determining a first prediction value of the sub-block based on the first motion vector comprises: Determine a sample matrix; wherein the sample matrix includes a brightness sample matrix and a chrominance sample matrix; The first prediction value is determined according to the prediction reference mode, the sub-block size parameter, the sample matrix and the motion vector set.

8. The method according to claim 1, characterized in that The determining the center position and the second motion vector deviation according to the horizontal deviation, the vertical deviation and a preset deviation threshold comprises: If the first absolute value and the second absolute value are both smaller than the preset deviation threshold, the first motion vector deviation is determined as the second motion vector deviation, and each pixel position is determined as the center position.

9. The method according to claim 1, characterized in that: The determining the center position and the second motion vector deviation according to the horizontal deviation, the vertical deviation and a preset deviation threshold comprises: If the first absolute value is greater than or equal to the preset deviation threshold, and the second absolute value is less than the preset deviation threshold, determining a first adjustment direction according to the horizontal deviation; wherein the first adjustment direction includes left and right sides; The center position and the second motion vector deviation are determined according to the first adjustment direction.

10. The method according to claim 9, characterized in that The determining the center position and the second motion vector deviation according to the first adjustment direction includes: If the first adjustment direction is the left side, the adjacent left pixel position of any pixel position is used as the center position; Add (1, 0) to the first motion vector deviation to obtain the second motion vector deviation.

11. The method according to claim 9, characterized in that The determining the center position and the second motion vector deviation according to the first adjustment direction includes: If the first adjustment direction is right, the adjacent right pixel position of any pixel position is used as the center position; Subtract (1, 0) from the first motion vector deviation to obtain the second motion vector deviation.

12. The method according to claim 1, characterized in that The determining the center position and the second motion vector deviation according to the horizontal deviation, the vertical deviation and a preset deviation threshold comprises: If the second absolute value is greater than or equal to the preset deviation threshold, and the first absolute value is less than the preset deviation threshold, determining a second adjustment direction according to the vertical deviation; wherein the second adjustment direction includes an upper side and a lower side; According to the second adjustment direction, the center position and the second motion vector deviation are determined.

13. The method according to claim 12, characterized in that The determining the center position and the second motion vector deviation according to the second adjustment direction includes: If the second adjustment direction is upward, the upper pixel position adjacent to any pixel position is used as the center position; Add (0, 1) to the first motion vector deviation to obtain the second motion vector deviation.

14. The method according to claim 12, characterized in that The determining the center position and the second motion vector deviation according to the second adjustment direction includes: If the second adjustment direction is downward, the adjacent downward pixel position of any pixel position is used as the center position; Subtract (0, 1) from the first motion vector deviation to obtain the second motion vector deviation.

15. The method according to claim 1, characterized in that The determining the center position and the second motion vector deviation according to the horizontal deviation, the vertical deviation and a preset deviation threshold comprises: If the first absolute value and the second absolute value are both greater than or equal to the preset deviation threshold, determining a third adjustment direction according to the horizontal deviation and the vertical deviation; wherein the third adjustment direction includes the upper right side, the lower right side, the upper left side and the lower left side; The center position and the second motion vector deviation are determined according to the third adjustment direction.

16. The method according to any one of claims 1, 8 to 15, characterized in that: The preset deviation threshold is k pixels; wherein k is greater than 0.5 and less than or equal to 1.

17. The method according to claim 1, characterized in that Before determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, the method further includes: The first motion vector deviation is limited according to a preset deviation range; wherein the preset deviation range includes a lower deviation limit value and an upper deviation limit value.

18. The method according to claim 17, characterized in that The limiting processing of the first motion vector deviation according to a preset deviation range includes: If the horizontal deviation and / or the vertical deviation is less than the deviation lower limit value, the horizontal deviation and / or the vertical deviation is set to the deviation lower limit value; If the horizontal deviation and / or the vertical deviation is greater than the upper limit of the deviation, the horizontal deviation and / or the vertical deviation is set to the upper limit of the deviation.

19. The method according to claim 1, characterized in that After determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, the method further includes: If the center position does not belong to the current block, the center position is re-determined according to the pixel positions in the current block.

20. The method according to claim 1, characterized in that After determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, the method further includes: If the center position does not belong to the current block, each pixel position is directly determined as the center position corresponding to each pixel position.

21. The method according to claim 1, characterized in that The method of performing secondary prediction or PROF processing based on the first prediction value according to the center position and the second motion vector deviation to determine a second prediction value of the sub-block, and determining the second prediction value as the inter-frame prediction value of the sub-block includes: Parse the code stream and obtain PROF parameters; When the PROF parameter indicates to perform PROF processing, determining a pixel horizontal gradient and a pixel vertical gradient corresponding to the center position based on the first prediction value; Calculating a deviation value corresponding to each pixel position according to the pixel horizontal gradient, the pixel vertical gradient and the second motion vector deviation; Based on the first prediction value and the deviation value, obtaining a prediction value of a pixel at each pixel position; The second prediction value is determined using the prediction value of the pixel at each pixel position.

22. The method according to claim 1, characterized in that The method of performing secondary prediction or PROF processing based on the first prediction value according to the center position and the second motion vector deviation to determine a second prediction value of the sub-block, and determining the second prediction value as the inter-frame prediction value of the sub-block includes: Parsing the bitstream to obtain secondary prediction parameters; When the secondary prediction parameter indicates the use of secondary prediction, determining a filter coefficient of a two-dimensional filter according to the second motion vector deviation; wherein the two-dimensional filter is used to perform secondary prediction processing according to a preset shape; Determining a predicted value of a pixel at each pixel position based on the filter coefficient and the first predicted value; The second prediction value is determined using the prediction value of the pixel at each pixel position.

23. The method according to claim 22, characterized in that The two-dimensional filter is used for performing secondary prediction using a plurality of adjacent pixel positions forming the preset shape.

24. The method according to claim 23, characterized in that The preset shape is a rectangle, a rhombus, or any symmetrical shape.

25. The method according to claim 24, characterized in that If the value of the affine mode parameter is 1, it indicates that the affine mode is used; If the value of the affine mode parameter is 0, or the affine mode parameter is not obtained through parsing, it indicates that the affine mode is not used.

26. The method according to claim 2, characterized in that The determining of the sub-block size parameter comprises: Parsing the code stream to obtain a sub-block size flag; If the value of the sub-block size flag is 1, the sub-block size parameter is determined to be 8×8; If the value of the sub-block size flag is 0, or the sub-block size flag is not obtained by parsing, the sub-block size parameter is determined to be 4×4.

27. The method according to claim 2, characterized in that The control point mode includes a 4-parameter mode and a 6-parameter mode.

28. An inter-frame prediction method, characterized in that: Applied to an encoder, the method comprises: Determining prediction mode parameters for the current block; When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a sub-block of the current block; wherein the current block includes a plurality of sub-blocks; determining a first prediction value of the sub-block based on the first motion vector and a first motion vector deviation between each pixel position in the sub-block and the sub-block; Determine a horizontal deviation and a vertical deviation of the first motion vector deviation; and determine a center position corresponding to each pixel position and a second motion vector deviation according to a first absolute value of the horizontal deviation, a second absolute value of the vertical deviation, and a preset deviation threshold; Based on the first prediction value, secondary prediction or PROF processing is performed according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and the second prediction value is determined as the inter-frame prediction value of the sub-block.

29. The method according to claim 28, characterized in that: The determining of the prediction mode parameters of the current block includes: Pre-encoding the current block using multiple prediction modes to obtain a rate-distortion cost value corresponding to each prediction mode; A minimum rate-distortion cost value is selected from the obtained multiple rate-distortion cost values, and a prediction mode parameter of the current block is determined according to the prediction mode corresponding to the minimum rate-distortion cost value.

30. The method according to claim 28, characterized in that The determining a first motion vector of a sub-block of the current block comprises: Determining affine mode parameters and a prediction reference mode of the current block; When the affine mode parameter indicates to use the affine mode, determining a control point mode and a sub-block size parameter; wherein the control point mode is used to determine the number of control points; The first motion vector is determined according to the prediction reference mode, the control point mode and the sub-block size parameter.

31. The method according to claim 30, characterized in that The determining the first motion vector according to the prediction reference mode, the control point mode and the sub-block size parameter includes: Determine a control point motion vector group according to the prediction reference mode; The first motion vector is determined according to the control point motion vector group, the control point mode and the sub-block size parameter.

32. The method according to claim 31, characterized in that The determining the first motion vector according to the control point motion vector group, the control point mode and the sub-block size parameter comprises: Based on the size parameter of the current block and the control point mode, performing subtraction on the motion vectors in the control point motion vector group to determine a difference variable; Determining a sub-block position based on the prediction mode parameter and the sub-block size parameter; The first motion vector of the current sub-block is determined by using the difference variable and the sub-block position.

33. The method according to any one of claims 30 to 32, characterized in that The method further comprises: Each sub-block of the current block is traversed, and a motion vector set is constructed according to the first motion vector of each sub-block.

34. The method according to claim 33, characterized in that The determining the first motion vector according to the control point motion vector group, the control point mode and the sub-block size parameter comprises: Based on the size parameter of the current block and the control point mode, performing subtraction on the motion vectors in the control point motion vector group to determine a difference variable; Determining a sub-block position based on the prediction mode parameter and the sub-block size parameter; The first motion vector of the sub-block is determined using the difference variable and the sub-block position.

35. The method according to claim 33, characterized in that The determining a first prediction value of the sub-block based on the first motion vector comprises: Determine a sample matrix; wherein the sample matrix includes a brightness sample matrix and a chrominance sample matrix; The first prediction value is determined according to the prediction reference mode, the sub-block size parameter, the sample matrix and the motion vector set.

36. The method according to claim 28, characterized in that The determining the center position and the second motion vector deviation according to the horizontal deviation, the vertical deviation and a preset deviation threshold comprises: If the first absolute value and the second absolute value are both smaller than the preset deviation threshold, the first motion vector deviation is determined as the second motion vector deviation, and each pixel position is determined as the center position.

37. The method according to claim 36, characterized in that The determining the center position and the second motion vector deviation according to the horizontal deviation, the vertical deviation and a preset deviation threshold comprises: If the first absolute value is greater than or equal to the preset deviation threshold, and the second absolute value is less than the preset deviation threshold, determining a first adjustment direction according to the horizontal deviation; wherein the first adjustment direction includes left and right sides; The center position and the second motion vector deviation are determined according to the first adjustment direction.

38. The method according to claim 37, characterized in that The determining the center position and the second motion vector deviation according to the first adjustment direction includes: If the first adjustment direction is the left side, the adjacent left pixel position of any pixel position is used as the center position; Add (1, 0) to the first motion vector deviation to obtain the second motion vector deviation.

39. The method according to claim 37, characterized in that The determining the center position and the second motion vector deviation according to the first adjustment direction includes: If the first adjustment direction is right, the adjacent right pixel position of any pixel position is used as the center position; Subtract (1, 0) from the first motion vector deviation to obtain the second motion vector deviation.

40. The method according to claim 28, characterized in that The determining the center position and the second motion vector deviation according to the horizontal deviation, the vertical deviation and a preset deviation threshold comprises: If the second absolute value is greater than or equal to the preset deviation threshold, and the first absolute value is less than the preset deviation threshold, determining a second adjustment direction according to the vertical deviation; wherein the second adjustment direction includes an upper side and a lower side; According to the second adjustment direction, the center position and the second motion vector deviation are determined.

41. The method according to claim 40, characterized in that The determining the center position and the second motion vector deviation according to the second adjustment direction includes: If the second adjustment direction is upward, the upper pixel position adjacent to any pixel position is used as the center position; Add (0, 1) to the first motion vector deviation to obtain the second motion vector deviation.

42. The method according to claim 40, characterized in that The determining the center position and the second motion vector deviation according to the second adjustment direction includes: If the second adjustment direction is downward, the adjacent downward pixel position of any pixel position is used as the center position; Subtract (0, 1) from the first motion vector deviation to obtain the second motion vector deviation.

43. The method according to claim 28, characterized in that The determining the center position and the second motion vector deviation according to the horizontal deviation, the vertical deviation and a preset deviation threshold comprises: If the first absolute value and the second absolute value are both greater than or equal to the preset deviation threshold, determining a third adjustment direction according to the horizontal deviation and the vertical deviation; wherein the third adjustment direction includes the upper right side, the lower right side, the upper left side and the lower left side; The center position and the second motion vector deviation are determined according to the third adjustment direction.

44. The method according to any one of claims 28, 37 to 43, characterized in that The preset deviation threshold is k pixels; wherein k is greater than 0.5 and less than or equal to 1.

45. The method according to claim 28, characterized in that Before determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, the method further includes: The first motion vector deviation is limited according to a preset deviation range; wherein the preset deviation range includes a lower deviation limit value and an upper deviation limit value.

46. ​​The method according to claim 45, characterized in that The limiting processing of the first motion vector deviation according to a preset deviation range includes: If the horizontal deviation and / or the vertical deviation is less than the deviation lower limit value, the horizontal deviation and / or the vertical deviation is set to the deviation lower limit value; If the horizontal deviation and / or the vertical deviation is greater than the upper limit of the deviation, the horizontal deviation and / or the vertical deviation is set to the upper limit of the deviation.

47. The method according to claim 28, characterized in that After determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, the method further includes: If the center position does not belong to the current block, the center position is re-determined according to the pixel positions in the current block.

48. The method according to claim 28, characterized in that After determining the center position and the second motion vector deviation corresponding to each pixel position according to the first motion vector deviation, the method further includes: If the center position does not belong to the current block, each pixel position is directly determined as the center position corresponding to each pixel position.

49. The method according to claim 28, characterized in that The method of performing secondary prediction or PROF processing based on the first prediction value according to the center position and the second motion vector deviation to determine a second prediction value of the sub-block, and determining the second prediction value as the inter-frame prediction value of the sub-block includes: Determine PROF parameters; When the PROF parameter indicates to perform PROF processing, determining a pixel horizontal gradient and a pixel vertical gradient corresponding to the center position based on the first prediction value; Calculating a deviation value corresponding to each pixel position according to the pixel horizontal gradient, the pixel vertical gradient and the second motion vector deviation; Based on the first prediction value and the deviation value, obtaining a prediction value of a pixel at each pixel position; The second prediction value is determined using the prediction value of the pixel at each pixel position.

50. The method of claim 28, wherein: The method of performing secondary prediction or PROF processing based on the first prediction value according to the center position and the second motion vector deviation to determine a second prediction value of the sub-block, and determining the second prediction value as the inter-frame prediction value of the sub-block includes: Determine secondary prediction parameters; When the secondary prediction parameter indicates the use of secondary prediction, determining a filter coefficient of a two-dimensional filter according to the second motion vector deviation; wherein the two-dimensional filter is used to perform secondary prediction processing according to a preset shape; Determining a predicted value of a pixel at each pixel position based on the filter coefficient and the first predicted value; The second prediction value is determined using the prediction value of the pixel at each pixel position.

51. The method according to claim 50, characterized in that The two-dimensional filter is used for performing secondary prediction using a plurality of adjacent pixel positions forming the preset shape.

52. The method according to claim 51, characterized in that The preset shape is a rectangle, a rhombus, or any symmetrical shape.

53. The method according to claim 52, characterized in that If the value of the affine mode parameter is 1, it indicates that the affine mode is used; If the value of the affine mode parameter is 0, or the affine mode parameter is not obtained through parsing, it indicates that the affine mode is not used.

54. The method according to claim 30, characterized in that If the sub-block size parameter is 8×8, the sub-block size flag is set to 1, and the sub-block size flag is written into the bitstream; If the sub-block size parameter is 4×4, the sub-block size flag is set to 0 and the sub-block size flag is written into the bitstream.

55. The method of claim 30, wherein: The control point mode includes a 4-parameter mode and a 6-parameter mode.

56. The method of claim 30, wherein: The prediction mode parameters, the affine mode parameters, and the prediction reference mode are written into a bitstream.

57. The method according to claim 49, characterized in that Write the PROF parameters into the bitstream.

58. The method according to claim 50, characterized in that The secondary prediction parameters are written into the bitstream.

59. A decoder, characterized in that The decoder comprises a parsing unit, a first determining unit; The parsing unit is used to parse the bitstream and obtain the prediction mode parameters of the current block; The first determination unit is used to determine the first motion vector of a subblock of the current block when the prediction mode parameter indicates that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode; wherein the current block includes multiple sub-blocks; and determine the first prediction value of the subblock and the first motion vector deviation between each pixel position in the subblock and the subblock based on the first motion vector; and determine the horizontal deviation and vertical deviation of the first motion vector deviation; and determine the center position and the second motion vector deviation corresponding to each pixel position according to the first absolute value of the horizontal deviation, the second absolute value of the vertical deviation and a preset deviation threshold; and based on the first prediction value, perform secondary prediction or PROF processing according to the center position and the second motion vector deviation to determine the second prediction value of the subblock, and determine the second prediction value as the inter-frame prediction value of the subblock.

60. A decoder, characterized in that The decoder includes a first processor and a first memory storing instructions executable by the first processor. When the instructions are executed, the first processor implements the method according to any one of claims 1 to 27.

61. An encoder, characterized in that The encoder comprises a second determination unit; The second determination unit is used to determine the prediction mode parameters of the current block; and when the prediction mode parameters indicate that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode, determine the first motion vector of the sub-block of the current block; wherein the current block includes multiple sub-blocks; and determine the first prediction value of the sub-block based on the first motion vector, and the first motion vector deviation between each pixel position in the sub-block and the sub-block; and; determine the horizontal deviation and vertical deviation of the first motion vector deviation; and determine the center position and the second motion vector deviation corresponding to each pixel position according to the first absolute value of the horizontal deviation, the second absolute value of the vertical deviation and a preset deviation threshold; and based on the first prediction value, perform secondary prediction or PROF processing according to the center position and the second motion vector deviation to determine the second prediction value of the sub-block, and determine the second prediction value as the inter-frame prediction value of the sub-block.

62. An encoder, characterized in that The encoder includes a second processor and a second memory storing instructions executable by the second processor. When the instructions are executed, the second processor implements the method according to any one of claims 28 to 58.

63. A computer storage medium, characterized in that The computer storage medium stores a computer program, which implements the method according to any one of claims 1 to 27 when executed by a first processor, or implements the method according to any one of claims 28 to 58 when executed by a second processor.

Images