Inter-frame prediction method, encoder, decoder and computer storage medium

Abstract

The embodiment of the present application discloses an inter-frame prediction method, an encoder, a decoder and a computer storage medium, including: parsing a code stream to obtain a prediction mode parameter of a current block; when the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a current sub-block of the current block; determining a first prediction value of the current sub-block based on the first motion vector, and determining a target pixel position corresponding to the current pixel position; the current pixel position is the position of a pixel point in the current sub-block, and the target pixel position is the position of a pixel point for performing secondary prediction or PROF processing on the pixel point at the current pixel position; if the target pixel position does not belong to the current sub-block, updating the target pixel position according to the current sub-block to obtain an updated pixel position; determining a second prediction value corresponding to the current sub-block based on the first prediction value and the updated pixel position, and determining the second prediction value as the inter-frame prediction value of the current sub-block.

Description

[0001] This application is a divisional application of the PCT international patent application PCT / CN2021 / 106589 with an application date of July 15, 2021, which entered the Chinese national phase with Chinese patent application number 202180005743.6 and the invention name “Inter-frame prediction method, encoder, decoder and computer storage medium”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to a Chinese patent application filed with the China Patent Office on August 20, 2020, with application number 202010845318.5 and application name “Inter-frame prediction method, encoder, decoder and computer storage medium”, the entire contents of which are incorporated by reference in this application. Technical Field

[0004] The present application relates to the technical field of video coding and decoding, and in particular to an inter-frame prediction method, an encoder, a decoder and a computer storage medium. Background Art

[0005] In the field of video coding and decoding, in order to balance performance and cost, in general, affine prediction in Versatile Video Coding (VVC) and Audio Videocoding Standard Workgroup of China (AVS) is implemented based on sub-blocks. At present, on the one hand, prediction refinement with optical flow (PROF) is proposed to use the optical flow principle to correct the affine prediction based on sub-blocks; on the other hand, it is proposed to use secondary prediction after sub-block-based prediction to obtain more accurate prediction values, thereby achieving improvement in affine prediction. Specifically, PROF relies on the horizontal and vertical gradients of the reference position to correct the affine prediction results, while the secondary prediction uses a filter to predict a pixel position in the sub-block again.

[0006] However, both the secondary prediction and PROF require the use of other pixel positions around the current pixel position. Once one or more pixel positions in the other pixel positions are not in the same sub-block as the current pixel position, the accuracy of the calculation may be greatly reduced. In other words, the existing PROF corrected prediction value method and secondary prediction method are not rigorous. When improving the affine prediction, they cannot be well applied to all scenarios, and the encoding performance needs to be improved. Summary of the invention

[0007] The present application proposes an inter-frame prediction method, an encoder, a decoder and a computer storage medium, which can greatly improve the encoding performance, thereby improving the encoding and decoding efficiency.

[0008] The technical solution of this application is implemented as follows:

[0009] In a first aspect, an embodiment of the present application provides an inter-frame prediction method, which is applied to a decoder. The method includes:

[0010] Parse the bitstream and obtain the prediction mode parameters of the current block;

[0011] When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a current sub-block of the current block; wherein the current block includes a plurality of sub-blocks;

[0012] Determine a first prediction value of the current sub-block based on the first motion vector, and determine a target pixel position corresponding to a current pixel position; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for which a secondary prediction or PROF processing is performed on a pixel point at the current pixel position;

[0013] If the target pixel position does not belong to the current sub-block, updating the target pixel position according to the current sub-block to obtain an updated pixel position;

[0014] Based on the first prediction value and the updated pixel position, a second prediction value corresponding to the current sub-block is determined, and the second prediction value is determined as the inter-frame prediction value of the current sub-block.

[0015] In a second aspect, an embodiment of the present application provides an inter-frame prediction method, which is applied to an encoder, and the method includes:

[0016] Determining prediction mode parameters for the current block;

[0017] When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a current sub-block of the current block; wherein the current block includes a plurality of sub-blocks;

[0018] Determine a first prediction value of the current sub-block based on the first motion vector, and determine a target pixel position corresponding to a current pixel position; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for which a secondary prediction or PROF processing is performed on a pixel point at the current pixel position;

[0019] If the target pixel position does not belong to the current sub-block, updating the target pixel position according to the current sub-block to obtain an updated pixel position;

[0020] Based on the first prediction value and the updated pixel position, a second prediction value corresponding to the current sub-block is determined, and the second prediction value is determined as the inter-frame prediction value of the current sub-block.

[0021] In a third aspect, an embodiment of the present application provides a decoder, the decoder comprising a parsing part, a first determining part, a first updating part,

[0022] The parsing part is configured to parse the bitstream and obtain the prediction mode parameters of the current block;

[0023] The first determination part is configured to determine a first motion vector of a current subblock of the current block when the prediction mode parameter indicates that an inter-frame prediction mode is used to determine an inter-frame prediction value of the current block; wherein the current block includes a plurality of subblocks; determine the first prediction value of the current subblock based on the first motion vector, and determine a target pixel position corresponding to a current pixel position; wherein the current pixel position is a position of a pixel point in the current subblock, and the target pixel position is a position of a pixel point for which a secondary prediction or PROF processing is performed on a pixel point at the current pixel position;

[0024] The first updating part is configured to update the target pixel position according to the current sub-block to obtain an updated pixel position if the target pixel position does not belong to the current sub-block;

[0025] The first determination part is further configured to determine a second prediction value corresponding to the current sub-block based on the first prediction value and the updated pixel position, and determine the second prediction value as the inter-frame prediction value of the current sub-block.

[0026] In a fourth aspect, an embodiment of the present application provides a decoder, comprising a first processor and a first memory storing instructions executable by the first processor, wherein when the instructions are executed, the first processor implements the inter-frame prediction method as described above.

[0027] In a fifth aspect, an embodiment of the present application provides an encoder, the encoder comprising a second determining part, a second updating part,

[0028] The second determination part is configured to determine a prediction mode parameter of the current block; when the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determine a first motion vector of a current sub-block of the current block; wherein the current block includes a plurality of sub-blocks; determine the first prediction value of the current sub-block based on the first motion vector, and determine a target pixel position corresponding to a current pixel position; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for which a secondary prediction or PROF processing is performed on a pixel point at the current pixel position;

[0029] The second updating part is configured to update the target pixel position according to the current sub-block to obtain an updated pixel position if the target pixel position does not belong to the current sub-block;

[0030] The second determination part is further configured to determine a second prediction value corresponding to the current sub-block based on the first prediction value and the updated pixel position, and determine the second prediction value as the inter-frame prediction value of the current sub-block.

[0031] In a sixth aspect, an embodiment of the present application provides an encoder, comprising a second processor and a second memory storing instructions executable by the second processor, wherein when the instructions are executed, the second processor implements the inter-frame prediction method as described above.

[0032] In a seventh aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a first processor and a second processor, the inter-frame prediction method as described above is implemented.

[0033] The embodiments of the present application provide an inter-frame prediction method, an encoder, a decoder and a computer storage medium. The decoder parses a bit stream to obtain a prediction mode parameter of a current block; when the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, a first motion vector of a current sub-block of the current block is determined; wherein the current block includes multiple sub-blocks; based on the first motion vector, a first prediction value of the current sub-block is determined, and a target pixel position corresponding to the current pixel position is determined; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for performing secondary prediction or PROF processing on the pixel point at the current pixel position; if the target pixel position does not belong to the current sub-block, the target pixel position is updated according to the current sub-block to obtain an updated pixel position; based on the first prediction value and the updated pixel position, a second prediction value corresponding to the current sub-block is determined, and the second prediction value is determined as the inter-frame prediction value of the current sub-block. That is to say, the inter-frame prediction method proposed in the present application, after the prediction based on the sub-block, if the pixel positions required for the secondary prediction or PROF processing are not in the same sub-block, the pixel positions required for the secondary prediction or PROF processing can be restricted to the same sub-block by extending the boundary of the current sub-block, redefining the target pixel position beyond the boundary of the current sub-block, and other methods, thereby solving the problem of reduced prediction performance due to disconnected pixels, reducing the prediction error, greatly improving the encoding performance, and thus improving the encoding and decoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is an illustration of an affine model. Figure 1 ;

[0035] Figure 2 This is an illustration of an affine model. Figure 2 ;

[0036] Figure 3 Schematic diagram of pixel interpolation;

[0037] Figure 4 Interpolation for sub-blocks Figure 1 ;

[0038] Figure 5 Interpolation for sub-blocks Figure 2 ;

[0039] Figure 6 A schematic diagram of the motion vector for each sub-block;

[0040] Figure 7 Schematic diagram of sample location;

[0041] Figure 8 A schematic diagram of performing secondary prediction on the current pixel position;

[0042] Fig. 9 This indicates that the pixel position does not belong to the same sub-block. Figure 1 ;

[0043] Fig.10 This indicates that the pixel position does not belong to the same sub-block. Figure 2 ;

[0044] Fig.11 A schematic diagram of using PROF for the current pixel position;

[0045] Fig.12 This indicates that the pixel position does not belong to the same sub-block. Figure 3 ;

[0046] Fig.13 A schematic block diagram of a video encoding system provided in an embodiment of the present application;

[0047] Fig.14 A schematic block diagram of a video decoding system provided in an embodiment of the present application;

[0048] Fig.15 Schematic diagram of the implementation process of the inter-frame prediction method Figure 1 ;

[0049] Fig.16 Schematic diagram of the implementation process of the inter-frame prediction method Figure 2 ;

[0050] Fig.17 A schematic diagram for expanding the current sub-block;

[0051] Fig.18 Schematic diagram of a two-dimensional filter Figure 1 ;

[0052] Fig.19 Schematic diagram of a two-dimensional filter Figure 2 ;

[0053] Fig. 20 Schematic diagram of the implementation process of the inter-frame prediction method Figure 3 ;

[0054] Fig.21 is a schematic diagram of a 4x4 sub-block;

[0055] Fig. 22 Indicates the extension sub-block Figure 1 ;

[0056] Fig.23 Indicates the extension sub-block Figure 2 ;

[0057] Fig.24 A schematic diagram of replacing pixels;

[0058] Fig.25 Schematic diagram of the implementation process of the inter-frame prediction method Figure 4 ;

[0059] Fig.26 is a schematic diagram of alternative pixel positions;

[0060] Fig. 27 The structure of the decoder is shown in Figure 2. Figure 1 ;

[0061] Fig.28 The structure of the decoder is shown in Figure 2. Figure 2 ;

[0062] Fig.29 The structure of the encoder is shown in Figure 2. Figure 1 ;

[0063] Fig.30 The structure of the encoder is shown in Figure 2. Figure 2 . DETAILED DESCRIPTION

[0064] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the related applications, rather than to limit the applications. It should also be noted that, for the convenience of description, only the parts related to the related applications are shown in the accompanying drawings.

[0065] At present, the common video coding standards are based on a block-based hybrid coding framework. Each frame in the video image is divided into square largest coding units (LCU) of the same size (such as 128×128, 64×64, etc.), and each largest coding unit can be divided into rectangular coding units (CU) according to rules; and the coding unit may be divided into smaller prediction units (PU). Specifically, the hybrid coding framework may include prediction, transform, quantization, entropy coding, in loop filter and other modules; among them, the prediction module may include intra-frame prediction and inter-frame prediction, and inter-frame prediction may include motion estimation and motion compensation. Since there is a strong correlation between adjacent pixels in a frame of a video image, the intra-frame prediction method used in video coding and decoding technology can eliminate the spatial redundancy between adjacent pixels; however, since there is also a strong similarity between adjacent frames in a video image, the inter-frame prediction method is used in video coding and decoding technology to eliminate the temporal redundancy between adjacent frames, thereby improving the coding efficiency. The following application will be described in detail with inter-frame prediction.

[0066] Inter-frame prediction is to use the frames that have been encoded / decoded to predict the part of the current frame that needs to be encoded / decoded. In the block-based coding and decoding framework, the part that needs to be encoded / decoded is usually a coding unit or a prediction unit. Here, the coding unit or prediction unit that needs to be encoded / decoded is collectively referred to as the current block. Translational motion is a common and simple motion mode in video, so the prediction of translation is also a traditional prediction method in video coding and decoding. Translational motion in video can be understood as a part of the content moving from a certain position on one frame to a certain position on another frame over time. A simple unidirectional prediction of translation can be represented by a motion vector (motion vector, MV) between a certain frame and the current frame. The certain frame mentioned here is a reference frame of the current frame. The current block can find a reference block on the reference frame with the same size as the current block through the motion information containing the reference frame and the motion vector, and use this reference block as the prediction block of the current block. In an ideal translational motion, the content of the current block does not have deformation, rotation, brightness, color, etc. between different frames. However, the content in the video does not always meet this ideal situation. Bidirectional prediction can solve the above problems to a certain extent. Usually, bidirectional prediction refers to bidirectional translation prediction. Bidirectional prediction is to use the motion information of two reference frames and motion vectors to find two reference blocks of the same size as the current block from two reference frames (the two reference frames may be the same reference frame), and use these two reference blocks to generate a prediction block of the current block. The generation methods include averaging, weighted averaging, and some other calculations.

[0067] In the present application, prediction can be considered as a part of motion compensation. Some documents refer to the prediction in the present application as motion compensation. For example, the affine prediction mentioned in the present application may be referred to as affine motion compensation in some documents.

[0068] Rotation, enlargement, reduction, distortion, deformation, etc. are also common changes in videos. However, ordinary translation prediction cannot handle such changes well, so the affine prediction model is applied to video encoding and decoding, such as the affine in VVC and AVS, where the affine prediction models of VVC and AVS3 are similar. In changes such as rotation, enlargement, reduction, distortion, and deformation, it can be considered that not all points in the current block use the same MV, so it is necessary to derive the MV of each point. The affine prediction model uses a small number of parameters to derive the MV of each point through calculation. The affine prediction models of VVC and AVS3 both use 2 control point (4 parameters) and 3 control point (6 parameters) models. 2 control points are 2 control points at the upper left and upper right corners of the current block, and 3 control points are 3 control points at the upper left, upper right, and lower left corners of the current block. For example, Figure 1 This is an illustration of an affine model. Figure 1 , Figure 2 This is an illustration of an affine model. Figure 2 ,like Figure 1 and 2 As shown in Figure 2, each MV consists of an x ​​component and a y component, so there are 4 parameters for 2 control points and 6 parameters for 3 control points.

[0069] According to the affine prediction model, an MV can be derived for each pixel position. Each pixel position can find its corresponding position in the reference frame. If this position is not an integer pixel position, then the value of this sub-pixel position needs to be obtained by interpolation. The interpolation method used in the current video codec standard is usually implemented by a finite impulse response (FIR) filter, and the complexity (cost) of implementing this method is very high. For example, in AVS3, an 8-tap interpolation filter is used for the brightness component, and the sub-pixel accuracy of the normal mode is 1 / 4 pixel, and the sub-pixel accuracy of the affine mode is 1 / 16 pixel. For each sub-pixel point that meets the 1 / 16 pixel accuracy, 8 integer pixels in the horizontal direction and 8 integer pixels in the vertical direction, that is, 64 integer pixels, need to be interpolated. Figure 3 is a schematic diagram of pixel interpolation, such as Figure 3 As shown, the circular pixel is the desired sub-pixel point, the dark square pixel is the position of the whole pixel corresponding to the sub-pixel, the vector between the two is the motion vector of the sub-pixel, and the light square pixel is the pixel needed to interpolate the circular sub-pixel position. To obtain the value of the sub-pixel position, the pixel values ​​of these 8x8 light square pixel areas need to be interpolated, including the dark pixel position.

[0070] In the traditional translation prediction, the MV of each pixel position of the current block is the same. If the concept of sub-block is further introduced, the size of the sub-block is 4x4, 8x8, etc. Figure 4 Interpolation for sub-blocks Figure 1 , the pixel area required for 4x4 block interpolation is as follows Figure 4 shown. Figure 5 Interpolation for sub-blocks Figure 2 , the pixel area required for 8x8 block interpolation is as follows Figure 5 shown.

[0071] If the MV of each pixel position in a sub-block is the same, then the pixel positions in a sub-block can be interpolated together, thereby sharing the bandwidth, using filters with the same phase, and sharing the intermediate values ​​of the interpolation process. However, if each pixel uses an MV, the bandwidth will increase, and filters with different phases may be used and the intermediate values ​​of the interpolation process cannot be shared.

[0072] Point-based affine prediction is very expensive. Therefore, in order to balance performance and cost, affine prediction in VVC and AVS3 is implemented based on sub-blocks. The sub-block size in AVS3 is 4x4 and 8x8, and the sub-block size of 4x4 is used in VVC. Each sub-block has an MV, and the pixel positions inside the sub-block share the same MV. Therefore, all pixel positions inside the sub-block are uniformly interpolated. Through the above method, the motion compensation complexity of sub-block-based affine prediction is similar to that of other sub-block-based prediction methods.

[0073] It can be seen that in the affine prediction method based on sub-blocks, the pixel positions inside the sub-block share the same MV, and the method for determining this shared MV is to take the MV of the center of the current sub-block. For sub-blocks such as 4x4 and 8x8, which have at least one even number of pixels in the horizontal and vertical directions, their centers actually fall on non-integer pixel positions. The current standards all take an integer pixel position. For example, for a 4x4 sub-block, take the pixel position with a distance from the upper left corner position of (2, 2). For an 8x8 sub-block, take the pixel position with a distance from the upper left corner position of (4, 4).

[0074] The affine prediction model can derive the MV of each pixel position based on the control points (2 control points or 3 control points) used by the current block. In the sub-block-based affine prediction, the MV of this position is calculated based on the pixel position in the previous paragraph as the MV of the sub-block. Figure 6 The schematic diagram of the motion vector for each sub-block is shown in FIG6 . In order to derive the motion vector of each sub-block, the motion vector of the center sample of each sub-block is rounded to 1 / 16 precision as shown in the figure, and then motion compensation is performed.

[0075] With the development of technology, a method called PROF, an improved prediction technique using optical flow, has been proposed. This technology can improve the prediction value of block-based affine prediction without increasing bandwidth. After the affine prediction based on the sub-block is completed, the horizontal and vertical gradients are calculated for each pixel point that has completed the affine prediction based on the sub-block. In VVC, PROF uses a 3-tap filter [-1, 0, 1] to calculate the gradient, and its calculation method is the same as that of bidirectional optical flow (BDOF). After that, for each pixel position, its motion vector deviation is calculated. The motion vector deviation is the difference between the motion vector of the current pixel position and the MV used by the entire sub-block. These motion vector deviations can be calculated according to the formula of the affine prediction model. Due to the characteristics of the formula, the motion vector deviations of the same positions of some sub-blocks are the same. For these sub-blocks, only one set of motion vector deviations needs to be calculated, and other sub-blocks can directly reuse these values. For each pixel position, the horizontal pixel vertical gradient and motion vector deviation (including horizontal deviation and vertical deviation) of the point are used to calculate the correction value of the predicted value of the pixel position, and then the original prediction value, that is, the prediction value based on the affine prediction of the sub-block, is added to the correction value of the prediction value to obtain the corrected prediction value.

[0076] When calculating the gradients in the horizontal and vertical directions, a [-1, 0, 1] filter is used, that is, for the current pixel position, the predicted values ​​of the pixel position with a distance of 1 to the left and the pixel position with a distance of 1 to the right will be used in the horizontal direction, and the predicted values ​​of the pixel position with a distance of 1 to the top and the pixel position with a distance of 1 to the bottom will be used in the vertical direction. If the current pixel position is the boundary position of the current block, then some of the above-mentioned pixel positions will exceed the boundary of the current block by one pixel. The predicted value of the boundary of the current block is used to fill the position of one pixel distance outward to satisfy the gradient calculation, so there is no need to add the predicted value of one pixel distance beyond the boundary of the current block. Since the gradient calculation only needs to use the predicted value based on the affine prediction of the sub-block, there is no need to add additional bandwidth.

[0077] In the VVC standard text, the MV of each sub-block of the current block and the motion vector deviation of each pixel position in the sub-block are derived based on the MV of the control point. The pixel positions used as the sub-block MV of each sub-block in VVC are the same, so only one set of motion vector deviations of sub-blocks need to be derived, and other sub-blocks can reuse the sub-blocks. Furthermore, in the description of the PROF process in the VVC standard text, the calculation of the motion vector deviation by PROF is included in the above process.

[0078] AVS3's affine prediction has the same basic principle as VVC when calculating the MV of a sub-block, but AVS3 has special processing for the upper left sub-block A, upper right sub-block B and lower left sub-block C of the current block.

[0079] The following is the description of the derivation of the affine motion unit sub-block motion vector array in the AVS3 standard text:

[0080] If there are 3 motion vectors in the affine control point motion vector group, the motion vector group can be expressed as mvsAffine (mv0, mv1, mv2); if there are 2 motion vectors in the affine control point motion vector group, the motion vector group can be expressed as mvsAffine (mv0, mv1). Then, the affine motion unit sub-block motion vector array can be derived according to the following steps:

[0081] 1. Calculate the variables dHorX, dVerX, dHorY and dVerY:

[0082] dHorX=(mv1_x-mv0_x)<<(7-Log(width));

[0083] dHorY=(mv1_y-mv0_y)<<(7-Log(width));

[0084] If the motion vector group is mvsAffine(mv0, mv1, mv2), then:

[0085] dVerX=(mv2_x-mv0_x)<<(7-Log(height));

[0086] dVerY=(mv2_y-mv0_y)<<(7-Log(height));

[0087] If the motion vector group is mvsAffine(mv0, mv1), then:

[0088] dVerX=-dHorY;

[0089] dVerY = dHorX;

[0090] It should be noted that Figure 7 is a schematic diagram of the sample location, such as Figure 7As shown, (xE, yE) is the position of the upper left corner sample of the brightness prediction block of the current prediction unit in the brightness sample matrix of the current image, the width and height of the current prediction unit are width and height respectively, the width and height of each sub-block are subwidth and subheight respectively, the sub-block where the upper left corner sample of the brightness prediction block of the current prediction unit is located is A, the sub-block where the upper right corner sample is located is B, and the sub-block where the lower left corner sample is located is C.

[0091] 2.1. If the prediction reference mode of the current prediction unit is 'Pred_List01' or AffineSubblockSizeFlag is equal to 1 (AffineSubblockSizeFlag is used to indicate the size of the subblock), subwidth and subheight are both equal to 8, (x, y) is the coordinate of the upper left corner of the subblock of size 8x8, then the motion vector mvE (mvE_x, mvE_y) of each 8x8 luminance subblock can be calculated:

[0092] If the current sub-block is A, both xPos and yPos are equal to 0;

[0093] If the current sub-block is B, xPos is equal to width and yPos is equal to 0;

[0094] If the current sub-block is C and there are 3 motion vectors in mvsAffine, xPos is equal to 0 and yPos is equal to height;

[0095] Otherwise, xPos is equal to (x-xE)+4, and yPos is equal to (y-yE)+4;

[0096] Therefore, the motion vector mvE of the current 8x8 sub-block is:

[0097] mvE_x=Clip3(-131072, 131071, Rounding((mv0_x<<7)+dHorX×xPos+dVerX×yPos, 7));

[0098] mvE_y=Clip3(-131072, 131071, Rounding((mv0_y<<7)+dHorY×xPos+dVerY×yPos, 7));

[0099] 2.2. If the prediction reference mode of the current prediction unit is 'Pred_List0' or 'Pred_List1', and AffineSubblockSizeFlag is equal to 0, then subwidth and subheight are both equal to 4, (x, y) is the coordinate of the upper left corner of the sub-block of size 4x4, and the motion vector mvE (mvE_x, mvE_y) of each 4x4 luminance sub-block is calculated:

[0100] If the current sub-block is A, both xPos and yPos are equal to 0;

[0101] If the current sub-block is B, xPos is equal to width and yPos is equal to 0;

[0102] If the current sub-block is C and there are 3 motion vectors in mvAffine, xPos is equal to 0 and yPos is equal to height;

[0103] Otherwise, xPos is equal to (x-xE)+2, and yPos is equal to (y-yE)+2;

[0104] Therefore, the motion vector mvE of the current 4x4 sub-block is:

[0105] mvE_x=Clip3(-131072, 131071, Rounding((mv0_x<<7)+dHorX×xPos+dVerX×yPos, 7));

[0106] mvE_y=Clip3(-131072, 131071, Rounding((mv0_y<<7)+dHorY×xPos+dVerY×yPos, 7)).

[0107] The following is the description of AVS3 text for affine prediction sample derivation and brightness and chrominance sample interpolation:

[0108] If the position of the upper left corner sample of the brightness prediction block of the current prediction unit in the brightness sample matrix of the current image is (xE, yE).

[0109] If the prediction reference mode of the current prediction unit is 'PRED_List0' and the value of AffineSubblockSizeFlag is 0, mv0E0 is the LO motion vector of the 4x4 unit at the position (xE+x, yE+y) of the MvArrayL0 motion vector set. The value of the element predMatrixL0[x][y] in the luma prediction sample matrix predMatrixL0 is the sample value at position (((xE+x)<<4)+mv0E0_x, ((yE+y)<<4)+mv0E0_y) in the 1 / 16 precision luma sample matrix with reference index RefIdxL0 in reference picture queue 0. The value of the element predMatrixL0[x][y] in the chroma prediction sample matrix predMatrixL0 is the sample value at position (((xE+2x)<<4)+MvC_x, ((yE+2y)<<4)+MvC_y) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL0 in reference picture queue 0. Among them, x1=((xE+2x)>>3)<<3, y1=((yE+2y)>>3)<<3, mv1E0 is the LO motion vector of the 4x4 unit of the MvArrayL0 motion vector set at the position (x1, y1), mv2E0 is the LO motion vector of the 4x4 unit of the MvArrayL0 motion vector set at the position (x1+4, y1), mv3E0 is the LO motion vector of the 4x4 unit of the MvArrayL0 motion vector set at the position (x1, y1+4), and mv4E0 is the LO motion vector of the 4x4 unit of the MvArrayL0 motion vector set at the position (x1+4, y1+4).

[0110] MvC_x=(mv1E0_x+mv2E0_x+mv3E0_x+mv4E0_x+2)>>2

[0111] MvC_y=(mv1E0_y+mv2E0_y+mv3E0_y+mv4E0_y+2)>>2

[0112] If the prediction reference mode of the current prediction unit is 'PRED_List0' and the value of AffineSubblockSizeFlag is 1, mv0E0 is the LO motion vector of the 8x8 unit at the position (xE+x, yE+y) of the MvArrayL0 motion vector set. The value of the element predMatrixL0[x][y] in the luma prediction sample matrix predMatrixL0 is the sample value at position (((xE+x)<<4)+mv0E0_x, ((yE+y)<<4)+mv0E0_y) in the 1 / 16 precision luma sample matrix with reference index RefIdxL0 in reference picture queue 0. The value of the element predMatrixL0[x][y] in the chroma prediction sample matrix predMatrixL0 is the sample value at position (((xE+2x)<<4)+MvC_x, ((yE+2y)<<4)+MvC_y) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL0 in reference picture queue 0. Among them, MvC_x is equal to mv0E0_x, and MvC_y is equal to mv0E0.

[0113] If the prediction reference mode of the current prediction unit is 'PRED_List1' and the value of AffineSubblockSizeFlag is 0, mv0E1 is the L1 motion vector of the 4x4 unit at the position (xE+x, yE+y) of the MvArrayL1 motion vector set. The value of the element predMatrixL1[x][y] in the luma prediction sample matrix predMatrixL1 is the sample value at position (((xE+x)<<4)+mv0E1_x, ((yE+y)<<4)+mv0E1_y) in the 1 / 16 precision luma sample matrix with reference index RefIdxL1 in reference image queue 1. The value of the element predMatrixL1[x][y] in the chroma prediction sample matrix predMatrixL1 is the sample value at position (((xE+2x)<<4)+MvC_x, ((yE+2y)<<4)+MvC_y) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL1 in reference image queue 1. Among them, x1=((xE+2x)>>3)<<3, y1=((yE+2y)>>3)<<3, mv1E1 is the L1 motion vector of the 4x4 unit of the MvArrayL1 motion vector set at the position (x1, y1), mv2E1 is the L1 motion vector of the 4x4 unit of the MvArrayL1 motion vector set at the position (x1+4, y1), mv3E1 is the L1 motion vector of the 4x4 unit of the MvArrayL1 motion vector set at the position (x1, y1+4), and mv4E1 is the L1 motion vector of the 4x4 unit of the MvArrayL1 motion vector set at the position (x1+4, y1+4).

[0114] MvC_x=(mv1E1_x+mv2E1_x+mv3E1_x+mv4E1_x+2)>>2

[0115] MvC_y=(mv1E1_y+mv2E1_y+mv3E1_y+mv4E1_y+2)>>2

[0116] If the prediction reference mode of the current prediction unit is 'PRED_List1' and the value of AffineSubblockSizeFlag is 1, mv0E1 is the L1 motion vector of the 8x8 unit at the position (xE+x, yE+y) of the MvArrayL1 motion vector set. The value of the element predMatrixL1[x][y] in the luma prediction sample matrix predMatrixL1 is the sample value at position (((xE+x)<<4)+mv0E1_x, ((yE+y)<<4)+mv0E1_y) in the 1 / 16 precision luma sample matrix with reference index RefIdxL1 in reference picture queue 1. The value of the element predMatrixL1[x][y] in the chroma prediction sample matrix predMatrixL1 is the sample value at position (((xE+2x)<<4)+MvC_x, ((yE+2y)<<4)+MvC_y) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL1 in reference picture queue 1. Where MvC_x is equal to mv0E1_x, and MvC_y is equal to mv0E1.

[0117] If the prediction mode of the current prediction unit is 'PRED_List01', mv0E0 is the L0 motion vector of the 8x8 unit at position (xE+x, yE+y) of the MvArrayL0 motion vector set, and mv0E1 is the L1 motion vector of the 8x8 unit at position (x, y) of the MvArrayL1 motion vector set. The value of the element predMatrixL0[x][y] in the luminance prediction sample matrix predMatrixL0 is the sample value at position (((xE+x)<<4)+mv0E0_x, ((yE+y)<<4)+mv0E0_y) in the 1 / 16 precision luminance sample matrix with reference index RefIdxL0 in reference image queue 0. The value of the element predMatrixL0[x][y] in the chrominance prediction sample matrix predMatrixL0 is the sample value at position (((xE+2x)<<4)+MvC0 in the 1 / 32 precision chrominance sample matrix with reference index RefIdxL0 in reference image queue 0. _x, ((yE+2y)<<4)+MvC0_y), the value of the element predMatrixL1[x][y] in the luminance prediction sample matrix predMatrixL1 is the sample value at position ((((xE+x)<<4)+mv0E1_x, ((yE+y)<<4)+mv0E1_y)) in the 1 / 16 precision luminance sample matrix with reference index RefIdxL1 in the reference picture queue 1, and the value of the element predMatrixL1[x][y] in the chrominance prediction sample matrix predMatrixL1 is the sample value at position (((xE+2x)<<4)+MvC1_x, ((yE+2y)<<4)+MvC1_y) in the 1 / 32 precision chrominance sample matrix with reference index RefIdxL1 in the reference picture queue 1. Among them, MvC0_x is equal to mv0E0_x, MvC0_y is equal to mv0E0_y, MvC1_x is equal to mv0E1_x, and MvC1_y is equal to mv0E1_y.

[0118] The element values ​​of each position in the 1 / 16 precision sample matrix of luminance and 1 / 32 precision sample matrix of chrominance of the reference image are obtained by the interpolation method defined by the following affine luminance sample interpolation process and affine chrominance sample interpolation process. Integer samples outside the reference image should be replaced by the integer samples (edge ​​or corner samples) closest to the samples in the image, that is, the motion vector can point to samples outside the reference image.

[0119] Specifically, the affine brightness sample interpolation process is as follows:

[0120] A, B, C, D are adjacent integer pixel samples, dx and dy are the horizontal and vertical distances between the sub-pixel sample a (dx, dy) around the integer pixel sample A and A, dx is equal to fx & 15, dy is equal to fy & 15, where (fx, fy) is the coordinate of the sub-pixel sample in the 1 / 16 precision brightness sample matrix. Integer pixel A x，y There are 255 sub-pixel samples a around x，y (dx, dy).

[0121] Specifically, the sample position a x，0 (x=1-15) is obtained by filtering the 8 integer values ​​closest to the interpolation point in the horizontal direction. The prediction value is obtained as follows:

[0122] a x，0 =Clip1((fL[x][0]×A -3，0 +fL[x][1]×A -2，0 +fL[x][2]×A -1，0 +fL[x][3]×A 0，0 +fL[x][4]×A 1，0 +fL[x][5]×A 2，0 +fL[x][6]×A 3，0 +fL[x][7]×A 4，0 +32)>>6).

[0123] Specifically, the sample position a 0，y (y=1-15) is obtained by filtering the 8 integer values ​​closest to the interpolation point in the vertical direction. The predicted value is obtained as follows:

[0124] a 0，y =Clip1((fL[y][0]×A 0，-3 +fL[y][1]×A -2，0 +fL[y][2]×A -1，0 +fL[y][3]×A 0，0 +fL[y][4]×A 1，0 +fL[y][5]×A 2，0 +fL[y][6]×A3,0+fL[y][7]×A -4，0 +32)>>6).

[0125] Specifically, the sample position a x，y The method for obtaining the predicted value of (x=1-15, y=1-15) is as follows:

[0126] a x，y = Clip1((fL[y][0]×a' x，y-3 +fL[y][1]×a' x，y-2+fL[y][2]×a'x, y-1+fL[y][3]×a'x, y+fL[y][4]×a'x, y+1+fL[y][5]×a'x, y+2+fL[y][6]×a'x, y+3+fL[y][7]×a'x, y+4+(1<<(19-BitDepth)))>>(20-BitDepth)).

[0127] in:

[0128] a' x，y =(fL[x][0]×A -3，y +fL[x][1]×A -2，y +fL[x][2]×A -1，y +fL[x][3]×A 0，y +fL[x][4]×A 1，y +fL[x][5]×A 2，y +fL[x][6]×A 3，y +fL[x][7]×A 4，y +((1<<(BitDepth-8))>>1))>>(BitDepth-8).

[0129] The brightness interpolation filter coefficients are shown in Table 1:

[0130] Table 1

[0131]

[0132] Specifically, the affine chromaticity sample interpolation process is as follows:

[0133] A, B, C, D are adjacent integer pixel samples, dx and dy are the horizontal and vertical distances between the sub-pixel sample a (dx, dy) around the integer pixel sample A and A, dx is equal to fx & 31, dy is equal to fy & 31, where (fx, fy) is the coordinate of the sub-pixel sample in the chroma sample matrix with 1 / 32 accuracy. Integer pixel A x，y There are 1023 sub-pixel samples around a x，y (dx, dy).

[0134] Specifically, for sub-pixel points where dx is equal to 0 or dy is equal to 0, the chrominance integer pixel interpolation can be directly used. For points where dx is not equal to 0 and dy is not equal to 0, the sub-pixels on the integer pixel row (dy is equal to 0) are used for calculation:

[0135]

[0136] Among them, a' x，y (dx, 0) is the temporary value of the sub-pixel on the integer pixel row, defined as: a'x，y (dx, 0) = (fC[dx][0] × A x-1，y +fC[dx][1]×A x，y +fC[dx][2]×A x+1，y +fC[dx][3]×A x+2，y +((1<<(BitDepth-8))>>1))>>(BitDepth-8).

[0137] The chroma interpolation filter coefficients are shown in Table 2:

[0138] Table 2

[0139]

[0140]

[0141] Common affine prediction methods can include the following steps:

[0142] Step 101: Determine the motion vector of the control point.

[0143] Step 102: Determine the motion vector of the sub-block according to the motion vector of the control point.

[0144] Step 103: Predict the sub-block according to the motion vector of the sub-block.

[0145] Currently, when improving the prediction value of block-based affine prediction through PROF, the following steps may be specifically included:

[0146] Step 101: Determine the motion vector of the control point.

[0147] Step 102: Determine the motion vector of the sub-block according to the motion vector of the control point.

[0148] Step 103: Predict the sub-block according to the motion vector of the sub-block.

[0149] Step 104: Determine the motion vector deviation between each position in the sub-block and the sub-block according to the motion vector of the control point and the motion vector of the sub-block.

[0150] Step 105: Determine the motion vector of the sub-block according to the motion vector of the control point.

[0151] Step 106: Use the sub-block-based prediction value to derive the gradients in the horizontal and vertical directions for each position.

[0152] Step 107: Using the optical flow principle, according to the motion vector deviation and horizontal and vertical gradients of each position, calculate the deviation value of the predicted value at each position.

[0153] Step 108: For each position, add the prediction value based on the sub-block to the deviation value of the prediction value to obtain a corrected prediction value.

[0154] Currently, when improving the prediction value of block-based affine prediction through PROF, the following steps may be specifically included:

[0155] Step 101: Determine the motion vector of the control point.

[0156] Step 109: Determine the motion vector of the sub-block and the deviation between each position in the sub-block and the motion vector of the sub-block according to the motion vector of the control point.

[0157] Step 103: Predict the sub-block according to the motion vector of the sub-block.

[0158] Step 106: Use the sub-block-based prediction value to derive the gradients in the horizontal and vertical directions for each position.

[0159] Step 107: Using the optical flow principle, according to the motion vector deviation and horizontal and vertical gradients of each position, calculate the deviation value of the predicted value at each position.

[0160] Step 108: For each position, add the prediction value based on the sub-block to the deviation value of the prediction value to obtain a corrected prediction value.

[0161] PROF can use the optical flow principle to correct the affine prediction based on sub-blocks, thereby improving the compression performance. However, the application of PROF is based on the situation that the deviation between the motion vector of the pixel position in the sub-block and the sub-block motion vector is very small. In other words, the optical flow calculation method using PROF is effective when the deviation between the motion vector of the pixel position in the sub-block and the sub-block motion vector is very small. However, since PROF relies on the horizontal and vertical gradients of the reference position, when the actual position is far from the reference position, the horizontal and vertical gradients of the reference position cannot truly reflect the horizontal and vertical gradients between the reference position and the actual position. Therefore, when the deviation between the motion vector of the pixel position in the sub-block and the sub-block motion vector is large, this method is not particularly effective.

[0162] At this time, a secondary prediction method is proposed to overcome the defects of PROF. The method for the decoder to perform secondary prediction may include the following steps:

[0163] Step 201: predict the sub-block according to the motion vector of the sub-block to obtain a prediction value.

[0164] Step 202: Determine the motion vector deviation between each position in the sub-block and the sub-block.

[0165] Step 203: According to the motion vector deviation of each position, the predicted value is filtered using a two-dimensional filter to obtain a predicted value of secondary prediction.

[0166] That is to say, after the sub-block-based prediction, the secondary prediction method can perform point-based secondary prediction on the basis of the sub-block-based prediction for the pixel position where the motion vector deviates from the motion vector of the sub-block, and finally complete the correction of the predicted value to obtain a new predicted value, that is, the predicted value of the secondary prediction.

[0167] Specifically, the point-based secondary prediction uses a two-dimensional filter. The two-dimensional filter is a filter composed of adjacent points that form a preset shape. The adjacent points that form the preset shape can be 9 points. For a pixel position, the result of the filter processing is the predicted value of the secondary prediction of the position. Among them, the filter coefficient of the two-dimensional filter is determined by the motion vector deviation of each position, the input of the two-dimensional filter is the predicted value, and the output is the predicted value of the secondary prediction.

[0168] Furthermore, in the present application, if the affine mode is used for the current block in the present application, the motion vector of the control point needs to be determined first. The method for the decoder to perform secondary prediction may include the following steps:

[0169] Step 204: Determine the motion vector of the control point.

[0170] Step 205: Determine the motion vector of the sub-block according to the motion vector of the control point.

[0171] Step 201: predict the sub-block according to the motion vector of the sub-block to obtain a prediction value.

[0172] Step 202: Determine the motion vector deviation between each position in the sub-block and the sub-block.

[0173] Step 203: According to the motion vector deviation of each position, the predicted value is filtered using a two-dimensional filter to obtain a predicted value of secondary prediction.

[0174] It can be seen that in the embodiments of the present application, after the motion vector of the control point is determined, the motion vector of the control point can be used to perform prediction processing on the sub-block, and after determining the deviation between the pixel position in the sub-block and the motion vector of the sub-block, for the pixel position where the motion vector deviates from the motion vector of the sub-block, a point-based secondary prediction is performed on the basis of the sub-block-based prediction, and finally the prediction value is corrected to obtain a new prediction value, i.e., the prediction value of the secondary prediction.

[0175] Furthermore, in the present application, the two steps of determining the motion vector of the sub-block and determining the deviation of the motion vector of each position in the sub-block from the sub-block can also be performed simultaneously. The method for the decoder to perform secondary prediction may include the following steps:

[0176] Step 204: Determine the motion vector of the control point.

[0177] Step 206: Determine the motion vector of the sub-block according to the motion vector of the control point, and the deviation of the motion vector of each position in the sub-block from the sub-block.

[0178] Step 201: predict the sub-block according to the motion vector of the sub-block to obtain a prediction value.

[0179] Step 203: According to the motion vector deviation of each position, the predicted value is filtered using a two-dimensional filter to obtain a predicted value of secondary prediction.

[0180] It can be seen that in the embodiments of the present application, after the motion vector of the control point is determined, the motion vector of the sub-block and the deviation of each position in the sub-block from the motion vector of the sub-block can be determined simultaneously, and then for the pixel positions where the motion vector deviates from the motion vector of the sub-block, a point-based secondary prediction is performed on the basis of the sub-block-based prediction, and finally the prediction value is corrected to obtain a new prediction value, i.e., the prediction value of the secondary prediction.

[0181] It should be noted that, because the affine model can clearly calculate the motion vector of each pixel position, or the deviation between each pixel position in the sub-block and the sub-block motion vector, the secondary prediction method can be used to improve the affine prediction. Of course, the secondary prediction can also be applied to improve other sub-block-based predictions. That is to say, the sub-block-based prediction proposed in this application includes but is not limited to the affine sub-block-based prediction.

[0182] Furthermore, the secondary prediction method may be based on the AVS3 standard, and may also be applied to the VVC standard, which is not specifically limited in this application.

[0183] When the secondary prediction method is used to perform secondary prediction on the current pixel position in the sub-block, the pixel values ​​of the adjacent pixel positions of the current pixel position in the prediction block (current block) based on the prediction of the sub-block are used. For example, the two-dimensional filter used is a 3x3 rectangular filter, and the adjacent pixel positions include the pixel position of one pixel distance to the left of the current pixel position, the pixel position of one pixel distance to the right, the pixel position of one pixel distance above, the pixel position of one pixel distance below, the pixel position of one pixel distance to the left, the pixel position of one pixel distance to the top, the pixel position of one pixel distance to the right, the pixel position of one pixel distance to the bottom, and the pixel position of one pixel distance to the right. If filters of other shapes are used, corresponding other pixel positions may also be included.

[0184] If the other pixel positions adjacent to the current pixel position used by the filter do not belong to the same sub-block as the current pixel position, that is, the pixel position used by the filter may belong to more than one sub-block, and there is a cross-sub-block situation, it is often assumed that the adjacent pixels between adjacent sub-blocks are continuous. However, in actual situations, since multiple sub-blocks of the current block may be predicted based on different mvs, the adjacent pixels between adjacent sub-blocks may not be connected, or they are actually not adjacent, or they are not adjacent in the reference image. If the gap between the adjacent sub-block mvs of the current block is relatively small, this disconnection may not be obvious, but if the gap between the adjacent sub-block mvs is relatively large, this disconnection will be reflected.

[0185] It can be seen that when adjacent pixels between adjacent sub-blocks of the current block are not connected, the correlation between them will become weaker. At this time, if the filter uses pixels in non-connected pixel positions that do not belong to the same sub-block for filtering, the effect will be poor.

[0186] Figure 8 It is a schematic diagram of secondary prediction of the current pixel position, such as Figure 8 As shown, when the secondary prediction method is used to perform secondary prediction on the basis of the prediction block based on the prediction of the sub-block, it is assumed that the current pixel position is position 1, that is, a 3x3 rectangular filter is used to perform secondary prediction on position 1. At this time, 8 pixel positions adjacent to position 1 are needed, among which position 2 is the left adjacent pixel of position 1, that is, the pixel position with a distance of 1 pixel to the left of position 1, and position 2 is not in the same sub-block as position 1, position 1 belongs to sub-block 1, and position 2 belongs to sub-block 2. In general, it can be considered that the adjacent pixels between adjacent sub-blocks 1 and 2 are continuous, that is, position 1 and position 2 are connected, but in the reference image, their positional relationship may not be so.

[0187] For example, Fig. 9 This indicates that the pixel position does not belong to the same sub-block. Figure 1 ,like Fig. 9 As shown in FIG. 1 , a possible relative position of two adjacent sub-blocks, sub-block 1 and sub-block 2, in the reference image, wherein the MVs of sub-block 1 and sub-block 2 are not much different, and the positions in the reference image are slightly misaligned, then the actual position of position 2 adjacent to position 1 in the reference image is position 3 in sub-block 2. It can be seen that position 3 in sub-block 2 is not the position one pixel to the left of position 1 in sub-block 1, and there is a position deviation between the two pixel positions.

[0188] For example, Fig.10 This indicates that the pixel position does not belong to the same sub-block. Figure 2 ,like Fig.10As shown, another possible relative position of two adjacent sub-blocks, sub-block 1 and sub-block 2, in the reference image, wherein there is no pixel position in sub-block 2 that is adjacent to the left of position 1 in sub-block 1. If the current secondary prediction method is used, position 3 in sub-block 2 will be selected as the position one pixel to the left of position 1 in sub-block 1. It can be seen that position 1 and position 3 are far apart. If position 3 is used as the adjacent pixel position of position 1 for filtering, the prediction effect of the final secondary prediction will be greatly reduced.

[0189] Correspondingly, the situation of PROF is similar. PROF needs to calculate the gradients in the horizontal and vertical directions. The gradient calculation in the horizontal direction needs to use the pixel position at one pixel to the left and one pixel to the right of the current pixel position in the current block based on the sub-block prediction. The gradient calculation in the vertical direction needs to use the pixel position at one pixel above and one pixel below the current pixel position in the previous block based on the sub-block prediction. When one or more of these positions that need to be used are not in the same sub-block as the current pixel position, if they are not connected in the reference image, the accuracy of the gradient calculation will decrease.

[0190] For example, Fig.11 This is a schematic diagram of using PROF for the current pixel position, such as Fig.11 As shown, when PROF calculates the horizontal gradient, it is assumed that the current pixel position is position 1. At this time, two pixel positions horizontally adjacent to position 1 are needed, among which position 2 is the left adjacent pixel of position 1, that is, the pixel position with a distance of 1 pixel to the left of position 1, and position 2 is not in the same sub-block as position 1. Position 1 belongs to sub-block 1, and position 2 belongs to sub-block 2. In general, it can be considered that the adjacent pixels between adjacent sub-blocks 1 and 2 are continuous, that is, position 1 and position 2 are connected, but in the reference image, their positional relationship may not be so.

[0191] Fig.12 This indicates that the pixel position does not belong to the same sub-block. Figure 3 ,like Fig.12 As shown in the figure, the MVs of sub-block 1 and sub-block 2 are not much different, and their positions in the reference image are slightly misaligned. Then, the actual position of position 2 adjacent to position 1 in the reference image is position 3 in sub-block 2. It can be seen that position 3 in sub-block 2 is not the position one pixel to the left of position 1 in sub-block 1, and there is a position deviation between the two pixel positions.

[0192] It can be seen from this that when the pixel positions needed for secondary prediction or PROF are not in the same sub-block, the prediction performance may be degraded due to the disconnected pixel positions. That is, the existing PROF corrected prediction value method and secondary prediction method are not rigorous. When improving affine prediction, they cannot be well applied to all scenarios, and the encoding performance needs to be improved.

[0193] In order to solve the defects existing in the prior art, in an embodiment of the present application, after the sub-block-based prediction, if the pixel positions required for the secondary prediction or PROF processing are not in the same sub-block, the pixel positions required for the secondary prediction or PROF processing can be restricted to the same sub-block by extending the boundary of the current sub-block, redefining the target pixel position beyond the boundary of the current sub-block, and other methods, thereby solving the problem of reduced prediction performance caused by disconnected pixels, reducing the prediction error, greatly improving the encoding performance, and thus improving the encoding and decoding efficiency.

[0194] It should be understood that the embodiment of the present application provides a video encoding system. Fig.13 A schematic block diagram of a video encoding system provided in an embodiment of the present application is shown in FIG. Fig.13As shown, the video coding system 11 may include: a transform unit 111, a quantization unit 112, a mode selection and coding control logic unit 113, an intra-frame prediction unit 114, an inter-frame prediction unit 115 (including: motion compensation and motion estimation), an inverse quantization unit 116, an inverse transform unit 117, a loop filtering unit 118, a coding unit 119 and a decoded image buffer unit 110; for the input original video signal, a coding tree block (Coding A video reconstruction block can be obtained by dividing the video reconstruction block into a plurality of blocks, and the coding mode is determined by the mode selection and coding control logic unit 113. Then, the residual pixel information obtained after intra-frame or inter-frame prediction is transformed by the transform unit 111 and the quantization unit 112, including transforming the residual information from the pixel domain to the transform domain, and quantizing the obtained transform coefficients to further reduce the bit rate; the intra-frame prediction unit 114 is used to perform intra-frame prediction on the video reconstruction block; wherein the intra-frame prediction unit 114 is used to determine the optimal intra-frame prediction mode (i.e., the target prediction mode) of the video reconstruction block; the inter-frame prediction unit 115 is used to perform inter-frame prediction coding of the received video reconstruction block relative to one or more blocks in one or more reference frames to provide temporal prediction information; wherein the motion estimation is The process of generating a motion vector, which can estimate the motion of the video reconstruction block, and then motion compensation is performed based on the motion vector determined by the motion estimation; after determining the inter-frame prediction mode, the inter-frame prediction unit 115 is also used to provide the selected inter-frame prediction data to the encoding unit 119, and the calculated and determined motion vector data is also sent to the encoding unit 119; in addition, the inverse quantization unit 116 and the inverse transformation unit 117 are used to reconstruct the video reconstruction block, reconstruct the residual block in the pixel domain, and the reconstructed residual block is removed by the loop filter unit 118. The reconstructed residual block is then added to a predictive block in the frame of the decoded image cache unit 110 to generate a reconstructed video reconstruction block; the encoding unit 119 is used to encode various encoding parameters and quantized transform coefficients. The decoded image cache unit 110 is used to store the reconstructed video reconstruction block for prediction reference. As the video image encoding proceeds, new reconstructed video reconstruction blocks will be continuously generated, and these reconstructed video reconstruction blocks will be stored in the decoded image cache unit 110.

[0195] The embodiment of the present application also provides a video decoding system, Fig.14 A schematic block diagram of a video decoding system provided in an embodiment of the present application is shown in FIG. Fig.14As shown, the video decoding system 12 may include: a decoding unit 121, an inverse transform unit 127, an inverse quantization unit 122, an intra-frame prediction unit 123, a motion compensation unit 124, a loop filter unit 125 and a decoded image cache unit 126; after the input video signal is encoded by the video encoding system 11, a code stream of the video signal is output; the code stream is input into the video decoding system 12, and first passes through the decoding unit 121 to obtain a decoded transform coefficient; the transform coefficient is processed by the inverse transform unit 127 and the inverse quantization unit 122 to generate a residual block in the pixel domain; the intra-frame prediction unit 123 can be used to generate a prediction number of the current video decoding block based on the determined intra-frame prediction direction and the data of the previously decoded block from the current frame or picture. The motion compensation unit 124 determines prediction information for the video decoding block by analyzing the motion vector and other associated syntax elements, and uses the prediction information to generate a predictive block of the video decoding block being decoded; a decoded video block is formed by summing the residual block from the inverse transform unit 127 and the inverse quantization unit 122 with the corresponding predictive block generated by the intra-frame prediction unit 123 or the motion compensation unit 124; the decoded video signal passes through the loop filtering unit 125 to remove the block effect artifacts, which can improve the video quality; the decoded video block is then stored in the decoded image cache unit 126, which stores the reference image used for subsequent intra-frame prediction or motion compensation, and is also used for the output of the video signal to obtain the restored original video signal.

[0196] An inter-frame prediction method provided in an embodiment of the present application mainly acts on the inter-frame prediction unit 215 of the video encoding system 11 and the inter-frame prediction unit of the video decoding system 12, namely, the motion compensation unit 124; that is, if the video encoding system 11 can obtain a better prediction effect through the inter-frame prediction method provided in an embodiment of the present application, then, correspondingly, in the video decoding system 12, the video decoding recovery quality can also be improved.

[0197] Based on this, the technical solution of the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. Before the detailed description, it should be noted that the "first", "second", "third", etc. mentioned throughout the specification are only for distinguishing different features and do not have the function of limiting priority, sequence, size relationship, etc.

[0198] It should be noted that this embodiment is exemplified based on the AVS3 standard. The inter-frame prediction method proposed in this application can also be applied to other coding standard technologies such as VVC, and the application does not make any specific limitation on this.

[0199] The embodiment of the present application provides an inter-frame prediction method, which is applied to a video decoding device, i.e., a decoder. The function implemented by the method can be implemented by calling a computer program by a first processor in the decoder, and of course the computer program can be stored in a first memory. It can be seen that the decoder includes at least a first processor and a first memory.

[0200] Furthermore, in the embodiments of the present application, Fig.15 Schematic diagram of the implementation process of the inter-frame prediction method Figure 1 ,like Fig.15 As shown, the method for the decoder to perform inter-frame prediction may include the following steps:

[0201] Step 301: parse the bitstream to obtain prediction mode parameters of the current block.

[0202] In an embodiment of the present application, the decoder may first parse the binary code stream to obtain prediction mode parameters of the current block, wherein the prediction mode parameters may be used to determine the prediction mode used by the current block.

[0203] It should be noted that the image to be decoded can be divided into multiple image blocks, and the current image block to be decoded can be called a current block (which can be represented by CU), and the image blocks adjacent to the current block can be called adjacent blocks; that is, in the image to be decoded, there is an adjacent relationship between the current block and the adjacent blocks. Here, each current block can include a first image component, a second image component, and a third image component, that is, the current block represents an image block in the image to be decoded that is currently to be predicted for the first image component, the second image component, or the third image component.

[0204] Among them, assuming that the current block predicts the first image component, and the first image component is a luminance component, that is, the image component to be predicted is a luminance component, then the current block can also be called a luminance block; or, assuming that the current block predicts the second image component, and the second image component is a chrominance component, that is, the image component to be predicted is a chrominance component, then the current block can also be called a chrominance block.

[0205] Further, in an embodiment of the present application, the prediction mode parameter may not only indicate the prediction mode adopted by the current block, but also indicate parameters related to the prediction mode.

[0206] It can be understood that, in the embodiments of the present application, the prediction mode may include an inter-frame prediction mode, a traditional intra-frame prediction mode, a non-traditional intra-frame prediction mode, and the like.

[0207] That is to say, on the encoding side, the encoder can select the optimal prediction mode to pre-encode the current block. In this process, the prediction mode of the current block can be determined, and then the prediction mode parameters used to indicate the prediction mode can be determined, so that the corresponding prediction mode parameters are written into the bitstream and transmitted from the encoder to the decoder.

[0208] Accordingly, on the decoder side, the decoder can directly obtain the prediction mode parameters of the current block by parsing the bitstream, and determine the prediction mode used by the current block and the relevant parameters corresponding to the prediction mode according to the prediction mode parameters obtained by parsing.

[0209] Further, in an embodiment of the present application, after parsing and obtaining the prediction mode parameters, the decoder may determine whether the current block uses the inter-frame prediction mode based on the prediction mode parameters.

[0210] Step 302: When the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, determine a first motion vector of a current sub-block of the current block; wherein the current block includes a plurality of sub-blocks.

[0211] In an embodiment of the present application, after the decoder parses and obtains the prediction mode parameters, if the parse and obtain prediction mode parameters indicate that the current block uses the inter-frame prediction mode to determine the inter-frame prediction value of the current block, the decoder can first determine the first motion vector of each sub-block of the current block. One sub-block corresponds to one first motion vector.

[0212] It should be noted that in the embodiment of the present application, the current block is an image block to be decoded in the current frame, and the current frame is decoded in the form of image blocks in a certain order, and the current block is an image block to be decoded at the next moment in the current frame in the order. The current block can have a variety of specifications, such as 16×16, 32×32 or 32×16, where the numbers represent the number of rows and columns of pixels on the current block.

[0213] Furthermore, in an embodiment of the present application, the current block can be divided into a plurality of sub-blocks, wherein each sub-block of the current block is a predicted sub-block based on the prediction of the sub-block, and each sub-block has the same size, and the sub-block is a set of pixels of a smaller size. The size of the sub-block can be 8×8 or 4×4.

[0214] Exemplarily, in the present application, the size of the current block is 16×16, and can be divided into 4 sub-blocks of 8×8 size each.

[0215] It can be understood that in the embodiment of the present application, when the decoder parses the code stream and obtains the prediction mode parameter indicating the use of the inter-frame prediction mode to determine the inter-frame prediction value of the current block, the inter-frame prediction method provided in the embodiment of the present application can continue to be used.

[0216] In an embodiment of the present application, further, when the prediction mode parameter indicates that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode, the method for the decoder to determine the first motion vector of the current sub-block of the current block may include the following steps:

[0217] Step 302a: parse the bitstream to obtain the affine mode parameters and prediction reference mode of the current block.

[0218] Step 302b: When the affine mode parameter indicates to use the affine mode, determine the control point mode and sub-block size parameters.

[0219] Step 302c: determine a first motion vector according to the prediction reference mode, the control point mode and the sub-block size parameter.

[0220] In an embodiment of the present application, after the decoder parses and obtains the prediction mode parameters, if the prediction mode parameters obtained by parsing indicate that the current block uses the inter-frame prediction mode to determine the inter-frame prediction value of the current block, then the decoder can obtain the affine mode parameters and the prediction reference mode by parsing the bitstream.

[0221] It should be noted that, in the embodiment of the present application, the affine mode parameter is used to indicate whether to use the affine mode. Specifically, the affine mode parameter may be an affine motion compensation enable flag affine_enable_flag, and the decoder may further determine whether to use the affine mode by determining the value of the affine mode parameter.

[0222] That is, in the present application, the affine mode parameter may be a binary variable. If the value of the affine mode parameter is 1, it indicates that the affine mode is used; if the value of the affine mode parameter is 0, it indicates that the affine mode is not used.

[0223] It can be understood that, in the present application, if the decoder parses the code stream and fails to obtain the affine mode parameters, it can be understood as an indication that the affine mode is not used.

[0224] Exemplarily, in the present application, the value of the affine mode parameter may be equal to the value of the affine motion compensation enable flag affine_enable_flag. If the value of affine_enable_flag is '1', it indicates that affine motion compensation can be used; if the value of affine_enable_flag is '0', it indicates that affine motion compensation should not be used.

[0225] Further, in an embodiment of the present application, if the affine mode parameters obtained by the decoder when parsing the code stream indicate the use of the affine mode, the decoder can obtain the control point mode and sub-block size parameters.

[0226] It should be noted that, in the embodiment of the present application, the control point mode is used to determine the number of control points. In the affine model, a sub-block may have 2 control points or 3 control points, and accordingly, the control point mode may be a control point mode corresponding to 2 control points, or a control point mode corresponding to 3 control points. That is, the control point mode may include a 4-parameter mode and a 6-parameter mode.

[0227] It can be understood that in the embodiments of the present application, for the AVS3 standard, if the current block uses the affine mode, the decoder also needs to determine the number of control points of the current block in the affine mode, so as to determine whether the 4-parameter (2 control point) mode or the 6-parameter (3 control point) mode is used.

[0228] Further, in an embodiment of the present application, if the affine mode parameter obtained by the decoder when parsing the bitstream indicates that the affine mode is used, the decoder can further obtain the sub-block size parameter by parsing the bitstream.

[0229] Specifically, the sub-block size parameter can be determined by the affine prediction sub-block size flag affine_subblock_size_flag. The decoder obtains the sub-block size flag by parsing the bitstream, and determines the size of the current sub-block of the current block according to the value of the sub-block flag. The size of the sub-block can be 8×8 or 4×4. Specifically, in the present application, the sub-block size flag can be a binary variable. If the value of the sub-block size flag is 1, it indicates that the sub-block size parameter is 8×8; if the value of the sub-block size flag is 0, it indicates that the sub-block size parameter is 4×4.

[0230] Exemplarily, in the present application, the value of the subblock size flag may be equal to the value of the affine prediction subblock size flag affine_subblock_size_flag. If the value of affine_subblock_size_flag is '1', the current block is divided into subblocks of size 8×8; if the value of affine_subblock_size_flag is '0', the current block is divided into subblocks of size 4×4.

[0231] It is understood that in this application, if the decoder parses the bitstream and does not parse the subblock size flag, it can be understood that the current block is divided into 4×4 subblocks. In other words, if affine_subblock_size_flag does not exist in the bitstream, the value of the subblock size flag can be directly set to 0.

[0232] Furthermore, in an embodiment of the present application, after determining the control point mode and the sub-block size parameters, the decoder can further determine the first motion vector of the current sub-block in the current block according to the prediction reference mode, the control point mode and the sub-block size parameters.

[0233] Specifically, in an embodiment of the present application, the decoder may first determine a control point motion vector group according to a prediction reference mode; and then determine a first motion vector of the current sub-block based on the control point motion vector group, the control point mode and the sub-block size parameter.

[0234] It can be understood that, in the embodiment of the present application, the control point motion vector group can be used to determine the motion vector of the control point.

[0235] It should be noted that in an embodiment of the present application, the decoder can traverse each sub-block in the current block according to the above method, and use the control point motion vector group, control point mode and sub-block size parameters of each sub-block to determine the first motion vector of each sub-block, so that a motion vector set can be constructed based on the first motion vector of each sub-block.

[0236] It can be understood that, in the embodiment of the present application, the motion vector set of the current block may include the first motion vector of each sub-block of the current block.

[0237] Furthermore, in an embodiment of the present application, when the decoder determines the first motion vector based on the control point motion vector group, the control point mode and the sub-block size parameters, it can first determine the difference variable based on the control point motion vector group, the control point mode and the size parameters of the current block; then it can determine the sub-block position based on the prediction mode parameters and the sub-block size parameters; finally, it can use the difference variable and the sub-block position to determine the first motion vector of the sub-block, and then obtain a set of motion vectors of multiple sub-blocks of the current block.

[0238] Exemplarily, in the present application, the difference variable may include four variables, namely dHorX, dVerX, dHorY and dVerY. When calculating the difference variable, the decoder needs to first determine the control point motion vector group, where the control point motion vector group can characterize the motion vector of the control point.

[0239] Specifically, if the control point mode is a 6-parameter mode, that is, there are 3 control points, then the control point motion vector group can be a motion vector group including 3 motion vectors, expressed as mvsAffine(mv0, mv1, mv2); if the control point mode is a 4-parameter mode, that is, there are 2 control points, then the control point motion vector group can be a motion vector group including 2 motion vectors, expressed as mvsAffine(mv0, mv1).

[0240] The decoder can then use the control point motion vector group to calculate the difference variable:

[0241] dHorX=(mv1_x-mv0_x)<<(7-Log(width));

[0242] dHorY=(mv1_y-mv0_y)<<(7-Log(width));

[0243] If the motion vector group is mvsAffine(mv0, mv1, mv2), then:

[0244] dVerX=(mv2_x-mv0_x)<<(7-Log(height));

[0245] dVerY=(mv2_y-mv0_y)<<(7-Log(height));

[0246] If the motion vector group is mvsAffine(mv0, mv1), then:

[0247] dVerX=-dHorY;

[0248] dVerY=dHorX.

[0249] Wherein, width and height are respectively the width and height of the current block, that is, the size parameter of the current block. Specifically, the size parameter of the current block may be obtained by the decoder by parsing the bitstream.

[0250] Further, in an embodiment of the present application, after determining the difference value variable, the decoder can then determine the sub-block position based on the prediction mode parameter and the sub-block size parameter. Specifically, the decoder can determine the size of the sub-block through the word block size flag, and can determine which prediction mode to use through the prediction mode parameter, and then can determine the sub-block position according to the size of the sub-block and the prediction mode used.

[0251] Exemplarily, in the present application, if the prediction reference mode of the current block is 2, that is, the third reference mode 'Pred_List01', or the sub-block size flag is 1, that is, the width subwidth and height subheight of the sub-block are both equal to 8, then (x, y) is the coordinate of the upper left corner position of the 8×8 sub-block, then the coordinates xPos and yPos of the sub-block position can be determined by the following method:

[0252] If the current sub-block is the control point of the upper left corner of the current block, xPos and yPos are both equal to 0;

[0253] If the current sub-block is the control point of the upper right corner of the current block, xPos is equal to width and yPos is equal to 0;

[0254] If the current sub-block is the control point of the lower left corner of the current block, the control point motion vector group may be a motion vector group including 3 motion vectors, then xPos is equal to 0, and yPos is equal to height;

[0255] Otherwise, xPos is equal to (x-xE)+4, and yPos is equal to (y-yE)+4.

[0256] Exemplarily, in the present application, if the prediction reference mode of the current block takes a value of 0 or 1, that is, the first reference mode 'Pred_List0' or the second reference mode 'Pred_List1', and the sub-block size flag takes a value of 0, that is, the width subwidth and height subheight of the sub-block are both equal to 4, (x, y) is the coordinate of the upper left corner position of the 4×4 sub-block, then the coordinates xPos and yPos of the sub-block position can be determined by the following method:

[0257] If the current sub-block is the control point of the upper left corner of the current block, xPos and yPos are both equal to 0;

[0258] If the current sub-block is the control point of the upper right corner of the current block, xPos is equal to width and yPos is equal to 0;

[0259] If the current sub-block is the control point of the lower left corner of the current block, the control point motion vector group may be a motion vector group including 3 motion vectors, then xPos is equal to 0, and yPos is equal to height;

[0260] Otherwise, xPos is equal to (x-xE)+2, and yPos is equal to (y-yE)+2.

[0261] Furthermore, in an embodiment of the present application, after the decoder calculates the sub-block position, it can then determine the first motion vector of the current sub-block based on the sub-block position and the difference variable, and finally, traverse each sub-block of the current block to obtain the first motion vector of each sub-block, so as to construct a motion vector set of multiple sub-blocks of the current block.

[0262] Exemplarily, in the present application, after determining the sub-block positions xPos and yPos, the decoder may determine the first motion vector mvE (mvE_x, mvE_y) of the sub-block in the following manner:

[0263] mvE_x=Clip3(-131072, 131071, Rounding((mv0_x<<7)+dHorX×xPos+dVerX×yPos, 7));

[0264] mvE_y=Clip3(-131072, 131071, Rounding((mv0_y<<7)+dHorY×xPos+dVerY×yPos, 7)).

[0265] It should be noted that in the present application, when determining the deviation of each position in a sub-block from the motion vector of the sub-block, if the current block uses an affine prediction model, the motion vector of each position in the sub-block can be calculated according to the formula of the affine prediction model, and their deviations can be obtained by subtracting the motion vector of the sub-block. If the motion vectors of the sub-blocks all select the motion vectors at the same position in the sub-block, such as a 4x4 block using the position (2, 2) from the upper left corner, and an 8x8 block using the position (4, 4) from the upper left corner, according to the affine model used in current standards including VVC and AVS3, the motion vector deviations at the same position of each sub-block are the same. However, AVS has the same deviations in the upper left corner, upper right corner, and lower left corner with three control points (such as in the above AVS3 text). Figure 7 The A, B, C positions shown) are different from the positions used by other blocks, and accordingly the motion vector deviations of the sub-blocks in the upper left corner, upper right corner, and lower left corner in the case of 3 control points are calculated differently from other blocks.

[0266] Step 303, determine the first prediction value of the current sub-block based on the first motion vector, and determine the target pixel position corresponding to the current pixel position; wherein the current pixel position is the position of a pixel point in the current sub-block, and the target pixel position is the position of the pixel point for secondary prediction or PROF processing of the pixel point at the current pixel position.

[0267] In an embodiment of the present application, after determining the first motion vector of each sub-block of the current block, the decoder may first determine the first prediction value of the current sub-block based on the first motion vector of the current sub-block, and then determine the target pixel position corresponding to the current pixel position in the current sub-block. The target pixel position is a pixel position adjacent to the current pixel position.

[0268] It should be noted that, in the embodiment of the present application, the current pixel position is the position of a pixel in the current sub-block of the current block, wherein the current pixel position can represent the position of the pixel to be processed. Specifically, the current pixel position can be the position of the pixel to be secondary predicted, or the position of the pixel to be processed by PROF.

[0269] Further, in an embodiment of the present application, the target pixel position is a pixel position around the current pixel point and adjacent to the current pixel position. Specifically, if the current pixel position is the position of a pixel point to be secondary predicted, then based on the shape of the filter used for secondary prediction, the pixel position adjacent to the current pixel position in each direction can be determined as the target pixel position; if the current pixel position is the position of a pixel point to be processed by PROF, then the pixel position adjacent to the current pixel position in the horizontal direction and the vertical direction can be determined as the target pixel position.

[0270] Exemplarily, in the present application, if a rectangular filter is used to perform secondary prediction on the current pixel position, then the target pixel position is the pixel position adjacent to the current pixel position in the eight directions of up, down, left, right, upper left, lower left, upper right, and lower right. If PROF processing is performed on the current pixel position, then the target pixel position is the two pixel positions adjacent to the current pixel position in the horizontal direction and the left and right, and the two pixel positions adjacent to the current pixel position in the vertical direction and the top and bottom.

[0271] It should be noted that, in the embodiment of the present application, all positions in the target pixel position may belong to the current sub-block corresponding to the current pixel position, and some positions in the target pixel position may not belong to the current sub-block corresponding to the current pixel position. In other words, the target pixel position and the current pixel position may belong to the same sub-block or different sub-blocks.

[0272] It can be understood that, in the embodiment of the present application, step 303 may specifically include:

[0273] Step 303a: Determine a first prediction value of the current sub-block based on the first motion vector.

[0274] Step 303b: determine the target pixel position corresponding to the current pixel position.

[0275] Among them, the inter-frame prediction method proposed in the embodiment of the present application does not limit the order in which the decoder executes step 303a and step 303b. That is to say, in the present application, after determining the first motion vector of each sub-block of the current block, the decoder can first execute step 303a and then execute step 303b, or first execute step 303b and then execute step 303a, or execute step 303a and step 303b at the same time.

[0276] Further, in an embodiment of the present application, when the decoder determines the first prediction value of the current sub-block based on the first motion vector, it can first determine the sample matrix; wherein the sample matrix includes a luminance sample matrix and a chrominance sample matrix; and then the first prediction value can be determined based on the prediction reference mode, the sub-block size parameter, the sample matrix, and the motion vector set.

[0277] It should be noted that, in an embodiment of the present application, when the decoder determines the first prediction value based on the prediction reference mode, sub-block size parameters, sample matrix and motion vector set, it can first determine the target motion vector from the motion vector set based on the prediction reference mode and sub-block size parameters; then, the prediction sample matrix can be determined using the reference image queue and reference index sample matrix corresponding to the prediction reference mode and the target motion vector; wherein the prediction sample matrix includes the first prediction values ​​of multiple sub-blocks.

[0278] Specifically, in an embodiment of the present application, the sample matrix may include a luma sample matrix and a chroma sample matrix, and accordingly, the prediction sample matrix determined by the decoder may include a luma prediction sample matrix and a chroma prediction sample matrix, wherein the luma prediction sample matrix includes first luma prediction values ​​of multiple sub-blocks, and the chroma prediction sample matrix includes first chroma prediction values ​​of multiple sub-blocks, and the first luma prediction value and the first chroma prediction value constitute the first prediction value of the sub-block.

[0279] Exemplarily, in the present application, it is assumed that the position of the upper left corner sample of the current block in the brightness sample matrix of the current image is (xE, yE). If the prediction reference mode of the current block is 0, that is, the first reference mode 'PRED_List0' is used, and the sub-block size flag is 0, that is, the sub-block size parameter is 4×4, then the target motion vector mv0E0 is the first motion vector of the 4×4 sub-block at the position (xE+x, yE+y) of the motion vector set of the current block. The value of the element predMatrixL0[x][y] in the luma prediction sample matrix predMatrixL0 is the sample value at position (((xE+x)<<4)+mv0E0_x, ((yE+y)<<4)+mv0E0_y) in the 1 / 16 precision luma sample matrix with reference index RefIdxL0 in reference picture queue 0. The value of the element predMatrixL0[x][y] in the chroma prediction sample matrix predMatrixL0 is the sample value at position (((xE+2×x)<<4)+MvC_x, ((yE+2×y)<<4)+MvC_y) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL0 in reference picture queue 0. Among them, x1=((xE+2×x)>>3)<<3, y1=((yE+2×y)>>3)<<3, mv1E0 is the first motion vector of the 4×4 unit of the motion vector set of the current block at the position (x1, y1), mv2E0 is the first motion vector of the 4×4 unit of the motion vector set of the current block at the position (x1+4, y1), mv3E0 is the first motion vector of the 4×4 unit of the motion vector set of the current block at the position (x1, y1+4), and mv4E0 is the first motion vector of the 4×4 unit of the motion vector set of the current block at the position (x1+4, y1+4).

[0280] Specifically, MvC_x and MvC_y can be determined by:

[0281] MvC_x=(mv1E0_x+mv2E0_x+mv3E0_x+mv4E0_x+2)>>2

[0282] MvC_y=(mv1E0_y+mv2E0_y+mv3E0_y+mv4E0_y+2)>>2

[0283] Exemplarily, in the present application, it is assumed that the position of the upper left corner sample of the current block in the brightness sample matrix of the current image is (xE, yE). If the prediction reference mode of the current block is 0, that is, the first reference mode 'PRED_List0' is used, and the sub-block size flag is 1, that is, the sub-block size parameter is 8×8, then the target motion vector mv0E0 is the first motion vector of the 8×8 unit of the motion vector set of the current block at the position (xE+x, yE+y). The value of the element predMatrixL0[x][y] in the luma prediction sample matrix predMatrixL0 is the sample value at position (((xE+x)<<4)+mv0E0_x, ((yE+y)<<4)+mv0E0_y) in the 1 / 16 precision luma sample matrix with reference index RefIdxL0 in reference picture queue 0, and the value of the element predMatrixL0[x][y] in the chroma prediction sample matrix predMatrixL0 is the sample value at position (((xE+2×x)<<4)+MvC_x, ((yE+2×y)<<4)+MvC_y) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL0 in reference picture queue 0. Among them, MvC_x is equal to mv0E0_x, and MvC_y is equal to mv0E0.

[0284] Exemplarily, in the present application, it is assumed that the position of the upper left corner sample of the current block in the brightness sample matrix of the current image is (xE, yE). If the prediction reference mode of the current block is 1, that is, the second reference mode 'PRED_List1' is used, and the sub-block size flag is 0, that is, the sub-block size parameter is 4×4, then the target motion vector mv0E1 is the first motion vector of the 4×4 unit of the motion vector set of the current block at the position (xE+x, yE+y). The value of the element predMatrixL1[x][y] in the luma prediction sample matrix predMatrixL1 is the sample value at position (((xE+x)<<4)+mv0E1_x, ((yE+y)<<4)+mv0E1_y) in the 1 / 16 precision luma sample matrix with reference index RefIdxL1 in reference image queue 1. The value of the element predMatrixL1[x][y] in the chroma prediction sample matrix predMatrixL1 is the sample value at position (((xE+2×x)<<4)+MvC_x, ((yE+2×y)<<4)+MvC_y) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL1 in reference image queue 1. Among them, x1=((xE+2×x)>>3)<<3, y1=((yE+2×y)>>3)<<3, mv1E1 is the first motion vector of the 4×4 unit of the MvArray first motion vector set at the position (x1, y1), mv2E1 is the first motion vector of the 4×4 unit of the MvArray first motion vector set at the position (x1+4, y1), mv3E1 is the first motion vector of the 4×4 unit of the MvArray first motion vector set at the position (x1, y1+4), and mv4E1 is the first motion vector of the 4×4 unit of the MvArray first motion vector set at the position (x1+4, y1+4).

[0285] Specifically, MvC_x and MvC_y can be determined by:

[0286] MvC_x=(mv1E1_x+mv2E1_x+mv3E1_x+mv4E1_x+2)>>2

[0287] MvC_y=(mv1E1_y+mv2E1_y+mv3E1_y+mv4E1_y+2)>>2

[0288] Exemplarily, in the present application, it is assumed that the position of the upper left corner sample of the current block in the brightness sample matrix of the current image is (xE, yE). If the prediction reference mode of the current block is 1, that is, the second reference mode 'PRED_List1' is used, and the sub-block size flag is 1, that is, the sub-block size parameter is 8×8, then the target motion vector mv0E1 is the first motion vector of the 8×8 unit of the motion vector set of the current block at the position (xE+x, yE+y). The value of the element predMatrixL1[x][y] in the luma prediction sample matrix predMatrixL1 is the sample value at position (((xE+x)<<4)+mv0E1_x, ((yE+y)<<4)+mv0E1_y) in the 1 / 16 precision luma sample matrix with reference index RefIdxL1 in reference picture queue 1, and the value of the element predMatrixL1[x][y] in the chroma prediction sample matrix predMatrixL1 is the sample value at position (((xE+2×x)<<4)+MvC_x, ((yE+2×y)<<4)+MvC_y) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL1 in reference picture queue 1. Where MvC_x is equal to mv0E1_x, and MvC_y is equal to mv0E1.

[0289] Exemplarily, in the present application, it is assumed that the position of the upper left corner sample of the current block in the luma sample matrix of the current image is (xE, yE). If the prediction reference mode of the current block takes a value of 2, that is, the third reference mode 'PRED_List01' is used, then the target motion vector mv0E0 is the first motion vector of the 8×8 unit of the motion vector set of the current block at the position (xE+x, yE+y), and the target motion vector mv0E1 is the first motion vector of the 8×8 unit of the motion vector set of the current block at the position (x, y). The value of the element predMatrixL0[x][y] in the luma prediction sample matrix predMatrixL0 is the sample value at the position (((xE+x)<<4)+mv0E0_x, ((yE+y)<<4)+mv0E0_y)) in the 1 / 16 precision luma sample matrix with reference index RefIdxL0 in the reference image queue 0, and the chroma prediction sample matrix predMatrixL0 The value of the element predMatrixL0[x][y] in is the sample value at position (((xE+2×x)<<4)+MvC0_x, ((yE+2×y)<<4)+MvC0_y) in the 1 / 32 precision chroma sample matrix with reference index RefIdxL0 in reference picture queue 0. The value of the element predMatrixL1[x][y] in the brightness prediction sample matrix predMatrixL1 is the 1 / 16 precision brightness sample matrix with reference index RefIdxL1 in reference picture queue 1. The sample value at position ((((xE+x)<<4)+mv0E1_x, ((yE+y)<<4)+mv0E1_y)) in the sample matrix, the value of the element predMatrixL1[x][y] in the chrominance prediction sample matrix predMatrixL1 is the sample value at position (((xE+2×x)<<4)+MvC1_x, ((yE+2×y)<<4)+MvC1_y) in the 1 / 32 precision chrominance sample matrix with reference index RefIdxL1 in reference picture queue 1. Among them, MvC0_x is equal to mv0E0_x, MvC0_y is equal to mv0E0_y, MvC1_x is equal to mv0E1_x, and MvC1_y is equal to mv0E1_y.

[0290] It should be noted that, in the embodiment of the application, the brightness sample matrix in the sample matrix can be a brightness sample matrix with 1 / 16 precision, and the chrominance sample matrix in the sample matrix can be a chrominance sample matrix with 1 / 32 precision.

[0291] It can be understood that, in the embodiments of the present application, for different prediction reference modes, the reference picture queues and reference indexes obtained by the decoder through parsing the bitstream are different.

[0292] Furthermore, in an embodiment of the present application, when the decoder determines the sample matrix, it can first obtain the luminance interpolation filter coefficients and the chrominance interpolation filter coefficients; then the luminance sample matrix can be determined based on the luminance interpolation filter coefficients, and at the same time, the chrominance sample matrix can be determined based on the chrominance interpolation filter coefficients.

[0293] Exemplarily, in the present application, when the decoder determines the brightness sample matrix, the brightness interpolation filter coefficients obtained are as shown in Table 1 above, and then the brightness sample matrix is ​​calculated according to the pixel position and the sample position.

[0294] Specifically, the sample position a x，0 (x=1-15) is obtained by filtering the 8 integer values ​​closest to the interpolation point in the horizontal direction. The prediction value is obtained as follows:

[0295] a x，0 =Clip1((fL[x][0]×A -3，0 +fL[x][1]×A -2，0 +fL[x][2]×A -1，0 +fL[x][3]×A 0，0 +fL[x][4]×A 1，0 +fL[x][5]×A 2，0 +fL[x][6]×A 3，0 +fL[x][7]×A 4，0 +32)>>6).

[0296] Specifically, the sample position a 0，y (y=1-15) is obtained by filtering the 8 integer values ​​closest to the interpolation point in the vertical direction. The predicted value is obtained as follows:

[0297] a 0，y =Clip1((fL[y][0]×A 0，-3 +fL[y][1]×A -2，0 +fL[y][2]×A -1，0 +fL[y][3]×A 0，0 +fL[y][4]×A 1，0 +fL[y][5]×A 2，0 +fL[y][6]×A 3，0 +fL[y][7]×A -4，0 +32)>>6).

[0298] Specifically, the sample position a x，y The method for obtaining the predicted value of (x=1-15, y=1-15) is as follows:

[0299] a x，y= Clip1((fL[y][0]×a' x，y-3 +fL[y][1]×a' x，y-2 +fL[y][2]×a'x, y-1+fL[y][3]×a'x, y+fL[y][4]×a'x, y+1+fL[y][5]×a'x, y+2+fL[y][6]×a'x, y+3+fL[y][7]×a'x, y+4+(1<<(19-BitDepth)))>>(20-BitDepth)).

[0300] in:

[0301] a' x，y =(fL[x][0]×A -3，y +fL[x][1]×A -2，y +fL[x][2]×A -1，y +fL[x][3]×A 0，y +fL[x][4]×A 1，y +fL[x][5]×A 2，y +fL[x][6]×A 3，y +fL[x][7]×A 4，y +((1<<(BitDepth-8))>>1))>>(BitDepth-8).

[0302] Exemplarily, in the present application, when determining the chroma sample matrix, the decoder may first parse the bitstream to obtain the chroma interpolation filter coefficients as shown in Table 2 above, and then calculate the chroma sample matrix according to the pixel position and sample position.

[0303] Specifically, for sub-pixel points where dx is equal to 0 or dy is equal to 0, the chrominance integer pixel interpolation can be directly used. For points where dx is not equal to 0 and dy is not equal to 0, the sub-pixels on the integer pixel row (dy is equal to 0) are used for calculation:

[0304]

[0305] Among them, a' x，y (dx, 0) is the temporary value of the sub-pixel on the integer pixel row, defined as: a' x，y (dx, 0) = (fC[dx][0] × A x-1，y +fC[dx][1]×A x，y +fC[dx][2]×A x+1，y +fC[dx][3]×A x+2，y +((1<<(BitDepth-8))>>1))>>(BitDepth-8).

[0306] Step 304: If the target pixel position does not belong to the current sub-block, the target pixel position is updated according to the current sub-block to obtain an updated pixel position.

[0307] In an embodiment of the present application, after the decoder determines the first prediction value of the current sub-block based on the first motion vector and determines the target pixel position corresponding to the current pixel position, if the target pixel position does not belong to the current sub-block, that is, the target pixel position and the current pixel position belong to different sub-blocks, then the decoder can update the target pixel position according to the current sub-block to obtain the updated pixel position.

[0308] It should be noted that, in an embodiment of the present application, a current pixel position corresponds to multiple target pixel positions. If the current pixel position corresponds to multiple target pixel positions that all belong to the current sub-block, then it is determined that the target pixel position and the current pixel position both belong to the current sub-block; if there is at least one position among the multiple target pixel positions that does not belong to the current sub-block, then it is determined that the target pixel position and the current pixel position belong to different sub-blocks.

[0309] In the embodiments of the present application, further, Fig.16 Schematic diagram of the implementation process of the inter-frame prediction method Figure 2 ,like Fig.16 As shown, if the target pixel position does not belong to the current sub-block, the decoder updates the target pixel position according to the current sub-block. Before obtaining the updated pixel position, that is, before step 304, the method for the decoder to perform inter-frame prediction may further include the following steps:

[0310] Step 306: Determine whether the current sub-block satisfies a preset restriction condition; wherein the preset restriction condition is used to determine whether to restrict the target pixel position within the current sub-block.

[0311] Step 307: If the preset restriction condition is met, determine whether the target pixel position belongs to the current sub-block.

[0312] Step 308: If the preset restriction condition is not met, a second prediction value is determined based on the first prediction value and the target pixel position, and the second prediction value is determined as an inter-frame prediction value.

[0313] In an embodiment of the present application, the decoder may first determine whether the current sub-block satisfies a preset restriction condition. If the preset restriction condition is satisfied, the decoder may continue to determine whether the target pixel position belongs to the current sub-block, thereby further determining whether to perform restriction processing on the target pixel position, that is, whether to perform an update processing process on the target pixel position; if the preset restriction condition is not satisfied, the decoder may not need to perform restriction processing on the target pixel position, but directly determine the corrected prediction value corresponding to the current pixel position based on the first prediction value and the target pixel position, and after traversing each pixel position in the current sub-block and obtaining the corrected prediction value corresponding to each pixel position, determine the second prediction value corresponding to the current sub-block, thereby determining the inter-frame prediction value of the current sub-block.

[0314] It should be noted that, in an embodiment of the present application, the preset restriction condition is pre-set by the decoder and is used to determine whether to restrict the target pixel position to the current sub-block. That is to say, in the present application, the decoder can use the preset restriction condition to determine whether to restrict the current pixel position and the target pixel position to belong to the same sub-block.

[0315] It can be understood that in the embodiments of the present application, if the preset restriction conditions are met, it can be considered that the current pixel position and the target pixel position need to be restricted to the same sub-block, and therefore it is necessary to continue to determine whether the target pixel position belongs to the current sub-block corresponding to the current pixel position.

[0316] It can be understood that in the embodiments of the present application, if the preset restriction conditions are not met, it can be considered that there is no need to restrict the current pixel position and the target pixel position to the same sub-block. Therefore, the target pixel position can be directly used to perform secondary prediction or PROF processing on the current pixel position, and finally obtain the inter-frame prediction value corresponding to the current sub-block.

[0317] That is to say, in the present application, the decoder can restrict whether all pixel positions used in the secondary prediction or PROF processing belong to the same sub-block based on the preset restriction conditions. Specifically, if the preset restriction conditions are met, the pixel values ​​required for the secondary prediction or PROF processing all use the pixel values ​​of the current sub-block, that is, all pixel positions are required to be data of the current sub-block, and the affine prediction effect of the current pixel position can be improved according to the method proposed in steps 305 to 307 to obtain the corresponding inter-frame prediction value; if the preset restriction conditions are not met, the pixel values ​​required for the secondary prediction or PROF processing can use the pixel values ​​of other sub-blocks, that is, all pixel positions are not required to be data of the current sub-block, and the affine prediction effect of the current pixel position can be improved according to the method proposed in step 308 to obtain the corresponding inter-frame prediction value.

[0318] Further, in the embodiment of the present application, the decoder can determine whether the current sub-block meets the preset restriction condition in a variety of ways. Specifically, the decoder can further determine whether the preset restriction condition is met through the motion vector.

[0319] Exemplarily, in the present application, when the decoder determines whether the current sub-block meets the preset restriction condition, it can first determine the first motion vector deviation based on the control point motion vector group; then compare the first motion vector deviation with the preset deviation threshold; if the first motion vector deviation is greater than or equal to the preset deviation threshold, then it can be determined that the preset restriction condition is met; if the first motion vector deviation is less than the preset deviation threshold, then it is determined that the preset restriction condition is not met.

[0320] It can be understood that in the present application, the first motion vector deviation can be the difference between the motion vectors of the two control points of the current block. In the affine mode, if the 4-parameter (2 control points) mode is used, the decoder can directly determine the difference between the motion vectors mv0 and mv1 of the two control points in the control point motion vector group mvsAffine (mv0, mv1) as the first motion vector deviation; if the 6-parameter (3 control points) mode is used, the decoder can determine the difference between the motion vectors of any two control points in the control point motion vector group mvsAffine (mv0, mv1, mv2) as the first motion vector deviation, or the maximum difference of the motion vectors in the control point motion vector group mvsAffine (mv0, mv1, mv2) can be determined as the first motion vector deviation.

[0321] It should be noted that, in an embodiment of the present application, if a first motion vector deviation representing the difference between the motion vectors of two control points of the current block is used to determine whether a preset restriction condition is met, the setting of the preset deviation threshold may be related to the size of the current block, or may be related to the size of the sub-block of the current block used for the current affine prediction.

[0322] Further, in the present application, if a first motion vector deviation representing the difference between the motion vectors of two control points of the current block is used to determine whether the preset restriction conditions are met, based on the setting of a preset deviation threshold, all sub-blocks of the current block meet the preset restriction conditions, or all sub-blocks of the current block do not meet the preset restriction conditions.

[0323] Exemplarily, in the present application, when the decoder determines whether the current sub-block meets the preset restriction condition, it can first determine the second motion vector deviation based on the first motion vector of each sub-block; then compare the second motion vector deviation with the preset deviation threshold; if the second motion vector deviation is greater than or equal to the preset deviation threshold, then it is determined that the preset restriction condition is met; if the second motion vector deviation is less than the preset deviation threshold, then it is determined that the preset restriction condition is not met.

[0324] It can be understood that in the present application, the second motion vector deviation can be the difference between the motion vectors of any two or more sub-blocks in the current block. In AVS3, for different sub-blocks, the relative positions of the pixel positions as the motion vectors of the sub-blocks are not all exactly the same in the sub-blocks, such as Figure 7 The three sub-blocks shown in the figure are A upper left corner, B upper right corner, and C lower left corner. Therefore, the difference in motion vectors between different sub-blocks is not exactly the same. The difference in motion vectors between different sub-blocks of the current block can be used as the second motion vector deviation to determine whether the current sub-block meets the preset restriction conditions.

[0325] It should be noted that, in an embodiment of the present application, if a second motion vector deviation representing the motion vector difference between different sub-blocks of the current block is used to determine whether the preset restriction conditions are met, the setting of the preset deviation threshold may be related to the size of the sub-block of the current block used for the current affine prediction.

[0326] Further, in the present application, if the second motion vector deviation representing the motion vector difference between different sub-blocks of the current block is used to determine whether the preset restriction condition is met, each sub-block of the current block can determine whether the current sub-block meets the preset restriction condition based on the preset deviation threshold and the second motion vector deviation corresponding thereto, wherein, except for the following Figure 7 The sub-blocks other than A, B, C and the sub-blocks connected thereto all conform to the same rules, and therefore, the other sub-blocks can share the same conditional judgment result.

[0327] Exemplarily, in the present application, when the decoder determines whether the current sub-block satisfies the preset restriction condition, it can first determine the third motion vector deviation between the preset pixel position in the current sub-block and the current sub-block; then compare the third motion vector deviation with the preset deviation threshold; if the third motion vector deviation is greater than or equal to the preset deviation threshold, then it is determined that the preset restriction condition is met; if the third motion vector deviation is less than the preset deviation threshold, then it is determined that the preset restriction condition is not met.

[0328] It is understandable that in the present application, the third motion vector deviation may be the difference between the motion vector of the preset pixel position in the current sub-block and the current sub-block. In AVS3, for different pixel positions in a sub-block, the corresponding motion vector may be different. Generally, the farther the preset pixel position is from the pixel position of the sub-block motion vector, the greater the corresponding third motion vector deviation is, and the greater the possibility that the pixel positions of other sub-blocks adjacent to the preset pixel position are not connected.

[0329] Further, in the present application, if a third motion vector deviation representing the motion vector difference between a preset pixel position in the current sub-block and the current sub-block is used to determine whether the preset restriction condition is satisfied, each pixel position in the sub-block of the current block can be determined based on the preset deviation threshold to determine whether the preset restriction condition is satisfied, wherein, except for the following Figure 7 The sub-blocks other than A, B, C and the sub-blocks connected thereto all conform to the same rules, and therefore, the other sub-blocks can share the same conditional judgment result.

[0330] It can be understood that in the present application, if a third motion vector deviation representing the difference between the motion vector of a preset pixel position in the current sub-block and the current sub-block is used to determine whether the preset restriction condition is met, the decoder can select the difference between the motion vector of one or several specific pixel positions and the motion vector of the corresponding sub-block as the third motion vector deviation, for example, Figure 7 The A-type sub-block shown in the figure uses the motion vector of the pixel position in the upper left corner of the sub-block as the motion vector of the sub-block. The position of the pixel position in the sub-block is (0, 0), the width of the sub-block is sub_w, and the height of the sub-block is sub_h. Then the pixel position in the lower right corner of the sub-block is (sub_w-1, sub_h-1). Since the pixel position in the lower right corner is farthest from the position (0, 0) in both the horizontal and vertical directions, the difference between the motion vector of the pixel position in the lower right corner and the motion vector of the A-type sub-block can be used as the third motion vector deviation corresponding to the A-type sub-block, or the difference can be used as the third motion vector deviation corresponding to the current block. Similarly, for Figure 7 As shown in B, the difference between the motion vector of the pixel position at the lower left corner of the sub-block and the motion vector of its corresponding sub-block can be selected as the third motion vector deviation; Figure 7 As shown in C, the difference between the motion vector of the pixel position at the upper right corner of the sub-block and the motion vector of its corresponding sub-block can be selected as the third motion vector deviation.

[0331] It should be noted that, in the present application, the preset deviation threshold may be determined according to the size parameter of the current block and / or the size parameter of the sub-block.

[0332] Further, in an embodiment of the present application, if the target pixel position does not belong to the current sub-block, the decoder can update the target pixel position according to the current sub-block, thereby obtaining the updated pixel position. The decoder can obtain the updated pixel position in a variety of ways.

[0333] It can be understood that in the present application, when the decoder updates the target pixel position according to the current sub-block to obtain the updated pixel position, it can first expand the current sub-block to obtain the expanded sub-block; then the updated pixel position corresponding to the target pixel position can be determined in the expanded sub-block.

[0334] That is to say, in the present application, in order to make all the required pixel values ​​use the pixel values ​​of the current sub-block during secondary prediction or PROF processing, that is, the pixel positions that are restricted to use belong to the current sub-block, the decoder can first expand the predicted sub-block based on the prediction of the sub-block of the current sub-block, that is, first perform expansion processing on the current sub-block. Specifically, since the distance between the target pixel position used in the secondary prediction or PROF processing and the current pixel position is 1 pixel position, even if the target pixel position does not belong to the current sub-block, it only exceeds the boundary of the current sub-block by one pixel position. It can be seen that when the decoder expands the current sub-block, it only needs to expand the current sub-block based on the prediction of the sub-block by one or two rows of pixels, and / or, one or two columns of pixels, and finally the expanded sub-block can be obtained.

[0335] For example, in this application, Fig.17 To expand the schematic diagram of the current sub-block, Fig.17 As shown, taking the 9-point rectangular filter as an example, the predicted current sub-block of the sub-block can be expanded by one pixel position at its upper, lower, left and right boundaries. A simple expansion method is that the pixels expanded on the left copy the pixel values ​​of the left boundary corresponding to its horizontal direction, the pixels expanded on the right copy the pixel values ​​of the right boundary corresponding to its horizontal direction, the pixels expanded on the top copy the pixel values ​​of the upper boundary corresponding to its vertical direction, and the pixels expanded on the bottom copy the pixel values ​​of the lower boundary corresponding to its vertical direction. The four expanded vertices can copy the pixel values ​​of their corresponding vertices.

[0336] It can be understood that in the embodiments of the present application, when the decoder performs expansion processing on the current sub-block to obtain the expanded sub-block, it can choose to use all boundary positions of the current sub-block for expansion processing to obtain the expanded sub-block; it can also choose to use the boundary positions of the rows and / or columns corresponding to the target pixel positions in the current sub-block for expansion processing to obtain the expanded sub-block.

[0337] That is to say, in the present application, when the current sub-block is expanded, the four upper, lower, left and right boundaries of the current sub-block can be expanded, or only one or two boundaries corresponding to the target pixel can be expanded. For example, if a target pixel position belongs to the sub-block to the left of the current sub-block, then the left boundary of the current sub-block can be expanded, while the other three side boundaries of the current sub-block are not expanded.

[0338] It can be understood that in the present application, when the decoder updates the target pixel position according to the current sub-block to obtain the updated pixel position, it can also use the pixel position adjacent to the target pixel position in the current sub-block to replace the target pixel position, thereby obtaining the updated pixel position.

[0339] That is to say, in the present application, if the pixel position corresponding to a certain position of the secondary prediction filter or the pixel position used for PROF processing exceeds the boundary of the current sub-block, then the corresponding pixel position can be adjusted to a pixel position within the current sub-block. For example, the upper left corner pixel position of the current sub-block is (0, 0), the width of the current sub-block is sub_width, and the height of the current sub-block is sub_height, then the horizontal range of the current sub-block is 0 to (sub_width-1), and the vertical range of the current sub-block is 0 to (sub_height-1). If the pixel position to be used by the secondary prediction filter or PROF processing is (x, y), if x is less than 0, then set x to 0. If x is greater than sub_width-1, then set x to sub_width-1. If y is less than 0, then set y to 0, and if y is greater than sub_height-1, then set y to sub_height-1.

[0340] Step 305: Determine a second prediction value corresponding to the current sub-block based on the first prediction value and the updated pixel position, and determine the second prediction value as the inter-frame prediction value of the current sub-block.

[0341] In an embodiment of the present application, if the target pixel position does not belong to the current sub-block, the decoder updates the target pixel position according to the current sub-block, and after obtaining the updated pixel position, it can determine the corrected prediction value corresponding to the current pixel position based on the first prediction value and the updated pixel position, and after traversing each pixel position in the current sub-block and obtaining the corrected prediction value corresponding to each pixel position, determine the second prediction value corresponding to the current sub-block, and then the second prediction value can be determined as the inter-frame prediction value of the current sub-block.

[0342] It should be noted that in an embodiment of the present application, since the target pixel position is to perform secondary prediction or PROF processing on the pixel point of the current pixel position, after obtaining the updated pixel position corresponding to the target pixel position, the decoder can perform secondary prediction or PROF processing on the current pixel position based on the first prediction value and the updated pixel position, thereby obtaining the corresponding second prediction value.

[0343] Further, in an embodiment of the present application, a method for a decoder to determine a second prediction value corresponding to a current sub-block based on the first prediction value and the updated pixel position may include the following steps:

[0344] Step 305a, parse the code stream and obtain PROF parameters;

[0345] Step 305b: when the PROF parameter indicates to perform PROF processing, determine the pixel horizontal gradient and the pixel vertical gradient between the current pixel position and the updated pixel position based on the first prediction value;

[0346] Step 305c, determining a fourth motion vector deviation between the updated pixel position and the current sub-block;

[0347] Step 305d, calculating a deviation value corresponding to the current pixel position according to the pixel horizontal gradient, the pixel vertical gradient and the fourth motion vector deviation;

[0348] Step 305e: Obtain a second prediction value based on the first prediction value and the deviation value.

[0349] In an embodiment of the present application, the decoder may first parse the bitstream to obtain the PROF parameter. If the PROF parameter indicates that the PROF processing is to be performed, the decoder may determine the pixel horizontal gradient and the pixel vertical gradient between the current pixel position and the updated pixel position based on the first prediction value; wherein the pixel horizontal gradient is the gradient value between the pixel value corresponding to the current pixel position and the pixel value corresponding to the updated pixel position in the horizontal direction; the pixel vertical gradient is the gradient value between the pixel value corresponding to the current pixel position and the pixel value corresponding to the updated pixel position in the vertical direction. At the same time, the decoder may also determine the fourth motion vector deviation between the updated pixel position and the current sub-block, and the fourth motion vector deviation is the difference between the motion vector of the updated pixel position and the first motion vector of the current sub-block.

[0350] Further, in an embodiment of the present application, the decoder can calculate the deviation value corresponding to the current pixel position according to the pixel horizontal gradient, pixel vertical gradient and fourth motion vector deviation corresponding to the current pixel position, wherein the deviation value can be used to correct the predicted value of the pixel value at the current pixel position.

[0351] It should be noted that, in an embodiment of the present application, the decoder can further obtain a corrected prediction value corresponding to the current pixel position based on the first prediction value and the deviation value, and after traversing each pixel position in the current sub-block and obtaining the corrected prediction value corresponding to each pixel position, use the corrected prediction values ​​corresponding to all pixel positions to determine the second prediction value corresponding to the current sub-block, thereby determining the corresponding inter-frame prediction value. Specifically, in the present application, after completing the prediction based on the sub-block, the first prediction value of the current sub-block is used as the prediction value of the current pixel position, and then, the first prediction value is added to the deviation value corresponding to the current pixel position, that is, the correction processing of the prediction value of the current pixel position can be completed to obtain the corrected prediction value, so that the second prediction value of the current sub-block can be further obtained, and the second prediction value is used as the inter-frame prediction value corresponding to the current sub-block.

[0352] Further, in an embodiment of the present application, the method for the decoder to determine the second prediction value corresponding to the current sub-block based on the first prediction value and the updated pixel position may include the following steps:-

[0353] Step 305f, parsing the bitstream to obtain secondary prediction parameters;

[0354] Step 305g: when the secondary prediction parameter indicates to use secondary prediction, determine a fourth motion vector deviation between the updated pixel position and the current sub-block;

[0355] Step 305h, determining a filter coefficient of a two-dimensional filter according to the fourth motion vector deviation; wherein the two-dimensional filter is used to perform secondary prediction processing according to a preset shape;

[0356] Step 305i: determine a second prediction value based on the filter coefficient and the first prediction value, and determine the second prediction value as an inter-frame prediction value.

[0357] In an embodiment of the present application, the decoder can first parse the code stream to obtain secondary prediction parameters. If the secondary prediction parameters indicate the use of secondary prediction, the decoder can determine the fourth motion vector deviation between the updated pixel position and the current sub-block. The fourth motion vector deviation is the difference between the motion vector of the updated pixel position and the first motion vector of the current sub-block, so that the filter coefficient of the two-dimensional filter can be determined according to the fourth motion vector deviation; wherein the two-dimensional filter is used to perform secondary prediction processing according to a preset shape.

[0358] Further, in an embodiment of the present application, when the decoder determines the fourth third motion vector deviation between the target pixel position and the current sub-block, the fourth third motion vector deviation between the current sub-block and each pixel position can be determined based on the difference variable.

[0359] Specifically, in an embodiment of the present application, when the decoder determines the fourth motion vector deviation between the current sub-block and each pixel position based on the difference variable, it can determine the four difference variables dHorX, dVerX, dHorY and dVerY according to the control point motion vector group, the control point mode and the size parameters of the current block according to the method proposed in the above step 302, and then use the difference variables to further determine the fourth motion vector deviation corresponding to each pixel position in the sub-block.

[0360] Exemplarily, in this application, width and height are the width and height of the current block obtained by the decoder, respectively, and the width subwidth and height subheight of the subblock are determined by the subblock size parameter. Assuming (i, j) is the coordinate of any pixel point inside the subblock, where the value range of i is 0 to (subwidth-1), and the value range of j is 0 to (subheight-1), then the fourth motion vector deviation of each pixel (i, j) position inside the 4 different types of subblocks can be calculated by the following method:

[0361] If the current sub-block is the control point A at the upper left corner of the current block, then the fourth motion vector deviation dMvA[i][j] of the (i, j) pixel is:

[0362] dMvA[i][j][0]=dHorX×i+dVerX×j

[0363] dMvA[i][j][1]=dHorY×i+dVerY×j;

[0364] If the current sub-block is the control point B at the upper right corner of the current block, then the fourth motion vector deviation dMvB[i][j] of the (i, j) pixel is:

[0365] dMvB[i][j][0]=dHorX×(i-subwidth)+dVerX×j

[0366] dMvB[i][j][1]=dHorY×(i-subwidth)+dVerY×j;

[0367] If the current sub-block is the control point C at the lower left corner of the current block, the control point motion vector group may be a motion vector group including three motion vectors, then the fourth motion vector deviation dMvC[i][j] of the (i, j) pixel is:

[0368] dMvC[i][j][0]=dHorX×i+dVerX×(j-subheight)

[0369] dMvC[i][j][1]=dHorY×i+dVerY×(j-subheight);

[0370] Otherwise, the fourth motion vector deviation dMvN[i][j] of the (i, j) pixel is:

[0371] dMvN[i][j][0]=dHorX×(i–(subwidth>>1))+dVerX×(j–(subheight>>1))

[0372] dMvN[i][j][1]=dHorY×(i–(subwidth>>1))+dVerY×(j–(subheight>>1)).

[0373] Wherein, dMvX[i][j][0] represents the deviation value of the fourth motion vector deviation in the horizontal component, and dMvX[i][j][1] represents the deviation value of the fourth motion vector deviation in the vertical component. X is A, B, C or N.

[0374] It can be understood that in the embodiment of the present application, after the decoder determines the fourth motion vector deviation between the current sub-block and each pixel position based on the difference variable, it can use all the fourth motion vector deviations corresponding to all the pixel positions in the sub-block to construct a motion vector deviation matrix corresponding to the current sub-block. It can be seen that the motion vector deviation matrix includes the fourth motion vector deviation between the sub-block and any pixel point inside.

[0375] It should be noted that, in the embodiment of the present application, the filter coefficient of the two-dimensional filter is related to the fourth motion vector deviation corresponding to the target pixel position. In other words, for different target pixel positions, if the corresponding fourth motion vector deviation is different, the filter coefficient of the two-dimensional filter used is also different.

[0376] It is understandable that in the embodiment of the present application, the two-dimensional filter is used to perform secondary prediction using a plurality of adjacent pixel positions forming a preset shape, wherein the preset shape is a rectangle, a diamond or any symmetrical shape.

[0377] That is to say, in the present application, the two-dimensional filter used for secondary prediction is a filter composed of adjacent points constituting a preset shape. The adjacent points constituting the preset shape may include multiple points, for example, 9 points. The preset shape may be a symmetrical shape, for example, the preset shape may include a rectangle, a rhombus, or any other symmetrical shape.

[0378] Exemplarily, in the present application, the two-dimensional filter is a rectangular filter, specifically, the two-dimensional filter is a filter composed of 9 adjacent pixel positions forming a rectangle. Among the 9 pixel positions, the pixel position at the center is the pixel position of the pixel that currently needs secondary prediction, that is, the current pixel position.

[0379] Furthermore, in an embodiment of the present application, when the decoder determines the filter coefficient of the two-dimensional filter based on the fourth motion vector deviation, it can first parse the bit stream to obtain the scale parameter, and then determine the filter coefficient corresponding to the pixel position based on the scale parameter and the fourth motion vector deviation.

[0380] It should be noted that, in an embodiment of the present application, the scale parameter may include at least one scale value, and the fourth motion vector deviation includes a horizontal deviation and a vertical deviation; wherein at least one scale value is a non-zero real number.

[0381] Specifically, in the present application, when the two-dimensional filter uses 9 adjacent pixel positions forming a rectangle for secondary prediction, the pixel position located at the center of the rectangle is the position to be predicted, that is, the current pixel position, and the other 8 target pixel positions are located in the 8 directions of upper left, upper, upper right, right, lower right, lower, lower left, and left of the current pixel position, respectively.

[0382] Accordingly, in the present application, the decoder can calculate 9 filter coefficients corresponding to 9 adjacent pixel positions according to a preset calculation rule based on at least one proportional value and a fourth motion vector deviation of the position to be predicted.

[0383] It should be noted that, in the present application, the preset calculation rules may include a variety of different calculation methods, such as addition, subtraction, multiplication, etc. Among them, for different pixel positions, different calculation methods may be used to calculate the filter coefficients.

[0384] It can be understood that in the present application, when the decoder calculates multiple filter coefficients corresponding to multiple pixel positions according to different calculation methods in the preset calculation rules, some filter coefficients can be a linear function of the fourth motion vector deviation, that is, the two are in a linear relationship, or can be a quadratic function or a higher-order function of the fourth motion vector deviation, that is, the two are in a non-linear relationship.

[0385] That is to say, in the present application, any one of the multiple filter coefficients corresponding to multiple adjacent pixel positions can be a linear function, a quadratic function, or a higher-order function of the fourth motion vector deviation.

[0386] Exemplarily, in the present application, it is assumed that the fourth motion vector deviation of the pixel position is (dmv_x, dmv_y), where if the coordinates of the target pixel position are (i, j), then dmv_x can be expressed as dMvX[i][j][0], which represents the deviation value of the fourth motion vector deviation in the horizontal component, and dmv_y can be expressed as dMvX[i][j][1], which represents the deviation value of the fourth motion vector deviation in the vertical component.

[0387] Correspondingly, Table 3 shows the filter coefficients obtained based on the fourth motion vector deviation (dmv_x, dmv_y). As shown in Table 3, for a two-dimensional filter, according to the fourth motion vector deviation of the pixel position (the horizontal deviation is dmv_x, the vertical deviation is dmv_y) and different scale parameters, such as m and n, 9 filter coefficients corresponding to 9 adjacent pixel positions can be obtained, wherein the decoder can directly set the filter coefficient of the current pixel position in the center to 1.

[0388] Table 3

[0389] Pixel location Filter coefficients Top left (-dmv_x-dmv_y)×m Left -dmv_x×n Lower left (-dmv_x+dmv_y)×m superior -dmv_y×n center 1 Down dmv_y×n Top right (dmv_x-dmv_y)×m right dmv_x×n Bottom right (dmv_x+dmv_y)×m

[0390] Among them, the scale parameters m and n are generally decimals or fractions. One possible case is that m and n are powers of 2, such as 1 / 2, 1 / 4, 1 / 8, etc. Here, dmv_x and dmv_y are their actual sizes, that is, 1 in dmv_x and dmv_y represents the distance of 1 pixel, and dmv_x and dmv_y are decimals or fractions.

[0391] It should be noted that, in the embodiment of the present application, compared with the existing 8-tap filter, the motion vectors of the integer pixel position and the sub-pixel position corresponding to the currently common 8-tap filter are non-negative in the horizontal and vertical directions, and the sizes are between 0 pixels and 1 pixel, that is, dmv_x and dmv_y cannot be negative. In the present application, the motion vectors of the integer pixel position and the sub-pixel position corresponding to the filter can be negative in the horizontal and vertical directions, that is, dmv_x and dmv_y can be negative.

[0392] For example, in the embodiment of the present application, if the ratio parameter m is 1 / 16 and n is 1 / 2, then the above Table 3 can be expressed as the following Table 4:

[0393] Table 4

[0394] Pixel location Filter coefficients Top left (-dmv_x-dmv_y) / 16 Left -dmv_x / 2 Lower left (-dmv_x+dmv_y) / 16 superior -dmv_y / 2 center 1 Down dmv_y / 2 Top right (dmv_x-dmv_y) / 16 right dmv_x / 2 Bottom right (dmv_x+dmv_y) / 16

[0395] It is understandable that in the embodiments of the present application, in video coding and decoding technology and standards, magnification is usually used to avoid decimals, floating point operations, and then the calculated result is reduced by an appropriate multiple to obtain the correct result. Left shift is usually used when magnifying, and right shift is usually used when reducing. Therefore, when performing secondary prediction through a two-dimensional filter, it will be written in the following form in actual application:

[0396] Assuming that the fourth motion vector deviation of the pixel position is (dmv_x, dmv_y), after left shifting by shift1, (dmv_x', dmv_y') is obtained. Based on the above Table 4, the coefficients of the two-dimensional filter can be expressed as the following Table 5:

[0397] Table 5

[0398] Pixel location Filter coefficients Top left -dmv_x' -dmv_y' Left -dmv_x'×8 Lower left -dmv_x'+dmv_y' superior -dmv_y'×8 center 16<<shift1 Down dmv_y'×8 Top right dmv_x'-dmv_y' right dmv_x×8 Bottom right dmv_x'+dmv_y'

[0399] Fig.18 Schematic diagram of a two-dimensional filter Figure 1 ,like Fig.18 As shown in the figure, the result of the prediction based on the sub-block is used as the basis for the secondary prediction. The light-colored square is the integer pixel position of the filter, that is, the position obtained based on the prediction of the sub-block. The circle is the sub-pixel position that needs to be predicted twice, that is, the position of the pixel position. The dark square is the integer pixel position corresponding to the sub-pixel position. When interpolating to obtain this sub-pixel position, 9 integer pixel positions as shown in the figure are required.

[0400] Fig.19 Schematic diagram of a two-dimensional filter Figure 2 ,like Fig.19 As shown in the figure, the result of the prediction based on the sub-block is used as the basis for the secondary prediction. The light-colored square is the integer pixel position of the filter, that is, the position obtained based on the prediction of the sub-block. The circle is the sub-pixel position that needs to be predicted twice, that is, the position of the pixel position. The dark square is the integer pixel position corresponding to the sub-pixel position. When interpolating to obtain this sub-pixel position, 13 integer pixel positions are required as shown in the figure.

[0401] Furthermore, in an embodiment of the present application, after the decoder determines the filter coefficient of the two-dimensional filter according to the fourth motion vector deviation, it can determine the second prediction value of the current sub-block based on the filter coefficient and the first prediction value, thereby realizing secondary prediction of the current sub-block.

[0402] It can be understood that in the embodiment of the present application, the decoder determines the filter coefficient using the fourth motion vector deviation corresponding to the pixel position, so that the first prediction value can be corrected by a two-dimensional filter according to the filter coefficient to obtain a corrected second prediction value of the current sub-block. It can be seen that the second prediction value is a correction value based on the first prediction value.

[0403] Furthermore, in an embodiment of the present application, when the decoder determines the second prediction value of the current sub-block based on the filter coefficient and the first prediction value, it can first multiply the filter coefficient and the first prediction value to obtain a product result, and after traversing all pixel positions in the current sub-block, add the product results of all pixel positions in the current sub-block to obtain a sum result, and finally normalize the addition result to finally obtain the corrected second prediction value of the current sub-block.

[0404] It should be noted that in the embodiments of the present application, before performing secondary prediction, the first prediction value of the current sub-block where the pixel position is located is generally used as the prediction value of the pixel position before correction. Therefore, when filtering through a two-dimensional filter, the filter coefficient can be multiplied by the prediction value of the corresponding pixel position, that is, the first prediction value, and the product results corresponding to each pixel position are accumulated and then normalized.

[0405] It is understandable that in the present application, the decoder can perform normalization processing in a variety of ways, for example, the result of multiplying the filter coefficient with the prediction value of the corresponding pixel position and then accumulating it can be right-shifted by 4+shift1. Alternatively, the result of multiplying the filter coefficient with the prediction value of the corresponding pixel position and then accumulating it can be added (1<<(3+shift1)), and then right-shifted by 4+shift1.

[0406] It can be seen that in the present application, after obtaining the fourth motion vector deviation corresponding to the pixel position within the current sub-block, for each sub-block and each pixel position in each sub-block, a two-dimensional filter can be used for filtering based on the first prediction value of motion compensation of the current sub-block according to the fourth motion vector deviation to complete the secondary prediction of the current sub-block and obtain a new second prediction value.

[0407] Furthermore, in the embodiment of the present application, the two-dimensional filter can be understood as performing secondary prediction using a plurality of adjacent pixel positions forming a preset shape, wherein the preset shape can be a rectangle, a diamond or any symmetrical shape.

[0408] Specifically, in an embodiment of the present application, when a two-dimensional filter uses nine adjacent pixel positions forming a rectangle for secondary prediction, it can first determine a prediction sample matrix of the current block and a motion vector deviation matrix of the current sub-block of the current block; wherein the motion vector deviation matrix includes a fourth motion vector deviation corresponding to all pixel positions; and then based on the nine adjacent pixel positions forming a rectangle, the prediction sample matrix and the motion vector deviation matrix are used to determine the sample matrix of the current block after secondary prediction.

[0409] For example, in this application, if the width and height of the current block are width and height respectively, the width and height of each sub-block are subwidth and subheight respectively. Figure 7 As shown, the sub-block where the upper left corner sample of the brightness prediction sample matrix of the current block is located is A, the sub-block where the upper right corner sample is located is B, the sub-block where the lower left corner sample is located is C, and the sub-blocks where other positions are located are other sub-blocks.

[0410] For each sub-block in the current block, the motion vector deviation matrix of the sub-block can be recorded as dMv, then:

[0411] 1. If the sub-block is A, dMv is equal to dMvA;

[0412] 2. If the sub-block is B, dMv is equal to dMvB;

[0413] 3. If the sub-block is C, and there are 3 motion vectors in the control point motion vector group mvAffine of the current sub-block, dMv is equal to dMvC;

[0414] 4. If the sub-block is other than A, B, and C, dMv is equal to dMvN.

[0415] Further, assuming that (x, y) is the coordinate of the upper left corner of the current sub-block, (i, j) is the coordinate of the pixel inside the luminance sub-block, the value range of i is 0 to (subwidth-1), the value range of j is 0 to (subheight-1), the prediction sample matrix based on the sub-block is PredMatrixSb, and the prediction sample matrix of the secondary prediction is PredMatrixS, the prediction sample PredMatrixS[x+i][y+j] of the secondary prediction of (x+i, y+j) can be calculated according to the following method:

[0416] PredMatrixS[x+i][y+j]=

[0417] (UPLEFT(x+i,y+j)×(-dMv[i][j][0]-dMv[i][j][1])+

[0418] UP(x+i,y+j)×((-dMv[i][j][1])<<3)+

[0419] UPRIGHT(x+i,y+j)×(dMv[i][j][0]-dMv[i][j][1])+

[0420] LEFT(x+i,y+j)×((-dMv[i][j][0])<<3)+

[0421] CENTER(x+i,y+j)×(1<<15)+

[0422] RIGHT(x+i,y+j)×(dMv[i][j][0]<<3)+

[0423] DOWNLEFT(x+i,y+j)×(-dMv[i][j][0]+dMv[i][j][1])+

[0424] DOWN(x+i,y+j)×(dMv[i][j][1]<<3)+

[0425] DOWNRIGHT(x+i,y+j)×(dMv[i][j][0]+dMv[i][j][1])+

[0426] (1<<14))>>15

[0427] PredMatrixS[x+i][y+j]=Clip3(0, (1< <BitDepth)-1,PredMatrixS[x+i][y+j])。

[0428] Further, in the present application, if the absolute value of the maximum value of dMv is less than a preset threshold, for example, the preset threshold may be equal to 2048 converted by 1 integer pixel, that is, when the absolute value of the maximum value of dMv is less than 2048,

[0429] UPLEFT(x+i,y+j)=PredMatrixSb[max(0,x+i-1)][max(0,y+j-1)]

[0430] UP(x+i,y+j)=PredMatrixSb[x+i][max(0,y+j-1)]

[0431] UPRIGHT(x+i,y+j)=PredMatrixSb[min(width-1,x+i+1)][max(0,y+j-1)]

[0432] LEFT(x+i,y+j)=PredMatrixSb[max(0,x+i-1)][y+j]

[0433] CENTER(x+i,y+j)=PredMatrixSb[x+i][y+j]

[0434] RIGHT(x+i,y+j)=PredMatrixSb[min(width-1,x+i+1)][y+j]

[0435] DOWNLEFT(x+i, y+j) = PredMatrixSb[max(0, x+i-1)][min(height-1, y+j+1)]

[0436] DOWN(x+i, y+j) = PredMatrixSb[x+i][min(height-1, y+j+1)]

[0437] DOWNRIGHT(x+i, y+j) = PredMatrixSb[min(width-1, x+i+1)][min(height-1, y+j+1)]

[0438] Otherwise:

[0439] UPLEFT(x+i, y+j) = PredMatrixSb[max(x, x+i-1)][max(y, y+j-1)]

[0440] UP(x+i, y+j) = PredMatrixSb[x+i][max(y, y+j-1)]

[0441] UPRIGHT(x+i, y+j) = PredMatrixSb[min(x+subwidth-1, x+i+1)][max(y, y+j-1)]

[0442] LEFT(x+i, y+j) = PredMatrixSb[max(x, x+i-1)][y+j]

[0443] CENTER(x+i, y+j) = PredMatrixSb[x+i][y+j]

[0444] RIGHT(x+i, y+j) = PredMatrixSb[min(x+subwidth-1, x+i+1)][y+j]

[0445] DOWNLEFT(x+i, y+j) = PredMatrixSb[max(x, x+i-1)][min(y+subheight-1, y+j+1)]

[0446] DOWN(x+i, y+j) = PredMatrixSb[x+i][min(y+subheight-1, y+j+1)]

[0447] DOWNRIGHT(x+i,y+j)=PredMatrixSb[min(x+subwidth-1,x+i+1)][min(y+subheight-1,y+j+1)]

[0448] Among them, max(a, b) can be understood as taking the larger value of a and b, and min(a, b) can be understood as taking the smaller value of a and b.

[0449] It can be understood that in the embodiment of the present application, the CENTER (x+i, y+j) pixel position can be the center position of the above 9 adjacent pixel positions that form a rectangle, and then the secondary prediction process can be performed based on the (x+i, y+j) pixel position and the other 8 pixel positions adjacent to it. Specifically, the other 8 pixel positions are UP (up), UPRIGHT (upper right), LEFT (left), RIGHT (right), DOWNLEFT (lower left), DOWN (lower), DOWNRIGHT (lower right), UPLEFT (upper left).

[0450] It should be noted that in this application, the precision of the calculation formula of PredMatrixS[x+i][y+j] can use lower precision. For example, each term on the right side of the multiplication is shifted right, such as dMv[i][j][0] and dMv[i][j][1] are shifted right by shift3 positions. Accordingly, 1<<15 becomes 1<<(15-shift3), ...+(1<<10))>>11 becomes ...+(1<<(10-shift3)))>>(11-shift3).

[0451] Exemplarily, the size of the motion vector may be limited to a reasonable range, such as the positive and negative values ​​of the motion vector used above in the horizontal and vertical directions do not exceed 1 pixel, 1 / 2 pixel, 1 / 4 pixel, etc.

[0452] It is understandable that if the prediction reference mode of the current block is 'Pred_List01', the decoder averages multiple prediction sample matrices of each component to obtain the final prediction sample matrix of the component. For example, two brightness prediction sample matrices are averaged to obtain a new brightness prediction sample matrix.

[0453] Furthermore, in an embodiment of the present application, after obtaining the prediction sample matrix of the current block, if the current block has no transform coefficients, then the prediction matrix is ​​used as the decoding result of the current block. If the current block still has transform coefficients, then the transform coefficients can be decoded first, and the residual matrix can be obtained through inverse transformation and inverse quantization, and the residual matrix is ​​added to the prediction matrix to obtain the decoding result.

[0454] In summary, through the inter-frame prediction method proposed in steps 301 to 308, after the prediction based on the sub-block, if the pixel positions required for the secondary prediction or PROF processing are not in the same sub-block, in order to avoid the decrease in prediction performance due to the disconnection of pixels, the decoder can update the target pixel positions that do not belong to the current sub-block to obtain the updated pixel positions, so that the secondary prediction or PROF processing can be performed based on the updated pixel positions. Among them, the above-mentioned update process can be understood as the extension of the boundary of the current sub-block, and can also be understood as the redefinition of the target pixel position that exceeds the boundary of the current sub-block, so as to limit the pixel positions required for the secondary prediction or PROF processing to the same sub-block, thereby improving the prediction performance by ensuring the connection between pixels.

[0455] It is understandable that, in the present application, the pixel positions required for secondary prediction are not in the same sub-block, which means that for a certain pixel position in the current sub-block, i.e., the current pixel position, the target pixel positions required for filtering by secondary prediction do not all belong to the current sub-block. Correspondingly, the pixel positions required for PROF processing are not in the same sub-block, which means that for a certain pixel position in the current sub-block, i.e., the current pixel position, the target pixel positions required for gradient calculation by PROF processing do not all belong to the current sub-block.

[0456] It should be noted that the inter-frame prediction method proposed in the embodiment of the present application can be applied to the entire coding unit or prediction unit, that is, to the current block, to each sub-block in the current block, or to each pixel position in any sub-block. This application does not make any specific limitation.

[0457] Further, in the embodiment of the present application, if the inter-frame prediction method acts on the current block, it can also reduce the data dependency between each sub-block in the current block, so that the parallel execution of the sub-block-based secondary prediction or PROF processing can be realized, that is, when performing secondary prediction or PROF processing on a sub-block, there is no need to wait for the prediction value of the sub-block-based prediction of other sub-blocks. Specifically, the parallel processing of the sub-block-based prediction can be realized, and the parallel processing of the point-based prediction (secondary prediction or PROF processing) can also be realized on the basis of the sub-block-based prediction.

[0458] It should be noted that the inter-frame prediction method proposed in the present application can be applied to any image component. In the present embodiment, the secondary prediction scheme is used for the luminance component, but it can also be used for the chrominance component or any component in other formats. The inter-frame prediction method proposed in the present application can also be applied to any video format, including but not limited to the YUV format, including but not limited to the luminance component in the YUV format.

[0459] The present embodiment provides an inter-frame prediction method. After the prediction based on the sub-block, if the pixel positions required for the secondary prediction or PROF processing are not in the same sub-block, the pixel positions required for the secondary prediction or PROF processing can be restricted to the same sub-block by extending the boundary of the current sub-block, redefining the target pixel positions beyond the boundary of the current sub-block, and other methods. This solves the problem of reduced prediction performance due to disconnected pixels, reduces prediction errors, greatly improves encoding performance, and thus improves encoding and decoding efficiency.

[0460] Based on the above embodiment, in yet another embodiment of the present application, further, Fig. 20 Schematic diagram of the implementation process of the inter-frame prediction method Figure 3 ,like Fig. 20 As shown, the method for the decoder to perform inter-frame prediction may also include the following steps:

[0461] Step 309: When the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, determine the extended sub-block of the current sub-block.

[0462] Step 3010: determine a third prediction value of the extended sub-block based on the first motion vector, and determine a target pixel position corresponding to the current pixel position.

[0463] Step 3011: If the target pixel position does not belong to the current sub-block, determine a second prediction value based on the third prediction value and the target pixel position, and determine the second prediction value as an inter-frame prediction value.

[0464] In an embodiment of the present application, after the decoder parses and obtains the prediction mode parameters, if the prediction mode parameters indicate that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, the decoder may first determine the extended sub-block of the current sub-block.

[0465] It should be noted that, in the embodiment of the present application, the extended sub-block is determined after the current sub-block is extended in the current block. Specifically, since the distance between the target pixel position used in the secondary prediction or PROF processing and the current pixel position is 1 pixel position, even if the target pixel position does not belong to the current sub-block, it only exceeds the boundary of the current sub-block by one pixel position. Therefore, when the decoder makes predictions based on sub-blocks, it can first directly predict the extended sub-block corresponding to the current sub-block, thereby solving the problem of poor prediction effect caused by the target pixel position not belonging to the current sub-block.

[0466] Further, in an embodiment of the present application, compared with the current sub-block, the extended sub-block corresponding to the current sub-block may add one row (column) of pixels on the upper side, lower side, left side and right side of the current sub-block.

[0467] It should be noted that in the embodiment of the present application, based on the sub-block size parameter, it is determined that the size of the current sub-block may be 8×8 or 4×4, then the size parameter of the extended sub-block, that is, the size of the corresponding extended sub-block may be 10x10 or 6x6.

[0468] Furthermore, in an embodiment of the present application, after determining the extended sub-block of the current sub-block, the decoder can determine the third prediction value of the extended sub-block based on the first motion vector of the current sub-block, and at the same time, can determine the target pixel position corresponding to the current pixel position.

[0469] It can be understood that, in the present application, since the center point of the extended sub-block is the same as the center point of the corresponding current sub-block, the motion vectors of the extended sub-block and the corresponding current sub-block are the same.

[0470] It should be noted that, in the embodiment of the present application, step 3010 may specifically include:

[0471] Step 3010a: Determine a third prediction value of the extended sub-block based on the first motion vector.

[0472] Step 3010b: determine the target pixel position corresponding to the current pixel position.

[0473] Among them, the inter-frame prediction method proposed in the embodiment of the present application does not limit the order in which the decoder executes step 3010a and step 3010b. That is to say, in the present application, after determining the first motion vector corresponding to each sub-block of the current block, the decoder can first execute step 3010a and then step 3010b, or first execute step 3010b and then step 3010a, or execute step 3010a and step 3010b at the same time.

[0474] Further, in an embodiment of the present application, when the decoder determines the third prediction value of the extended sub-block based on the first motion vector, it can first determine the sample matrix; wherein the sample matrix includes a luminance sample matrix and a chrominance sample matrix; and then the third prediction value can be determined based on the prediction reference mode, the size parameter of the extended sub-block, the sample matrix, and the motion vector set.

[0475] It can be seen that in the present application, in the process of performing sub-block-based prediction, the pixel values ​​of the pixel positions required for secondary prediction or PROF processing can be predicted in advance. Since the distance between the target pixel position used in the secondary prediction or PROF processing and the current pixel position is 1 pixel position, the secondary prediction or PROF processing only needs to use the pixel values ​​of the pixel positions of one row above and one column below and one column on the left and right of the current sub-block. According to this principle, the corresponding extended sub-block can be determined. For example, if the current sub-block is a sub-block of size 4x4, then the secondary prediction or PROF processing needs to use the sub-block-based prediction value of the extended sub-block of size 6x6. If the current sub-block is a sub-block of size 8x8, then the secondary prediction or PROF processing needs to use the sub-block-based prediction value of the extended sub-block of size 10x10. That is to say, the decoder predicts the pixel values ​​of the pixel positions required for secondary prediction or PROF processing in advance during the sub-block-based prediction process. Therefore, in the subsequent secondary prediction or PROF processing, there will be no problem of the current pixel position and the target pixel position being disconnected.

[0476] In an embodiment of the present application, further, after the decoder obtains the third prediction value of the extended sub-block by advance prediction, if the target pixel position does not belong to the current sub-block, the decoder can determine the second prediction value corresponding to the current sub-block based on the third prediction value and the target pixel position, and then the second prediction value can be determined as the inter-frame prediction value.

[0477] It can be understood that in the present application, even if the target pixel position does not belong to the current sub-block, it must belong to the extended sub-block corresponding to the current sub-block. Therefore, after determining that the target pixel position does not belong to the current sub-block, the decoder can use the third prediction value of the extended sub-block and the target pixel position to further perform a secondary prediction on the current pixel position, obtain the corrected prediction value corresponding to the current pixel position, traverse each pixel position in the current sub-block, obtain the corrected prediction value corresponding to each pixel position, and finally determine the second prediction value of the current sub-block, and then determine the second prediction value as the inter-frame prediction value of the current block.

[0478] For example, in this application, if the width and height of the current block are width and height respectively, the width and height of each sub-block are subwidth and subheight respectively. Figure 7 As shown, the sub-block where the upper left corner sample of the brightness prediction sample matrix of the current block is located is A, the sub-block where the upper right corner sample is located is B, the sub-block where the lower left corner sample is located is C, and the sub-blocks where other positions are located are other sub-blocks.

[0479] For each sub-block in the current block, the motion vector deviation matrix of the sub-block can be recorded as dMv, then:

[0480] 1. If the sub-block is A, dMv is equal to dMvA;

[0481] 2. If the sub-block is B, dMv is equal to dMvB;

[0482] 3. If the sub-block is C, and there are 3 motion vectors in the control point motion vector group mvAffine of the current sub-block, dMv is equal to dMvC;

[0483] 4. If the sub-block is other than A, B, and C, dMv is equal to dMvN.

[0484] Further, assuming that (x, y) is the coordinate of the upper left corner of the current sub-block, (i, j) is the coordinate of the pixel inside the luminance sub-block, the value range of i is 0 to (subwidth-1), the value range of j is 0 to (subheight-1), the prediction sample matrix of the current sub-block based on the sub-block is PredMatrixTmp, PredMatrixTmp is the prediction sample matrix corresponding to the extended sub-block (subwidth+2)*(subheight+2), the prediction sample matrix of the secondary prediction is PredMatrixS, and the prediction sample PredMatrixS[x+i][y+j] of the secondary prediction of (x+i, y+j) can be calculated according to the following method:

[0485] PredMatrixS[x+i][y+j]=

[0486] (UPLEFT(x+i,y+j)×(-dMv[i][j][0]-dMv[i][j][1])+

[0487] UP(x+i,y+j)×((-dMv[i][j][1])<<3)+

[0488] UPRIGHT(x+i,y+j)×(dMv[i][j][0]-dMv[i][j][1])+

[0489] LEFT(x+i,y+j)×((-dMv[i][j][0])<<3)+

[0490] CENTER(x+i,y+j)×(1<<15)+

[0491] RIGHT(x+i,y+j)×(dMv[i][j][0]<<3)+

[0492] DOWNLEFT(x + i, y + j) × (-dMv[i][j][0] + dMv[i][j][1]) +

[0493] DOWN(x + i, y + j) × (dMv[i][j][1] << 3) +

[0494] DOWNRIGHT(x + i, y + j) × (dMv[i][j][0] + dMv[i][j][1]) +

[0495] (1 << 14)) >> 15

[0496] PredMatrixS[x + i][y + j] = Clip3(0, (1 << BitDepth) - 1, PredMatrixS[x + i][y + j]).

[0497] Where, UPLEFT(x + i, y + j) = PredMatrixTmp[i][j]

[0498] UP(x + i, y + j) = PredMatrixTmp[i + 1][j]

[0499] UPRIGHT(x + i, y + j) = PredMatrixTmp[i + 2][j]

[0500] LEFT(x + i, y + j) = PredMatrixTmp[i][j + 1]

[0501] CENTER(x + i, y + j) = PredMatrixTmp[i + 1][j + 1]

[0502] RIGHT(x + i, y + j) = PredMatrixTmp[i + 2][j + 1]

[0503] DOWNLEFT(x + i, y + j) = PredMatrixTmp[i][j + 2]

[0504] DOWN(x + i, y + j) = PredMatrixTmp[i + 1][j + 2]

[0505] DOWNRIGHT(x + i, y + j) = PredMatrixTmp[i + 2][j + 2]

[0506] It can be understood that in the present application, the prediction sample matrix PredMatrixTmp corresponding to the extended sub-block with a size parameter of 6x6 or 10x10 can be a prediction matrix that adds one row of prediction values ​​above and below and one column of prediction values ​​on the left and right sides on the basis of the original sub-block prediction matrix based on sub-block prediction. Among them, PredMatrixTmp is a matrix corresponding to the sub-block, and PredMatrixS is a matrix corresponding to the entire prediction unit, that is, the matrix of the current block, so their indexes are different. Compared with the current sub-block, the extended sub-block has one more column on the left and one more row on the top, so the index of the position corresponding to the sub-block is increased by 1 in the horizontal direction and 1 in the vertical direction respectively.

[0507] Further, in an embodiment of the present application, before executing step 3010a, that is, before determining the third prediction value of the extended sub-block based on the first motion vector, the decoder may also first determine whether the current sub-block meets the preset restriction conditions according to the method proposed in the above steps 306 to 308. Only when the preset restriction conditions are met, the decoder will determine the third prediction value of the extended sub-block, and use the third prediction value for secondary prediction or PROF processing to obtain the inter-frame prediction value of the current sub-block; if the current sub-block does not meet the preset restriction conditions, the decoder can directly use the first prediction value of the current sub-block and the target pixel position to determine the second prediction value, and then determine the second prediction value as the inter-frame prediction value.

[0508] That is to say, in the present application, the preset restriction condition can also be used to determine whether to perform a third prediction value determination of the extended sub-block on the current sub-block.

[0509] It can be understood that in the embodiments of the present application, in addition to using the above-mentioned first motion vector deviation, second motion vector deviation or third motion vector deviation to determine whether the preset restriction conditions are met, the decoder can also directly use the sub-block size parameters corresponding to the current sub-block to determine whether the preset restriction conditions are met.

[0510] Exemplarily, in the present application, if the sub-block size parameter corresponding to the current sub-block is 8x8, then it can be determined that the preset restriction condition is met; if the sub-block size parameter corresponding to the current sub-block is 4x4, then it can be determined that the preset restriction condition is not met.

[0511] Since sub-block-based prediction requires interpolation filtering, in general, 8-tap filters in the horizontal and vertical directions can be used for interpolation. If the number of interpolation points is changed from the original 4x4 to 6x6, or from the original 8x8 to 10x10, additional bandwidth will be required and the amount of calculation will also increase. Therefore, it is necessary to determine and use the third prediction value of the extended sub-block corresponding to the current sub-block based on the preset constraints. For example, if the size of the current sub-block is 8x8, then it is determined that the preset constraints are met, and the third prediction value of the extended sub-block is determined and used for secondary prediction or PROF processing; if the size of the current sub-block is 4x4, then it is determined that the preset constraints are not met, and there is no need to determine the third prediction value of the extended sub-block, but directly use the first prediction value of the current sub-block for secondary prediction or PROF processing.

[0512] Since adding new prediction values ​​according to the existing interpolation method, such as the prediction values ​​of one row above and one column below and one column below and one column below, will increase the bandwidth, in order to solve the problem of increased bandwidth, a filter with less than 8 taps can also be used for interpolation, so that the bandwidth can be not increased when determining the prediction value of the additional pixel position, and further, the nearest reference pixel value can be directly used as the prediction value of the corresponding additional pixel position. This application is not specifically limited.

[0513] For example, in this application, Fig.21 is a schematic diagram of a 4x4 sub-block, such as Fig.21 As shown in the figure, if an 8-tap interpolation filter is used for a 4x4 sub-block, 11x11 reference pixels around the sub-block need to be used for interpolation as shown in the figure. In other words, for a 4x4 sub-block, if an 8-tap interpolation filter is used, 3 more reference pixels are needed on the left and top sides, and 4 more reference pixels are needed on the right and bottom sides for interpolation.

[0514] It can be understood that in the present application, the problem of increased bandwidth can be solved by using a filter with fewer taps for the extended sub-block, for example, using an n-tap interpolation filter for the extended sub-block to obtain a third prediction value; wherein n is any one of the following values: 6, 5, 4, 3, 2.

[0515] Fig. 22 Indicates the extension sub-block Figure 1 ,like Fig. 22As shown, circle 1 can represent the sub-pixel in the reference image corresponding to the pixel position inside the sub-block, that is, it can represent the predicted value of the sub-block based on the prediction of the sub-block, and circle 2 can represent the additional predicted value obtained in advance and in the extended sub-block beyond the atomic block boundary. Among them, the reference pixels required for the interpolation of circle 2 using the same method as circle 1 will exceed the original 11x11 range. However, using a filter with fewer taps, such as a 6-, 5-, 4-, 3-, or 2-tap filter, can keep within this range and thus not increase the bandwidth. A 2-tap filter can be the average or weighted average of two adjacent integer pixels of the sub-pixel point. The above-mentioned filters are all horizontal or vertical. If the position to be interpolated is a sub-pixel in both the horizontal and vertical directions, it is necessary to superimpose horizontal and vertical filters.

[0516] It can be understood that in the present application, in order to solve the problem of increased bandwidth, the reference pixels corresponding to the current sub-block can be expanded first to obtain expanded reference pixels; then based on the expanded reference pixels, the interpolation filter corresponding to the current sub-block is used to obtain the third prediction value.

[0517] Fig.23 Indicates the extension sub-block Figure 2 ,like Fig.23 As shown, in order not to increase the bandwidth, the outermost circle of the original interpolated reference pixels can also be expanded, for example, the 11x11 reference pixels can be expanded to 13x13 reference pixels, and the expansion method proposed in the above step 304 can be used. In this way, a unified filter can be used.

[0518] Exemplarily, another method without increasing the bandwidth is proposed, which is not to expand the pixels, but to choose to use pixels in a nearby range instead when the pixels to be used by the filter exceed the original pixel range.

[0519] It is understandable that, in the present application, in order to solve the problem of increased bandwidth, the adjacent pixel positions corresponding to the extended sub-block may be first determined in the current block; and then the third prediction value may be determined according to the integer pixel values ​​of the adjacent pixel positions.

[0520] Fig.24 is a schematic diagram of replacing pixels, such as Fig.24As shown, in order not to increase the bandwidth, the value of the nearest integer pixel can also be directly used as the prediction value that needs to be used and is beyond the sub-block. Specifically, the pixel value of the square 3 can be directly used as the prediction value of the circle 2. If the sub-pixel MV in the horizontal direction is less than or equal to (or less than) 1 / 2 pixel, the integer pixel on the left of the sub-pixel is used, if the sub-pixel MV in the horizontal direction is greater than (or greater than or equal to) 1 / 2 pixel, the integer pixel on the right of the sub-pixel is used, if the sub-pixel MV in the vertical direction is less than or equal to (or less than) 1 / 2 pixel, the integer pixel above the sub-pixel is used, if the sub-pixel MV in the vertical direction is greater than (or greater than or equal to) 1 / 2 pixel, the integer pixel below the sub-pixel is used.

[0521] In the embodiments of the present application, further, Fig.25 Schematic diagram of the implementation process of the inter-frame prediction method Figure 4 ,like Fig.25 As shown, after determining the first prediction value of the current sub-block based on the first motion vector and determining the target pixel position corresponding to the current pixel position, that is, after step 303, the method for the decoder to perform inter-frame prediction may further include the following steps:

[0522] Step 3012: If the target pixel position does not belong to the current sub-block, determine a neighboring pixel position in the current block; wherein the neighboring pixel position is adjacent to the target pixel position in the current block.

[0523] Step 3013: determine a second prediction value based on the first prediction value and the neighboring pixel positions, and determine the second prediction value as the inter-frame prediction value.

[0524] In an embodiment of the present application, after the decoder determines the target pixel position corresponding to the current pixel position, if the target pixel position does not belong to the current sub-block, that is, the current pixel position and the target pixel position are not in the same sub-block, then the decoder can first determine the neighboring pixel position that is closest to the target pixel position among all the pixel positions in the current block; wherein the neighboring pixel position is adjacent to the target pixel position in the current block.

[0525] Further, in an embodiment of the present application, if the pixel value required for secondary prediction or PROF processing is the pixel value of the target pixel position outside the current sub-block, the decoder can re-determine the pixel value. Specifically, the decoder can calculate and obtain a neighboring pixel position in the current block that is closest to the target pixel position, and then use the pixel value of the neighboring pixel position for secondary prediction or PROF processing.

[0526] For example, in this application, Fig.26 is a schematic diagram of alternative pixel positions, such as Fig.26As shown, there is no pixel position in sub-block 2 that is adjacent to the left of position 1 in sub-block 1. If the current secondary prediction method is used, position 3 in sub-block 2 will be selected as the position one pixel away from the left of position 1 in sub-block 1. It can be seen that position 1 and position 3 are far apart. If position 3 is used as the adjacent pixel position of position 1 for filtering, the prediction effect of the secondary prediction obtained in the end will be greatly reduced. At this time, position 4 can be selected in the current block to replace position 3, that is, position 4 is the adjacent pixel position of position 2. Specifically, position 4 is the pixel position closest to position 2, and the deviation between the two is small. Using position 4 as the adjacent pixel position of position 1 for filtering will obtain more accurate results than using position 3.

[0527] It can be understood that in an embodiment of the present application, when the decoder determines the neighboring pixel position in other sub-blocks in the current block, it needs to use the first motion vector of the current sub-block for calculation, and the result obtained may also be that the neighboring pixel position does not exist in other sub-blocks in the current block. If the neighboring pixel position does not exist, the decoder can choose to continue using the pixel value of the target pixel position for secondary prediction or PROF processing.

[0528] That is to say, the inter-frame prediction method proposed in the embodiment of the present application, after the prediction based on the sub-block, if the pixel position required for the secondary prediction or PROF processing is not in the same sub-block, in order to avoid the decrease in prediction performance due to the disconnection of pixels, based on the principle of limiting the pixel position used to the same sub-block, the decoder can update the target pixel position that does not belong to the current sub-block to obtain the updated pixel position, so that the secondary prediction or PROF processing can be performed based on the updated pixel position. Among them, the above-mentioned update process can be understood as the extension of the boundary of the current sub-block, and can also be understood as the redefinition of the target pixel position beyond the boundary of the current sub-block, so as to limit the pixel position required for the secondary prediction or PROF processing to the same sub-block, thereby improving the prediction performance by ensuring the connection between pixels. Among them, the inter-frame prediction method can act on the current block, each sub-block in the current block, and each pixel position in any sub-block. It can be seen that the inter-frame prediction method proposed in the embodiment of the present application can reduce the prediction error and thus improve the encoding performance by avoiding taking obviously non-adjacent pixels in the reference image as adjacent pixels for secondary prediction or PROF processing.

[0529] The present embodiment provides an inter-frame prediction method. After the prediction based on the sub-block, if the pixel positions required for the secondary prediction or PROF processing are not in the same sub-block, the pixel positions required for the secondary prediction or PROF processing can be restricted to the same sub-block by extending the boundary of the current sub-block, redefining the target pixel positions beyond the boundary of the current sub-block, and other methods. This solves the problem of reduced prediction performance due to disconnected pixels, reduces prediction errors, greatly improves encoding performance, and thus improves encoding and decoding efficiency.

[0530] The embodiment of the present application provides an inter-frame prediction method, which is applied to a video encoding device, i.e., an encoder. The function implemented by the method can be implemented by calling a computer program by a second processor in the encoder, and of course the computer program can be stored in a second memory. It can be seen that the encoder includes at least a second processor and a second memory.

[0531] Furthermore, in an embodiment of the present application, the method for an encoder to perform inter-frame prediction may include the following steps:

[0532] Step 401: Determine prediction mode parameters of the current block.

[0533] In an embodiment of the present application, the encoder may first determine the prediction mode parameters of the current block. Specifically, the encoder may first determine the prediction mode used by the current block, and then determine the corresponding prediction mode parameters based on the prediction mode. The prediction mode parameters may be used to determine the prediction mode used by the current block.

[0534] It should be noted that, in the embodiment of the present application, the image to be encoded can be divided into multiple image blocks, the current image block to be encoded can be called the current block, and the image blocks adjacent to the current block can be called adjacent blocks; that is, in the image to be encoded, there is an adjacent relationship between the current block and the adjacent blocks. Here, each current block may include a first image component, a second image component, and a third image component; that is, the current block is an image block in the image to be encoded that is currently to be predicted for the first image component, the second image component, or the third image component.

[0535] Among them, assuming that the current block predicts the first image component, and the first image component is a luminance component, that is, the image component to be predicted is a luminance component, then the current block can also be called a luminance block; or, assuming that the current block predicts the second image component, and the second image component is a chrominance component, that is, the image component to be predicted is a chrominance component, then the current block can also be called a chrominance block.

[0536] It should be noted that, in the embodiments of the present application, the prediction mode parameters indicate the prediction mode used by the current block and the parameters related to the prediction mode. Here, for determining the prediction mode parameters, a simple decision strategy can be adopted, such as determining according to the size of the distortion value; or a complex decision strategy can be adopted, such as determining according to the result of rate distortion optimization (RDO), which is not limited in the embodiments of the present application. Generally speaking, the RDO method can be used to determine the prediction mode parameters of the current block.

[0537] Specifically, in some embodiments, when determining the prediction mode parameters of the current block, the encoder may first perform pre-encoding processing on the current block using multiple prediction modes to obtain rate-distortion cost values ​​corresponding to each prediction mode; then select the minimum rate-distortion cost value from the multiple rate-distortion cost values ​​obtained, and determine the prediction mode parameters of the current block according to the prediction mode corresponding to the minimum rate-distortion cost value.

[0538] That is to say, on the encoder side, multiple prediction modes can be used for the current block to perform precoding processing on the current block respectively. Here, the multiple prediction modes generally include inter-frame prediction mode, traditional intra-frame prediction mode and non-traditional intra-frame prediction mode; among which, the traditional intra-frame prediction mode may include direct current (DC) mode, plane (PLANAR) mode and angle mode, etc., the non-traditional intra-frame prediction mode may include matrix-based intra-frame prediction (Matrix-based IntraPrediction, MIP) mode, cross-component linear model prediction (Cross-componentLinearmodelPrediction, CCLM) mode, intra-frame block copy (Intra Block Copy, IBC) mode and PLT (Palette) mode, etc., and the inter-frame prediction mode may include ordinary inter-frame prediction mode, GPM mode and AWP mode, etc.

[0539] In this way, after pre-encoding the current block using multiple prediction modes, the rate-distortion cost value corresponding to each prediction mode can be obtained; then the minimum rate-distortion cost value is selected from the multiple rate-distortion cost values ​​obtained, and the prediction mode corresponding to the minimum rate-distortion cost value is determined as the prediction mode parameter of the current block. In addition, after pre-encoding the current block using multiple prediction modes, the distortion value corresponding to each prediction mode can be obtained; then the minimum distortion value is selected from the multiple distortion values ​​obtained, and then the prediction mode corresponding to the minimum distortion value is determined as the prediction mode used by the current block, and the corresponding prediction mode parameters are set according to the prediction mode. In this way, the current block is finally encoded using the determined prediction mode parameters, and in this prediction mode, the prediction residual can be made smaller, which can improve the coding efficiency.

[0540] That is to say, on the encoding side, the encoder can select the optimal prediction mode to pre-encode the current block. In this process, the prediction mode of the current block can be determined, and then the prediction mode parameters used to indicate the prediction mode can be determined, so that the corresponding prediction mode parameters are written into the bitstream and transmitted from the encoder to the decoder.

[0541] Accordingly, on the decoder side, the decoder can directly obtain the prediction mode parameters of the current block by parsing the bitstream, and determine the prediction mode used by the current block and the relevant parameters corresponding to the prediction mode according to the prediction mode parameters obtained by parsing.

[0542] Step 402: When the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, determine a first motion vector of a current sub-block of the current block; wherein the current block includes a plurality of sub-blocks.

[0543] In an embodiment of the present application, if the prediction mode parameter indicates that the current block uses the inter-frame prediction mode to determine the inter-frame prediction value of the current block, the encoder may first determine the first motion vector of each sub-block of the current block, wherein one sub-block corresponds to one first motion vector.

[0544] It should be noted that in the embodiment of the present application, the current block is an image block to be encoded in the current frame, the current frame is encoded in the form of image blocks in a certain order, and the current block is the image block to be encoded at the next moment in the current frame in the order. The current block can have a variety of specifications, such as 16×16, 32×32 or 32×16, where the numbers represent the number of rows and columns of pixels on the current block.

[0545] Furthermore, in an embodiment of the present application, the current block can be divided into a plurality of sub-blocks, wherein each sub-block has the same size, and the sub-block is a set of pixels of a smaller size. The size of the sub-block can be 8×8 or 4×4.

[0546] Exemplarily, in the present application, the size of the current block is 16×16, and can be divided into 4 sub-blocks of 8×8 size each.

[0547] It can be understood that in the embodiment of the present application, when the encoder determines that the prediction mode parameter indicates the use of the inter-frame prediction mode to determine the inter-frame prediction value of the current block, the inter-frame prediction method provided in the embodiment of the present application can continue to be used.

[0548] In an embodiment of the present application, further, when the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, when the encoder determines the first motion vector of the current sub-block of the current block, the affine mode parameter and the prediction reference mode of the current block can be determined. When the affine mode parameter indicates that the affine mode is used, the control point mode and the sub-block size parameter are determined. Finally, the first motion vector can be determined according to the prediction reference mode, the control point mode and the sub-block size parameter.

[0549] It should be noted that, in the embodiment of the present application, the affine mode parameter is used to indicate whether to use the affine mode. Specifically, the affine mode parameter may be an affine motion compensation enable flag affine_enable_flag, and the encoder may further determine whether to use the affine mode by determining the value of the affine mode parameter.

[0550] That is, in the present application, the affine mode parameter may be a binary variable. If the value of the affine mode parameter is 1, it indicates that the affine mode is used; if the value of the affine mode parameter is 0, it indicates that the affine mode is not used.

[0551] It should be noted that, in the embodiment of the present application, the control point mode is used to determine the number of control points. In the affine model, a sub-block may have 2 control points or 3 control points, and accordingly, the control point mode may be a control point mode corresponding to 2 control points, or a control point mode corresponding to 3 control points. That is, the control point mode may include a 4-parameter mode and a 6-parameter mode.

[0552] It can be understood that in the embodiments of the present application, for the AVS3 standard, if the current block uses the affine mode, the encoder also needs to determine the number of control points of the current block in the affine mode, so as to determine whether the 4-parameter (2 control point) mode or the 6-parameter (3 control point) mode is used.

[0553] Further, in an embodiment of the present application, if the affine mode parameter determined by the encoder indicates to use the affine mode, the encoder may further determine a sub-block size parameter.

[0554] Specifically, the sub-block size parameter can be represented by the affine prediction sub-block size flag affine_subblock_size_flag, and the encoder can set the value of the sub-block size flag to indicate the sub-block size parameter, that is, to indicate the size of the sub-block of the current block. The size of the sub-block can be 8×8 or 4×4.

[0555] Exemplarily, in the present application, if the sub-block size parameter is 8×8, the sub-block size flag is set to 1 and the sub-block size flag is written into the code stream; if the sub-block size parameter is 4×4, the sub-block size flag is set to 0 and the sub-block size flag is written into the code stream.

[0556] Specifically, in an embodiment of the present application, the encoder may first determine a control point motion vector group according to a prediction reference mode; and then determine a first motion vector of the sub-block based on the control point motion vector group, the control point mode and the sub-block size parameter.

[0557] It can be understood that, in the embodiment of the present application, the control point motion vector group can be used to determine the motion vector of the control point.

[0558] It should be noted that in an embodiment of the present application, the encoder can traverse each sub-block in the current block according to the above method, and use the control point motion vector group, control point mode and sub-block size parameters of each sub-block to determine the first motion vector of each sub-block, so that a motion vector set can be constructed based on the first motion vector of each sub-block.

[0559] It can be understood that, in the embodiment of the present application, the motion vector set of the current block may include the first motion vector of each sub-block of the current block.

[0560] Furthermore, in an embodiment of the present application, when the encoder determines the first motion vector based on the control point motion vector group, the control point mode and the sub-block size parameters, it can first determine the difference variable based on the control point motion vector group, the control point mode and the size parameters of the current block; then it can determine the sub-block position based on the prediction mode parameters and the sub-block size parameters; finally, it can use the difference variable and the current sub-block position to determine the first motion vector of the current sub-block, and then obtain a set of motion vectors of multiple sub-blocks of the current block.

[0561] Step 403, determine the first prediction value of the current sub-block based on the first motion vector, and determine the target pixel position corresponding to the current pixel position; wherein the current pixel position is the position of a pixel point in the current sub-block, and the target pixel position is the position of the pixel point for secondary prediction or PROF processing of the pixel point at the current pixel position.

[0562] In an embodiment of the present application, after determining the first motion vector of each sub-block of the current block, the encoder may first determine the first prediction value of the current sub-block based on the first motion vector of the current sub-block, and then determine the target pixel position corresponding to the current pixel position in the current sub-block. The target pixel position is a pixel position adjacent to the current pixel position.

[0563] It should be noted that, in the embodiment of the present application, the current pixel position is the position of a pixel in the current sub-block of the current block, wherein the current pixel position can represent the position of the pixel to be processed. Specifically, the current pixel position can be the position of the pixel to be secondary predicted, or the position of the pixel to be processed by PROF.

[0564] It can be understood that, in the embodiment of the present application, step 403 may specifically include:

[0565] Step 403a: Determine a first prediction value of the sub-block based on the first motion vector.

[0566] Step 403b: Determine the target pixel position corresponding to the current pixel position.

[0567] Among them, the inter-frame prediction method proposed in the embodiment of the present application does not limit the order in which the encoder executes step 403a and step 403b. That is to say, in the present application, after determining the first motion vector of each sub-block of the current block, the encoder can first execute step 403a and then step 403b, or first execute step 403b and then step 403a, or execute step 403a and step 403b at the same time.

[0568] Step 404: If the target pixel position does not belong to the current sub-block, the target pixel position is updated according to the current sub-block to obtain an updated pixel position.

[0569] In an embodiment of the present application, after the encoder determines the first prediction value of the current sub-block based on the first motion vector and determines the target pixel position corresponding to the current pixel position, if the target pixel position does not belong to the current sub-block, that is, the target pixel position and the current pixel position belong to different sub-blocks, then the encoder can update the target pixel position according to the current sub-block to obtain the updated pixel position.

[0570] Further, in an embodiment of the present application, if the target pixel position does not belong to the current sub-block, the encoder updates the target pixel position according to the current sub-block, and before obtaining the updated pixel position, that is, before step 404, the method for the encoder to perform inter-frame prediction may further include the following steps:

[0571] Step 406: Determine whether the current sub-block satisfies a preset restriction condition; wherein the preset restriction condition is used to determine whether to restrict the target pixel position within the current sub-block.

[0572] Step 407: If the preset restriction condition is met, determine whether the target pixel position belongs to the current sub-block.

[0573] Step 408: If the preset restriction condition is not met, a second prediction value is determined based on the first prediction value and the target pixel position, and the second prediction value is determined as an inter-frame prediction value.

[0574] In an embodiment of the present application, the encoder may first determine whether the current sub-block satisfies a preset restriction condition. If the preset restriction condition is satisfied, the encoder may continue to determine whether the target pixel position belongs to the current sub-block, thereby further determining whether to perform restriction processing on the target pixel position, that is, whether to perform an update processing process on the target pixel position; if the preset restriction condition is not satisfied, the encoder may not need to perform restriction processing on the target pixel position, but directly determine the corrected prediction value corresponding to the current pixel position based on the first prediction value and the target pixel position, and after traversing each pixel position in the current sub-block and obtaining the corrected prediction value corresponding to each pixel position, determine the second prediction value corresponding to the current sub-block, thereby determining the inter-frame prediction value of the current sub-block.

[0575] Further, in the embodiment of the present application, the encoder may determine whether the current sub-block meets the preset restriction condition in a variety of ways. Specifically, the encoder may further determine whether the preset restriction condition is met through the motion vector.

[0576] Exemplarily, in the present application, when the encoder determines whether the current sub-block meets the preset restriction condition, it can first determine the first motion vector deviation based on the control point motion vector group; then compare the first motion vector deviation with the preset deviation threshold; if the first motion vector deviation is greater than or equal to the preset deviation threshold, then it can be determined that the preset restriction condition is met; if the first motion vector deviation is less than the preset deviation threshold, then it is determined that the preset restriction condition is not met.

[0577] Exemplarily, in the present application, when the encoder determines whether the current sub-block meets the preset restriction condition, it can first determine the second motion vector deviation based on the first motion vector of each sub-block; then compare the second motion vector deviation with the preset deviation threshold; if the second motion vector deviation is greater than or equal to the preset deviation threshold, then it is determined that the preset restriction condition is met; if the second motion vector deviation is less than the preset deviation threshold, then it is determined that the preset restriction condition is not met.

[0578] Exemplarily, in the present application, when the encoder determines whether the current sub-block satisfies the preset restriction condition, it can first determine the third motion vector deviation between the preset pixel position in the current sub-block and the current sub-block; then compare the third motion vector deviation with the preset deviation threshold; if the third motion vector deviation is greater than or equal to the preset deviation threshold, then it is determined that the preset restriction condition is met; if the third motion vector deviation is less than the preset deviation threshold, then it is determined that the preset restriction condition is not met.

[0579] It should be noted that, in the present application, the preset deviation threshold may be determined according to the size parameter of the current block and / or the size parameter of the sub-block.

[0580] It can be understood that in the present application, when the encoder updates the target pixel position according to the current sub-block to obtain the updated pixel position, it can first expand the current sub-block to obtain the expanded sub-block; then the updated pixel position corresponding to the target pixel position can be determined in the expanded sub-block.

[0581] It can be understood that in an embodiment of the present application, when the encoder performs expansion processing on the current sub-block to obtain the expanded sub-block, it can choose to use all boundary positions of the current sub-block for expansion processing to obtain the expanded sub-block; it can also choose to use the boundary positions of the rows and / or columns corresponding to the target pixel positions in the current sub-block for expansion processing to obtain the expanded sub-block.

[0582] Step 405: Determine a second prediction value corresponding to the current sub-block based on the first prediction value and the updated pixel position, and determine the second prediction value as the inter-frame prediction value of the current sub-block.

[0583] In an embodiment of the present application, if the target pixel position does not belong to the current sub-block, the encoder updates the target pixel position according to the current sub-block. After obtaining the updated pixel position, the encoder can determine the corrected prediction value corresponding to the current pixel position based on the first prediction value and the updated pixel position, and after traversing each pixel position in the current sub-block and obtaining the corrected prediction value corresponding to each pixel position, determine the second prediction value corresponding to the current sub-block, and then the second prediction value can be determined as the inter-frame prediction value of the current sub-block.

[0584] Further, in an embodiment of the present application, a method for an encoder to determine a second prediction value corresponding to a current sub-block based on the first prediction value and the updated pixel position may include the following steps:

[0585] Step 405a, determining PROF parameters;

[0586] Step 405b: when the PROF parameter indicates to perform PROF processing, determine the pixel horizontal gradient and the pixel vertical gradient between the current pixel position and the updated pixel position based on the first prediction value;

[0587] Step 405c, determining a fourth motion vector deviation between the updated pixel position and the current sub-block;

[0588] Step 405d, calculating a deviation value corresponding to the current pixel position according to the pixel horizontal gradient, the pixel vertical gradient and the fourth motion vector deviation;

[0589] Step 405e: Obtain a second prediction value based on the first prediction value and the deviation value.

[0590] Further, in an embodiment of the present application, the method for the encoder to determine the second prediction value corresponding to the current sub-block based on the first prediction value and the updated pixel position may include the following steps:-

[0591] Step 405f, determining secondary prediction parameters;

[0592] Step 405g: when the secondary prediction parameter indicates to use secondary prediction, determine a fourth motion vector deviation between the updated pixel position and the current sub-block;

[0593] Step 405h, determining a filter coefficient of a two-dimensional filter according to the fourth motion vector deviation; wherein the two-dimensional filter is used to perform secondary prediction processing according to a preset shape;

[0594] Step 405i: determine a second prediction value based on the filter coefficient and the first prediction value, and determine the second prediction value as an inter-frame prediction value.

[0595] In summary, through the inter-frame prediction method proposed in steps 401 to 408, after the prediction based on the sub-block, if the pixel positions required for the secondary prediction or PROF processing are not in the same sub-block, in order to avoid the decrease in prediction performance due to the disconnection of pixels, the encoder can update the target pixel positions that do not belong to the current sub-block to obtain the updated pixel positions, so that the secondary prediction or PROF processing can be performed based on the updated pixel positions. Among them, the above-mentioned update process can be understood as the extension of the boundary of the current sub-block, and can also be understood as the redefinition of the target pixel position that exceeds the boundary of the current sub-block, so as to limit the pixel positions required for the secondary prediction or PROF processing to the same sub-block, thereby improving the prediction performance by ensuring the connection between pixels.

[0596] Furthermore, in an embodiment of the present application, the method for the encoder to perform inter-frame prediction may further include the following steps:

[0597] Step 409: When the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, determine the extended sub-block of the current sub-block.

[0598] Step 4010: determine a third prediction value of the extended sub-block based on the first motion vector, and determine a target pixel position corresponding to the current pixel position.

[0599] Step 4011: If the target pixel position does not belong to the current sub-block, determine a second prediction value based on the third prediction value and the target pixel position, and determine the second prediction value as an inter-frame prediction value.

[0600] Further, in an embodiment of the present application, before executing step 4010a, that is, before determining the third prediction value of the extended sub-block based on the first motion vector, the encoder may also first determine whether the current sub-block meets the preset restriction conditions according to the method proposed in the above steps 406 to 408. Only when the preset restriction conditions are met, the third prediction value of the extended sub-block will be determined, and the third prediction value will be used for secondary prediction or PROF processing to obtain the inter-frame prediction value of the current sub-block; if the current sub-block does not meet the preset restriction conditions, the decoder can directly use the first prediction value of the current sub-block and the target pixel position to determine the second prediction value, and then determine the second prediction value as the inter-frame prediction value.

[0601] That is to say, in the present application, the preset restriction condition can also be used to determine whether to perform a third prediction value determination of the extended sub-block on the current sub-block.

[0602] It is understandable that in the present application, the problem of increased bandwidth can be solved by using a filter with fewer taps for the extended sub-block, for example, using an n-tap interpolation filter for the extended sub-block to obtain a third prediction value; wherein n is any one of the following values: 6, 5, 4, 3, 2. That is to say, in the present application, using filters with fewer taps, such as 6 or 5 or 4 or 3 or 2 tap filters, can keep within this range and thus not increase the bandwidth. The 2-tap filter can be the average or weighted average of the two adjacent integer pixels of the sub-pixel point. The above-mentioned filters are all horizontal or vertical. If the position to be interpolated is sub-pixel in both the horizontal and vertical directions, it is necessary to superimpose horizontal and vertical filters.

[0603] It is understandable that in the present application, in order to solve the problem of increased bandwidth, the reference pixels corresponding to the current sub-block can be expanded first to obtain the expanded reference pixels; then, based on the expanded reference pixels, the interpolation filter corresponding to the current sub-block is used to obtain the third prediction value. That is to say, in the present application, the outermost circle of the original interpolated reference pixels can be expanded, for example, the 11x11 reference pixels can be expanded to 13x13 reference pixels. Specifically, the expansion method proposed in the above step 304 can be adopted. In this way, a unified filter can be used. At the same time, another method that does not increase the bandwidth is proposed. Instead of expanding the pixels, when the pixels that the filter needs to use exceed the original pixel range, pixels in the nearest range are selected to be used instead.

[0604] It can be understood that, in the present application, in order to solve the problem of increased bandwidth, the adjacent pixel positions corresponding to the extended sub-block can also be determined in the current block first; and then the third prediction value can be determined according to the integer pixel values ​​of the adjacent pixel positions. That is, in the present application, the value of the most adjacent integer pixel can also be directly used as the prediction value that needs to be used and exceeds the sub-block. Specifically, the pixel value of the square 3 can be directly used as the prediction value of the circle 2. If the sub-pixel MV in the horizontal direction is less than or equal to (or less than) 1 / 2 pixel, the integer pixel on the left of the sub-pixel is used. If the sub-pixel MV in the horizontal direction is greater than (or greater than or equal to) 1 / 2 pixel, the integer pixel on the left of the sub-pixel is used. If the sub-pixel MV in the vertical direction is less than or equal to (or less than) 1 / 2 pixel, the integer pixel on the top of the sub-pixel is used. If the sub-pixel MV in the vertical direction is greater than (or greater than or equal to) 1 / 2 pixel, the integer pixel below the sub-pixel is used.

[0605] Further, in an embodiment of the present application, after determining the first prediction value of the current sub-block based on the first motion vector and determining the target pixel position corresponding to the current pixel position, that is, after step 203, the method for the encoder to perform inter-frame prediction may further include the following steps:

[0606] Step 4012: If the target pixel position does not belong to the current sub-block, determine a neighboring pixel position in the current block; wherein the neighboring pixel position is adjacent to the target pixel position in the current block.

[0607] Step 4013: determine a second prediction value based on the first prediction value and the neighboring pixel positions, and determine the second prediction value as the inter-frame prediction value.

[0608] It should be noted that the inter-frame prediction method proposed in the embodiment of the present application can be applied to the entire coding unit or prediction unit, that is, to the current block, to each sub-block in the current block, or to each pixel position in any sub-block. This application does not make any specific limitation.

[0609] It is understandable that in the present application, the encoder may write prediction mode parameters, affine mode parameters, and prediction reference mode into the bitstream, and may also write PROF parameters and secondary prediction parameters into the bitstream.

[0610] The present embodiment provides an inter-frame prediction method. After the prediction based on the sub-block, if the pixel positions required for the secondary prediction or PROF processing are not in the same sub-block, the pixel positions required for the secondary prediction or PROF processing can be restricted to the same sub-block by extending the boundary of the current sub-block, redefining the target pixel positions beyond the boundary of the current sub-block, and other methods. This solves the problem of reduced prediction performance due to disconnected pixels, reduces prediction errors, greatly improves encoding performance, and thus improves encoding and decoding efficiency.

[0611] Based on the above embodiment, in yet another embodiment of the present application, Fig. 27 The structure of the decoder is shown in Figure 2. Figure 1 ,like Fig. 27 As shown, the decoder 300 proposed in the embodiment of the present application may include a parsing part 301 , a first determining part 302 and a first updating part 303 .

[0612] The parsing part 301 is configured to parse the bitstream and obtain the prediction mode parameters of the current block;

[0613] The first determination part 302 is configured to determine a first motion vector of a current subblock of the current block when the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode; wherein the current block includes a plurality of subblocks; determine the first prediction value of the current subblock based on the first motion vector, and determine a target pixel position corresponding to a current pixel position; wherein the current pixel position is a position of a pixel point in the current subblock, and the target pixel position is a position of a pixel point for which a secondary prediction or PROF processing is performed on the pixel point at the current pixel position;

[0614] The first updating part 303 is configured to update the target pixel position according to the current sub-block to obtain an updated pixel position if the target pixel position does not belong to the current sub-block;

[0615] The first determining part 302 is further configured to determine a second prediction value corresponding to the current sub-block based on the first prediction value and the updated pixel position, and determine the second prediction value as the inter-frame prediction value of the current sub-block.

[0616] Fig.28 The structure of the decoder is shown in Figure 2. Figure 2 ,like Fig.28 As shown, the decoder 300 proposed in the embodiment of the present application may also include a first processor 304, a first memory 305 storing executable instructions of the first processor 304, a first communication interface 306, and a first bus 307 for connecting the first processor 304, the first memory 305 and the first communication interface 306.

[0617] Further, in an embodiment of the present application, the above-mentioned first processor 304 is used to parse the code stream and obtain the prediction mode parameters of the current block; when the prediction mode parameters indicate that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode, determine the first motion vector of the current sub-block of the current block; wherein the current block includes multiple sub-blocks; determine the first prediction value of the current sub-block based on the first motion vector, and determine the target pixel position corresponding to the current pixel position; wherein the current pixel position is the position of a pixel point in the current sub-block, and the target pixel position is the position of the pixel point for performing secondary prediction or PROF processing on the pixel point at the current pixel position; if the target pixel position does not belong to the current sub-block, update the target pixel position according to the current sub-block to obtain an updated pixel position; based on the first prediction value and the updated pixel position, determine the second prediction value corresponding to the current sub-block, and determine the second prediction value as the inter-frame prediction value of the current sub-block.

[0618] Fig.29 The structure of the encoder is shown in Figure 2. Figure 1 ,like Fig.29 As shown, the encoder 400 proposed in the embodiment of the present application may include a second determining part 401 and a second updating part 402.

[0619] The second determination part 401 is configured to determine a prediction mode parameter of the current block; when the prediction mode parameter indicates that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode, determine a first motion vector of a current sub-block of the current block; wherein the current block includes a plurality of sub-blocks; determine the first prediction value of the current sub-block based on the first motion vector, and determine a target pixel position corresponding to a current pixel position; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for which a secondary prediction or PROF processing is performed on the pixel point at the current pixel position;

[0620] The second updating part 402 is configured to update the target pixel position according to the current sub-block to obtain an updated pixel position if the target pixel position does not belong to the current sub-block;

[0621] The second determination part 401 is further configured to determine a second prediction value corresponding to the current sub-block based on the first prediction value and the updated pixel position, and determine the second prediction value as the inter-frame prediction value of the current sub-block.

[0622] Fig.30 The structure of the encoder is shown in Figure 2. Figure 2 ,like Fig.30 As shown, the encoder 400 proposed in the embodiment of the present application may also include a second processor 403, a second memory 404 storing executable instructions of the second processor 403, a second communication interface 405, and a second bus 406 for connecting the second processor 403, the second memory 404 and the second communication interface 405.

[0623] Further, in an embodiment of the present application, the above-mentioned second processor 403 is configured to determine a prediction mode parameter of the current block; when the prediction mode parameter indicates that the inter-frame prediction value of the current block is determined using the inter-frame prediction mode, determine the first motion vector of the current sub-block of the current block; wherein the current block includes multiple sub-blocks; determine the first prediction value of the current sub-block based on the first motion vector, and determine the target pixel position corresponding to the current pixel position; wherein the current pixel position is the position of a pixel point in the current sub-block, and the target pixel position is the position of the pixel point for which secondary prediction or PROF processing is performed on the pixel point at the current pixel position; if the target pixel position does not belong to the current sub-block, update the target pixel position according to the current sub-block to obtain an updated pixel position; based on the first prediction value and the updated pixel position, determine the second prediction value corresponding to the current sub-block, and determine the second prediction value as the inter-frame prediction value of the current sub-block.

[0624] The embodiments of the present application provide a decoder and an encoder. After the prediction based on the sub-block, if the pixel positions required for the secondary prediction or PROF processing are not in the same sub-block, the pixel positions required for the secondary prediction or PROF processing can be restricted to the same sub-block by extending the boundary of the current sub-block, redefining the target pixel positions beyond the boundary of the current sub-block, and other methods, thereby solving the problem of reduced prediction performance due to disconnected pixels, reducing the prediction error, greatly improving the encoding performance, and thus improving the encoding and decoding efficiency.

[0625] The embodiments of the present application provide a computer-readable storage medium and a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the method described in the above embodiments is implemented.

[0626] Specifically, the program instructions corresponding to an inter-frame prediction method in this embodiment may be stored in a storage medium such as an optical disk, a hard disk, or a USB flash drive. When the program instructions corresponding to an inter-frame prediction method in the storage medium are read or executed by an electronic device, the following steps are included:

[0627] Parse the bitstream and obtain the prediction mode parameters of the current block;

[0628] When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a current sub-block of the current block; wherein the current block includes a plurality of sub-blocks;

[0629] Determine a first prediction value of the current sub-block based on the first motion vector, and determine a target pixel position corresponding to a current pixel position; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for which a secondary prediction or PROF processing is performed on a pixel point at the current pixel position;

[0630] If the target pixel position does not belong to the current sub-block, updating the target pixel position according to the current sub-block to obtain an updated pixel position;

[0631] Based on the first prediction value and the updated pixel position, a second prediction value corresponding to the current sub-block is determined, and the second prediction value is determined as the inter-frame prediction value of the current sub-block.

[0632] Specifically, the program instructions corresponding to an inter-frame prediction method in this embodiment may be stored in a storage medium such as an optical disk, a hard disk, or a USB flash drive. When the program instructions corresponding to an inter-frame prediction method in the storage medium are read or executed by an electronic device, the following steps are included:

[0633] Determining prediction mode parameters for the current block;

[0634] When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a current sub-block of the current block; wherein the current block includes a plurality of sub-blocks;

[0635] Determine a first prediction value of the current sub-block based on the first motion vector, and determine a target pixel position corresponding to a current pixel position; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for which a secondary prediction or PROF processing is performed on a pixel point at the current pixel position;

[0636] If the target pixel position does not belong to the current sub-block, updating the target pixel position according to the current sub-block to obtain an updated pixel position;

[0637] Based on the first prediction value and the updated pixel position, a second prediction value corresponding to the current sub-block is determined, and the second prediction value is determined as the inter-frame prediction value of the current sub-block.

[0638] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0639] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0640] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0641] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0642] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0643] Industrial Applicability

[0644] The embodiments of the present application provide an inter-frame prediction method, an encoder, a decoder and a computer storage medium. The decoder parses a bit stream to obtain a prediction mode parameter of a current block; when the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, a first motion vector of a current sub-block of the current block is determined; wherein the current block includes multiple sub-blocks; based on the first motion vector, a first prediction value of the current sub-block is determined, and a target pixel position corresponding to the current pixel position is determined; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for performing secondary prediction or PROF processing on the pixel point at the current pixel position; if the target pixel position does not belong to the current sub-block, the target pixel position is updated according to the current sub-block to obtain an updated pixel position; based on the first prediction value and the updated pixel position, a second prediction value corresponding to the current sub-block is determined, and the second prediction value is determined as the inter-frame prediction value of the current sub-block. That is to say, the inter-frame prediction method proposed in the present application, after the prediction based on the sub-block, if the pixel positions required for the secondary prediction or PROF processing are not in the same sub-block, the pixel positions required for the secondary prediction or PROF processing can be restricted to the same sub-block by extending the boundary of the current sub-block, redefining the target pixel position beyond the boundary of the current sub-block, and other methods, thereby solving the problem of reduced prediction performance due to disconnected pixels, reducing the prediction error, greatly improving the encoding performance, and thus improving the encoding and decoding efficiency.

Claims

1. An inter-frame prediction method, applied to a decoder, the method comprising: Parse the bitstream and obtain the prediction mode parameters of the current block; When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a current sub-block of the current block; wherein the current block includes a plurality of sub-blocks; Determine a first prediction value of the current sub-block based on the first motion vector, and determine a target pixel position corresponding to a current pixel position; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for secondary prediction of a pixel point at the current pixel position; If the target pixel position does not belong to the current sub-block, updating the target pixel position according to the current sub-block to obtain an updated pixel position; Determine, based on the first prediction value and the updated pixel position, a second prediction value corresponding to the current sub-block, and determine the second prediction value as the inter-frame prediction value of the current sub-block; Wherein, the method further comprises: When the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, determining an extended sub-block of the current sub-block; Determine a third prediction value of the extended sub-block based on the first motion vector, and determine the target pixel position corresponding to the current pixel position; If the target pixel position does not belong to the current sub-block, the second prediction value is determined based on the third prediction value and the target pixel position, and the second prediction value is determined as the inter-frame prediction value.

2. The method according to claim 1, wherein: The updating process of the target pixel position according to the current sub-block to obtain an updated pixel position includes: Performing expansion processing on the current sub-block to obtain an expanded sub-block; The updated pixel position corresponding to the target pixel position is determined in the expanded sub-block.

3. The method according to claim 1, wherein: Determining adjacent pixel positions corresponding to the extended sub-block; The third prediction value is determined based on the integer pixel values ​​of the adjacent pixel positions.

4. The method according to claim 1, wherein: The determining a first motion vector of a current sub-block of the current block comprises: Parsing the bitstream to obtain affine mode parameters and prediction reference mode of the current block; When the affine mode parameter indicates to use the affine mode, determining a control point mode and a sub-block size parameter; The first motion vector is determined according to the prediction reference mode, the control point mode and the sub-block size parameter.

5. The method according to claim 4, wherein: The determining the first motion vector according to the prediction reference mode, the control point mode and the sub-block size parameter includes: Determine a control point motion vector group according to the prediction reference mode; The first motion vector is determined according to the control point motion vector group, the control point mode and the sub-block size parameter.

6. The method according to claim 5, wherein: The determining the first motion vector according to the control point motion vector group, the control point mode and the sub-block size parameter comprises: Determining a difference variable according to the control point motion vector group, the control point mode and a size parameter of the current block; Determining a sub-block position based on the prediction mode parameter and the sub-block size parameter; The first motion vector of the current sub-block is determined by using the difference variable and the sub-block position.

7. The method according to any one of claims 4 to 6, wherein: The method further comprises: Each sub-block of the current block is traversed, and a motion vector set is constructed according to the first motion vector of each sub-block.

8. The method according to claim 7, wherein: The determining a first prediction value of the current sub-block based on the first motion vector comprises: Determine a sample matrix; wherein the sample matrix includes a brightness sample matrix and a chrominance sample matrix; The first prediction value is determined according to the prediction reference mode, the sub-block size parameter, the sample matrix and the motion vector set.

9. An inter-frame prediction method, applied to an encoder, the method comprising: Determining prediction mode parameters for the current block; When the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determining a first motion vector of a current sub-block of the current block; wherein the current block includes a plurality of sub-blocks; Determine a first prediction value of the current sub-block based on the first motion vector, and determine a target pixel position corresponding to a current pixel position; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for secondary prediction of a pixel point at the current pixel position; If the target pixel position does not belong to the current sub-block, updating the target pixel position according to the current sub-block to obtain an updated pixel position; Determine, based on the first prediction value and the updated pixel position, a second prediction value corresponding to the current sub-block, and determine the second prediction value as the inter-frame prediction value of the current sub-block; Wherein, the method further comprises: When the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, determining an extended sub-block of the current sub-block; Determine a third prediction value of the extended sub-block based on the first motion vector, and determine the target pixel position corresponding to the current pixel position; If the target pixel position does not belong to the current sub-block, the second prediction value is determined based on the third prediction value and the target pixel position, and the second prediction value is determined as the inter-frame prediction value.

10. The method according to claim 9, wherein: The updating process of the target pixel position according to the current sub-block to obtain an updated pixel position includes: Performing expansion processing on the current sub-block to obtain an expanded sub-block; The updated pixel position corresponding to the target pixel position is determined in the expanded sub-block.

11. The method according to claim 9, wherein: Determining adjacent pixel positions corresponding to the extended sub-block; The third prediction value is determined based on the integer pixel values ​​of the adjacent pixel positions.

12. The method according to claim 9, wherein: The determining a first motion vector of a current sub-block of the current block comprises: Determining affine mode parameters and a prediction reference mode of the current block; When the affine mode parameter indicates to use the affine mode, determining a control point mode and a sub-block size parameter; Determining the first motion vector according to the prediction reference mode, the control point mode and the sub-block size parameter; The determining the first motion vector according to the prediction reference mode, the control point mode and the sub-block size parameter includes: Determine a control point motion vector group according to the prediction reference mode; The first motion vector is determined according to the control point motion vector group, the control point mode and the sub-block size parameter.

13. The method according to claim 12, wherein: The determining the first motion vector according to the control point motion vector group, the control point mode and the sub-block size parameter comprises: Determining a difference variable according to the control point motion vector group, the control point mode and a size parameter of the current block; Determining a sub-block position based on the prediction mode parameter and the sub-block size parameter; The first motion vector of the current sub-block is determined by using the difference variable and the sub-block position.

14. The method according to claim 12 or 13, wherein: The method further comprises: Traversing each sub-block of the current block, and constructing a motion vector set according to the first motion vector of each sub-block; The determining the first prediction value of the current sub-block based on the first motion vector includes: Determine a sample matrix; wherein the sample matrix includes a brightness sample matrix and a chrominance sample matrix; The first prediction value is determined according to the prediction reference mode, the sub-block size parameter, the sample matrix and the motion vector set.

15. A decoder comprising a parsing part, a first determining part, a first updating part, The parsing part is configured to parse the bitstream and obtain the prediction mode parameters of the current block; The first determining part is configured to determine a first motion vector of a current subblock of the current block when the prediction mode parameter indicates that an inter-frame prediction mode is used to determine an inter-frame prediction value of the current block; wherein, The current block includes a plurality of sub-blocks; a first prediction value of the current sub-block is determined based on the first motion vector, and a target pixel position corresponding to a current pixel position is determined; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for secondary prediction of a pixel point at the current pixel position; The first updating part is configured to update the target pixel position according to the current sub-block to obtain an updated pixel position if the target pixel position does not belong to the current sub-block; The first determining part is further configured to determine a second prediction value corresponding to the current sub-block based on the first prediction value and the updated pixel position, and determine the second prediction value as the inter-frame prediction value of the current sub-block; Wherein, the first determining part is further configured as: When the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, determining an extended sub-block of the current sub-block; Determine a third prediction value of the extended sub-block based on the first motion vector, and determine the target pixel position corresponding to the current pixel position; If the target pixel position does not belong to the current sub-block, the second prediction value is determined based on the third prediction value and the target pixel position, and the second prediction value is determined as the inter-frame prediction value.

16. A decoder, comprising a first processor and a first memory storing instructions executable by the first processor, wherein when the instructions are executed, the first processor implements the method according to any one of claims 1 to 8.

17. An encoder comprising a second determining part, a second updating part, The second determination part is configured to determine a prediction mode parameter of the current block; when the prediction mode parameter indicates that an inter-frame prediction value of the current block is determined using an inter-frame prediction mode, determine a first motion vector of a current sub-block of the current block; wherein, The current block includes a plurality of sub-blocks; a first prediction value of the current sub-block is determined based on the first motion vector, and a target pixel position corresponding to a current pixel position is determined; wherein the current pixel position is a position of a pixel point in the current sub-block, and the target pixel position is a position of a pixel point for secondary prediction of a pixel point at the current pixel position; The second updating part is configured to update the target pixel position according to the current sub-block to obtain an updated pixel position if the target pixel position does not belong to the current sub-block; The second determining part is further configured to determine a second prediction value corresponding to the current sub-block based on the first prediction value and the updated pixel position, and determine the second prediction value as the inter-frame prediction value of the current sub-block; Wherein, the second determining part is further configured as: When the prediction mode parameter indicates that the inter-frame prediction mode is used to determine the inter-frame prediction value of the current block, determining an extended sub-block of the current sub-block; Determine a third prediction value of the extended sub-block based on the first motion vector, and determine the target pixel position corresponding to the current pixel position; If the target pixel position does not belong to the current sub-block, the second prediction value is determined based on the third prediction value and the target pixel position, and the second prediction value is determined as the inter-frame prediction value.

18. An encoder, comprising a second processor and a second memory storing instructions executable by the second processor, wherein when the instructions are executed, the second processor implements the method according to any one of claims 9 to 14.

Images