Optical flow-based video inter-frame prediction

Abstract

To provide a method and apparatus for bi-predictive inter prediction using optical flow.SOLUTION: A method includes: obtaining a pair of motion vectors for a current coding block of a video signal; generating predictions of the current block using the pair of motion vectors from respective reference frames; determining an optical flow for the current coding block based on sample values in the predictions; determining one component of the optical flow based on the other component of the optical flow; and executing Bi-predictive optical flow (BPOF) on the current coding block using the determined optical flow.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to Indian Provisional Application No. IN201931010751, filed March 19, 2019, entitled "Encoder, Decoder, and Corresponding Method for Optical Flow Based Inter-Frame Prediction," the contents of which are incorporated herein by reference in their entirety.

[0002] [Technical field] FIELD OF THE DISCLOSURE This disclosure relates to video encoding and decoding, and more particularly to methods and apparatus for bi-predictive inter-frame prediction using optical flow. [Background technology]

[0003] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time interactive applications such as video chat and video conferencing, DVDs and Blu-ray discs, video content collection and editing systems, and video cameras in security applications. The amount of video data required to represent even a relatively short video can be substantial, which can pose challenges when the data is streamed or otherwise communicated over communication networks with limited bandwidth capacity. Therefore, video data is typically compressed before being communicated over modern telecommunications networks. Video size can also be an issue when storing video on storage devices, because memory resources can be limited.

[0004] Video compression devices often use software and / or hardware at the source to code video data before transmission or storage, thereby reducing the amount of data required to represent a digital video image. The compressed data is then received at the destination by a video decompression device, which decodes the video data. Due to limited network resources and an ever-increasing demand for higher video quality, improved compression and decompression techniques that improve compression ratios without sacrificing much image quality are desirable.

[0005] In video compression, inter-frame prediction is the process of using reconstructed samples of a previously decoded reference frame by specifying motion vectors for the current block. These motion vectors can be coded as prediction residuals using spatial or temporal motion vector predictors. The motion vectors can be made sub-pixel accurate. An interpolation filter is applied to derive sub-pixel accurate pixel values ​​in the reference frame from the reconstructed integer position values.

[0006] Bidirectional prediction refers to a process in which a prediction for a current block is derived as a weighted combination of two prediction blocks derived using two motion vectors from two reference video regions. In this case, in addition to the motion vectors, it is also necessary to code the reference indexes of the reference videos from which the two prediction blocks are derived. It is also possible to derive a motion vector for the current block by a merging process in which spatially adjacent motion vectors and reference indexes are inherited without coding any motion vector residuals. In addition to spatial neighbors, motion vectors of previously coded reference frames are also stored, taking into account their distance to the reference frame relative to that of the current block, and are used as a temporal merging option with appropriate scaling of the motion vectors.

[0007] Bidirectional predictive optical flow (BPOF) is a sample-by-sample motion refinement performed in addition to block-by-block motion compensation for bidirectional prediction. Conventional estimation of optical flow involves complexity issues or compression efficiency gaps, so there is a need for improved devices and methods for inter-frame prediction based on optical flow. Summary of the Invention

[0008] The present disclosure provides an inter-frame prediction apparatus and method for encoding and decoding images by bidirectional inter-frame prediction using optical flow, which can improve the efficiency of video signals. The present disclosure is detailed by the examples and claims included in this application.

[0009] According to a first aspect, the present disclosure provides a method for bidirectional optical flow (BDOF) based inter-frame prediction for a current block of a video signal, the method comprising: the horizontal motion offset v of the current block x and vertical movement offset v y wherein the vertical motion offset is determined based on the horizontal motion offset and a fifth variable s5; the fifth variable s5 indicates a sum of a plurality of terms, each of which is obtained from a sign of an element of a second matrix and an element of a first matrix, and the elements of the first matrix correspond to the elements of the second matrix; Each element of the first matrix is ​​obtained from the sum of a first horizontal predicted sample gradient corresponding to a first reference frame of the current block and a second horizontal predicted sample gradient corresponding to a second reference frame of the current block, and the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to the element of the first matrix; each element of the second matrix is ​​obtained from the sum of a first vertical predicted sample gradient corresponding to the first reference frame of the current block and a second vertical predicted sample gradient corresponding to the second reference frame of the current block, and the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to the element of the second matrix; determining predicted sample values ​​in the current block using predicted sample values ​​corresponding to the first reference frame, predicted sample values ​​corresponding to the second reference frame, the horizontal motion offset, and the vertical motion offset.

[0010] It should be noted that the current block may be a block of any size, such as a 4x4 block. The current block may also be a sub-block of a frame of a video signal. A pixel of the current block may be referenced using the pixel's absolute position relative to the upper-left corner of the frame (e.g., the top-left pixel), e.g., (xBlock+i, yBlock+j), or the pixel's relative position with respect to the upper-left corner of the block (e.g., the top-left pixel), e.g., (xBlock, yBlock) where (xBlock, yBlock) are the coordinates of the upper-left corner of the block (e.g., the top-left pixel) with respect to the upper-left corner of the frame (e.g., the top-left pixel).

[0011] It can be understood that in this disclosure the terms "predicted pixel value / predicted sample value", "sample / pixel" and "sample location / pixel location" may be used interchangeably.

[0012] The first matrix and the second matrix may be any two-dimensional array including rows and columns, and elements of the array may be indicated using (i, j), where x is the horizontal / row index and y is the vertical / column index. The range of i and j may be, for example, i = xBlock-1, ..., xBlock+4 and j = yBlock-1, ..., yBlock+4. The first matrix and the second matrix correspond to the current block or are determined for the current block. In some examples, the size of the first matrix is ​​the same as the size of the second matrix, which may be larger than the size of the current block. For example, the size of the first matrix and the second matrix may be 6x6, while the size of the current block is 4x4.

[0013] If the position (x, y) of the first element in the first matrix is ​​the same as the position (p, q) of the second element in the second matrix, i.e., (x, y) = (p, q), then the element (first element) of the first matrix corresponds to the element (second element) of the second matrix. The first horizontal predicted sample gradient corresponds to the first reference frame of the current block, meaning that the first horizontal predicted sample gradient is generated based on samples in the first reference frame of the current block. The second horizontal predicted sample gradient corresponds to the second reference frame of the current block, meaning that the second horizontal predicted sample gradient is generated based on samples in the second reference frame of the current block. The first horizontal predicted sample gradient corresponds to an element of the first matrix, meaning that the first horizontal predicted sample gradient is generated for the position (x, y) of that element in the first matrix. Similarly, a second horizontal predicted sample gradient corresponds to an element of a first matrix, meaning that the second horizontal predicted sample gradient is generated for the element's position (x, y) in the first matrix.

[0014] The first vertical predicted sample gradient corresponding to the first reference frame of the current block means that the first vertical predicted sample gradient is generated based on samples in the first reference frame of the current block. The second vertical predicted sample gradient corresponding to the second reference frame of the current block means that the second vertical predicted sample gradient is generated based on samples in the second reference frame of the current block. The first vertical predicted sample gradient corresponding to an element of the second matrix means that the first vertical predicted sample gradient is generated for the position (p, q) of that element in the second matrix. Similarly, the second vertical predicted sample gradient corresponding to an element of the second matrix means that the second vertical predicted sample gradient is generated for the position (p, q) of that element in the second matrix.

[0015] When each element of a matrix is ​​obtained from the sum of two terms, it means that the element may be determined as the sum of the two terms itself, or as the value after processing the sum of the two terms. The processing may include left shifting, right shifting, clipping, or a combination thereof. Similarly, when a term is obtained from the sign of an element of a second matrix and an element of a first matrix, it means that the term may take the value of the element of the first matrix itself, or the value after the element of the first matrix is ​​processed, and then the sign of the element of the second matrix is ​​applied. The processing of the element of the first matrix may include left shifting, right shifting, clipping, or a combination thereof. The sign of element x is

number

[0016] The technique presented herein adjusts the bidirectionally predicted sample values ​​of the current block based on a horizontal motion offset and a vertical motion offset. The vertical motion offset is calculated based on a fifth variable s5, which merely involves the summation of terms obtained from the signs of elements of a second matrix and elements of a first matrix. Applying the sign of one element to another element does not involve a multiplication operation. Similarly, the summation also does not involve any multiplication. As a result, the BDOF-based inter-frame prediction technique presented herein eliminates multiplication operations. Compared to conventional approaches, the multiplication operations are replaced by sign decisions, thereby reducing the bit depth of the fifth variable s5. This means that the horizontal motion offset v x and vertical movement offset v y This leads to a reduction in the bit depth of the .DELTA..times ...

[0017] In one possible implementation form of a method according to any of the preceding implementations of the first aspect, the vertical motion offset is derived based on the horizontal motion offset, a second variable s2, a fourth variable s4, and the fifth variable s5, where the second variable s2 indicates a sum of absolute values ​​of elements of the second matrix, and the fourth variable s4 indicates a sum of multiple terms, each of which is obtained from a sign of an element of the second matrix and an element of a third matrix, the element of the third matrix corresponding to the element of the second matrix, and each element of the third matrix is ​​a difference obtained from a first predicted sample of the first reference frame corresponding to the element of the third matrix and a second predicted sample of the second reference frame corresponding to the element of the third matrix.

[0018] It should be noted that if the position (k, l) of the first element in the third matrix is ​​the same as the position (p, q) of the second element in the second matrix, i.e., (k, l) = (p, q), the element (first element) of the third matrix corresponds to the element (second element) of the second matrix. The fact that the first predicted sample corresponds to the first reference frame of the current block means that the first predicted sample exists in the first reference frame of the current block. The fact that the second predicted sample corresponds to the second reference frame of the current block means that the second predicted sample exists in the second reference frame of the current block. The fact that the first predicted sample corresponds to an element of the third matrix means that the first predicted sample exists at the element position (k, l) of the third matrix. Similarly, the fact that the second predicted sample corresponds to an element of the third matrix means that the second predicted sample exists at the element position (k, l) of the third matrix.

[0019] The additional values ​​involved in the calculation of the vertical motion offset of the current block, the second variable s2 and the fourth variable s4, also do not involve multiplication operations. Similar to the fifth variable s5, the calculation of the fourth variable s4 only involves the summation of terms obtained from the signs of the elements of the second matrix and the elements of the third matrix. Applying the sign of one element to another element does not involve a multiplication operation. Furthermore, the summation also does not involve any multiplication. Compared to conventional approaches, the bit depth of the fourth variable s4 is reduced because the multiplication operation is replaced by a sign decision. Similarly, the calculation of the second variable s2 involves taking the absolute value of the elements of the second matrix and does not involve any multiplication. In this way, the bit depth of s2 is also reduced. As a result, when calculating the vertical motion offset, the bit depth of the result is also reduced, and the computational complexity is significantly reduced.

[0020] In one possible implementation form of a method according to any preceding implementation of the first aspect or the first aspect itself, the horizontal movement offset is derived based on a first variable s1 and a third variable s3, where the first variable s1 indicates a sum of absolute values ​​of elements of the first matrix and the third variable s3 indicates a sum of multiple terms, each of which is obtained from a sign of an element of the first matrix and an element of the third matrix, and the element of the third matrix corresponds to the element of the first matrix.

[0021] The fact that an element of the third matrix (the third element) corresponds to an element of the first matrix (the first element) means that the position (x,y) of the first element in the first matrix is ​​the same as the position (k,l) of the third element in the third matrix, i.e., (x,y)=(k,l).

[0022] An additional reduction in computational complexity can be achieved by deriving the horizontal motion offset based on the first variable s1 and the third variable s3. The calculation of both the first variable s1 and the third variable s3 does not involve multiplications, but rather simply involves taking absolute values, signing, and summation operations.

[0023] In one possible implementation of the method according to any preceding implementation of the first aspect or the first aspect itself, the horizontal motion offset is:

number

[0024] This means that v x Here, we show one possible way to determine s1 and s3. s1 and s3 are determined without multiplication operations, and by not performing multiplication operations, v xThis significantly reduces the computational complexity of the process for determining horizontal motion offset. Thus, by calculating the horizontal motion offset in this manner, it is possible to efficiently determine the horizontal motion offset.

[0025] In one possible implementation of the method according to any preceding implementation of the first aspect or the first aspect itself, the vertical motion offset v y teeth,

number

[0026] This is based on the autocorrelation and cross-correlation terms s2, s4, and s5, y s2, s4, and s5 are determined without multiplication operations, and by not performing multiplication operations, v y This significantly reduces the computational complexity of the process for determining the vertical motion offset. Thus, by calculating the vertical motion offset in this manner, it is possible to efficiently determine the vertical motion offset.

[0027] In one possible implementation of the method according to any preceding implementation of the first aspect or the first aspect itself, s1, s2, s3, s4, and s5 are

number

[0028] In some instances, I (0) is a block containing predicted samples in the first reference frame surrounding the sub-block corresponding to the current block. For example, if the current block is a 4x4 block, then I (0) may be a 6x6 block in the first reference frame surrounding the 4x4 block corresponding to the 4x4 current block. (1) is a block containing predicted samples in the second reference frame surrounding the sub-block corresponding to the current block. For example, if the current block is a 4x4 block, then I (1) may be a 6x6 block in the second reference frame that surrounds the 4x4 block corresponding to the 4x4 current block.

[0029] The current block is a 4x4 block and I (0) and I (1) If both of the blocks are 6x6, then G x0 and G x1 Each of these is a 6x6 block.

[0030] It should be noted that when calculating the autocorrelation and cross-correlation terms s1, s2, s3, s4, and s5, a shift operation may be applied to adjust the precision and / or bit depth of s1, s2, s3, s4, and s5.

[0031] Furthermore, s1, s2, s3, s4, and s5 are determined without performing multiplication operations, and by not performing multiplication operations, v x and v y It should be noted that this significantly reduces the computational complexity of the process for determining x and v y is derived based on the autocorrelation and cross-correlation terms shown above for s1, s2, s3, s4, and s5. (I (1) -I (0) ) term is (I (0) -I (1) ), if it is changed to v x and v y teeth,

number

number

[0032] In one possible implementation of the method according to any preceding implementation of the first aspect or the first aspect itself, x0 is determined as the difference obtained from two predicted samples corresponding to the first reference frame along the horizontal direction, and y0 is determined as the difference obtained from two predicted samples corresponding to said first reference frame along the vertical direction.

[0033] In some examples, two predicted samples corresponding to the first reference frame along the horizontal direction have the same vertical coordinate but different horizontal coordinates. Two predicted samples corresponding to the first reference frame along the vertical direction have the same horizontal coordinate but different vertical coordinates. Before calculating the difference, each of the two predicted samples may be processed by right shifting, left shifting, clipping, etc.

[0034] In one possible implementation of the method according to any preceding implementation of the first aspect or the first aspect itself, x1 is determined as the difference obtained from two predicted samples corresponding to the second reference frame along the horizontal direction, and y1 is determined as the difference obtained from two predicted samples corresponding to said second reference frame along the vertical direction.

[0035] In some examples, two predicted samples corresponding to the second reference frame along the horizontal direction have the same vertical coordinate but different horizontal coordinates. Two predicted samples corresponding to the second reference frame along the vertical direction have the same horizontal coordinate but different vertical coordinates. Before calculating the difference, each of the two predicted samples may be processed by right shifting, left shifting, clipping, etc.

[0036] In one possible implementation form of a method according to any preceding implementation of the first aspect or the first aspect itself, the predicted sample value corresponding to the first reference frame and the predicted sample value corresponding to the second reference frame are obtained from the first reference frame and the second reference frame, respectively, using a pair of motion vectors for the current block with respect to the first reference frame and the second reference frame.

[0037] In one possible implementation form of a method according to any of the previous implementations of the first aspect or the first aspect itself, the predicted sample values ​​for the current block are bidirectionally predicted sample values ​​based on bidirectional optical flow (BDOF) prediction.

[0038] According to a second aspect, the disclosure provides a device for encoding video data, comprising: a video data memory; a video encoder, the video encoder comprising: The horizontal motion offset v of the current block of the video signal x and vertical movement offset v y the vertical motion offset is determined based on the horizontal motion offset and a fifth variable s5; the fifth variable s5 indicates a sum of a plurality of terms, each of which is obtained from a sign of an element of a second matrix and an element of a first matrix, and the elements of the first matrix correspond to the elements of the second matrix; Each element of the first matrix is ​​obtained from the sum of a first horizontal predicted sample gradient corresponding to a first reference frame of the current block and a second horizontal predicted sample gradient corresponding to a second reference frame of the current block, and the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to the element of the first matrix; Each element of the second matrix is ​​obtained from the sum of a first vertical predicted sample gradient corresponding to the first reference frame of the current block and a second vertical predicted sample gradient corresponding to the second reference frame of the current block, and the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to the element of the second matrix; and determining a predicted sample value in the current block using a predicted sample value corresponding to the first reference frame, a predicted sample value corresponding to the second reference frame, the horizontal motion offset, and the vertical motion offset.

[0039] It should be noted that the current block may be a block of any size, such as a 4x4 block. The current block may also be a sub-block of a frame of a video signal. A pixel in the current block may be referenced using the pixel's absolute position relative to the top left corner of the frame, e.g., (x, y), or using the pixel's relative position with respect to the top left corner of the block, e.g., (xBlock+i, yBlock+j), where (xBlock, yBlock) are the coordinates of the block's top left corner relative to the top left corner of the frame.

[0040] The first matrix and the second matrix may be any two-dimensional array including rows and columns, and elements of the array may be indicated using (i, j), where x is the horizontal / row index and y is the vertical / column index. The range of i and j may be, for example, i = xBlock-1, ..., xBlock+4 and j = yBlock-1, ..., yBlock+4. The first matrix and the second matrix correspond to the current block or are determined for the current block. In some examples, the size of the first matrix is ​​the same as the size of the second matrix, which may be larger than the size of the current block. For example, the size of the first matrix and the second matrix may be 6x6, while the size of the current block is 4x4.

[0041] If the position (x, y) of the first element in the first matrix is ​​the same as the position (p, q) of the second element in the second matrix, i.e., (x, y) = (p, q), then the element (first element) of the first matrix corresponds to the element (second element) of the second matrix. The first horizontal predicted sample gradient corresponds to the first reference frame of the current block, meaning that the first horizontal predicted sample gradient is generated based on samples in the first reference frame of the current block. The second horizontal predicted sample gradient corresponds to the second reference frame of the current block, meaning that the second horizontal predicted sample gradient is generated based on samples in the second reference frame of the current block. The first horizontal predicted sample gradient corresponds to an element of the first matrix, meaning that the first horizontal predicted sample gradient is generated for the position (x, y) of that element in the first matrix. Similarly, a second horizontal predicted sample gradient corresponds to an element of the first matrix, meaning that the second horizontal predicted sample gradient is generated for the position (x, y) of that element in the first matrix.

[0042] The first vertical predicted sample gradient corresponding to the first reference frame of the current block means that the first vertical predicted sample gradient is generated based on samples in the first reference frame of the current block. The second vertical predicted sample gradient corresponding to the second reference frame of the current block means that the second vertical predicted sample gradient is generated based on samples in the second reference frame of the current block. The first vertical predicted sample gradient corresponding to an element of a second matrix means that the first vertical predicted sample gradient is generated for the position (p, q) of that element in the second matrix. Similarly, the second vertical predicted sample gradient corresponding to an element of a second matrix means that the second vertical predicted sample gradient is generated for the position (p, q) of that element in the second matrix.

[0043] When each element of a matrix is ​​obtained from the sum of two terms, it means that the element may be determined as the sum of the two terms itself, or as the value after processing the sum of the two terms. The processing may include left shifting, right shifting, clipping, or a combination thereof. Similarly, when a term is obtained from the sign of an element of a second matrix and an element of a first matrix, it means that the term may take the value of the element of the first matrix itself, or the value after the element of the first matrix is ​​processed, and then the sign of the element of the second matrix is ​​applied. The processing of the element of the first matrix may include left shifting, right shifting, clipping, or a combination thereof. The sign of element x is

number

[0044] The technique presented herein adjusts the bidirectionally predicted sample values ​​of the current block based on a horizontal motion offset and a vertical motion offset. The vertical motion offset is calculated based on a fifth variable s5, which merely involves the summation of terms obtained from the signs of elements of a second matrix and elements of a first matrix. Applying the sign of one element to another element does not involve a multiplication operation. Similarly, the summation also does not involve any multiplication. As a result, the BDOF-based inter-frame prediction technique presented herein eliminates multiplication operations. Compared to conventional approaches, the multiplication operations are replaced by sign decisions, thereby reducing the bit depth of the fifth variable s5. This means that the horizontal motion offset v x and vertical movement offset v y This leads to a reduction in the bit depth of the .DELTA..times ...

[0045] According to a third aspect, the disclosure provides a device for decoding video data, the device comprising: a video data memory; a video decoder, the video decoder comprising: The horizontal motion offset v of the current block of the video signal x and vertical movement offset v y the vertical motion offset is determined based on the horizontal motion offset and a fifth variable s5; the fifth variable s5 indicates a sum of a plurality of terms, each of which is obtained from a sign of an element of a second matrix and an element of a first matrix, and the elements of the first matrix correspond to the elements of the second matrix; Each element of the first matrix is ​​obtained from the sum of a first horizontal predicted sample gradient corresponding to a first reference frame of the current block and a second horizontal predicted sample gradient corresponding to a second reference frame of the current block, and the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to the element of the first matrix; Each element of the second matrix is ​​obtained from the sum of a first vertical predicted sample gradient corresponding to the first reference frame of the current block and a second vertical predicted sample gradient corresponding to the second reference frame of the current block, and the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to the element of the second matrix; and determining a predicted sample value in the current block using a predicted sample value corresponding to the first reference frame, a predicted sample value corresponding to the second reference frame, the horizontal motion offset, and the vertical motion offset.

[0046] It should be noted that the current block may be a block of any size, such as a 4x4 block. The current block may also be a sub-block of a frame of a video signal. A pixel in the current block may be referenced using the pixel's absolute position relative to the top left corner of the frame, e.g., (x, y), or using the pixel's relative position with respect to the top left corner of the block, e.g., (xBlock+i, yBlock+j), where (xBlock, yBlock) are the coordinates of the block's top left corner relative to the top left corner of the frame.

[0047] The first matrix and the second matrix may be any two-dimensional array including rows and columns, and elements of the array may be indicated using (i, j), where x is the horizontal / row index and y is the vertical / column index. The range of i and j may be, for example, i = xBlock-1, ..., xBlock+4 and j = yBlock-1, ..., yBlock+4. The first matrix and the second matrix correspond to the current block or are determined for the current block. In some examples, the size of the first matrix is ​​the same as the size of the second matrix, which may be larger than the size of the current block. For example, the size of the first matrix and the second matrix may be 6x6, while the size of the current block is 4x4.

[0048] If the position (x, y) of the first element in the first matrix is ​​the same as the position (p, q) of the second element in the second matrix, i.e., (x, y) = (p, q), then the element (first element) of the first matrix corresponds to the element (second element) of the second matrix. The first horizontal predicted sample gradient corresponds to the first reference frame of the current block, meaning that the first horizontal predicted sample gradient is generated based on samples in the first reference frame of the current block. The second horizontal predicted sample gradient corresponds to the second reference frame of the current block, meaning that the second horizontal predicted sample gradient is generated based on samples in the second reference frame of the current block. The first horizontal predicted sample gradient corresponds to an element of the first matrix, meaning that the first horizontal predicted sample gradient is generated for the position (x, y) of that element in the first matrix. Similarly, a second horizontal predicted sample gradient corresponds to an element of the first matrix, meaning that the second horizontal predicted sample gradient is generated for the position (x, y) of that element in the first matrix.

[0049] The first vertical predicted sample gradient corresponding to the first reference frame of the current block means that the first vertical predicted sample gradient is generated based on samples in the first reference frame of the current block. The second vertical predicted sample gradient corresponding to the second reference frame of the current block means that the second vertical predicted sample gradient is generated based on samples in the second reference frame of the current block. The first vertical predicted sample gradient corresponding to an element of a second matrix means that the first vertical predicted sample gradient is generated for the position (p, q) of that element in the second matrix. Similarly, the second vertical predicted sample gradient corresponding to an element of a second matrix means that the second vertical predicted sample gradient is generated for the position (p, q) of that element in the second matrix.

[0050] When each element of a matrix is ​​obtained from the sum of two terms, it means that the element may be determined as the sum of the two terms itself, or as the value after processing the sum of the two terms. The processing may include left shifting, right shifting, clipping, or a combination thereof. Similarly, when a term is obtained from the sign of an element of a second matrix and an element of a first matrix, it means that the term may take the value of the element of the first matrix itself, or the value after the element of the first matrix is ​​processed, and then the sign of the element of the second matrix is ​​applied. The processing of the element of the first matrix may include left shifting, right shifting, clipping, or a combination thereof. The sign of element x is

number

[0051] The technique presented herein adjusts the bidirectionally predicted sample values ​​of the current block based on a horizontal motion offset and a vertical motion offset. The vertical motion offset is calculated based on a fifth variable s5, which merely involves the summation of terms obtained from the signs of elements of a second matrix and elements of a first matrix. Applying the sign of one element to another element does not involve a multiplication operation. Similarly, the summation also does not involve any multiplication. As a result, the BDOF-based inter-frame prediction technique presented herein eliminates multiplication operations. Compared to conventional approaches, the multiplication operations are replaced by sign decisions, thereby reducing the bit depth of the fifth variable s5. This means that the horizontal motion offset v x and vertical movement offset v y This leads to a reduction in the bit depth of the .DELTA..times ...

[0052] In one possible implementation of a device according to any preceding implementation of the second and third aspects or the second and third aspects themselves, the vertical motion offset is derived based on the horizontal motion offset, a second variable s2, a fourth variable s4, and the fifth variable s5; the second variable s2 indicates the sum of the absolute values ​​of the elements of the second matrix; The fourth variable s4 indicates a sum of multiple terms, each of which is obtained from the sign of an element of the second matrix and an element of a third matrix, the element of the third matrix corresponding to the element of the second matrix, and each element of the third matrix being the difference obtained from a first predicted sample of the first reference frame corresponding to the element of the third matrix and a second predicted sample of the second reference frame corresponding to the element of the third matrix.

[0053] It should be noted that if the position (k, l) of the first element in the third matrix is ​​the same as the position (p, q) of the second element in the second matrix, i.e., (k, l) = (p, q), the element (first element) of the third matrix corresponds to the element (second element) of the second matrix. The fact that the first predicted sample corresponds to the first reference frame of the current block means that the first predicted sample exists in the first reference frame of the current block. The fact that the second predicted sample corresponds to the second reference frame of the current block means that the second predicted sample exists in the second reference frame of the current block. The fact that the first predicted sample corresponds to an element of the third matrix means that the first predicted sample exists at the element position (k, l) of the third matrix. Similarly, the fact that the second predicted sample corresponds to an element of the third matrix means that the second predicted sample exists at the element position (k, l) of the third matrix.

[0054] The additional values ​​involved in the calculation of the vertical motion offset of the current block, the second variable s2 and the fourth variable s4, also do not involve multiplication operations. Similar to the fifth variable s5, the calculation of the fourth variable s4 only involves the summation of terms obtained from the signs of the elements of the second matrix and the elements of the third matrix. Applying the sign of one element to another element does not involve a multiplication operation. Furthermore, the summation also does not involve any multiplication. Compared to conventional approaches, the bit depth of the fourth variable s4 is reduced because the multiplication operation is replaced by a sign decision. Similarly, the calculation of the second variable s2 involves taking the absolute value of the elements of the second matrix and does not involve any multiplication. In this way, the bit depth of s2 is also reduced. As a result, when calculating the vertical motion offset, the bit depth of the result is also reduced, and the computational complexity is significantly reduced.

[0055] In one possible implementation of a device according to any preceding implementation of the second and third aspects or the second and third aspects themselves, the horizontal motion offset is derived based on a first variable s1 and a third variable s3; the first variable s1 indicates the sum of the absolute values ​​of the elements of the first matrix; The third variable s3 indicates a sum of multiple terms, each of which is obtained from the sign of an element of the first matrix and an element of the third matrix, and the elements of the third matrix correspond to the elements of the first matrix.

[0056] The fact that an element of the third matrix (the third element) corresponds to an element of the first matrix (the first element) means that the position (x,y) of the first element in the first matrix is ​​the same as the position (k,l) of the third element in the third matrix, i.e., (x,y)=(k,l).

[0057] An additional reduction in computational complexity can be achieved by deriving the horizontal motion offset based on the first variable s1 and the third variable s3. The calculation of both the first variable s1 and the third variable s3 does not involve multiplications, but rather simply involves taking absolute values, signing, and summation operations.

[0058] In one possible implementation of a device according to any preceding implementation of the second and third aspects or the second and third aspects themselves, the horizontal motion offset is:

number

[0059] This means that v xHere, we show one possible way to determine s1 and s3. s1 and s3 are determined without multiplication operations, and by not performing multiplication operations, v x This significantly reduces the computational complexity of the process for determining horizontal motion offset. Thus, by calculating the horizontal motion offset in this manner, it is possible to efficiently determine the horizontal motion offset.

[0060] In one possible implementation of a device according to any preceding implementation of the second and third aspects or the second and third aspects themselves, the vertical motion offset v y teeth,

number

[0061] This is based on the autocorrelation and cross-correlation terms s2, s4, and s5, y s2, s4, and s5 are determined without multiplication operations, and by not performing multiplication operations, v y This significantly reduces the computational complexity of the process for determining the vertical motion offset. Thus, by calculating the vertical motion offset in this manner, it is possible to efficiently determine the vertical motion offset.

[0062] In one possible implementation of a device according to any preceding implementation of the second and third aspects or the second and third aspects themselves, s1, s2, s3, s4, and s5 are

number

[0063] In some instances, I (0) is a block containing predicted samples in the first reference frame surrounding the sub-block corresponding to the current block. For example, if the current block is a 4x4 block, then I (0) may be a 6x6 block in the first reference frame surrounding the 4x4 block corresponding to the 4x4 current block. (1) is a block containing predicted samples in the second reference frame surrounding the sub-block corresponding to the current block. For example, if the current block is a 4x4 block, then I (1) may be a 6x6 block in the second reference frame that surrounds the 4x4 block corresponding to the 4x4 current block.

[0064] The current block is a 4x4 block and I (0) and I (1) If both of the blocks are 6x6, then G x0 and G x1 Each of these is a 6x6 block.

[0065] It should be noted that when calculating the autocorrelation and cross-correlation terms s1, s2, s3, s4, and s5, a shift operation may be applied to adjust the precision and / or bit depth of s1, s2, s3, s4, and s5.

[0066] Furthermore, s1, s2, s3, s4, and s5 are determined without performing multiplication operations, and by not performing multiplication operations, v x and v y It should be noted that this significantly reduces the computational complexity of the process for determining x and v y is derived based on the autocorrelation and cross-correlation terms shown above for s1, s2, s3, s4, and s5. (I (1) -I (0) ) term is (I (0) -I (1) ), if it is changed to v x and v y teeth,

number

number

[0067] In one possible implementation of a device according to any preceding implementation of the second and third aspects or the second and third aspects themselves, x0 is determined as the difference obtained from two predicted samples corresponding to the first reference frame along the horizontal direction, and y0 is determined as the difference obtained from two predicted samples corresponding to said first reference frame along the vertical direction.

[0068] In some examples, two predicted samples corresponding to the first reference frame along the horizontal direction have the same vertical coordinate but different horizontal coordinates. Two predicted samples corresponding to the first reference frame along the vertical direction have the same horizontal coordinate but different vertical coordinates. Before calculating the difference, each of the two predicted samples may be processed by right shifting, left shifting, clipping, etc.

[0069] In one possible implementation of a device according to any preceding implementation of the second and third aspects or the second and third aspects themselves, x1 is determined as the difference obtained from two predicted samples corresponding to the second reference frame along the horizontal direction, and y1 is determined as the difference obtained from two predicted samples corresponding to said second reference frame along the vertical direction.

[0070] In some examples, two predicted samples corresponding to the second reference frame along the horizontal direction have the same vertical coordinate but different horizontal coordinates. Two predicted samples corresponding to the second reference frame along the vertical direction have the same horizontal coordinate but different vertical coordinates. Before calculating the difference, each of the two predicted samples may be processed by right shifting, left shifting, clipping, etc.

[0071] In one possible implementation of a device according to any of the previous implementations of the second and third aspects or the second and third aspects themselves, the predicted sample value corresponding to the first reference frame and the predicted sample value corresponding to the second reference frame are obtained from the first reference frame and the second reference frame, respectively, using a pair of motion vectors for the current block with respect to the first reference frame and the second reference frame.

[0072] In one possible implementation of a device according to any preceding implementation of the second and third aspects or the second and third aspects themselves, the predicted sample values ​​for the current block are bidirectionally predicted sample values ​​based on bidirectional optical flow (BDOF) prediction.

[0073] According to a fourth aspect, the disclosure provides an apparatus for performing bidirectional optical flow (BDOF) based inter-frame prediction for a current block of a video signal, the apparatus comprising: the horizontal motion offset v of the current block x and vertical movement offset v y wherein the vertical motion offset is determined based on the horizontal motion offset and a fifth variable s5; the fifth variable s5 indicates a sum of a plurality of terms, each of which is obtained from a sign of an element of a second matrix and an element of a first matrix, and the elements of the first matrix correspond to the elements of the second matrix; Each element of the first matrix is ​​obtained from the sum of a first horizontal predicted sample gradient corresponding to a first reference frame of the current block and a second horizontal predicted sample gradient corresponding to a second reference frame of the current block, and the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to the element of the first matrix; a determining unit, wherein each element of the second matrix is ​​obtained from a sum of a first vertical predicted sample gradient corresponding to the first reference frame of the current block and a second vertical predicted sample gradient corresponding to the second reference frame of the current block, and the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to the element of the second matrix; a prediction processing unit configured to predict predicted sample values ​​in the current block using predicted sample values ​​corresponding to the first reference frame, predicted sample values ​​corresponding to the second reference frame, the horizontal motion offset, and the vertical motion offset.

[0074] It should be noted that the current block may be a block of any size, such as a 4x4 block. The current block may also be a sub-block of a frame of a video signal. A pixel in the current block may be referenced using the pixel's absolute position relative to the top left corner of the frame, e.g., (x, y), or using the pixel's relative position with respect to the top left corner of the block, e.g., (xBlock+i, yBlock+j), where (xBlock, yBlock) are the coordinates of the block's top left corner relative to the top left corner of the frame.

[0075] The first matrix and the second matrix may be any two-dimensional array including rows and columns, and elements of the array may be indicated using (i, j), where x is the horizontal / row index and y is the vertical / column index. The range of i and j may be, for example, i = xBlock-1, ..., xBlock+4 and j = yBlock-1, ..., yBlock+4. The first matrix and the second matrix correspond to the current block or are determined for the current block. In some examples, the size of the first matrix is ​​the same as the size of the second matrix, which may be larger than the size of the current block. For example, the size of the first matrix and the second matrix may be 6x6, while the size of the current block is 4x4.

[0076] If the position (x, y) of the first element in the first matrix is ​​the same as the position (p, q) of the second element in the second matrix, i.e., (x, y) = (p, q), then the element (first element) of the first matrix corresponds to the element (second element) of the second matrix. The first horizontal predicted sample gradient corresponds to the first reference frame of the current block, meaning that the first horizontal predicted sample gradient is generated based on samples in the first reference frame of the current block. The second horizontal predicted sample gradient corresponds to the second reference frame of the current block, meaning that the second horizontal predicted sample gradient is generated based on samples in the second reference frame of the current block. The first horizontal predicted sample gradient corresponds to an element of the first matrix, meaning that the first horizontal predicted sample gradient is generated for the position (x, y) of that element in the first matrix. Similarly, a second horizontal predicted sample gradient corresponds to an element of a first matrix, meaning that the second horizontal predicted sample gradient is generated for the element's position (x, y) in the first matrix.

[0077] The first vertical predicted sample gradient corresponding to the first reference frame of the current block means that the first vertical predicted sample gradient is generated based on samples in the first reference frame of the current block. The second vertical predicted sample gradient corresponding to the second reference frame of the current block means that the second vertical predicted sample gradient is generated based on samples in the second reference frame of the current block. The first vertical predicted sample gradient corresponding to an element of a second matrix means that the first vertical predicted sample gradient is generated for the position (p, q) of that element in the second matrix. Similarly, the second vertical predicted sample gradient corresponding to an element of a second matrix means that the second vertical predicted sample gradient is generated for the position (p, q) of that element in the second matrix.

[0078] When each element of a matrix is ​​obtained from the sum of two terms, it means that the element may be determined as the sum of the two terms itself, or as the value after processing the sum of the two terms. The processing may include left shifting, right shifting, clipping, or a combination thereof. Similarly, when a term is obtained from the sign of an element of a second matrix and an element of a first matrix, it means that the term may take the value of the element of the first matrix itself, or the value after the element of the first matrix is ​​processed, and then the sign of the element of the second matrix is ​​applied. The processing of the element of the first matrix may include left shifting, right shifting, clipping, or a combination thereof. The sign of element x is

number

[0079] The technique presented herein adjusts the bidirectionally predicted sample values ​​of the current block based on a horizontal motion offset and a vertical motion offset. The vertical motion offset is calculated based on a fifth variable s5, which merely involves the summation of terms obtained from the signs of elements of a second matrix and elements of a first matrix. Applying the sign of one element to another element does not involve a multiplication operation. Similarly, the summation also does not involve any multiplication. As a result, the BDOF-based inter-frame prediction technique presented herein eliminates multiplication operations. Compared to conventional approaches, the multiplication operations are replaced by sign decisions, thereby reducing the bit depth of the fifth variable s5. This means that the horizontal motion offset v x and vertical movement offset v y This leads to a reduction in the bit depth of the .DELTA..times ...

[0080] In one possible implementation of a device according to any preceding implementation of the fourth aspect or the fourth aspect itself, the determination unit is configured to determine the vertical motion offset based on the horizontal motion offset, a second variable s2, a fourth variable s4, and the fifth variable s5; the second variable s2 indicates the sum of the absolute values ​​of the elements of the second matrix; The fourth variable s4 indicates a sum of multiple terms, each of which is obtained from the sign of an element of the second matrix and an element of a third matrix, the element of the third matrix corresponding to the element of the second matrix, and each element of the third matrix being the difference obtained from a first predicted sample of the first reference frame corresponding to the element of the third matrix and a second predicted sample of the second reference frame corresponding to the element of the third matrix.

[0081] It should be noted that if the position (k, l) of the first element in the third matrix is ​​the same as the position (p, q) of the second element in the second matrix, i.e., (k, l) = (p, q), the element (first element) of the third matrix corresponds to the element (second element) of the second matrix. The fact that the first predicted sample corresponds to the first reference frame of the current block means that the first predicted sample exists in the first reference frame of the current block. The fact that the second predicted sample corresponds to the second reference frame of the current block means that the second predicted sample exists in the second reference frame of the current block. The fact that the first predicted sample corresponds to an element of the third matrix means that the first predicted sample exists at the element position (k, l) of the third matrix. Similarly, the fact that the second predicted sample corresponds to an element of the third matrix means that the second predicted sample exists at the element position (k, l) of the third matrix.

[0082] The additional values ​​involved in the calculation of the vertical motion offset of the current block, the second variable s2 and the fourth variable s4, also do not involve multiplication operations. Similar to the fifth variable s5, the calculation of the fourth variable s4 only involves the summation of terms obtained from the signs of the elements of the second matrix and the elements of the third matrix. Applying the sign of one element to another element does not involve a multiplication operation. Furthermore, the summation also does not involve any multiplication. Compared to conventional approaches, the bit depth of the fourth variable s4 is reduced because the multiplication operation is replaced by a sign decision. Similarly, the calculation of the second variable s2 involves taking the absolute value of the elements of the second matrix and does not involve any multiplication. In this way, the bit depth of s2 is also reduced. As a result, when calculating the vertical motion offset, the bit depth of the result is also reduced, and the computational complexity is significantly reduced.

[0083] In one possible implementation of a device according to any preceding implementation of the fourth aspect or the fourth aspect itself, the determining unit is configured to determine the horizontal movement offset based on a first variable s1 and a third variable s3; the first variable s1 indicates the sum of the absolute values ​​of the elements of the first matrix; The third variable s3 indicates a sum of multiple terms, each of which is obtained from the sign of an element of the first matrix and an element of the third matrix, and the elements of the third matrix correspond to the elements of the first matrix.

[0084] The fact that an element of the third matrix (the third element) corresponds to an element of the first matrix (the first element) means that the position (x,y) of the first element in the first matrix is ​​the same as the position (k,l) of the third element in the third matrix, i.e., (x,y)=(k,l).

[0085] An additional reduction in computational complexity can be achieved by deriving the horizontal motion offset based on the first variable s1 and the third variable s3. The calculation of both the first variable s1 and the third variable s3 does not involve multiplications, but rather simply involves taking absolute values, signing, and summation operations.

[0086] In one possible implementation of a device according to any preceding implementation of the fourth aspect or the fourth aspect itself, the determining unit:

number

[0087] This means that v xHere, we show one possible way to determine s1 and s3. s1 and s3 are determined without multiplication operations, and by not performing multiplication operations, v x This significantly reduces the computational complexity of the process for determining horizontal motion offset. Thus, by calculating the horizontal motion offset in this manner, it is possible to efficiently determine the horizontal motion offset.

[0088] In one possible implementation of a device according to any preceding implementation of the fourth aspect or the fourth aspect itself, the determining unit:

number

[0089] This is based on the autocorrelation and cross-correlation terms s2, s4, and s5, y s2, s4, and s5 are determined without multiplication operations, and by not performing multiplication operations, v y This significantly reduces the computational complexity of the process for determining the vertical motion offset. Thus, by calculating the vertical motion offset in this manner, it is possible to efficiently determine the vertical motion offset.

[0090] In one possible implementation of a device according to any preceding implementation of the fourth aspect or the fourth aspect itself, s1, s2, s3, s4, and s5 are:

number

[0091] In some instances, I (0) is a block containing predicted samples in the first reference frame surrounding the sub-block corresponding to the current block. For example, if the current block is a 4x4 block, then I (0) may be a 6x6 block in the first reference frame surrounding the 4x4 block corresponding to the 4x4 current block. (1) is a block containing predicted samples in the second reference frame surrounding the sub-block corresponding to the current block. For example, if the current block is a 4x4 block, then I (1) may be a 6x6 block in the second reference frame that surrounds the 4x4 block corresponding to the 4x4 current block.

[0092] The current block is a 4x4 block and I (0) and I (1) If both of the blocks are 6x6, then G x0 and G x1 Each of these is a 6x6 block.

[0093] It should be noted that when calculating the autocorrelation and cross-correlation terms s1, s2, s3, s4, and s5, a shift operation may be applied to adjust the precision and / or bit depth of s1, s2, s3, s4, and s5.

[0094] Furthermore, s1, s2, s3, s4, and s5 are determined without performing multiplication operations, and by not performing multiplication operations, v x and v y It should be noted that this significantly reduces the computational complexity of the process for determining x and v y is derived based on the autocorrelation and cross-correlation terms shown above for s1, s2, s3, s4, and s5. (I (1) -I (0) ) term is (I (0) -I (1) ), if it is changed to v x and v y teeth,

number

number

[0095] In one possible implementation of a device according to any preceding implementation of the fourth aspect or the fourth aspect itself, x0 is determined as the difference obtained from two predicted samples corresponding to the first reference frame along the horizontal direction, and y0 is determined as the difference obtained from two predicted samples corresponding to said first reference frame along the vertical direction.

[0096] In some examples, two predicted samples corresponding to the first reference frame along the horizontal direction have the same vertical coordinate but different horizontal coordinates. Two predicted samples corresponding to the first reference frame along the vertical direction have the same horizontal coordinate but different vertical coordinates. Before calculating the difference, each of the two predicted samples may be processed by right shifting, left shifting, clipping, etc.

[0097] In one possible implementation of a device according to any preceding implementation of the fourth aspect or the fourth aspect itself, x1 is determined as the difference obtained from two predicted samples corresponding to the second reference frame along the horizontal direction, and y1 is determined as the difference obtained from two predicted samples corresponding to said second reference frame along the vertical direction.

[0098] In some examples, two predicted samples corresponding to the second reference frame along the horizontal direction have the same vertical coordinate but different horizontal coordinates. Two predicted samples corresponding to the second reference frame along the vertical direction have the same horizontal coordinate but different vertical coordinates. Before calculating the difference, each of the two predicted samples may be processed by right shifting, left shifting, clipping, etc.

[0099] In one possible implementation form of a device according to any preceding implementation of the fourth aspect or the fourth aspect itself, the predicted sample value corresponding to the first reference frame and the predicted sample value corresponding to the second reference frame are obtained from the first reference frame and the second reference frame, respectively, using a pair of motion vectors for the current block with respect to the first reference frame and the second reference frame.

[0100] In one possible implementation of a device according to any of the previous implementations of the fourth aspect or the fourth aspect itself, the predicted sample values ​​for the current block are bidirectionally predicted sample values ​​based on bidirectional optical flow (BDOF) prediction.

[0101] Methods according to some aspects of the invention may be performed by apparatus according to some of those aspects of the invention. Further features and implementations of methods according to some aspects of the invention result directly from the functionality of apparatus according to some of those aspects of the invention and its different implementations.

[0102] It should be noted that a coding device may be an encoding device or a decoding device.

[0103] According to another aspect, the invention relates to an apparatus for decoding a video stream, the apparatus comprising a processor and a memory, the memory storing a plurality of instructions that cause the processor to perform the method previously presented.

[0104] According to another aspect, the invention relates to an apparatus for encoding a video stream, the apparatus comprising a processor and a memory, the memory storing a plurality of instructions that cause the processor to perform the method previously presented.

[0105] According to another aspect, a computer-readable storage medium is proposed having stored thereon instructions that, when executed, configure one or more processors to code video data, the instructions causing the one or more processors to perform the previously presented method.

[0106] According to another aspect, there is provided a computer program product having a program code for performing the method set out previously, when the computer program is run by a computer.

[0107] The details of one or more embodiments are set forth below in the accompanying drawings and detailed description. Many other features, objects, and advantages will be apparent from the detailed description, the drawings, and the claims.

[0108] For clarity, any one of the above embodiments may be combined with any one or more of the other above embodiments to create new embodiments within the scope of the present disclosure.

[0109] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. [Brief explanation of the drawings]

[0110] Further embodiments of the present invention are described with respect to the following figures.

[0111] [Figure 1A] 1 shows a block diagram illustrating one example of a video coding system configured to implement embodiments presented herein. [Figure 1B] 1 shows a block diagram illustrating another example of a video coding system configured to implement several embodiments. [Figure 2] FIG. 1 is a block diagram illustrating one example configuration of a video encoder configured to implement embodiments presented herein. [Figure 3] FIG. 1 is a block diagram illustrating one example configuration of a video decoder configured to implement embodiments presented herein. [Figure 4]1 is a block diagram illustrating an example of an encoding device or a decoding device. [Figure 5] FIG. 10 is a block diagram illustrating another example of an encoding device or a decoding device. [Figure 6] FIG. 10 illustrates the relationship between a 6×6 window and 4×4 sub-blocks for computing gradient autocorrelation and cross-correlation. [Figure 7] FIG. 1 illustrates an example of bidirectional predictive optical flow. [Figure 8] 1 is a flowchart illustrating an example of a process for optical flow based inter-frame prediction according to an embodiment. [Figure 9] 1 is a flowchart illustrating one example of a method for bidirectional optical flow (BDOF)-based inter-frame prediction for a current block of a video signal. [Figure 10] 1 is a block diagram illustrating one exemplary configuration of an apparatus for bidirectional optical flow (BDOF)-based inter-frame prediction for a current block of a video signal. [Figure 11] 1 is a block diagram illustrating one exemplary configuration of a content delivery system that provides content distribution services. [Figure 12] FIG. 2 is a block diagram showing the configuration of an example of a terminal device.

[0112] The same reference numbers are used in the various figures for identical or functionally equivalent features. DETAILED DESCRIPTION OF THE INVENTION

[0113] In the following description, reference is made to the accompanying drawings, which form a part of this disclosure and which show, by way of illustration, specific aspects to which the present invention may be applied. It is understood that other aspects may be utilized and structural or logical changes may be made without departing from the scope of the present invention. Therefore, the following detailed description is not to be taken in a limiting sense, as the scope of the present invention is defined by the appended claims.

[0114] For example, it is understood that disclosure related to a described method may also apply to a corresponding device or system configured to perform that method, and vice versa. For example, when describing one or more particular method steps, a corresponding device may include one or more units, e.g., functional units (e.g., one unit that performs one or more steps, or multiple units that each perform one or more of the steps), to perform the described method step(s), even if such one or more units are not explicitly described or shown in the drawings. For example, on the other hand, when describing a particular apparatus based on one or more units, e.g., functional units, a corresponding method may include one step to perform the function(s) of the one or more units (e.g., one step that performs the function(s) of the one or more units, or multiple steps that each perform the function(s) of one or more units), even if such one or more steps are not explicitly described or shown in the drawings. Furthermore, it is understood that features of the various exemplary embodiments and / or aspects described herein may be combined with each other, unless otherwise stated.

[0115] The present disclosure provides a technique for calculating a second component of optical flow based on a calculated first component of optical flow without requiring any expensive multiplications. The technique can be adopted at both the encoding and decoding ends when BPOF is enabled. The technique provides an improved device and method for optical flow-based inter-frame prediction, thereby improving compression efficiency compared to conventional BPOF without increasing the computational complexity of BPOF.

[0116] To describe the present invention in detail, the following terms, abbreviations and notations are used. Picture Order Count for POC Display Order MV Motion Vector MCP Motion Compensated Prediction HEVC High Efficiency Video Coding Optical flow-based decoder-side correction of bidirectional prediction for BPOF MCP BDOF Bidirectional Optical Flow

[0117] As used herein, a video signal or video sequence is a set of subsequent frames that present a moving image. In other words, a video signal or video sequence is made up of a number of frames (also called pictures or images).

[0118] As used herein, a coding tree unit (CTU) refers to the root of the coding structure of a video sequence of a predefined size that comprises a portion of a frame (e.g., 64x64 pixels). A CTU may be partitioned into several CUs.

[0119] As used herein, a coding unit (CU) refers to a basic coding structure of a video sequence, which has a predefined size, contains a portion of a frame, and belongs to a CTU. A CU may be partitioned into further CUs.

[0120] As used herein, a prediction unit (PU) refers to a coding structure that is the result of partitioning a CU.

[0121] As used herein, the term co-located herein refers to a block or region in a second frame, i.e., a reference frame, that corresponds to an actual block or region in a first frame, i.e., a current frame.

[0122] Video coding typically refers to the processing of a video or a sequence of videos to form a video sequence. Instead of the word "video," the words "frame" or "image" may be used as synonyms in the field of video coding. Video coding (or coding in general) includes two parts: video encoding and video decoding. Video encoding is performed at the source side and typically involves processing the original video video (e.g., by compression) to reduce the amount of data needed to represent the video video (for more efficient storage and / or transmission). Video decoding is performed at the destination side and typically involves the reverse process compared to an encoder reconstructing the video video. It should be understood that embodiments referring to "coding" video video (or video in general) also relate to "encoding" or "decoding" the video video or respective video sequences. The combination of the encoding and decoding parts is also referred to as CODEC (Coding and Decoding).

[0123] In the case of lossless video coding, it is possible to reconstruct the original video image, i.e., the reconstructed video image has the same quality as the original video image (assuming there are no transmission or other data losses during storage or transmission). In the case of lossy video coding, further compression, e.g., by quantization, is performed to reduce the amount of data representing the video image, and it is not possible to completely reconstruct the video image at the decoder, i.e., the quality of the reconstructed video image is lower or worse than that of the original video image.

[0124] Some video coding standards belong to the group of "lossy hybrid video codecs" (i.e., they combine spatial and temporal prediction in the sample domain with two-dimensional transform coding to apply quantization in the transform domain). Each picture of a video sequence is typically partitioned into a set of non-overlapping blocks, and coding is typically performed at the block level. In other words, an encoder typically processes, or encodes, the video at the block (video block) level, e.g., by using spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to generate a predictive block, subtracting the predictive block from a current block (the block currently being processed / to be processed) to obtain a residual block, transforming the residual block, and quantizing the residual block in the transform domain to reduce the amount of data to be transmitted (compression). A decoder applies an inverse process to the coded or compressed block compared to the encoder to reconstruct the current block for representation. Furthermore, the encoder replicates the decoder processing loop, so that both the encoder and decoder will generate the same predictions (e.g., intra-picture predictions and inter-picture predictions) and / or reconstructions for processing, i.e., coding, of subsequent blocks.

[0125] 1A-3, several embodiments of a video coding system 10, a video encoder 20, and a video decoder 30 are described.

[0126] 1A is a schematic block diagram illustrating an example coding system 10, such as video coding system 10 (or, for short, coding system 10) capable of implementing the techniques presented herein. Video encoder 20 (or, for short, encoder 20) and video decoder 30 (or, for short, decoder 30) of video coding system 10 represent examples of devices that may be configured to perform the techniques in accordance with various examples described herein.

[0127] 1A, coding system 10 includes a source device 12 configured to provide encoded video data 21 to a destination device 14, for example, for decoding encoded video data 13. Source device 12 includes an encoder 20 and may additionally include a video source 16, a pre-processor (or pre-processing unit) 18, such as a video pre-processor 18, and a communications interface or unit 22.

[0128] Video source 16 may include any type of video capture device, such as a camera for capturing real-world video, or any type of video generation device, such as a computer graphics processor for generating computer-animated video. Video source 16 may also include any type of other device for obtaining and / or providing real-world video, computer-generated video (e.g., screen content, virtual reality (VR) video, etc.), and / or any combination thereof (e.g., augmented reality (AR) video, etc.). Video source 16 may also be any type of memory or storage device for storing any of the above videos.

[0129] To distinguish between the processing performed by the pre-processor 18 and the pre-processing unit 18, the image or image data 17 may also be referred to as raw image or raw image data 17. The pre-processor 18 is configured to receive the (raw) image data 17 and perform pre-processing on the image data 17 to obtain pre-processed image 19 or pre-processed image data 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal.

[0130] Video encoder 20 is configured to receive preprocessed video data 19 and provide encoded video data 21 (described in further detail below, e.g., with respect to FIG. 2). Communications interface 22 of source device 12 may be configured to receive encoded video data 21 and transmit the encoded video data 21 (or any further processed version of the encoded video data) via communications channel 13 to another device, such as destination device 14 or any other device, for storage or direct reconstruction. Destination device 14 includes a decoder 30 (e.g., video decoder 30) and may additionally include a communications interface or unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34. The communications interface 28 of the destination device 14 is configured to receive the encoded video data 21 (or any further processed version of the encoded video data), e.g., directly from the source device 12 or from any other source, e.g., a storage device, e.g., an encoded video data storage device, and to provide the encoded video data 21 to a decoder 30.

[0131] The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded video data 21 or the encoded data 21 via a direct communication link between the source device 12 and the destination device 14, such as a direct wired connection or a direct wireless connection, or via any type of network, such as a wired network or a wireless network or any combination thereof, or via any type of private network and public network, or any combination thereof.

[0132] For example, communications interface 22 may be configured to package the encoded video data 21 into an appropriate format, such as packets, and / or process the encoded video data using any type of transmission encoding or processing for transmission over a communications link or network. Communications interface 28, paired with communications interface 22, may be configured to receive the transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or unpackaging to obtain encoded video data 21.

[0133] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow of communication channel 13 in FIG. 1A pointing from source device 12 to destination device 14, or may be configured as bidirectional communication interfaces, e.g., to send and receive messages, e.g., to set up a connection, thereby confirming and exchanging any other information related to the communication link and / or data transmission, e.g., encoded video data transmission.

[0134] Decoder 30 of destination device 14 is configured to receive encoded video data 21 and provide decoded video data 31 or decoded video 31 (described in further detail below, e.g., with respect to FIG. 3 or FIG. 5). Post-processor 32 of destination device 14 is configured to post-process decoded video data 31 (also referred to as reconstructed video data), e.g., decoded video 31, to obtain post-processed video data 33, e.g., post-processed video 33. The post-processing performed by post-processing unit 32 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing, e.g., to prepare decoded video data 31 for display by display device 34.

[0135] A display device 34 of destination device 14 is configured to receive the post-processed video data 33, for example, to display the video to a user or viewer. Display device 34 may be or include any type of display for presenting the reconstructed video, such as, for example, an integrated display or an external display or monitor. Such displays may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0136] 1A depicts source device 12 and destination device 14 as separate devices, embodiments of the devices may include both devices or functionality of both devices, source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such embodiments, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0137] As will be clear to those skilled in the art from the description, the presence and (exact) distribution of different units or functions in source device 12 and / or destination device 14 shown in FIG. 1A may vary depending on the actual device and application.

[0138] As shown in FIG. 1B , encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30, may be implemented by processing circuitry such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated to video coding, or any combination thereof. Encoder 20 may be implemented by processing circuitry 46 for implementing various modules as described with respect to encoder 20 of FIG. 2 and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented by processing circuitry 46 for implementing various modules as described with respect to decoder 30 of FIG. 3 and / or any other decoder system or subsystem described herein. The processing circuitry may be configured to perform various operations as described below. 5, where the techniques are implemented in part by software, a device may store instructions for the software in a suitable non-transitory computer-readable storage medium and execute those instructions in hardware using one or more processors to perform the techniques of this disclosure. For example, as shown in FIG. 1B, either video encoder 20 and video decoder 30 may be integrated as part of a combined encoder / decoder (CODEC) in a single device.

[0139] The source device 12 and the destination device 14 may include any of a wide range of devices, including any type of handheld or fixed device, such as, for example, a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or content distribution server), a broadcast receiver device, or a broadcast transmitter device, and may use no operating system or any type of operating system. In some cases, the source device 12 and the destination device 14 may be capable of wireless communication. Thus, the source device 12 and the destination device 14 may be wireless communication devices.

[0140] 1A is just one example, and the techniques presented herein may be applied to video coding configurations (e.g., video encoding or video decoding) that do not necessarily involve data communication between encoding and decoding devices. In other examples, data may be retrieved from local memory, streamed over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by multiple devices that do not communicate with each other but simply encode data to memory and / or retrieve and decode data from memory.

[0141] For ease of explanation, embodiments of the present invention are described herein with reference to, for example, High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC) reference software, the next generation video coding standards developed by the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG) Joint Collaboration Team on Video Coding (JCT-VC). Those skilled in the art will understand that embodiments of the present invention are not limited to HEVC or VVC.

[0142] Encoder and encoding method

[0143] 2 shows a schematic block diagram of an exemplary video encoder 20 configured to implement the techniques presented herein. In the example of FIG. 2, the video encoder 20 includes an input 201 (or input interface 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output unit 272 (or output interface 272). The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a partitioning unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in FIG. 2 may also be referred to as a hybrid video encoder or a video encoder according to a hybrid video codec.

[0144] The residual calculation unit 204, the transform processing unit 206, the quantization unit 208, and the mode selection unit 260 may be considered to form a forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244, and the intra-frame prediction unit 254 may be considered to form a backward signal path of the video encoder 20, which corresponds to the signal path of a decoder (see video decoder 30 in FIG. 3 ). The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244, and the intra-frame prediction unit 254 may also be considered to form an “embedded decoder” of the video encoder 20.

[0145] Image and image segmentation (image and block)

[0146] Encoder 20 may be configured to receive, for example, via input 201, a picture 17 (or video data 17), such as, for example, a picture of a sequence of pictures forming a video or a video sequence. The received picture or video data may also be a preprocessed picture 19 (or preprocessed video data 19). For simplicity, the following description refers to picture 17. Picture 17 may also be referred to as a current picture or a picture to be coded (e.g., during video coding, in particular to distinguish the current picture from other pictures, such as previously coded and / or decoded pictures of the same video sequence, i.e., also a video sequence including the current picture).

[0147] A (digital) image may be considered as a two-dimensional array or matrix of samples with intensity values. A sample in the array may also be called a pixel or pel (an abbreviation for image element). The number of samples in the horizontal and vertical directions (or axes) of the array or image defines the size and / or resolution of the image. For color representation, three color components are typically used, i.e., the image may be represented by or contain three sample arrays. In an RBG format or color space, the image contains corresponding red, green, and blue sample arrays. On the other hand, in the case of video coding, each pixel is typically represented by a luminance and chromaticity format or color space, such as YCbCr, which contains a luminance component denoted by Y (also sometimes referred to as L) and two chromaticity components denoted by Cb and Cr. The luminance (or, abbreviated, luma) component Y represents lightness or gray-level intensity (e.g., as in a grayscale image), while the two chrominance (or, abbreviated, chroma) components Cb and Cr represent chromaticity or color information components. Thus, an image in YCbCr format includes a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). An image in RGB format may be transformed or converted to YCbCr format, and vice versa; this process is also known as color conversion or color transformation. If an image is monochrome, the image may include only a luminance sample array. Thus, an image may be, for example, an array of luma samples in monochrome format, or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0148] Embodiments of video encoder 20 may include a picture partitioning unit (not shown in FIG. 2 ) configured to partition picture 17 into (typically non-overlapping) picture blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), or coding tree blocks (CTBs) or coding tree units (CTUs) (H.265 / HEVC and VVC). The picture partitioning unit may be configured to use the same block size for all of the pictures in the video sequence and a corresponding grid that defines the block sizes, or may be configured to vary the block size among pictures or among subsets or groups of pictures and partition each picture into corresponding blocks.

[0149] In further embodiments, the video encoder may be configured to directly receive block 203 of video 17, such as one block, some of the blocks, or all of the blocks that form video 17. Video block 203 may also be referred to as the current video block or the video block to be coded.

[0150] Similar to image 17, image block 203 may also be considered a two-dimensional array or matrix of samples having intensity values ​​(sample values), albeit with smaller dimensions than image 17. In other words, for example, block 203 may include one sample array (e.g., a luma array for monochrome image 17, or a luma array or a chroma array for color image) or three sample arrays (e.g., a luma and two chroma arrays for color image 17), or any other number and / or type of array depending on the color format applied. The number of samples in the horizontal and vertical directions (or axes) of block 203 define the size of block 203. Thus, a block may be, for example, an M×N (M columns by N rows) array of samples or an M×N array of transform coefficients.

[0151] 2 may be configured to encode video 17 on a block-by-block basis, such as by performing encoding and prediction on a block-by-block basis 203. Embodiments of video encoder 20 shown in FIG. 2 may also be configured to partition and / or encode video using slices (also referred to as video slices), where video may be partitioned into or encoded using one or more (typically non-overlapping) slices, each of which may include one or more blocks (e.g., CTUs).

[0152] Embodiments of video encoder 20 shown in FIG. 2 may be configured to partition and / or encode video by using tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles), where video may be partitioned into or encoded using one or more (typically non-overlapping) tile groups, each of which may include, for example, one or more blocks (e.g., CTUs) or one or more tiles, and each tile may be, for example, rectangular in shape and may include, for example, one or more blocks (e.g., CTUs), such as complete blocks or fractional blocks.

[0153] Residual calculation

[0154] The residual calculation unit 204 may be configured to calculate the residual block 205 (also referred to as the residual 205) based on the video block 203 and the prediction block 265 (further details about the prediction block 265 will be described later), for example, by subtracting the sample values ​​of the prediction block 265 from the sample values ​​of the video block 203 on a sample-by-sample (pixel-by-pixel) basis to obtain the residual block 205 in the sample domain.

[0155] conversion

[0156] The transform processing unit 206 may be configured to apply a transform, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), to the sample values ​​of the residual block 205 to obtain transform coefficients 207 in a transform domain. The transform coefficients 207 may also be referred to as transform residual coefficients and represent the residual block 205 in the transform domain.

[0157] Transform processing unit 206 may be configured to apply an integer approximation of a DCT / DST, such as the transform specified for H.265 / HEVC. Compared to an orthogonal DCT transform, such an integer approximation is typically scaled by a factor. To preserve the norm of the residual blocks processed by the forward and inverse transforms, multiple additional scaling factors are applied as part of the transform process. These scaling factors are typically selected based on specific constraints, such as scaling factors that are powers of two for shift operations, the bit depth of the transform coefficients, a trade-off between accuracy and implementation cost, etc. A particular scaling factor may be specified for the inverse transform, e.g., by inverse transform processing unit 212 (and a corresponding inverse transform, e.g., by inverse transform processing unit 312 in video decoder 30), and a corresponding scaling factor for the forward transform, e.g., by transform processing unit 206, in encoder 20 may be specified accordingly.

[0158] Multiple embodiments of video encoder 20 (respectively, transform processing unit 206) may be configured to output transform parameters, e.g., one or more types of transform, that are encoded or compressed, e.g., directly by entropy coding unit 270 or by entropy coding unit 270, so that, for example, video decoder 30 may receive and use those transform parameters for decoding.

[0159] Quantization

[0160] The quantization unit 208 may be configured to quantize the transform coefficients 207, for example by applying scalar quantization or vector quantization, to obtain quantized coefficients 209, which may also be referred to as quantized transform coefficients 209 or quantized residual coefficients 209.

[0161] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, quantization may involve rounding down n-bit transform coefficients to m-bit transform coefficients, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, in the case of scalar quantization, different scaling may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to finer quantization, while a larger quantization step size corresponds to coarser quantization. The applicable quantization step size may be indicated by the quantization parameter (QP). The quantization parameter may, for example, be an index into a predefined set of applicable quantization step sizes. For example, a small quantization parameter may correspond to fine quantization (small quantization step size) and a large quantization parameter may correspond to coarse quantization (large quantization step size), or vice versa.

[0162] Quantization may involve division by a quantization step size, and corresponding and / or inverse dequantization by, for example, the inverse quantization unit 210 may involve multiplication by that quantization step size. For example, embodiments according to some standards, such as HEVC, may be configured to determine the quantization step size using a quantization parameter. Generally, the quantization step size may be calculated based on the quantization parameter using a fixed-point approximation of an equation that includes division. Additional scaling factors may be introduced for quantization and dequantization to restore the norm of the residual block, which may be modified due to the scaling used in the fixed-point approximation of the equations for the quantization step size and the quantization parameter. In one exemplary implementation, scaling of the inverse transform and dequantization may be combined. Alternatively, customized quantization tables may be used and signaled from the encoder to the decoder, for example, in the bitstream. Quantization is a lossy operation, and the loss increases as the quantization step size increases.

[0163] Multiple embodiments of video encoder 20 (respectively, quantization unit 208) may be configured, for example, to output a quantization parameter (QP) directly by entropy encoding unit 270 or to output a quantization parameter (QP) that has been encoded by entropy encoding unit 270, so that, for example, video decoder 30 may receive and apply the quantization parameter for decoding.

[0164] inverse quantization

[0165] The inverse quantization unit 210 is configured to apply the inverse quantization of the quantization unit 208 to the quantized coefficients, for example, by applying an inverse of the quantization scheme applied by the quantization unit 208, based on or using the same quantization step size as the quantization unit 208, to obtain dequantized coefficients 211. The dequantized coefficients 211 may also be referred to as dequantized residual coefficients 211 and typically correspond to the transform coefficients 207, although they are not the same as the transform coefficients due to quantization losses.

[0166] Inverse transformation

[0167] The inverse transform processing unit 212 is configured to apply an inverse transform of the transform applied by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), or other inverse transform, to obtain a reconstructed residual block 213 (or corresponding dequantized coefficients 213) in the sample domain. The reconstructed residual block 213 may also be referred to as a transform block 213.

[0168] Reconstruction

[0169] The reconstruction unit 214 (e.g., an adder or summing adder 214) is configured to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265, e.g., by adding the sample values ​​of the reconstructed residual block 213 and the sample values ​​of the prediction block 265 sample by sample, to obtain the reconstructed block 215 in the sample domain.

[0170] filtering

[0171] Loop filter unit 220 (or "loop filter" 220 for short) is configured to filter reconstructed block 215 to obtain filtered block 221, or generally, to filter reconstructed samples to obtain filtered samples. The loop filter unit is configured, for example, to smooth pixel transitions or otherwise improve video quality. Loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample adaptive offset (SAO) filter, or one or more other filters, such as a bilateral filter, an adaptive loop filter (ALF), a sharpening filter, a smoothing filter, or a collaborative filter, or any combination thereof. Although loop filter unit 220 is shown in FIG. 2 as an in-loop filter, in other configurations, loop filter unit 220 may be implemented as a post-loop filter. Filtered block 221 may also be referred to as filtered reconstructed block 221.

[0172] Some embodiments of video encoder 20 (respectively, loop filter unit 220) may be configured, for example, to output loop filter parameters (such as sample adaptive offset information) directly by entropy coding unit 270 or to output loop filter parameters that have been coded by entropy coding unit 270, so that, for example, decoder 30 may receive and apply the same loop filter parameters or the respective loop filters for decoding.

[0173] The video buffer being decoded

[0174] The decoded picture buffer (DPB) 230 may be a memory that stores a reference picture, or generally, reference picture data, for encoding video data by video encoder 20. The DPB 230 may be comprised of any of a variety of memory devices, such as dynamic random access memory (DRAM) or other types of memory devices, including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), and resistive RAM (RRAM). The decoded picture buffer (DPB) 230 may be configured to store one or more filtered blocks 221. The decoded picture buffer 230 may also be configured to store other previously filtered blocks, such as previously reconstructed and filtered blocks 221, of the same current picture or of different pictures, such as previously reconstructed pictures, and may provide a complete previously reconstructed picture, i.e., the picture being decoded (and corresponding reference blocks and samples) and / or a partially reconstructed current picture (and corresponding reference blocks and samples), for example, for inter-frame prediction. The decoded picture buffer (DPB) 230 may also be configured to store one or more reconstructed unfiltered blocks 215, for example in the case where the reconstructed blocks 215 have not been filtered by the loop filter unit 220, or, in general, reconstructed unfiltered samples, or any other further processed version of the reconstructed blocks or samples.

[0175] Mode Selection (Segmentation and Prediction)

[0176] The mode selection unit 260 includes a partitioning unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, and is configured to receive or obtain original video data, such as the original block 203 (the current block 203 of the current video 17), and reconstructed video data, such as reconstructed filtered and / or unfiltered samples or blocks of one or more previously decoded videos, for example from the video buffer 230 being decoded or another buffer (such as a line buffer, not shown), of the same (current) video. The reconstructed video data is used as reference video data for prediction, such as inter-frame prediction or intra-frame prediction, to obtain a prediction block 265 or predictor 265.

[0177] The mode selection unit 260 may be configured to determine or select a partitioning and a prediction mode (e.g., an intra-frame prediction mode or an inter-frame prediction mode) for a current block prediction mode (not including partitioning) and generate a corresponding prediction block 265, which is used for the calculation of the residual block 205 and for the reconstruction of the reconstructed block 215.

[0178] Embodiments of the mode selection unit 260 may be configured to select the partitioning and prediction mode (e.g., from the partitioning and prediction modes supported by or available to the mode selection unit 260) that provides the best match, i.e., the smallest residual (smallest residual means better compression for transmission or storage) or the smallest signaling overhead (smallest signaling overhead means better compression for transmission or storage), or that considers or balances both the partitioning and prediction mode. The mode selection unit 260 may also be configured to determine the partitioning and prediction mode based on rate-distortion optimization (RDO), i.e., select the prediction mode that provides the smallest rate-distortion. Terms such as "best," "minimum," "optimum," etc. in this context do not necessarily refer to the overall "best," "minimum," "optimum," etc., but rather to the achievement of termination or selection criteria, such as reducing complexity and processing time, while a value exceeds or falls below a threshold or other constraint that may lead to a "suboptimal selection."

[0179] In other words, the partitioning unit 262 may be configured to partition the block 203 into smaller block partitions or sub-blocks (which again form blocks), e.g., using quadtree partitioning (QT), binary partitioning (BT), ternary tree partitioning (TT), or any combination thereof iteratively, and then perform prediction for each of those block partitions or sub-blocks, e.g., where the mode selection includes selecting a tree structure for the block 203 to be partitioned, and a prediction mode is applied to each of those block partitions or sub-blocks.

[0180] The following description provides a more detailed explanation of the partitioning (eg, by partitioning unit 260) and prediction (eg, by inter-frame prediction unit 244 and intra-frame prediction unit 254) processes performed by exemplary video encoder 20.

[0181] compartmentalization

[0182] The partitioning unit 262 may partition (or distribute) the current block 203 into smaller partitions, such as smaller blocks of square or rectangular size. These smaller blocks (which may also be referred to as sub-blocks) may be further partitioned into even smaller partitions. This partitioning may also be referred to as tree partitioning or hierarchical tree partitioning, where the root block at root tree level 0 (hierarchical level 0, depth 0) may be partitioned recursively into two or more blocks at the next subtree level, such as a node at tree level 1 (hierarchical level 1, depth 1), which may then be partitioned again into two or more blocks at the next sublevel, such as tree level 2 (hierarchical level 2, depth 2), until the partitioning terminates, for example, because a termination criterion is met, such as reaching a maximum tree depth or a minimum block size. Blocks that are not further partitioned are also referred to as leaf blocks or leaf nodes of the tree. A tree that uses a partitioning into two partitions is called a binary tree (BT), a tree that uses a partitioning into three partitions is called a ternary tree (TT), and a tree that uses a partitioning into four partitions is called a quad tree (QT).

[0183] As mentioned above, the term "block" as used herein may refer to a portion of a picture, in particular a square or rectangular portion. For example, with reference to HEVC and VVC, a block may be or correspond to a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), and a transform unit (TU), and / or a corresponding block, such as a coding tree block (CTB), a coding block (CB), a transform block (TB), or a prediction block (PB).

[0184] For example, a coding tree unit (CTU) may be or include a CTB for luma samples, two corresponding CTBs for chroma samples for a video having a three-sample arrangement, or a CTB for monochrome video or video samples coded using three distinct color planes and syntax configurations used to code the samples. Correspondingly, a coding tree block (CTB) may be an N x N block of samples for some value of N, whereby the division of a component into multiple CTBs constitutes a partition. A coding unit (CU) may be or include a coding block for luma samples, two corresponding coding blocks for chroma samples for a video having a three-sample arrangement, or a coding block for monochrome video or video samples coded using three distinct color planes and syntax configurations used to code the samples. Correspondingly, a coding block (CB) may be an M x N block of samples for some values ​​of M and N, whereby the division of a CTB into multiple coding blocks constitutes a partition.

[0185] In some embodiments, for example, according to HEVC, coding tree units (CTUs) may be distributed into multiple CUs by using a quadtree structure, which is referred to as a coding tree. The decision of whether to code a video region using inter-frame (temporal) prediction or intra-frame (spatial) prediction is made at the CU level. Each CU may be further distributed into one, two, or four PUs according to a PU distribution type. Within a PU, the same prediction process is applied, and related information is transmitted to the decoder on a PU-by-PU basis. After obtaining residual blocks by applying a prediction process based on the PU distribution type, the CU may be partitioned into transform units (TUs) according to another quadtree structure similar to the coding tree for a CU.

[0186] In some embodiments, for example, according to the latest video coding standard currently under development, referred to as Versatile Video Coding (VVC), a combined quadtree and binary tree (QTBT) partitioning is used to partition coding blocks. In the QTBT block structure, CUs may have either a square or rectangular shape. For example, a coding tree unit (CTU) is first partitioned using a quadtree structure. The quadtree leaf nodes are further partitioned using a binary tree structure or a triplet (or three-fold) tree structure. The partitioned tree leaf nodes are called coding units (CUs), and their subdivisions are used for prediction and transform processing without any further partitioning. This means that CUs, PUs, and TUs have the same block size in the QTBT coding block structure. At the same time, multiple partitioning methods, such as ternary tree partitioning, may also be used with the QTBT block structure.

[0187] In some embodiments, a virtual pipeline data unit (VPDU) is defined to facilitate processing pipelining in hardware with limited internal memory compared to a CTU, such as in the Draft VVC Standard. A VPDU is a virtual partitioning of a CTU into uniform sub-blocks of luminance samples and corresponding chrominance samples in a particular processing order across multiple partitions within the CTU, such that processing of a given VPDU does not depend on processing of any other future VPDUs in the processing order. However, specific syntax elements may still be signaled in the bitstream at the CTU level and must apply to all of the VPDUs within the CTU. Specific constraints on partitioning may be imposed to ensure that a coding unit spans one or more VPDUs completely but cannot partially cover a VPDU. In one example, mode select unit 260 of video encoder 20 may be configured to perform any combination of the partitioning techniques described herein.

[0188] As described above, video encoder 20 is configured to determine or select a best or optimal prediction mode from a (predetermined) set of prediction modes, which may include, for example, intra-frame and / or inter-frame prediction modes.

[0189] Intra-frame prediction

[0190] The set of intra-frame prediction modes may include 35 different intra-frame prediction modes, such as non-directional modes such as DC (or average) mode and planar mode, or directional modes defined in HEVC, or may include 67 different intra-frame prediction modes, such as non-directional modes such as DC (or average) mode and planar mode, or directional modes defined in VVC.

[0191] The intra prediction unit 254 is configured to generate an intra prediction block 265 using a plurality of reconstructed samples of a plurality of neighboring blocks of the same current image according to an intra prediction mode from a set of intra prediction modes.

[0192] Intra prediction unit 254 (or, generally, mode selection unit 260) is further configured to output intra prediction parameters (or, generally, information indicating the selected intra prediction mode for the block) in the form of syntax element 266 to entropy coding unit 270 for inclusion in the video data 21 being encoded, so that, for example, video decoder 30 may receive and use the prediction parameters for decoding.

[0193] Inter-frame prediction

[0194] The set of inter-frame prediction modes (or possible inter-frame prediction modes) depends on the available reference picture (i.e., a previous, at least partially decoded picture stored, for example, in DBP 230) and other inter-frame prediction parameters, such as whether the whole or only a part of the reference picture, e.g., a search window area around the area of ​​the current block of the reference picture, is used to search for the best matching reference block, and / or whether pixel interpolation, e.g., half-pel / quasi-pel and / or quarter-pel interpolation, is applied. In addition to the above prediction modes, skip mode and / or direct mode may also be applied.

[0195] Inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both not shown in FIG. 2 ). The motion estimation unit may be configured to receive or obtain, for motion estimation, video block 203 (current video block 203 of current video 17) and at least one or more previously reconstructed blocks, such as reconstructed blocks of one or more other / different previously decoded videos 231, from the video being decoded 231. For example, a video sequence may include the current video and the previously decoded video 231; in other words, the current video and the previously decoded video 231 may be part of a sequence of videos that form a video sequence, or may form a sequence of videos that form a video sequence.

[0196] The encoder 20 may be configured to, for example, select a reference block from a plurality of reference blocks of the same or a different one of a plurality of other pictures, and provide the reference picture (or reference picture index) and / or an offset (spatial offset) between the position (x-coordinate, y-coordinate) of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit, this offset also being called a motion vector (MV).

[0197] The motion compensation unit is configured to obtain inter-frame prediction parameters, such as receiving inter-frame prediction parameters, and perform inter-frame prediction based on or using the inter-frame prediction parameters, thereby obtaining inter-frame prediction block 265. The motion compensation performed by the motion compensation unit may include retrieving or generating a prediction block based on a motion vector / block vector determined by motion estimation, and may perform interpolation up to sub-pixel accuracy. Interpolation filtering may generate additional pixel samples from known pixel samples, thus potentially increasing the number of candidate prediction blocks that may be used to code the video block. Upon receiving a motion vector for the PU of the current video block, the motion compensation unit may locate the prediction block to which the motion vector points in one of the reference picture lists.

[0198] The motion compensation unit may also generate blocks and syntax elements associated with the video slice for use by video decoder 30 in decoding the video blocks of the video slice. In addition to or as an alternative to slices and their respective syntax elements, the motion compensation unit may also generate or use tile groups and / or tiles and their respective syntax elements.

[0199] As described in more detail below, the embodiments presented herein provide improvements to the inter-frame prediction unit 244 by providing more accurate motion vector predictions for use by the inter-frame prediction unit when performing inter-frame prediction, such as bidirectional optical flow (BDOF)-based inter-frame prediction.

[0200] Entropy Coding

[0201] The entropy coding unit 270 is configured to apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context-adaptive VLC scheme (CAVLC), an arithmetic coding scheme, binarization, context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probabilistic interval partitioned entropy (PIPE) coding, or other entropy coding method or technique), or bypass (uncompressed), to, for example, the quantized coefficients 209, the inter-frame prediction parameters, the intra-frame prediction parameters, the loop filter parameters, and / or other syntax elements to obtain coded video data 21, which may be output via output 272, for example, in the form of coded bitstream 21, so that, for example, video decoder 30 can receive and use the parameters for decoding. The coded bitstream 21 may be transmitted to video decoder 30 or stored in memory for later transmission or retrieval by video decoder 30.

[0202] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform-based encoder 20 may quantize the residual signal directly for a particular block or frame, without using the transform processing unit 206. In other implementations, the encoder 20 may have the quantization unit 208 and the inverse quantization unit 210 combined into a single unit.

[0203] Decoder and decoding method

[0204] 3 shows an example of a video decoder 30 configured to implement techniques of this application. The video decoder 30 is configured to receive encoded video data 21 (e.g., encoded bitstream 21) encoded by, for example, encoder 20 to obtain decoded video 331. The encoded video data or bitstream includes information for decoding the encoded video data, such as data representing video blocks and associated syntax elements of a video slice (and / or tile group or tile) being encoded.

[0205] 3, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summation adder 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. Inter-frame prediction unit 344 may be or include a motion compensation unit. Video decoder 30, in some examples, may perform a decoding path that is generally reverse to the encoding path described with respect to video encoder 100 from FIG. 2.

[0206] As described with respect to encoder 20, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 344, and intra-frame prediction unit 354 are also considered to form an "embedded decoder" of video encoder 20. Accordingly, inverse quantization unit 310 may be functionally the same as inverse quantization unit 110, inverse transform processing unit 312 may be functionally the same as inverse transform processing unit 212, reconstruction unit 314 may be functionally the same as reconstruction unit 214, loop filter 320 may be functionally the same as loop filter 220, and decoded picture buffer 330 may be functionally the same as decoded picture buffer 230. Accordingly, the descriptions provided for the respective units and functions of video encoder 20 apply correspondingly to the respective units and functions of video decoder 30.

[0207] Entropy Decoding

[0208] The entropy decoding unit 304 is configured to analyze the bitstream 21 (or, generally, the coded video data 21) and, for example, perform entropy decoding on the coded video data 21 to obtain quantized coefficients 309 and / or decoded coding parameters (not shown in FIG. 3 ), such as any or all of inter-frame prediction parameters (e.g., reference picture indices and motion vectors), intra-frame prediction parameters (e.g., intra-frame prediction modes or indices), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 may be configured to apply a decoding algorithm or scheme corresponding to the coding scheme, as described with respect to the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 may further be configured to provide the inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice level and / or the video block level. In addition to or as an alternative to slices and their respective syntax elements, tile groups and / or tiles and their respective syntax elements may be received or used.

[0209] inverse quantization

[0210] Inverse quantization unit 310 may be configured to receive quantization parameters (QPs) (or, generally, information regarding inverse quantization) and quantized coefficients from encoded video data 21 (e.g., by parsing and / or decoding by entropy decoding unit 304), and to apply inverse quantization to quantized decoded coefficients 309 based on the quantization parameters to obtain dequantized coefficients 311, which may also be referred to as transform coefficients 311. The inverse quantization process may include using a quantization parameter determined by video encoder 20 for each video block in a video slice (or tile or tile group) to determine the degree of quantization, and similarly, the degree of inverse quantization that needs to be applied.

[0211] Inverse transformation

[0212] The inverse transform processing unit 312 may be configured to receive dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transform to those dequantized coefficients 311 to obtain, in the sample domain, reconstructed residual blocks 213. Those reconstructed residual blocks 213 may also be referred to as transform blocks 313. The transform may be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may further be configured to receive transform parameters or corresponding information from the coded video data 21 (e.g., by analyzing and / or decoding, by the entropy decoding unit 304) to determine the transform to apply to the dequantized coefficients 311.

[0213] Reconstruction

[0214] The reconstruction unit 314 (e.g., an adder or summing adder 314) may be configured to add the reconstructed residual block 313 to the prediction block 365, e.g., by adding the sample values ​​of the reconstructed residual block 313 and the sample values ​​of the prediction block 365, to obtain the reconstructed block 315 in the sample domain.

[0215] filtering

[0216] Loop filter unit 320 (either in the coding loop or after the coding loop) is configured to filter reconstructed block 315 to obtain filtered block 321, e.g., to smooth pixel transitions or otherwise improve video quality. Loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample adaptive offset (SAO) filter, or one or more other filters, such as, for example, a bilateral filter, an adaptive loop filter (ALF), a sharpening filter, a smoothing filter, or a collaborative filter, or any combination thereof. Although loop filter unit 320 is shown in FIG. 3 as being an in-loop filter, in other configurations, loop filter unit 320 may be implemented as a post-loop filter.

[0217] The video buffer being decoded

[0218] The decoded video blocks 321 of the picture are then stored in a decoded picture buffer 330, which stores the decoded picture 331 as a reference picture for subsequent motion compensation for other pictures and / or for respective display on the output. The decoder 30 is configured to output the decoded picture 311 for presentation or viewing to a user, for example, via an output 312.

[0219] prediction

[0220] The inter prediction unit 344 may be identical to the inter prediction unit 244 (in particular the motion compensation unit), and the intra prediction unit 354 may be functionally identical to the inter prediction unit 254, performing the distribution decision or the partitioning decision and prediction based on the partitioning parameters and / or prediction parameters or respective information received from the video data 21 being encoded (e.g., by parsing and / or decoding by the entropy decoding unit 304). The mode application unit 360 may be configured to perform prediction (intra prediction or inter prediction) for each block based on the (filtered or unfiltered) reconstructed video, block, or respective sample to obtain a prediction block 365.

[0221] When a video slice is coded as an intra-coded (I) slice, intra prediction unit 354 of mode application unit 360 is configured to generate a prediction block 365 for a video block of the current video slice based on data from previously decoded blocks of the current picture and a signaled intra prediction mode. When a video picture is coded as an inter-coded (i.e., B or P) slice, inter prediction unit 344 (e.g., a motion compensation unit) of mode application unit 360 is configured to generate a prediction block 365 for a video block of the current video slice based on motion vectors and other syntax elements received from entropy decoding unit 304. For inter prediction, the prediction block may be generated from one of multiple reference pictures in one of multiple reference picture lists. Video decoder 30 may construct the reference frame lists, List 0 and List 1, using a default construction technique based on the reference pictures stored in DPB 330. For example, the same or similar may apply to or among multiple embodiments using multiple tile groups (e.g., video tile groups) and / or multiple tiles (e.g., video tiles) in addition to or as an alternative to slices (e.g., video slices), such that it is possible to code video using I, P, or B tile groups and / or tiles.

[0222] As described in more detail below, the embodiments presented in this specification provide improvements to the inter-frame prediction unit 344 by providing more accurate motion vector predictions for use by the inter-frame prediction unit when performing inter-frame prediction, such as bidirectional optical flow (BDOF)-based inter-frame prediction.

[0223] Mode application unit 360 is configured to determine prediction information for video blocks of the current video slice by analyzing motion vectors or related information and other syntax elements, and use the prediction information to generate a prediction block for the current video block being decoded. For example, mode application unit 360 uses some of the received syntax elements to determine a prediction mode (e.g., intra prediction or inter prediction) to be used to code the video blocks of the video slice, an inter prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more of a plurality of reference picture lists for the slice, a motion vector for each inter coded video block of the slice, an inter prediction state for each inter coded video block of the slice, and other information for decoding video blocks in the current video slice. For example, the same or similar may apply to or for multiple embodiments using multiple tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles) in addition to or as an alternative to slices (e.g., video slices), such that it is possible to code video using I, P, or B tile groups and / or tiles.

[0224] Embodiments of video decoder 30 shown in FIG. 3 may be configured to partition and / or decode video using slices (also referred to as video slices), where video may be partitioned into or decoded using one or more (typically non-overlapping) slices, each of which may include one or more blocks (e.g., CTUs).

[0225] The embodiments of video decoder 30 shown in FIG. 3 may be configured to partition and / or decode video by using tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles), where the video may be partitioned into or decoded using one or more (typically non-overlapping) tile groups, each of which may include, for example, one or more blocks (e.g., CTUs) or one or more tiles, and each tile may be, for example, rectangular in shape and may include, for example, one or more blocks (e.g., CTUs), such as complete blocks or fractional blocks.

[0226] Other variations of the video decoder 30 may be used to decode the encoded video data 21. For example, the decoder 30 may generate an output video stream without using the loop filtering unit 320. For example, a non-transform-based decoder 30 may inverse quantize the residual signal directly for a particular block or frame, without using the inverse transform processing unit 312. In other implementations, the video decoder 30 may have the inverse quantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0227] It should be understood that the encoder 20 and decoder 30 may further process the processing result of the current step and then output the processing result to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations such as clipping or shifting may be performed on the processing result of the interpolation filtering, motion vector derivation, or loop filtering.

[0228] It should be noted that further operations may be applied to the derived motion vector of the current block (including, but not limited to, control point motion vectors in affine mode, sub-block motion vectors in affine, planar, and ATMVP modes, and temporal motion vectors). For example, the value of a motion vector is restricted to a predefined range according to its representation bits. If the representation bits of a motion vector are bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" denotes the exponential function. For example, if bitDepth is set equal to 16, the range is -32768 to 32767, and if bitDepth is set equal to 18, the range is -131072 to 131071. For example, the values ​​of derived motion vectors (such as the MVs of four 4x4 sub-blocks in an 8x8 block) are constrained so that the maximum difference between the integer parts of the MVs of those four 4x4 sub-blocks is less than or equal to N pixels, such as less than or equal to 1 pixel. This specification provides two methods for constraining motion vectors according to bitDepth.

[0229] Method 1: Operation ux=(mvx+2 bitDepth )%2 bitDepth (1) mvx=(ux>=2 bitDepth-1 ) ? (ux-2 bitDepth ):ux (2) uy=(mvy+2 bitDepth )%2 bitDepth (3) mvy=(uy>=2 bitDepth-1 ) ? (uy-2 bitDepth ):uy (4) The overflow MSB (Most Significant Bit) is removed by mvx = mvy + uy + uy + uy + mvx + mvy + uy + uy + mvx + mvy + uy + uy + mvx + mvy + uy + mvx + mvy + uy + mvy + mvx + mvy ...

[0230] For example, if the value of mvx is -32769, after applying equations (1) and (2), the resulting value will be 32767. In computer systems, decimal numbers are stored as two's complement numbers. The two's complement of -32769 is 1, 0111, 1111, 1111, 1111 (17 bits), and then the MSB is discarded, so the resulting two's complement becomes 0111, 1111, 1111, 1111 (decimal number is 32767), and the resulting two's complement is the same as the output by applying equations (1) and (2). ux=(mvpx+mvdx+2 bitDepth )%2 bitDepth (5) mvx=(ux>=2 bitDepth-1 ) ? (ux-2 bitDepth ):ux (6) uy=(mvpy+mvdy+2 bitDepth )%2 bitDepth (7) mvy=(uy>=2 bitDepth-1 ) ? (uy-2 bitDepth ):uy (8) As shown in equations (5) to (8), these operations may be applied when summing mvp and mvd.

[0231] Method 2: Remove the overflow MSB by clipping the value. vx=Clip3(-2 bitDepth-1 ,2 bitDepth-1 -1,vx) (9) vy=Clip3(-2 bitDepth-1 ,2 bitDepth-1 -1,vy) (10) vx is the horizontal component of the motion vector of the image block or sub-block, vy is the vertical component of the motion vector of the image block or sub-block, x, y, and z correspond to the three input values ​​of the MV clipping process, respectively, and the definition of the function Clip3 is:

number

[0232] 4 is a schematic diagram of a video coding device 400 according to one embodiment. The video coding device 400 is suitable for implementing several disclosed embodiments as described herein. In one embodiment, the video coding device 400 may be a decoder, such as the video decoder 30 of FIG. 1A, or an encoder, such as the video encoder 20 of FIG. 1A.

[0233] Video coding device 400 includes an ingress port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 for processing the data, a transmitter unit (Tx) 440 and an egress port 450 (or output port 450) for transmitting the data, and a memory 460 for storing the data. Video coding device 400 may also include optical-electrical (OE) and electro-optical (EO) components coupled to ingress port 410, receiver unit 420, transmitter unit 440, and egress port 450 for the egress or ingress of optical or electrical signals.

[0234] The processor 430 is implemented in hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., a multi-core processor), FPGA, ASIC, and DSP. The processor 430 communicates with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470. The coding module 470 implements multiple disclosed embodiments as described above. For example, the coding module 470 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the coding module 470 provides substantial improvements to the functionality of the video coding device 400 and provides transformations of the video coding device 400 into different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0235] Memory 460 may include one or more disks, tape drives, and solid state drives, and may be used as an overflow data storage device to store programs when they are selected for execution, and to store instructions and data retrieved during the execution of programs. Memory 460 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0236] 5 is a simplified block diagram of an apparatus 500 that may be used as either or both of the source device 12 and the destination device 14 of FIG. 1A according to one example embodiment. The processor 502 in the apparatus 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices now existing or later developed that are capable of manipulating or processing information. While the disclosed implementations may be implemented using a single processor, such as the processor 502 as shown, more than one processor may be used to achieve advantages in speed and efficiency.

[0237] In one implementation, the memory 504 in the apparatus 500 may be a read-only memory (ROM) device or a random-access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 that the processor 502 accesses using a bus 512. The memory 504 may further include an operating system 508 and application programs 510, which include at least one program that enables the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1-N, which further include a video coding application that performs the methods described herein.

[0238] The device 500 may also include one or more output devices, such as a display 518. The display 518, in one example, may be a touch-sensitive display that combines a display with a touch-sensitive element operable to detect touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0239] Although shown herein as a single bus, bus 512 of device 500 may be comprised of multiple buses. Additionally, secondary storage 514 may be directly coupled to other components of device 500 or may be accessed over a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 500 may be implemented in a wide variety of configurations.

[0240] Motion Vector Refinement (MVR)

[0241] Motion vectors are typically at least partially determined at the encoder side and transmitted to the decoder by signaling in the coded bitstream. On the other hand, motion vectors may be refined in the decoder (and also in the encoder) starting from an initial motion vector indicated in the bitstream. In such cases, the accuracy of the initial motion vector may be improved, for example, by using similarities between multiple partitions of already decoded pixels pointed to by the initial motion vector. Such motion refinement offers the advantage of reducing signaling overhead, i.e., improving the accuracy of the initial motion vector in the same way in both the encoder and the decoder, and therefore not requiring any additional signaling for refinement.

[0242] It should be noted that the initial motion vector before refinement may not be the best motion vector that results in the best prediction. Since the initial motion vector is signaled in the bitstream, it may not be possible to express the initial motion vector with very high accuracy (which would increase the bitrate), and therefore the motion vector refinement process is used to improve the initial motion vector. The initial motion vector may, for example, be the motion vector used to predict the neighboring block of the current block. In this case, the initial motion vector is sufficient to signal an indicator in the bitstream to indicate which neighboring block's motion vector is used by the current block. Such a prediction mechanism is very efficient in reducing the number of bits representing the initial motion vector. However, since the motion vectors of two neighboring blocks are generally not expected to be the same, the accuracy of the initial motion vector may be low.

[0243] To further improve the accuracy of the motion vectors without further increasing signaling overhead, it may be beneficial to further refine the motion vectors derived at the encoder side and provided in the bitstream (transmitted by signaling). The motion vector refinement can be performed at the decoder without assistance from the encoder. The encoder in the decoder loop can use the same refinement to obtain the corresponding refined motion vector that will be available at the decoder. Refinement for the current block being reconstructed in the current picture is performed by determining a template for the reconstructed sample, determining a search space around the initial motion information for the current block, and finding the reference picture portion within the search space that best matches the template. The best match determines the refined motion vector for the current block, and the refined motion vector is then used to obtain the inter-frame predicted sample for the current block, i.e., the current block being reconstructed. Motion vector refinement is part of the inter-frame prediction unit (244) in FIG. 2 and 344 in FIG. 3.

[0244] The refinement of the motion vector may be performed according to the following steps: Typically, the initial motion vector may be determined based on an index in the bitstream. For example, an index indicating a position in a list of candidate motion vectors may be signaled in the bitstream. In another example, a motion vector prediction index and a motion vector differential value may be signaled in the bitstream. The motion vector determined based on the index in the bitstream is defined as the initial motion vector. In the case of bidirectional prediction, the inter-frame prediction for the current block is obtained as a weighted combination of predicted blocks of samples determined according to two motion vectors MV0 and MV1. In this specification, MV0 is the initial motion vector of the first reference picture in list L0, and MV1 is the initial motion vector of the second reference picture in list L1.

[0245] The initial motion vectors are used to determine refinement candidate motion vector (MV) pairs. At least two refinement candidate pairs need to be determined. Typically, the refinement candidate motion vector pair is determined based on the initial motion vector pair (MV0, MV1). Further, the candidate MV pair is determined by adding a small motion vector difference to MV0 and MV1. For example, the candidate MV pair is (MV0,MV1) (MV0+(0,1),MV1+(0,-1)) (MV0+(1,0),MV1+(-1,0)) (MV0+(0,-1),MV1+(0,1)) (MV0+(-1,0),MV1+(1,0)) … In this specification, (1,-1) denotes a vector having a displacement of 1 in the horizontal (or x) direction and a displacement of -1 in the vertical (or y) direction. It should be noted that the above list of candidate pairs is merely an illustrative example, and the present invention is not limited to any particular list of candidates. In some examples, the search space of the motion vector refinement process includes refined candidate motion vector (MV) pairs.

[0246] In bidirectional prediction of the current block, the two prediction blocks obtained using the respective first motion vector for list L0 and the second motion vector for list L1 are combined into a single prediction signal that may provide a better adaptation to the original signal than unidirectional prediction, resulting in smaller residual information and possibly more efficient compression.

[0247] In motion vector refinement, two prediction blocks obtained using the first and second motion vectors of each candidate MV pair are compared based on a similarity metric for each refined candidate MV pair. The candidate MV pair that results in the highest similarity metric is selected as the refined motion vector. The refined motion vector for the first reference picture in list L0 and the refined motion vector for the second reference picture in list L1 are denoted as MV0' and MV1', respectively. In other words, predictions corresponding to the list L0 motion vector and the list L1 motion vector of the candidate motion vector pair are obtained, and then the list L0 motion vector and the list L1 motion vector are compared based on a similarity metric. The candidate motion vector pair with the highest relative similarity is selected as the refined MV pair.

[0248] Typically, the output of the refinement process is refined MVs. These refined MVs may be the same as or different from the initial MVs, depending on which candidate MV pair achieves the highest similarity, and the candidate MV pair formed by the initial MV is also an MV pair among multiple candidate MV pairs. In other words, if the highest candidate MV pair achieving the highest similarity is formed by the initial MV, the refined MV and the initial MV are equal to each other.

[0249] Instead of selecting the location that maximizes the similarity metric, another method is to select the location that minimizes the dissimilarity metric. The dissimilarity comparison measure may be SAD (Sum of absolute differences), MRSAD (mean removed sum of absolute differences), SSE (Sum of Squared Error), etc. The SAD between two prediction blocks may be obtained using a candidate MV pair (CMV0, CMV1), and the SAD is

number

[0250] Alternatively, the dissimilarity comparison measure may be obtained by evaluating only a subset of samples in the prediction block to reduce the number of calculations. One example is as follows, where rows of samples are alternatively included in the SAD calculation (such that every other row is evaluated):

number

[0251] One example of motion vector refinement is described in the JVET (ITU-T SG 16 WP3 and ISO / IEC JTC 1 / SC 29 / WG11) document JVET-M1001-v3 "Multipurpose Video Coding (Draft 4)", which is publicly available at http: / / phenix.it-sudparis.eu / jvet / . Section "8.4.3 Decoder-side motion vector refinement process" in that document illustrates motion vector refinement.

[0252] To reduce internal memory requirements for refinement, in some embodiments, the motion vector refinement process may be performed independently for multiple blocks of luma samples. A coded block of luma samples exceeding a certain predetermined width or height may be obtained by partitioning the coded block of luma samples into sub-blocks of luma samples of equal to or less than the predetermined width and height. The refined MV pair for each sub-block in the partitioned coded block may be different. Then, inter-frame prediction for both luma and chroma is performed for each sub-block using the refined MV pair of the sub-block.

[0253] Let max_sb_width and max_sb_height denote the maximum allowable sub-block width and maximum allowable sub-block height, respectively. A current coding unit of size cbWidth×cbHeight to which MVR is applied is typically partitioned into multiple sub-blocks numSbs, each of size sbWidth×sbHeight, as follows:

number

[0254] Typically, a two-dimensional interpolation of the reference integer sample grid values ​​being reconstructed is performed to obtain sample values ​​at fractional sample offset positions. The process of obtaining predicted samples from the reference picture being reconstructed using candidate MV pairs may be performed by one of the following methods: Round the fractional part of the initial MV pair to the nearest integer position and obtain integer grid values ​​for the reference image being reconstructed. · Perform a two-tap (e.g., bilinear) separable bilinear interpolation to obtain the predicted sample values ​​with fractional pixel accuracy indicated by the initial MV pair. · Perform separable interpolation of higher order taps (e.g., 8-tap or 6-tap) to obtain the predicted sample values ​​with fractional pixel accuracy indicated by the initial MV pair.

[0255] While the candidate MV pairs may have any sub-pixel offset relative to the initial MV pair, in some embodiments, to simplify the search, the candidate MV pairs are selected by integer-pixel distance relative to the initial MV pair. In such cases, predictions for a block of samples around the initial MV pair may be performed to obtain predicted samples across the multiple candidate MV pairs by covering all of the multiple refined positions around the initial MV pair.

[0256] In some embodiments, after evaluating dissimilarity cost values ​​for candidate MV pairs at an integer distance from the initial MV pair, additional candidate MV pairs with subpixel offsets from the best cost value location may be added and evaluated. A predicted sample is obtained for each of these locations using one of the methods described above, and the dissimilarity costs are evaluated and compared to obtain the lowest dissimilarity location. In other embodiments, to avoid this computationally expensive prediction process for each subpixel distance location around the best cost integer distance location, the evaluated integer distance cost values ​​are stored, and a parametric error surface is fitted in the vicinity of the best integer distance location. The minimum of this error surface is then analytically calculated and used as the location with the least dissimilarity. In such cases, the dissimilarity cost value is derived from the calculated integer distance cost value.

[0257] The application of motion vector refinement to a given coded block of samples may be conditioned on specific coding characteristics of the coded block of samples. Some examples of such coding characteristics include that the two reference pictures used for bidirectional prediction of the coded block of samples are equally spaced in picture number (when sampled at a uniform frame rate) from the current picture and are located on opposite sides of the current picture. The coding characteristics may also include that the initial dissimilarity between the two prediction blocks obtained using the initial MV pair is smaller than a predetermined per-sample threshold.

[0258] In some implementations, BPOF is applied to bidirectionally predicted blocks when the two predictions are from different reference pictures. BPOF is not applied in the case of affine transformation and weighted bidirectional motion compensated sub-block-based advanced temporal merge modes.

[0259] Bidirectional predictive optical flow refinement

[0260] Bidirectional optical flow refinement is a process that improves the accuracy of bidirectional prediction of a block without explicitly providing any additional signal in the bitstream other than the signal for bidirectional prediction. The bidirectional optical flow refinement can be implemented by the inter-frame prediction unit 244 in Figure 2 and the inter-frame prediction unit 344 in Figure 3. The input of the optical flow refinement process is prediction samples from two reference pictures, and the output of the optical flow refinement is a combined prediction (predBIO) calculated according to the optical flow equation.

[0261] In the case of bidirectional prediction, two inter-frame predictions are obtained from two reference frames according to two motion vectors, such as the motion vector pair MV0 and MV1 described above or a refined motion vector pair. The two predictions may be combined by weighted averaging or the like. The combined prediction can result in a reduction in residual energy because the quantization noise in the two predictions is canceled, thereby providing greater coding efficiency compared to unidirectional prediction (i.e., prediction using one motion vector). In one example, the weighted combination in bidirectional prediction may be performed as follows: Bi-prediction=Prediction1*W1+Prediction2*W2+K, (19) In the above equation, W1 and W2 are weighting coefficients, which may be signaled in the bitstream or may be predefined. K is an additional coefficient, which may also be signaled or may be predefined. As an example, Bi-prediction=(Prediction1+Prediction2) / 2, (20) In the above equation, W1 and W2 are set to 1 / 2 and K is set to 0.

[0262] The accuracy of bidirectional prediction can be improved by optical flow refinement. Optical flow is the pattern of apparent movement of image objects between two frames caused by object or camera motion. The optical flow refinement process improves the accuracy of bidirectional prediction by determining the optical flow between two reference frames and adjusting the bidirectional prediction based on the determined optical flow.

[0263] Consider a pixel I(x,y,t) in the first frame with x and y corresponding to spatial coordinates and t corresponding to the time dimension. That pixel I(x,y,t) has moved a distance (dx,dy) in the next frame, dt time later. Assuming that the pixels in the two frames are the same and that their intensity does not change in time dt, the optical flow equation is: I(x,y,t)=I(x+v x ,y+v y ,t+dt) (21) where I(x,y,t) specifies the intensity (i.e., sample value) of the pixel at the coordinates (x,y,t). Based on the assumption that the movement or displacement of the pixel is small and other assumptions such as higher order terms in the Taylor series expansion can be ignored, the optical flow equation is:

number

number

number

number

[0264] Optical flow refinement utilizes the principle shown in equation (22) to improve the quality of bidirectional prediction. In some implementations, optical flow refinement uses the sample gradients

number

number

number

[0265] The displacement (v x ,v y ) are used to solve the minimization problem mentioned above. Some methods minimize the sum of squared errors for different pixels in the reference frame patch. Other methods minimize the sum of absolute errors. x ,v y ), the prediction of the combination at a given position (x,y) is

number

[0266] From this equation, assuming τ0 and τ1 are 1, the offset determined based on the BDOF is

number

[0267] In some embodiments, to simplify the complexity of estimating displacements for each pixel, the displacements are estimated for groups of pixels. For example, the displacements may be estimated for blocks of 4x4 pixels, such as 4x4 luma samples, rather than for individual pixels. In these examples, to calculate improved bidirectional prediction for a block of 4x4 luma samples, the displacements are estimated using sample values ​​from neighboring blocks of 4x4 luma samples, such as a block of 8x8 luma samples centered on the 4x4 block of samples. When the width or height of a coding unit exceeds 16, the coding unit is partitioned into multiple sub-blocks. At the boundaries of a sub-block, multiple sample gradients are calculated using integer grid reference sample values ​​without any two-dimensional separable motion-compensated interpolation. Then, extended samples and sample gradients outside the sub-block positions are obtained by extending the nearest sample values ​​and sample gradient values ​​from the sub-block boundaries.

[0268] The input of the optical flow refinement process is the predicted samples from two reference videos, and the output of the optical flow refinement is the combined prediction (predBIO) calculated according to the optical flow equation.

[0269] The currently adopted version of BDOF calculates optical flow (v) based on the horizontal and vertical gradients of a 6x6 block of samples centered around the current 4x4 block of samples. x ,v y ) using the following equation:

number

[0270] Optical flow displacement (v x ,v y ) is also called "optical flow (vx ,v y )"v x and v y The division operation required to calculate x is simplified at the expense of precision by right-shifting the numerator using only the most significant bit position in the denominator. In other prior art, precision is improved by replacing the division with an N-bit lookup table containing the reciprocal values, using a variable shift representing the most significant bit position. However, the lookup table increases on-chip memory. An N-bit lookup table with M-bit precision for the reciprocal requires N*M bits of SRAM.

[0271] One example of a motion vector refinement is described in document JVET-M1001, Multi-Objective Video Coding (Draft 4), in 8.4.7.4 "Bidirectional Optical Flow Estimation Process".

[0272] As explained above, optical flow is calculated by the horizontal v x and vertical v y Compared to the method shown in equations (25)-(31), the horizontal v x and vertical v y The calculations presented here for the two components of eliminate multiplication operations and reduce the bit depth of the terms.

[0273] In particular, optical flow

number

number

number

number

number

[0274] It should be understood that in calculating the above cross-correlation and auto-correlation terms s1 through s5, one or more of the terms can be shifted to adjust the precision and bit depth of their values.

[0275] Furthermore, it should be noted that equations (32)-(38) listed above are for illustrative purposes only and should not be construed as limiting. Various terms in these equations may be pre-processed before being combined with other terms in these equations. For example, the term (G y1 (i,j)+G y0 (i,j)) or G x1 (i,j)+G x0 (i,j) may be shifted, inverted by changing sign, or otherwise manipulated before being used in the manner shown in the above equations to calculate s1 through s5. Similarly, the term (I (1) -I (0)) may also be pre-processed before being combined with other terms in the above equation. Similarly, the various values ​​determined in the above equation may also be pre-processed before being combined with other terms in the above equation. x and v y For example, the s determined above may be post-processed before being used to calculate the value for k (k=1,…,5) is s k may be post-processed by adding the low order bits of the value to the high order bits of the value to determine the final version of v. This final version may then be used as shown above to find v x and v y may be determined.

[0276] As can be seen from this implementation, in this example, a new number s5 is determined to represent the second component of the optical flow v y This facilitates the calculation of v. s5 is determined based on the sign of the sum of predicted sample gradients in the vertical direction across two reference frames and the sum of the products of the sum of predicted sample gradients in the horizontal direction across those two reference frames. It is possible to achieve the calculation of s5 without multiplication operations. For example, it can be performed by conditionally adding or subtracting the sum of predicted sample gradients in the horizontal direction at a given sample location based on the sign of the sum of predicted sample gradients in the vertical direction at that sample location. Then, v x Based on s5, s6, and s7, the vertical component of the optical flow v y In some implementations, it is possible to simplify the division by s1 or s2 in equations (32)-(38) by applying a right shift equal to the position of the most significant bit position in s1 and s2, respectively. As a result, the calculation of components s1 to s5 for the optical flow specified in equations (32)-(44), especially the calculation of component s5, reduces the bit depth because it is based on sign operations. In particular, since the calculation involves only absolute and sign operations, v x By reducing the bit depth of y In the calculation of vx It is possible to reduce the bit depth of the multiplier for v × s5. x and v y This significantly reduces the computational complexity of the calculations for

[0277] As can be seen from the above, BDOF therefore requires less computation, especially in terms of the number of multiplications and the size of the multipliers. In some examples, BDOF is used to refine the bidirectional predicted signal of a CU at the level of a 4x4 sub-block and is applied only to the luma component. The BDOF mode is based on the optical flow concept, which assumes that object motion is smooth. For each 4x4 sub-block, a motion refinement or motion offset is calculated by minimizing the difference between the L0 predicted sample and the L1 predicted sample. The motion refinement is then used to adjust the bidirectionally predicted sample values ​​in the 4x4 sub-block.

[0278] As explained above, the horizontal gradient and vertical gradient are calculated by calculating the difference between two adjacent samples in the corresponding reference frame.

number

number

[0279] Movement refinement (v x ,v y ) then,

number

number

number

number

number

number

[0280] 8 illustrates one example of a process 800 for performing bi-predictive optical flow refinement based on the optical flow calculations presented herein. One or more computing devices (e.g., encoding device 200 or decoding device 300) perform the operations illustrated in FIG. 8 by executing appropriate program code.

[0281] Block 810 corresponds to the first step described above. In this block, two motion vectors are received as input. An initial motion vector can be determined based on indication information in the bitstream. For example, an index indicating a position in a list of candidate motion vectors can be signaled in the bitstream. In another example, a motion vector predictor index and a motion vector differential value can be signaled in the bitstream. In another example, these motion vectors can be derived as refined motion vectors using motion vector refinement starting from an initial pair of motion vectors indicated in the bitstream. In another example, multiple reference frame indices can be obtained from the bitstream, indicating reference frames to which a given motion vector of the obtained motion vector pair is associated. For example, the indices can specify that a frame from a first reference frame list L0 is associated with motion vector MV0 of the motion vector pair, and a frame from a second reference frame list L1 is associated with motion vector MV1 of the motion vector pair.

[0282] Block 820 corresponds to the second step described above. In this block, unidirectional prediction can be obtained in each of two reference frames (i.e., reconstructed luminance samples) according to the obtained motion vector pair and K-tap interpolation filter. For example, when the motion vector corresponds to an integer sample position, the prediction obtains multiple reconstructed reference sample values. When the motion vector has a non-zero horizontal component but a zero vertical component, the prediction performs horizontal K-tap interpolation to obtain the predicted sample values. When the motion vector has a non-zero vertical component but a zero horizontal component, the prediction performs vertical K-tap interpolation to obtain the predicted sample values. When the motion vector has non-zero values ​​for both horizontal and vertical components, two-dimensional separable K-tap interpolation is performed to obtain the predicted sample values, with horizontal interpolation performed first followed by vertical interpolation. In this way, a first prediction is generated using MV0 among the reference frames from the first reference frame list L0, and a second prediction is generated using MV1 among the reference frames from the second reference frame list L1.

[0283] Block 830 corresponds to the third step described above. In this block, the optical flow is estimated for each sub-block in a given current coding unit using the prediction obtained in the second step in the reference frame. Consistent with the notation described above, the predicted samples from the prediction obtained in the first reference frame are denoted by I (0) The predicted sample obtained in the second criterion is shown as I (1)The horizontal sample gradient at a location (i,j) can be calculated by taking the difference between the predicted sample value to the right of this location and the sample value to the left of this location, i.e., ∂I / ∂x = I(x+1,y) - I(x-1,y). The vertical sample gradient at a location (i,j) can be calculated by taking the difference between the predicted sample value below this location and the sample value above this location, i.e., ∂I / ∂y = I(x,y+1) - I(x,y-1). It should be noted that in the case of an image or frame, the horizontal direction runs from left to right and the vertical direction runs from top to bottom. In some examples, for a set of locations within the current coding sub-block, ∂I (0) / ∂x and ∂I (0) / ∂y and ∂I (1) / ∂x and ∂I (1) Based on these determined sample gradients, the optical flow may be determined using the method described above using the respective equations (31)-(40), or the iterative optical flow estimation method described above using the respective equations (38)-(46) or (47)-(52).

[0284] Block 840 corresponds to the fourth step described above, in which the final inter-frame bidirectional predicted samples for the current coding block can be calculated according to equation (24), which takes into account the predicted sample values, the determined sample gradients, and the estimated optical flow.

[0285] FIG. 9 is a flowchart of an example method 900 for bidirectional optical flow (BDOF)-based inter-frame prediction for a current block of a video signal.

[0286] In step 910, the method 900 calculates the horizontal motion offset v of the current block. x and vertical movement offset vy The method includes determining a vertical motion offset based on the horizontal motion offset and a fifth variable s5, where the fifth variable s5 represents a sum of a plurality of terms, each of which is obtained from the signs of elements of the second matrix and elements of the first matrix, and the elements of the first matrix correspond to the elements of the second matrix.

[0287] Each element of the first matrix is ​​obtained from the sum of a first horizontal predicted sample gradient corresponding to the first reference frame of the current block and a second horizontal predicted sample gradient corresponding to the second reference frame of the current block. The first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to elements of the first matrix. Each element of the second matrix is ​​obtained from the sum of a first vertical predicted sample gradient corresponding to the first reference frame of the current block and a second vertical predicted sample gradient corresponding to the second reference frame of the current block. The first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to elements of the second matrix.

[0288] In step 920, the process 900 includes determining a predicted sample value in the current block using a predicted sample value corresponding to the first reference frame, a predicted sample value corresponding to the second reference frame, a horizontal motion offset, and a vertical motion offset.

[0289] It should be noted that the current block may be a block of any size, such as a 4x4 block. The current block may also be a sub-block of a frame of a video signal. Pixels of the current block may be referenced using the absolute position of the pixel relative to the top left corner of the frame, e.g., (x, y), or using the relative position of the pixel with respect to the top left corner of the block, e.g., (xBlock+i, yBlock+j). In this specification, (xBlock, yBlock) are the coordinates of the top left corner of the block with respect to the top left corner of the frame.

[0290] The first matrix and the second matrix may be any two-dimensional array including rows and columns, and elements of the array may be indicated using (i, j), where x is the horizontal / row index and y is the vertical / column index. The range of i and j may be, for example, i = xBlock-1, ..., xBlock+4 and j = yBlock-1, ..., yBlock+4. The first matrix and the second matrix correspond to the current block or are determined for the current block. In some examples, the size of the first matrix is ​​the same as the size of the second matrix, which may be larger than the size of the current block. For example, the size of the first matrix and the second matrix may be 6x6, while the size of the current block is 4x4.

[0291] If the position (x, y) of the first element in the first matrix is ​​the same as the position (p, q) of the second element in the second matrix, i.e., (x, y) = (p, q), then the element (first element) of the first matrix corresponds to the element (second element) of the second matrix. The first horizontal predicted sample gradient corresponds to the first reference frame of the current block, meaning that the first horizontal predicted sample gradient is generated based on samples in the first reference frame of the current block. The second horizontal predicted sample gradient corresponds to the second reference frame of the current block, meaning that the second horizontal predicted sample gradient is generated based on samples in the second reference frame of the current block. The first horizontal predicted sample gradient corresponds to an element of the first matrix, meaning that the first horizontal predicted sample gradient is generated for the position (x, y) of that element in the first matrix. Similarly, a second horizontal predicted sample gradient corresponds to an element of a first matrix, meaning that the second horizontal predicted sample gradient is generated for the element's position (x, y) in the first matrix.

[0292] The first vertical predicted sample gradient corresponding to the first reference frame of the current block means that the first vertical predicted sample gradient is generated based on samples in the first reference frame of the current block. The second vertical predicted sample gradient corresponding to the second reference frame of the current block means that the second vertical predicted sample gradient is generated based on samples in the second reference frame of the current block. The first vertical predicted sample gradient corresponding to an element of the second matrix means that the first vertical predicted sample gradient is generated for the position (p, q) of that element in the second matrix. Similarly, the second vertical predicted sample gradient corresponding to an element of the second matrix means that the second vertical predicted sample gradient is generated for the position (p, q) of that element in the second matrix.

[0293] When each element of a matrix is ​​obtained from the sum of two terms, it means that the element may be determined as the sum of the two terms itself, or as the value after processing the sum of the two terms. The processing may include left shifting, right shifting, clipping, or a combination thereof. Similarly, when a term is obtained from the sign of an element of a second matrix and an element of a first matrix, it means that the term may take the value of the element of the first matrix itself, or the value after the element of the first matrix is ​​processed, and then the sign of the element of the second matrix is ​​applied. The processing of the element of the first matrix may include left shifting, right shifting, clipping, or a combination thereof. The sign of element x is

number

[0294] The technique presented herein adjusts the bidirectionally predicted sample values ​​of the current block based on a horizontal motion offset and a vertical motion offset. The vertical motion offset is calculated based on a fifth variable s5, which merely involves the summation of terms obtained from the signs of elements of a second matrix and elements of a first matrix. Applying the sign of one element to another element does not involve a multiplication operation. Similarly, the summation also does not involve any multiplication. As a result, the BDOF-based inter-frame prediction technique presented herein eliminates multiplication operations. Compared to conventional approaches, the multiplication operations are replaced by sign decisions, thereby reducing the bit depth of the fifth variable s5. This means that the horizontal motion offset v x and vertical movement offset v y This leads to a reduction in the bit depth of the .DELTA..times ...

[0295] 10 illustrates a device 1000 for bidirectional optical flow (BDOF)-based inter-frame prediction for a current block of a video signal.

[0296] The horizontal movement offset v of the current block x and vertical movement offset v y wherein the vertical motion offset is determined based on the horizontal motion offset and a fifth variable s5, the fifth variable s5 indicating a sum of a plurality of terms, each of which is obtained from the signs of elements of the second matrix and elements of the first matrix, and the elements of the first matrix correspond to the elements of the second matrix.

[0297] The present invention includes a prediction processing unit 1003 configured to predict a predicted sample value in a current block using a predicted sample value corresponding to a first reference frame, a predicted sample value corresponding to a second reference frame, a horizontal motion offset, and a vertical motion offset.

[0298] The determining unit 1001 is further configured to obtain each element of the first matrix from a sum of a first horizontal predicted sample gradient corresponding to a first reference frame of the current block and a second horizontal predicted sample gradient corresponding to a second reference frame of the current block, where the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to elements of the first matrix.

[0299] The determining unit 1001 is further configured to obtain each element of the second matrix from a sum of a first vertical predicted sample gradient corresponding to a first reference frame of the current block and a second vertical predicted sample gradient corresponding to a second reference frame of the current block, and the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to elements of the second matrix.

[0300] Correspondingly, in one example, the exemplary configuration of device 1000 may correspond to encoder 200 in Figure 2. In another example, the exemplary configuration of device 1000 may correspond to decoder 300 in Figure 3. In another example, the exemplary configuration of device 1000 may correspond to inter-frame prediction unit 244 in Figure 2. In another example, the exemplary configuration of device 1000 may correspond to inter-frame prediction unit 344 in Figure 3.

[0301] The techniques presented herein for computing optical flow and bidirectionally predicted samples improve coding efficiency by computing a second component of the optical flow based on an independently computed first component of the optical flow. Because the dependent computation also does not require any multiplication operations, computational complexity remains low. By conditionally adding or subtracting a sum of multiple gradients in a first direction based on the sign of the sum of multiple gradients in a second direction, it is possible to achieve the sum of products of the signs of the sum of multiple gradients in a second direction and the sum of multiple gradients in a first direction without performing any multiplications. The techniques presented herein also achieve compression efficiency similar to those methods that use multiplication operations.

[0302] The present disclosure provides the following further aspects.

[0303] According to a first aspect, there is provided a method for optical flow based inter-frame prediction, the method comprising: - determining an optical flow for the current coding block, wherein a second component of the optical flow is determined or derived based on a first component of the optical flow according to a first formulation (such that in a bidirectionally predictive optical flow based on inter-frame bidirectional prediction, v y is based on v x , or v x is based on v y , etc.); - obtaining or deriving predicted sample values ​​(such as bidirectionally predicted sample values) for the current sub-block using the determined optical flow for the current coding block.

[0304] In one possible implementation of the method according to any preceding implementation of the first aspect or the first aspect itself, determining the optical flow for the current coding block comprises: Calculating an optical flow for the current coding block, wherein a second component of the optical flow is a first component of the optical flow; the sign and absolute value of the sum of corresponding predicted sample gradients (such as corresponding sample positions in the two predicted blocks) across two reference frames in the direction corresponding to the second component; the sum of corresponding predicted sample gradients across those two reference frames in the direction corresponding to the first component; The steps include:

[0305] In one possible implementation of the method according to any preceding implementation of the first aspect or the first aspect itself, the step of obtaining or deriving predicted sample values ​​(such as bidirectionally predicted sample values) for the current sub-block using the determined optical flow for the current coding block comprises: The method includes a step of obtaining bidirectionally predicted sample values ​​for the current coding block using predicted sample values ​​in two reference frames, the calculated optical flow, and a set of horizontal and vertical sample gradients, wherein the set of predicted sample values ​​is obtained in each of the two reference frames using a pair of motion vectors for the current coding block relative to the two reference frames.

[0306] In one possible implementation of a method according to any preceding implementation of the first aspect or the first aspect itself, the method comprises: - calculating an optical flow for the current coding block, the second component of the optical flow being: the first component of the optical flow being calculated; the sign and absolute value of the sum of the corresponding predicted sample gradients across the two reference frames in the direction corresponding to the second component; the sum of corresponding predicted sample gradients across two reference frames in the direction corresponding to the first component; The steps are calculated using - Obtaining bidirectionally predicted sample values ​​for the current coding block using predicted sample values ​​in two reference frames, the calculated optical flow, and a set of horizontal and vertical sample gradients, wherein the set of predicted sample values ​​is obtained in each of the two reference frames using a pair of motion vectors for the current coding block relative to the two reference frames.

[0307] According to a second aspect, there is provided a method for optical flow based inter-frame prediction, the method comprising: - obtaining a pair of motion vectors of the current coding block relative to two reference frames; - obtaining a set of prediction samples in each reference frame using the obtained motion vector pairs and the reconstructed luminance sample values ​​of the two reference frames; - calculating an optical flow for the current coding block using a first predicted sample difference between corresponding samples in two reference frames and a sum of corresponding horizontal sample gradients (sGx) and vertical sample gradients (sGy) in the two reference frames, wherein a second component of the optical flow is the first component of the optical flow being calculated; the sign and absolute value of the sum of the corresponding predicted sample gradients across the two references in the direction corresponding to the second component; the sum of the corresponding predicted sample gradients across the two references in the direction corresponding to the first component; The steps are calculated using - obtaining bidirectionally predicted sample values ​​for the current coding block based on a prediction equation for BDOF (Bi-Directional Optical Flow) using first predicted sample values ​​in two references, the calculated optical flow, and horizontal and vertical sample gradients.

[0308] In one possible implementation of a method according to any preceding implementation of the second aspect or the second aspect itself, the reconstructed reference luminance sample values ​​of the two reference frames include reconstructed adjacent luminance sample values ​​of the two reference frames.

[0309] In one possible implementation of the method according to any preceding implementation of the second aspect or the second aspect itself, the optical flow is calculated according to the optical flow equation

number

[0310] In one possible implementation of a method according to any preceding implementation of the second aspect or the second aspect itself, the method is used for bi-prediction; Correspondingly, the pair of motion vectors includes a first motion vector corresponding to the first reference frame list and a second motion vector corresponding to the second reference frame list; the obtained set of predicted samples includes a first set of predicted samples obtained according to a first motion vector and a second set of predicted samples obtained according to a second motion vector; the horizontal and vertical sample gradients include a first set of horizontal and vertical sample gradients calculated using the first set of predicted samples and a second set of horizontal and vertical sample gradients calculated using the second set of predicted samples; A motion offset is obtained based on the first and second sets of horizontal and vertical gradients and the first and second sets of predicted samples, and a predicted sample value for the current sub-block is obtained using the motion offset.

[0311] In one possible implementation of a method according to any preceding implementation of the second aspect or the second aspect itself, the second component of the optical flow (e.g., vy) is determined or derived based on the first component of the optical flow (e.g., vx) and one or more of the first value, the second value, the third value, the fourth value, and the fifth value; A first component (such as vx) of the optical flow is determined or derived based on one or more of the first value, the second value, the third value, the fourth value, and the fifth value.

[0312] In one possible implementation of the method according to any preceding implementation of the second aspect or the second aspect itself, vy is

number

[0313] In one possible implementation of a method according to any preceding implementation of the second aspect or the second aspect itself,

number

[0314] According to a fourth aspect, an encoder (20) includes processing circuitry for carrying out the method according to any one of the first and second aspects.

[0315] According to a fifth aspect, a decoder (30) includes processing circuitry for carrying out the method according to any one of the first and second aspects.

[0316] According to a sixth aspect, a computer program product comprises program code for carrying out the method according to any one of the first and second aspects.

[0317] According to a seventh aspect, there is provided a non-transitory computer-readable medium having program code thereon which, when executed by a computing device, causes the computing device to perform the method of any one of the first and second aspects.

[0318] According to an eighth aspect, there is provided a decoder comprising: one or more processors; and a non-transitory computer-readable storage medium coupled to the processors and storing programming for execution by the processors, the programming, when executed by the processors, configuring the decoder to perform a method according to any one of the first and second aspects.

[0319] According to a ninth aspect, there is provided an encoder comprising: one or more processors; and a non-transitory computer-readable storage medium coupled to the processors and storing programming for execution by the processors, the programming, when executed by the processors, configuring the encoder to perform a method according to any one of the first and second aspects.

[0320] According to a tenth aspect, there is provided an apparatus for optical flow based inter-frame prediction, the apparatus comprising: a determination unit configured to determine an optical flow for a current coding block, wherein a second component of the optical flow is determined or derived based on a first component of the optical flow (such as in bi-predictive optical flow-based inter-frame bi-directional prediction); and - an acquisition unit configured to acquire or derive predicted sample values ​​(e.g., bidirectionally predicted sample values) for the current sub-block using the optical flow determined for the current coding block.

[0321] The following describes several application examples of the encoding method and the decoding method shown in the above embodiments, and the system using them.

[0322] 11 is a block diagram illustrating a content delivery system 3100 for implementing a content distribution service. The content delivery system 3100 includes a capture device 3102 and a terminal device 3106, and optionally includes a display 3126. The capture device 3102 communicates with the terminal device 3106 via a communication link 3104. The communication link may include the communication channel 13 described above. The communication link 3104 may include, but is not limited to, WIFI, Ethernet, cable, wireless (3G / 4G / 5G), USB, or any combination thereof.

[0323] The capture device 3102 may generate data and encode it using the encoding method described in the above embodiments. Alternatively, the capture device 3102 may distribute the data to a streaming server (not shown), which encodes the data and transmits the encoded data to the terminal device 3106. The capture device 3102 may include, but is not limited to, a camera, a smartphone or iPad, a computer or laptop, a video conferencing system, a PDA, a vehicle-mounted device, or any combination thereof. For example, the capture device 3102 may include the source device 12 described above. When the data includes video, the video encoder 20 included in the capture device 3102 may actually perform the video encoding process. When the data includes audio (i.e., voice), the audio encoder included in the capture device 3102 may actually perform the audio encoding process. In some practical scenarios, the capture device 3102 distributes encoded video and audio data by multiplexing them together. In other practical scenarios, for example in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed: the capture device 3102 delivers the encoded audio data and the encoded video data separately to the terminal device 3106.

[0324] In the content delivery system 3100, the terminal device 3106 receives and plays the encoded data. The terminal device 3106 may be a device capable of receiving and restoring data, such as a smartphone or Pad 3108, a computer or laptop 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a TV 3114, a set-top box (STB) 3116, a video conferencing system 3118, a video surveillance system 3120, a personal digital assistant (PDA) 3122, an in-vehicle device 3124, or any combination thereof, capable of decoding the encoded data mentioned above. For example, the terminal device 3106 may include the destination device 14 described above. When the encoded data includes video, a video decoder 30 included in the terminal device is prioritized to perform video decoding. When the encoded data includes audio, an audio decoder included in the terminal device is prioritized to perform audio decoding.

[0325] For example, in the case of a terminal device having its own display, such as a smartphone or Pad 3108, a computer or laptop 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a TV 3114, a personal digital assistant (PDA) 3122, or an in-vehicle device 3124, the terminal device may provide the decoded data to its own display. For example, in the case of a terminal device not equipped with a display, such as an STB 3116, a video conferencing system 3118, or a video surveillance system 3120, an external display 3126 is contacted to receive and show the decoded data on the terminal device.

[0326] When each device in this system performs encoding or decoding, it may use the video encoding device or video decoding device shown in the above-mentioned embodiments.

[0327] 12 is a diagram illustrating an example configuration of the terminal device 3106. After the terminal device 3106 receives a stream from the capture device 3102, the protocol processing unit 3202 analyzes the transmission protocol of the stream, which may include, but is not limited to, Real Time Streaming Protocol (RTSP), HyperText Transfer Protocol (HTTP), HTTP Live Streaming Protocol (HLS), MPEG-DASH, Real Time Transport Protocol (RTP), Real Time Message Protocol (RTMP), or any combination thereof.

[0328] After the protocol processing unit 3202 processes the stream, it generates a stream file. The file is output to the demultiplexing unit 3204. The demultiplexing unit 3204 may separate the multiplexed data into encoded audio data and encoded video data. As described above, in some practical scenarios, such as in a video conferencing system, the encoded audio data and encoded video data are not multiplexed. In this situation, the encoded data is sent to the video decoder 3206 and the audio decoder 3208 without passing through the demultiplexing unit 3204.

[0329] The demultiplexing process generates a video element stream (ES), an audio ES, and optionally subtitles. As described in the above-mentioned embodiments, the video decoder 3206, including the video decoder 30, decodes the video ES using a decoding method to generate video frames, and provides this data to a synchronization unit 3212. The audio decoder 3208 decodes the audio ES to generate audio frames, and provides this data to a synchronization unit 3212. Alternatively, the video frames may be stored in a buffer (not shown in Figure Y) before being provided to the synchronization unit 3212. Similarly, the audio frames may be stored in a buffer (not shown in Figure Y) before being provided to the synchronization unit 3212.

[0330] The synchronization unit 3212 synchronizes the video and audio frames and provides the video / audio to the video / audio display 3214. For example, the synchronization unit 3212 synchronizes the presentation of video and audio information. The information may be coded according to a syntax using timestamps for the presentation of the audio and visual data being coded and for the delivery of the data stream itself.

[0331] If subtitles are included in the stream, the subtitle decoder 3210 decodes the subtitles, synchronizes the decoded subtitles with the video and audio frames, and provides the video / audio / subtitles to the video / audio / subtitle display 3216.

[0332] The present invention is not limited to the systems mentioned above, and any of the video encoding devices or video decoding devices in the embodiments mentioned above may be incorporated into other systems, such as, for example, vehicle systems.

[0333] mathematical operators

[0334] The mathematical operators used in this application are similar to those used in the C programming language, while the results of integer division and arithmetic shift operations are more precisely defined, and additional operations such as exponentiation and division of real values ​​are defined. Numbering and counting conventions generally start at 0, for example, "first" corresponds to 0, "second" corresponds to 1, etc.

[0335] Arithmetic operators

[0336] The following arithmetic operators are defined as follows:

number

[0337] Logical operators

[0338] The following logical operators are defined as follows:

number

[0339] Relational operators

[0340] The following relational operators are defined as follows:

number

[0341] Bitwise Operators

[0342] The following bitwise operators are defined as follows:

number

[0343] assignment operator

[0344] The following operators are defined as follows:

number

[0345] Range Notation

[0346] The following notation is used to specify a range of values:

number

[0347] mathematical functions

[0348] The following mathematical functions are defined:

number

[0349] The present invention has been described herein with reference to various embodiments. However, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that several means are recited in several different dependent claims does not indicate that a combination of these means cannot normally be used to advantage. A computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, or in other forms, such as via the Internet or other wired or wireless communication systems.

[0350] Those skilled in the art will appreciate that the "blocks" ("units") of the various figures (of methods and apparatus) represent or describe functions of multiple embodiments of the present invention (not necessarily individual "units" in hardware or software), and thus describe functions or features of method embodiments as well as apparatus embodiments (unit = step).

[0351] The term "unit" is used merely for the purpose of describing the functionality of multiple embodiments of the encoder / decoder and is not intended to limit the present disclosure.

[0352] It should be understood that in some of the embodiments provided by this application, the disclosed systems, devices, and methods may be implemented in other ways. For example, the described device embodiments are merely exemplary. For example, the division of units is merely a logical division of functions, and other divisions may be used in actual implementation. For example, multiple units or components may be combined or integrated into other systems, or some features may be ignored or not implemented. In addition, the shown or described mutual couplings, direct couplings, or communication connections may be implemented using some interfaces. Indirect couplings or communication connections between multiple devices or multiple units may be implemented in electronic, mechanical, or other forms.

[0353] The units described as separate parts may or may not be physically separated, and the parts shown as units may or may not be physical units, may be located in one place, or may be distributed across multiple network units. According to actual requirements, some or all of the units may be selected to achieve the objectives of the technical solutions of the embodiments.

[0354] Additionally, multiple functional units in multiple embodiments of the present invention may be integrated into a single processing unit, or each of the multiple units may exist physically alone, or two or more units may be integrated into a single unit.

[0355] Embodiments of the present invention may further include an apparatus, such as an encoder and / or decoder, that includes processing circuitry configured to perform any of the methods and / or processes described herein.

[0356] Although embodiments of the present invention have been described primarily in the context of video coding, it should be noted that embodiments of coding system 10, encoder 20, and decoder 30 (and, correspondingly, system 10), as well as other embodiments described herein, may also be configured for still picture processing or coding, i.e., processing or coding of individual pictures independent of any preceding or subsequent pictures, as in video coding. In general, when video processing coding is limited to a single picture 17, only inter-frame prediction units 244 (encoder) and 344 (decoder) may not be available. All of the other functions (also referred to as tools or techniques) of the video encoder 20 and the video decoder 30, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, partitioning 262 / 362, intra-frame prediction 254 / 354, and / or loop filtering 220, 320, and entropy coding 270 and entropy decoding 304, may equally be used for still image processing.

[0357] The embodiments of, for example, the encoder 20 and the decoder 30, and the functions described herein with reference to, for example, the encoder 20 and the decoder 30, may be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, the functions may be stored on a computer-readable medium or transmitted over a communication medium as one or more instructions or code and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium such as a data storage medium, or a communication medium, which includes any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. Thus, the computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that may be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described by this disclosure. A computer program product may include a computer-readable medium.

[0358] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that may be used to store required program code in the form of instructions or data structures and that may be accessed by a computer. Also, any connection is strictly referred to as a computer-readable medium. For example, if coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave are used to transmit instructions from a website, server, or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but rather relate to non-transitory and tangible storage media. As used herein, magnetic disks and disks include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks, where disks typically reproduce data magnetically, while disks reproduce data optically with a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0359] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuitry. Accordingly, the term "processor," as used herein, may refer to any of the above configurations or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a combined codec. Alternatively, the techniques may be implemented entirely by one or more circuit or logic elements.

[0360] The techniques of this disclosure may be implemented by a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). While this disclosure describes various components, modules, or units to emphasize functional aspects of devices configured to perform the disclosed techniques, implementation does not necessarily require multiple different hardware units. Rather, as noted above, the various units may be combined in a codec hardware unit or may be provided by a collection of interoperable hardware units including one or more processors as described above, along with appropriate software and / or firmware.

Claims

1. 1. A method for bidirectional optical flow (BDOF)-based inter-frame prediction for a current block of a video signal for use in a device for decoding video data, comprising: parsing the received bitstream to obtain indication information, said indication information indicating a pair of motion vectors for said current block; determining the pair of motion vectors for the current block with respect to a first reference frame and a second reference frame based on the indication; obtaining, from the first reference frame and the second reference frame, a predicted sample value corresponding to the first reference frame and a predicted sample value corresponding to the second reference frame, respectively, using the pair of motion vectors for the current block with respect to the first reference frame and the second reference frame; the horizontal motion offset v of the current block x and vertical movement offset v y wherein the vertical motion offset is determined by the horizontal motion offset and a fifth variable s 5 is determined based on The fifth variable s 5 denotes a sum of a plurality of first terms, each of which is obtained from a sign of an element of a second matrix and an element of a first matrix, and the element of the first matrix corresponds to the element of the second matrix; Each element of the first matrix is ​​obtained from the sum of a first horizontal predicted sample gradient corresponding to a first reference frame of the current block and a second horizontal predicted sample gradient corresponding to a second reference frame of the current block, and the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to the element of the first matrix; each element of the second matrix is ​​obtained from a sum of a first vertical predicted sample gradient corresponding to the first reference frame of the current block and a second vertical predicted sample gradient corresponding to the second reference frame of the current block, the first vertical predicted sample gradient and the second vertical predicted sample gradient corresponding to the element of the second matrix; determining a predicted sample value for the current block using the predicted sample value corresponding to the first reference frame, the predicted sample value corresponding to the second reference frame, the horizontal motion offset, and the vertical motion offset; A method comprising:

2. The vertical motion offset is a function of the horizontal motion offset, a second variable s 2 , the fourth variable s 4 , and the fifth variable s 5 is derived based on The second variable s 2 denotes the sum of the absolute values ​​of the elements of the second matrix, The fourth variable s 4 2. The method of claim 1 , wherein: denotes a summation of a plurality of second terms, each of the plurality of second terms being obtained from a sign of an element of the second matrix and an element of a third matrix, the element of the third matrix corresponding to the element of the second matrix, and each element of the third matrix being a difference obtained from a first predicted sample of the first reference frame corresponding to the element of the third matrix and a second predicted sample of the second reference frame corresponding to the element of the third matrix.

3. The horizontal motion offset is a first variable s 1 and a third variable s 3 is derived based on The first variable s 1 denotes the sum of the absolute values ​​of the elements of the first matrix, The third variable s 3 3. The method of claim 2, wherein: denotes a summation of a plurality of third terms, each of the plurality of third terms being obtained from a sign of an element of the first matrix and an element of the third matrix, the element of the third matrix corresponding to the element of the first matrix.

4. The horizontal motion offset is: [Number 58] is determined according to v x The method of claim 3 , wherein: represents the horizontal motion offset.

5. the vertical motion offset v y teeth, [Number 59] is determined in accordance with v x represents the horizontal movement offset, v y The method of claim 2 , wherein: represents the vertical motion offset.

6. s 1 , s 2 , s 3 , s 4 , and s 5 teeth, [Number 60] It is determined as I (0) is obtained from the predicted sample values ​​corresponding to the first reference frame, and I (1) is obtained from the predicted sample values ​​corresponding to the second reference frame, G x0 and G x1 denote the set of predicted sample gradients in the horizontal direction corresponding to the first and second reference frames, respectively; G y0 and G y1 denote the set of predicted sample gradients in the vertical direction corresponding to the first and second reference frames, respectively; 6. The method of claim 4, wherein i and j are integers, the value of i varying from −1 to 4, and the value of j varying from −1 to 4.

7. G x0 is determined as the difference obtained from two predicted samples corresponding to the first reference frame along the horizontal direction, and y0 The method of claim 6 , wherein {right arrow over (x)} is determined as the difference obtained from two predicted samples corresponding to the first reference frame along the vertical direction.

8. G x1 is determined as the difference obtained from two predicted samples corresponding to the second reference frame along the horizontal direction, and y1 The method of claim 6 , wherein {right arrow over (x)} is determined as the difference obtained from two predicted samples corresponding to the second reference frame along the vertical direction.

9. 9. A method according to any one of claims 1 to 8, wherein a reference frame index is also obtained by analyzing the bitstream, and the reference frame index indicates that the first reference frame from a first reference frame list L0 is associated with a motion vector MV0 of the pair of motion vectors, and the second reference frame from a second reference frame list L1 is associated with a motion vector MV1 of the pair of motion vectors.

10. The method of claim 1 , wherein the predicted sample values ​​for the current block are bidirectionally predicted sample values ​​based on bidirectional optical flow (BDOF) prediction.

11. 1. A device for decoding video data, comprising: a video data memory; a video encoder, the video encoder comprising: Parsing the received bitstream to obtain an indication of a pair of motion vectors for the current block; determining the pair of motion vectors for the current block with respect to a first reference frame and a second reference frame based on the indication; using the pair of motion vectors for the current block with respect to the first reference frame and the second reference frame to obtain, from the first reference frame and the second reference frame, a predicted sample value corresponding to the first reference frame and a predicted sample value corresponding to the second reference frame, respectively; the horizontal motion offset v of the current block x and vertical movement offset v y the vertical motion offset is determined by the horizontal motion offset and a fifth variable s 5 is determined based on The fifth variable s 5 denotes a sum of a plurality of first terms, each of which is obtained from a sign of an element of a second matrix and an element of a first matrix, and the element of the first matrix corresponds to the element of the second matrix; Each element of the first matrix is ​​obtained from the sum of a first horizontal predicted sample gradient corresponding to a first reference frame of the current block and a second horizontal predicted sample gradient corresponding to a second reference frame of the current block, and the first horizontal predicted sample gradient and the second horizontal predicted sample gradient correspond to the element of the first matrix; Each element of the second matrix is ​​obtained from the sum of a first vertical predicted sample gradient corresponding to the first reference frame of the current block and a second vertical predicted sample gradient corresponding to the second reference frame of the current block, and the first vertical predicted sample gradient and the second vertical predicted sample gradient correspond to the element of the second matrix; the video encoder is configured to determine a predicted sample value in the current block using the predicted sample value corresponding to the first reference frame, the predicted sample value corresponding to the second reference frame, the horizontal motion offset, and the vertical motion offset.

12. The vertical motion offset is a function of the horizontal motion offset, a second variable s 2 , the fourth variable s 4 , and the fifth variable s 5 is derived based on The second variable s 2 denotes the sum of the absolute values ​​of the elements of the second matrix, The fourth variable s 4 12. The device of claim 11, wherein x denotes a summation of multiple second terms, each of the multiple second terms being obtained from a sign of an element of the second matrix and an element of a third matrix, the element of the third matrix corresponding to the element of the second matrix, and each element of the third matrix being a difference obtained from a first predicted sample of the first reference frame corresponding to the element of the third matrix and a second predicted sample of the second reference frame corresponding to the element of the third matrix.

13. The horizontal motion offset is a first variable s 1 and a third variable s 3 is derived based on The first variable s 1 denotes the sum of the absolute values ​​of the elements of the first matrix, The third variable s 3 13. The device of claim 12, wherein: denotes a summation of multiple third terms, each of the multiple third terms being obtained from a sign of an element of the first matrix and an element of the third matrix, the element of the third matrix corresponding to the element of the first matrix.

14. The horizontal motion offset is: [Number 61] is determined according to v x The device of claim 13 , wherein: represents the horizontal motion offset.

15. the vertical motion offset v y teeth, [Number 62] is determined in accordance with v x represents the horizontal movement offset, v y The device of claim 12 , wherein: represents the vertical motion offset.

16. s 1 , s 2 , s 3 , s 4 , and s 5 teeth, [Number 63] It is determined as I (0) is obtained from the predicted sample values ​​corresponding to the first reference frame, and I (1) is obtained from the predicted sample values ​​corresponding to the second reference frame, G x0 and G x1 denote the set of predicted sample gradients in the horizontal direction corresponding to the first and second reference frames, respectively; G y0 and G y1 denote the set of predicted sample gradients in the vertical direction corresponding to the first and second reference frames, respectively; 16. The device of claim 14 or 15, wherein i and j are integers, the value of i varying from −1 to 4, and the value of j varying from −1 to 4.

17. G x0 is determined as the difference obtained from two predicted samples corresponding to the first reference frame along the horizontal direction, and y0 17. The device of claim 16, wherein {right arrow over (x)} is determined as a difference obtained from two predicted samples corresponding to the first reference frame along a vertical direction.

18. G x1 is determined as the difference obtained from two predicted samples corresponding to the second reference frame along the horizontal direction, and y1 17. The device of claim 16, wherein {right arrow over (x)} is determined as a difference obtained from two predicted samples corresponding to the second reference frame along a vertical direction.

19. 19. A device according to any one of claims 11 to 18, wherein a reference frame index is also obtained by analyzing the bitstream, and the reference frame index indicates that the first reference frame from a first reference frame list L0 is associated with a motion vector MV0 of the pair of motion vectors, and the second reference frame from a second reference frame list L1 is associated with a motion vector MV1 of the pair of motion vectors.

20. 20. The device of claim 11, wherein the predicted sample values ​​for the current block are bidirectionally predicted sample values ​​based on bidirectional optical flow (BDOF) prediction.

21. A decoder (20) including processing circuitry for carrying out the method of any one of claims 1 to 10.

22. A computer program comprising program code for performing the method according to any one of claims 1 to 10.

23. A non-transitory computer readable storage medium storing program code which, when executed by a computing device, causes the computing device to perform the method of any one of claims 1 to 10.

24. A decoder comprising: one or more processors; and a non-transitory computer-readable storage medium coupled to the processor and storing programming for execution by the processor, the programming, when executed by the processor, configuring the decoder to perform the method of any one of claims 1 to 10.