Method and apparatus for video decoding based on BI-directional optical flow(BDOF), method and apparatus for video encoding based on bdof

Abstract

Embodiments of the disclosure provide a video decoding apparatus and a method performed by the same, a storage medium, and a computer program product. The method includes: ddetermining residual motion related information of pixels in a current block; determining, based on the residual motion related information, whether to obtain a weighted sum of a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the current block; based on determining that the weighted sum is obtained, obtaining the optical flow compensation result of the current block using the weighted sum of the first optical flow compensation result and the second optical flow compensation result.

Description

METHOD AND APPARATUS FOR VIDEO DECODING BASED ON BI-DIRECTIONAL OPTICAL FLOW(BDOF), METHOD AND APPARATUS FOR VIDEO ENCODING BASED ON BDOF

[0001] The present disclosure relates to a field of video processing, and specifically to a method performed by an electronic apparatus, the electronic apparatus, a computer-readable storage medium, and a computer program product.

[0002] Since a codec as a whole adopts a block based encoding structure, existence of the residual motion problem within a block is very common. Residual motion is a relative displacement of a pixel within block with respect to a motion vector of the block obtained through motion estimation, which may result in a noticeable difference between a current block and a reference block thereof. Its existence eventually becomes a blurring and flickering problem for a fast moving object in video content visible to naked eyes. Since the existence of the residual motion increases the prediction error of the current block, the residual motion needs to be minimized as much as possible during encoding and decoding stages. When processing the same video content, a codec that is able to resolve the residual motion may obtain the same quality of video by transmitting fewer bitstream compared to a codec that is unable to resolve the residual motion. However, the residual motion problem can only be improved in very limited cases in existing codecs. In view of this, there is a need for a technology that can better improve the residual motion problem.

[0003] According to an embodiment of the present disclosure, it is provided a method performed by a video decoding apparatus, comprising: determining residual motion related information of pixels in a current block; determining, based on the residual motion related information, whether to obtain a weighted sum of a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the current block, wherein weights for the first optical flow compensation result and the second optical flow compensation result are determined based on a difference between a picture order count (POC) of the current block and a POC of the first reference block and a difference between the POC of the current block and a POC of the second reference block of the current block; based on determining that the weighted sum is obtained, obtaining the optical flow compensation result of the current block using the weighted sum of the first optical flow compensation result and the second optical flow compensation result.

[0004] The accompanying drawings herein are incorporated into and form part of the specification, illustrate embodiments consistent with the disclosure, which are used in conjunction with the specification to explain the principles of the disclosure and do not constitute an undue limitation of the disclosure.

[0005] FIG. 1 is a schematic diagram illustrating an inter prediction structure.

[0006] FIG. 2 is a schematic diagram illustrating an isometric bi-directional prediction.

[0007] FIG. 3 is a flowchart illustrating a method performed by a video decoding apparatus according to an embodiment of the present disclosure.

[0008] FIG. 4 is a schematic diagram illustrating an example of judgment of residual motion related information according to an embodiment of the present disclosure;

[0009] FIG. 5 is a schematic diagram illustrating a non-isometric bi-directional prediction.

[0010] FIG. 6 is a schematic diagram illustrating a structure of current BDOF technology.

[0011] FIG. 7 is a schematic diagram illustrating an example of judgment of pixel fluctuation related information according to an embodiment of the present disclosure.

[0012] FIG. 8 is a schematic diagram illustrating an example of obtaining an optical flow compensation result of a current block according to an embodiment of the present disclosure.

[0013] FIG. 9 is a flowchart illustrating a method performed by a video decoding apparatus or a video encoding apparatus according to an embodiment of the present disclosure.

[0014] FIG. 10 is a flowchart illustrating a method performed by a video decoding apparatus or a video encoding apparatus according to an embodiment of the present disclosure.

[0015] FIG. 11 is a flowchart illustrating a method performed by a video encoding apparatus according to an embodiment of the present disclosure.

[0016] FIG. 12 is a flowchart illustrating a method performed by a video encoding apparatus according to an embodiment of the present disclosure.

[0017] FIG. 13 is a flowchart illustrating a method performed by a video encoding apparatus according to an embodiment of the present disclosure.

[0018] FIG. 14 is a flowchart illustrating a method performed by a video decoding apparatus according to an embodiment of the present disclosure.

[0019] FIG. 15 is a flowchart illustrating a method performed by a video decoding apparatus according to an embodiment of the present disclosure.

[0020] FIG. 16 is a flowchart illustrating a method performed by a video decoding apparatus according to an embodiment of the present disclosure.

[0021] FIG. 17 is a block diagram illustrating a video decoding apparatus according to an embodiment of the present disclosure.

[0022] FIG. 18 is a block diagram illustrating a video encoding apparatus according to an embodiment of the present disclosure.

[0023] FIG. 19 is a schematic diagram illustrating a structure of an electronic apparatus according to an embodiment of the present disclosure.

[0024] According to an embodiment of the present disclosure, it is provided a method performed by a video decoding apparatus comprising: determining residual motion related information of pixels in a current block; determining (320), based on the residual motion related information, whether to obtain a weighted sum of a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the current block, wherein weights for the first optical flow compensation result and the second optical flow compensation result are determined based on a difference between a picture order count (POC) of the current block and a POC of the first reference block and a difference between the POC of the current block and a POC of the second reference block of the current block; based on determining that the weighted sum is obtained, obtaining (330) the optical flow compensation result of the current block using the weighted sum of the first optical flow compensation result and the second optical flow compensation result.

[0025] According to an embodiment of the present disclosure, wherein the obtaining of the optical flow compensation result of the current block, comprises: determining pixel fluctuation related information of the current block, wherein the pixel fluctuation related information represents a texture change situation in a region surrounding one pixel; determining whether a processing object of optical flow compensation is the current block or a sub-block of the current block, or a pixel in the current block, based on the pixel fluctuation related information; and obtaining, for the determined processing object, the optical flow compensation result of the processing object based on the first reference block and the second reference block.

[0026] According to an embodiment of the present disclosure, it is provided a method performed by a video decoding apparatus or a video encoding apparatus, comprising: determining residual motion related information of pixels in a current block; determining an optical flow compensation method for the current block based on the residual motion related information, wherein the optical flow compensation method comprises: fusing a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the current block, based on a picture order count (POC) of the current block and the POCs of the first reference block and the second reference block of the current block; obtaining the optical flow compensation result of the current block using the determined optical flow compensation method, based on the first reference block and the second reference block.

[0027] According to an embodiment of the present disclosure, the fusing of the first optical flow compensation result obtained based on the first reference block and the second optical flow compensation result obtained based on the second reference block comprises: performing a weighted sum on the first optical flow compensation result and the second optical flow compensation result, based on POC differences between the first reference block as well as the second reference block of the current block, respectively, and the current block.

[0028] According to an embodiment of the present disclosure, the obtaining of the optical flow compensation result of the current block using the determined optical flow compensation method, based on the first reference block and the second reference block, comprises: determining pixel fluctuation related information of the current block, wherein the pixel fluctuation related information represents a texture change situation in a region surrounding one pixel; determining whether a processing object of the optical flow compensation method is the current block or a sub-block of the current block, or a pixel in the current block, based on the pixel fluctuation related information; obtaining, for the determined processing object, the optical flow compensation result of the processing object using the optical flow compensation method, based on the first reference block and the second reference block.

[0029] According to an embodiment of the present disclosure, the determining of the pixel fluctuation related information of the current block comprises: determining the pixel fluctuation related information of the first reference block and the second reference block, respectively; determining the pixel fluctuation related information of the current block based on the pixel fluctuation related information of the first reference block and the second reference block.

[0030] According to an embodiment of the present disclosure, the determining of whether the processing object of the optical flow compensation method is the current block or the sub-block of the current block, or the pixel of the current block, based on the pixel fluctuation related information comprises: determining that the processing object of the optical flow compensation method is the pixel in the current block, if the pixel fluctuation related information of at least one of the first reference block and the second reference block exceeds a first threshold; determining that the processing object of the optical flow compensation method is the current block or the sub-block in the current block, if the pixel fluctuation related information of at least one of the first reference block and the second reference block does not exceed the first threshold.

[0031] According to an embodiment of the present disclosure, the determining of the residual motion related information of the pixels in the current block comprises: determining the residual motion related information of the pixels in the current block based on the first reference block and the second reference block of the current block.

[0032] According to an embodiment of the present disclosure, the determining of the residual motion related information of the pixels in the current block based on the first reference block and the second reference block of the current block comprises: determining the residual motion related information of the pixels in the current block based on a feature related to a pixel gradient of the first reference block and the second reference block.

[0033] According to an embodiment of the present disclosure, the determining of the residual motion related information of pixels in the current block based on the feature related to the pixel gradient of the first reference block and the second reference block comprises: calculating the feature of each pixel in the first reference block and the second reference block, respectively; determining the residual motion related information of the pixels in the current block based on number of pixels having a same coordinate position and the same feature in the first reference block and the second reference block.

[0034] According to an embodiment of the present disclosure, the feature comprises a value determined based on a vertical gradient and / or a horizontal gradient of each pixel in the first reference block and the second reference block.

[0035] According to an embodiment of the present disclosure, the residual motion related information of the pixels in the current block satisfies a first condition if the number of pixels having the same coordinate position and the same feature in the first reference block and the second reference block is greater than or equal to a second threshold, the residual motion related information of the pixels in the current block does not satisfy the first condition if the number of pixels having the same coordinate position and the same feature in the first reference block and the second reference block is less than the second threshold.

[0036] According to an embodiment of the present disclosure, weights used in the weighted sum are determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and a POC difference between the first reference block and the second reference block.

[0037] According to an embodiment of the present disclosure, the determining of the pixel fluctuation related information of the first reference block and the second reference block, respectively, comprises: determining gradients in a plurality of directions of each pixel of a plurality of pixels comprised in the first reference block and the second reference block, respectively; determining the pixel fluctuation related information of each of the first reference block and the second reference block based on the respectively determined gradients in the plurality of directions of each of the first reference block and the second reference block.

[0038] According to an embodiment of the present disclosure, the gradients in the plurality of directions comprises a gradient in a horizontal direction, a gradient in a vertical direction, and a gradient in at least one inclined direction different from the horizontal direction and vertical direction.

[0039] According to an embodiment of the present disclosure, it is provided a method performed by an electronic apparatus, comprising: determining a picture order count (POC) of a current block and the POCs of a first reference block and a second reference block of the current block; fusing a first optical flow compensation result of the current block obtained based on the first reference block and a second optical flow compensation result of the current block obtained based on the second reference block to obtain an optical flow compensation result of the current block, based on the determined POCs.

[0040] According to an embodiment of the present disclosure, the fusing of the first optical flow compensation result of the current block obtained based on the first reference block and the second optical flow compensation result of the current block obtained based on the second reference block to obtain the optical flow compensation result of the current block, based on the determined POCs, comprises: obtaining the optical flow compensation result of the current block by performing a weighted sum on the first optical flow compensation result and the second optical flow compensation result, based on POC differences between the first reference block as well as the second reference block, respectively, and the current block.

[0041] According to an embodiment of the present disclosure, the obtaining of the optical flow compensation result of the current block by performing the weighted sum on the first optical flow compensation result and the second optical flow compensation result, based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, comprises: determining a first weight corresponding to the first reference block and a second weight corresponding to the second reference block based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block; obtaining the optical flow compensation result of the current block by performing the weighted sum on the first optical flow compensation result and the second optical flow compensation result, using the first weight and the second weight.

[0042] According to an embodiment of the present disclosure, the determining of the first weight corresponding to the first reference block and the second weight corresponding to the second reference block based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, comprises: determining the first weight and the second weight based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and a POC difference between the first reference block and the second reference block.

[0043] According to an embodiment of the present disclosure, the determining of the first weight and the second weight based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and the POC difference between the first reference block and the second reference block, comprises: determining the first weight and the second weight based on a ratio of a first value to a second value, wherein the first value is determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and the second value is determined based on the POC difference between the first reference block and the second reference block.

[0044] According to an embodiment of the present disclosure, the method further comprises: determining optical flow vectors of the current block using POC differences between the first reference block as well as the second reference block, respectively, and the current block; obtaining the first optical flow compensation result and the second optical flow compensation result based on the optical flow vectors.

[0045] According to an embodiment of the present disclosure, the optical flow vectors of the current block comprises a horizontal direction optical flow vector and a vertical direction optical flow vector, wherein the determining of the optical flow vectors of the current block using the POC differences between the first reference block as well as the second reference block, respectively, and the current block, comprises: determining the horizontal direction optical flow vector and the vertical direction optical flow vector, respectively, based on gradient values of pixel values of the first reference block and the second reference block corresponding to a pixel value of the current block, in the horizontal direction and the vertical direction, respectively, and the POC differences between the first reference block as well as the second reference block, respectively, and the current block.

[0046] According to an embodiment of the present disclosure, the obtaining of the first optical flow compensation result and the second optical flow compensation result based on the optical flow vectors comprises: obtaining the first optical flow compensation result based on a POC difference between the first reference block and the current block, the optical flow vectors, and a pixel value of the first reference block corresponding to the pixel value of the current block; obtaining the second optical flow compensation result based on a POC difference between the second reference block and the current block, the optical flow vectors, and a pixel value of the second reference block corresponding to the pixel value of the current block.

[0047] According to an embodiment of the present disclosure, it is provided a method performed by an electronic apparatus, comprising: determining pixel fluctuation related information of a current block; determining whether a processing object of an optical flow compensation method is the current block or a sub-block of the current block, or a pixel in the current block, based on the pixel fluctuation related information; obtaining, for the determined processing object, an optical flow compensation result of the processing object using the optical flow compensation method, based on the first reference block and second reference block of the current block.

[0048] According to an embodiment of the present disclosure, the determining of the pixel fluctuation related information of the current block comprises: determining the pixel fluctuation related information of the first reference block and the second reference block, respectively; determining the pixel fluctuation related information of the current block based on the pixel fluctuation related information of the first reference block and the second reference block.

[0049] According to an embodiment of the present disclosure, the determining of the pixel fluctuation related information of the first reference block and the second reference block, respectively, comprises: determining gradients in a plurality of directions of each pixel of a plurality of pixels comprised in the first reference block and the second reference block, respectively; determining the pixel fluctuation related information of each of the first reference block and the second reference block based on the respectively determined gradients in the plurality of directions of each of the first reference block and the second reference block.

[0050] According to an embodiment of the present disclosure, the gradients in the plurality of directions comprises a gradient in a horizontal direction, a gradient in a vertical direction, and a gradient in at least one inclined direction different from the horizontal direction and vertical direction.

[0051] According to an embodiment of the present disclosure, the determining of the pixel fluctuation related information of each of the first reference block and the second reference block based on the respectively determined gradients in the plurality of directions of each of the first reference block and the second reference block comprises: determining, for the first reference block and the second reference block, the pixel fluctuation related information thereof by performing a sum or a weighted sum on the gradient in the horizontal direction, the gradient in the vertical direction, and the gradient in the at least one inclined direction of each pixel of the plurality of pixels comprised therein, respectively.

[0052] According to an embodiment of the present disclosure, it is provided a method performed by a video encoding apparatus, comprising: determining residual motion related information of pixels in a first encoded block; determining, based on the residual motion related information, whether to obtain a weighted sum of a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the first encoded block, wherein weights for the first optical flow compensation result and the second optical flow compensation result are determined based on a difference between a picture order count (POC) of the first encoded block and a POC of the first reference block and a difference between the POC of the first encoded block and a POC of the second reference block of the first encoded block; based on determining that the weighted sum is obtained, obtaining the optical flow compensation result of the first encoded block using the weighted sum of the first optical flow compensation result and the second optical flow compensation result; performing encoding on a second encoded block based on the optical flow compensation result; and generating a bitstream including encoding information of the second encoded block.

[0053] According to an embodiment of the present disclosure, the obtaining of the optical flow compensation result of the first encoded block, comprises: determining pixel fluctuation related information of the first encoded block, wherein the pixel fluctuation related information represents a texture change situation in a region surrounding one pixel; determining whether a processing object of optical flow compensation is the first encoded block or a sub-block of the first encoded block, or a pixel in the first encoded block, based on the pixel fluctuation related information; obtaining, for the determined processing object, the optical flow compensation result of the processing object based on the first reference block and the second reference block.

[0054] According to an embodiment of the present disclosure, the determining of the residual motion related information of the pixels in the first encoded block comprises: determining the residual motion related information of the pixels in the first encoded block based on the first reference block and the second reference block of the first encoded block.

[0055] According to an embodiment of the present disclosure, wherein the determining of the residual motion related information of the pixels in the first encoded block based on the first reference block and the second reference block of the first encoded block comprises: determining the residual motion related information of the pixels in the first encoded block based on a feature related to a pixel gradient of the first reference block and the second reference block.

[0056] According to an embodiment of the present disclosure, the determining of the pixel fluctuation related information of the first encoded block comprises: determining the pixel fluctuation related information of the first reference block and the second reference block, respectively; and determining the pixel fluctuation related information of the first encoded block based on the pixel fluctuation related information of the first reference block and the second reference block.

[0057] According to an embodiment of the present disclosure, the determining of the pixel fluctuation related information of the first reference block and the second reference block, respectively, comprises: determining gradients in a plurality of directions of each pixel of a plurality of pixels comprised in the first reference block and the second reference block, respectively; and determining the pixel fluctuation related information of each of the first reference block and the second reference block based on the respectively determined gradients in the plurality of directions of each of the first reference block and the second reference block.

[0058] According to an embodiment of the present disclosure, it is provided a method performed by an electronic apparatus, comprising: determining a picture order count (POC) of a first encoded block and the POCs of a first reference block and a second reference block of the first encoded block; fusing a first optical flow compensation result of the first encoded block obtained based on the first reference block and a second optical flow compensation result of the first encoded block obtained based on the second reference block to obtain an optical flow compensation result of the first encoded block, based on the determined POCs; performing encoding on a second encoded block based on the optical flow compensation result.

[0059] According to an embodiment of the present disclosure, it is provided a method performed by an electronic apparatus, comprising: determining pixel fluctuation related information of the first encoded block; determining whether a processing object of an optical flow compensation method is the first encoded block or a sub-block of the first encoded block, or a pixel in the first encoded block, based on the pixel fluctuation related information; obtaining, for the determined processing object, an optical flow compensation result of the processing object using the optical flow compensation method, based on a first reference block and a second reference block of the first encoded block; performing encoding on a second encoded block based on the optical flow compensation result.

[0060] According to an embodiment of the present disclosure, it is provided a method performed by an electronic apparatus, comprising: decoding a motion vector of a current block from a bitstream; determining a first reference block and a second reference block of the current block based on the motion vector of the current block; determining residual motion related information of pixels in the current block; determining an optical flow compensation method for the current block based on the residual motion related information, wherein the optical flow compensation method comprises: fusing a first optical flow compensation result obtained based on the first reference block and a second optical flow compensation result obtained based on the second reference block to obtain an optical flow compensation result of the current block, based on a picture order count (POC) of the current block and the POCs of the first reference block and the second reference block of the current block; obtaining the optical flow compensation result of the current block using the determined optical flow compensation method, based on the first reference block and the second reference block.

[0061] According to an embodiment of the present disclosure, it is provided a method performed by an electronic apparatus, comprising: decoding a motion vector of a current block from a bitstream; determining a first reference block and a second reference block of the current block based on the motion vector of the current block; determining picture order counts (POCs) of the current block, the first reference block and the second reference block; fusing, based on the determined POCs, a first optical flow compensation result of the current block obtained based on the first reference block and a second optical flow compensation result of the current block obtained based on the second reference block to obtain an optical flow compensation result of the current block.

[0062] According to an embodiment of the present disclosure, it is provided a method performed by an electronic apparatus, comprising: decoding a motion vector of a current block from a bitstream; determining a first reference block and a second reference block of the current block based on the motion vector of the current block; determining pixel fluctuation related information of the current block; determining whether a processing object of an optical flow compensation method is the current block or a sub-block of the current block, or a pixel in the current block, based on the pixel fluctuation related information; obtaining, for the determined processing object, an optical flow compensation result of the processing object using the optical flow compensation method, based on the first reference block and second reference block of the current block.

[0063] According to an embodiment of the present disclosure, it is a method of transmitting a bitstream, the bitstream being generated by any one of the methods performed by a video encoding apparatus as described above.

[0064] According to an embodiment of the present disclosure, it is provided a video decoding apparatus, comprising: a memory; and a processor coupled to the memory and configured to perform any one of the methods as described above.

[0065] According to an embodiment of the present disclosure, it is provided a video encoding apparatus, comprising: a memory; and a processor coupled to the memory and configured to perform any one of the methods as described above.

[0066] According to an embodiment of the present disclosure, it is provided a computer-readable storage medium, the computer-readable storage medium having stored thereon a computer program or instructions, the computer program or instructions, when executed by at least one processor, causing the at least one processor to perform any one of the methods as described above.

[0067] According to an embodiment of the present disclosure, it is provided a program product comprising a computer program, the computer program, when executed by a processor, implementing any one of the methods as described above.

[0068] According to a technical solution provided in embodiments of the present disclosure, by determining the residual motion related information of pixels in the current block and determining, based on the residual motion related information, the optical flow compensation method for the current block, which may include fusing the first optical flow compensation result obtained based on the first reference block and the second optical flow compensation result obtained based on the second reference block to obtain the optical flow compensation result of the current block, based on the POCs of the current block, the first reference block and the second reference block, it enables to obtain the optical flow compensation result of the current block without being constrained by a isometric bi-directional prediction condition. When the above technical solution is applied in encoding and decoding, the residual motion problem within the current block may be better improved in the encoding and decoding, and the performance of the encoding and decoding may be improved.

[0069] According to the technical solution provided in embodiments of the present disclosure, the optical flow compensation result of the current block is obtained by fusing the first optical flow compensation result of the current block obtained based on the first reference block and the second optical flow compensation result of the current block obtained based on the second reference block, based on the POC of current block, the POCs of the first reference block and the second reference of the current block, which enables to obtain a more accurate optical flow compensation result. When the above technical solution is applied in encoding and decoding, the residual motion problem of the current block may be better improved in the encoding and decoding, and the performance of the encoding and decoding may be improved.

[0070] According to the technical solution provided in embodiments of the present disclosure, by determining the pixel fluctuation related information of the current block, determining whether a processing object of an optical flow compensation method is the current block or a sub-block of the current block, or a pixel in the current block, based on the pixel fluctuation related information, and obtaining, for the determined processing object, an optical flow compensation result of the processing object using the optical flow compensation method, based on the first reference block and second reference block of the current block, it enables to balance both the reduction of the error of the optical flow compensation result and the reduction of the computational complexity. When the above technical solution is applied in encoding and decoding, it is possible to achieve a more accuracy and improvement of the residual motion problem with a lower computational complexity in the encoding and decoding, and to improve the performance of the encoding and decoding.

[0071] It should be understood that the above general description and the detailed descriptions that follow are merely exemplary and explanatory and do not limit the present disclosure.

[0072] The following description with reference to the accompanying drawings is provided to aid in a thorough understanding of various embodiments of the present disclosure as defined by claims and equivalents thereof. This description includes various specific details to aid in understanding but should only be considered exemplary. Accordingly, those ordinary skills in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known features and structures may be omitted for the sake of clarity and brevity.

[0073] The terms and phrases used in the claims and the following description are not limited to dictionary meaning thereof, but are used only by the inventor to enable a clear and consistent understanding of the present disclosure. Accordingly, it should be apparent to those skilled in the art that, the following description of the various embodiments of the present disclosure is provided for an illustrative purpose only and is not intended to a purpose of limiting the present disclosure as defined by the appended claims and equivalents thereof.

[0074] It should be understood that, "a", "an" and "the" in a singular form may also include a plural reference, unless the context clearly indicates otherwise. Thus, for example, a reference to a "part surface" includes a reference to one or more such surfaces. When it refers to one element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to a connection relationship between the one element and the other element established through an intermediate element. In addition, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled.

[0075] The term "include" or "may include" refers to the presence of a function, operation, or component of the corresponding disclosure that may be used in the various embodiments of the present disclosure, and does not limit the presence of one or more additional functions, operations, or features. In addition, the terms "include" or "have" may be interpreted to denote certain features, figures, steps, operations, constituent elements, components, or combinations thereof, but should not be interpreted to exclude the possibility of the presence of one or more other features, figures, steps, operations, constituent elements, components, or combinations thereof.

[0076] The term "or" as used in the various embodiments of the present disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing a plurality of (two or more) items, the plurality of items may refer to one, more, or all of the plurality of items if a relationship among the plurality of items is not explicitly defined. For example, for the description "a parameter A comprises A1, A2, A3", it may be implemented as parameter A comprising A1, A2 or A3, or as parameter A comprising at least two of the three items of the parameter A1, A2, A3.

[0077] All terms (including technical or scientific terms) used in the present disclosure have the same meaning as understood by those skilled in the art to which the present disclosure belongs, unless defined differently. Common terms as defined in dictionaries are interpreted to have a meaning consistent with the context in the relevant technology art and should not be interpreted in an idealized or overly formalistic manner, unless expressly so defined in the present disclosure.

[0078] Below, the technical solutions of the embodiments of the disclosure and the technical effects produced by the technical solutions of the disclosure will be explained by describing several optional embodiments. It should be noted that, the following embodiments may be referred to, imitated or combined with each other, and the same term, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0079] As described in the background, the residual motion problem can only be improved in very limited cases in existing codecs. The residual motion problem may be improved to some extent based on an optical flow compensation technology. Optical flow is instantaneous speed of motion of video frame content in the temporal domain, i.e., the speed of the motion of a pixel in the temporal domain. For two frames with a short time interval in the video, the optical flow is also expressed as the displacement of a same target point between the two frames, i.e., the motion vector of the pixel. For video frames that are close in a temporal domain, brightness of the same target usually does not change in different frames and a motion distance is small. Therefore, pixel motion information, i.e., the optical flow, between the video frames may be predicted by utilizing variation of brightness of the pixel in the temporal domain and space domain correlation of the pixel in image frames.

[0080] However, at present, the optical flow compensation technology, which is capable of improving the residual motion problem, can only be used under the very limited isometric bi-directional prediction condition, which severely constrains the use of the optical flow compensation technology and is not conducive to the improvement of the residual motion problem.

[0081] In the following, the inter prediction in encoding and decoding is taken as an example to illustrate the problem. However, it should be noted that the problem of residual motion improvement is not limited to being present in inter prediction, but may be present in other encoding and decoding modes as well.

[0082] The inter prediction may reduce the temporal redundancy of the encoded video stream, thereby enabling the codec to achieve a lower bit rate with the same image quality. In a video sequence, adjacent image contents are often extremely similar. For example, the content of the background portion of the video sequence usually has a small range of variation in brightness between multiple frames. Therefore, for those video contents that have a small content variation over multiple video frames, the use of inter prediction means can significantly reduce the bitstream for the purpose of improving the performance of the codec. The inter prediction may include motion estimation and motion compensation. For example, the motion estimation is usually categorized into three modes: a merge mode, an AMVP (Advanced Motion Vector Prediction) mode, and an affine mode. For the merge mode, the current encoded block completes the motion estimation by inheriting the motion vector (MV) of the adjacent block in the space domain. The merge mode is usually used to deal with large flat areas; and for the affine mode, the current encoded block completes the motion estimation by using the parameters related to the affine mode. The affine mode is usually used to deal with non-rigid body motion, such as zoom-in and zoom-out motion; for the AMVP mode, the current encoded block completes the motion estimation by constructing a MV candidate list. The AMVP mode is used to deal with other motions except for the merge mode and the affine mode. The motion compensation is the prediction of the current block using the reconstructed block of the reference frame and compensating for the current block. The motion compensation may include the motion compensation based on the optical flow, i.e., the optical flow compensation. In the video encoding process, directly encoding of the current block may be replaced with the encoding of the prediction residual. The purpose of the motion compensation is to minimize the difference between the finalized reference block and the current block to reduce the transmission bitstream overhead at the encoding and decoding sides, which thus improves the performance of the codec. For different motion estimation modes, the motion compensation stage can only adopt the compensation mode corresponding to the motion estimation stage for compensation.

[0083] For example, as shown in Fig. 1, after performing the motion vector (MV) estimation, e.g., the residual motion in the affine mode may use PROF (Prediction Refinement with optical flow) to improve the residual motion problem, whereas in the inter mode, after performing the MV estimation, e.g., after performing the MV estimation using the AMVP or the SMVD (Symmetric Motion Vector Difference), when improving the residual motion based on the optical flow compensation technology, the optical flow compensation technology can only be used under the isometric bi-directional prediction condition, for example, for the B frame, the BDOF (Bi-directional optical flow) compensation technology can only be used under the isometric bi-directional prediction condition.

[0084] FIG. 2 illustrates a schematic diagram of an isometric bi-directional prediction condition. As shown in Fig. 2, the isometric bi-directional prediction condition refers to that the current block is a bi-directional encoded or decoded block, reference blocks are located on different sides of the current block, and the POCs (picture order counts) from two reference blocks to the current block are the same, i.e., POC difference 1 is equal to POC difference 0 in Fig. 2.

[0085] With respect to the condition limitation on the use of the optical flow compensation technology in the residual motion improvement, in order to extend the use of optical prediction technology to better improve the residual motion problem, the present disclosure proposes a method performed by an electronic apparatus.

[0086] FIG. 3 is a flowchart illustrating a method performed by a video decoding apparatus according to an embodiment of the present disclosure.

[0087] As shown in FIG. 3, at step S310, residual motion related information of pixels in a current block is determined. Next, at step S320, an optical flow compensation method for the current block is determined based on the residual motion related information. According to an embodiment, the optical flow compensation method comprises: fusing a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the current block, based on a picture order count (POC) of the current block and the POCs of the first reference block and the second reference block of the current block. It is determined, based on the residual motion related information, whether to obtain a weighted sum of a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the current block. The weights for the first optical flow compensation result and the second optical flow compensation result are determined based on a difference between the POC of the current block and the POC of the first reference block and a difference between the POC of the current block and a POC of the second reference block. At step S330, the optical flow compensation result of the current block is obtained using the determined optical flow compensation method, based on the first reference block and the second reference block. In the present disclosure, fusion of the optical flow compensation results obtained based on the reference blocks may enable the optical flow compensation of the current block, and accordingly, the obtained optical flow compensation result of the current block is pixel values after the optical flow compensation is performed on the current block. Based on determining that the weighted sum is obtained at step 320, the optical flow compensation result of the current block is obtained using the weighted sum of the first optical flow compensation result and the second optical flow compensation result at step 330.

[0088] According to the method shown in FIG. 3, by determining the residual motion related information of the pixels in the current block and determining, based on the residual motion related information of the pixels in the current block, the optical flow compensation method for the current block, which may comprise: fusing, based on the POCs of the current block, the first reference block and the second reference block, the first optical flow compensation result obtained based on the first reference block and the second optical flow compensation result obtained based on the second reference block to obtain the optical flow compensation result of the current block, it enables to obtain the optical flow compensation result of the current block without being constrained by the isometric bi-directional prediction condition.

[0089] In the following, each of the steps illustrated in FIG. 3 is further described, respectively.

[0090] According to an embodiment, the step S310 may include: determining the residual motion related information of the pixels in the current block based on the first reference block and the second reference block of the current block. According to an embodiment, the first reference block and the second reference block may be located on two sides of the current block, for example, the first reference block and the second reference block may be a forward reference block and a backward reference block of the current block, respectively. Or, the first reference block and the second reference block may be located on the same side of the current block, for example, the first reference block and the second reference block may both be the forward reference blocks of the current block, or may both be the backward reference blocks of the current block.

[0091] According to an embodiment, the determining of the residual motion related information of the pixels in the current block based on the first reference block and the second reference block of the current block may include: determining the residual motion related information of the pixels in the current block based on a feature related to a pixel gradient of the first reference block and the second reference block. According to an embodiment, the feature of each pixel in the first reference block and the second reference block may first be calculated respectively, and then, the residual motion related information of the pixels in the current block is determined based on number of pixels having a same coordinate position and the same feature in the first reference block and the second reference block. However, the way of determining the residual motion related information of the pixels in the current block is not limited thereto.

[0092] According to an embodiment, the feature includes a value determined based on a vertical gradient and a horizontal gradient of each pixel in the first reference block and the second reference block, but is not limited thereto. For example, the feature may be a ratio of the vertical gradient to the horizontal gradient of the pixel, or, a ratio of the horizontal gradient to the vertical gradient, or any variation based on the ratios above, or any value obtained based on the vertical gradient and the horizontal gradient, or a value obtained based on the gradient of the pixel in any direction.

[0093] FIG. 4 is a schematic diagram illustrating an example of judgment of residual motion related information according to an embodiment of the present disclosure.

[0094] As shown in FIG. 4, assuming that the first reference block (reference block 0) and the second reference block (reference block 1) of the current block are encoded blocks of 8x8 size, they may be divided into four reference blocks of 4x4 size, respectively, such reference blocks may be referred to as "optical flow blocks" in the present disclosure. It should be noted that although in the example of FIG. 4, the reference block 0 and the reference block 1 are divided, the reference blocks may not be divided if the size of the reference blocks is appropriate. For each optical flow block divided from the reference block 0 and the reference block 1, the horizontal gradient and the vertical gradient of each pixel in the optical flow block are calculated respectively, and the ratio of the vertical gradient to the horizontal gradient, which may be referred to as a "gradient angle", is used as a feature. By comparing the features of the two reference blocks, the residual motion related information of the pixels in the current block may be determined. For example, the number of pixels having the same coordinate position and the same feature in the reference block 0 and the second reference block 1 may be calculated, and the residual motion related information of the pixels in the current block may be determined based on the number. For example, if the number is less than N pixels, the residual motion related information of the pixels in the current block is considered to be small, whereas if the number is not less than N pixels (i.e., at least N pixels) the residual motion related information of the pixels in the current block is considered to be large. In other words, if when the gradient angles of the pixels in the two reference blocks that have the same coordinate position as the current pixel in the current block are exactly the same, reference block consistency of the current pixel point is marked as consistency, then the current block is recognized as an encoded block with the reference block consistency when the number of pixels in the current block that are reference block consistency is greater than or equal to a predetermined threshold N, and the current block is recognized as an encoded block without the reference block consistency when the number of pixels in the current block that are reference block consistency is less than the predetermined threshold N. As an example, N may be set equal to 14, but is not limited thereto.

[0095] After determining the residual motion related information of the pixels in the current block, as described above, at step S320, the optical flow compensation method for the current block is determined based on the residual motion related information of the pixels in the current block. According to an embodiment, the optical flow compensation method for the current block may include: fusing a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the current block, based on a picture order count (POC) of the current block and the POCs of the first reference block and the second reference block of the current block. For example, the fusing of the first optical flow compensation result obtained based on the first reference block and the second optical flow compensation result obtained based on the second reference block includes: performing a weighted sum on the first optical flow compensation result and the second optical flow compensation result, based on POC differences between the first reference block as well as the second reference block of the current block, respectively, and the current block. According to an embodiment, weights used in the weighted sum may be determined based on the POC differences between the first reference block as well as the second reference block of the current block, respectively, and the current block, for example, the weights may be determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and the POC difference between the first reference block and the second reference block.

[0096] For example, step S320 may include: if the residual motion related information satisfies a first condition, determining that the optical flow compensation method for the current block is a first optical flow compensation method, wherein the first optical flow compensation method obtains the optical flow compensation result of the current block by averaging the first optical flow compensation result obtained based on the first reference block and the second optical flow compensation result obtained based on the second reference block; if the residual motion related information does not satisfy the first condition, the optical flow compensation method for the current block is determined to be a second optical flow compensation method, wherein the second optical flow compensation method obtains the optical flow compensation result of the current block by performing the weighted sum on the first optical flow compensation result and the second optical flow compensation result, wherein the weights used in the weighted sum is determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block.

[0097] According to an embodiment, for example, the residual motion related information of the pixels in the current block satisfies the first condition if the number of pixels having the same coordinate position and the same feature in the first reference block and the second reference block is greater than or equal to a second threshold, and the residual motion related information of the pixels in the current block does not satisfy the first condition if the number of pixels having the same coordinate position and the same feature in the first reference block and the second reference block is less than the second threshold. The number of pixels having the same coordinate position and the same feature in the first reference block and the second reference block is greater than or equal to a second threshold, that is, the number of pixels in the current block that are marked as with the reference block consistency is greater than or equal to the second threshold, and if this is the case, the residual motion related information of the pixels in the current block is considered to be small and thus the first condition is satisfied. Otherwise, if the number of pixels having the same coordinate position and the same feature in the first reference block and the second reference block is less than the second threshold, that is, the number of pixels in the current block that are marked as with the reference block consistency is less than the second threshold, it is considered that the residual motion related information of the pixels in the current block is large and thus the first condition is not satisfied.

[0098] If the residual motion related information satisfies the first condition, the optical flow compensation method for the current block is determined to be the first optical flow compensation method, otherwise the optical flow compensation method for the current block is determined to be the second optical flow compensation method.

[0099] For example, the first optical flow compensation method may be a first BDOF prediction method, and the second optical flow compensation method may be a second BDOF prediction method. In the following, the manner of obtaining the optical flow compensation result of the current block is described taking the first optical flow compensation method being the first BDOF prediction method and the second optical flow compensation method being the second BDOF prediction method as an example, however, the first optical flow compensation method and the second optical flow compensation method are not limited to being BDOF only.

[0100] In the following, the first BDOF prediction method is also simply referred to as a "traditional BDOF", and the second BDOF prediction method is also simply referred to as a "BDOF with the POC". For example, as shown in FIG. 4, if the number of pixels having the same coordinate position and the same feature in the respective optical flow blocks of the reference block 0 and the reference block 1 is less than N, that is, less than N pixels in the optical flow blocks have the same feature, then the residual motion of the pixels in the current block is considered to be large, and the "BDOF with the POC" is determined to be used for the current block. On the contrary, if the number of pixels having the same coordinate position and the same features in the respective optical flow blocks of reference block 0 and reference block 1 is greater than or equal to N, then the residual motion of the pixels in the current block is considered to be small, and the "traditional BDOF" is determined to be used for the current block.

[0101] The "traditional BDOF" is under the limitation of the isometric bi-directional prediction condition, which obtains the optical flow compensation result of the current block by averaging the first optical flow compensation result obtained based on the first reference block and the second optical flow compensation result obtained based on the second reference block. The "BDOF with the POC" is not under the limitation of the isometric bi-directional prediction condition, and may perform the optical flow compensation under the non-isometric bi-directional prediction condition, specifically, the optical flow compensation result of the current block may be obtained by performing a weighted sum on the first optical flow compensation result obtained based on the first reference block and the second optical flow compensation result obtained based on the second reference block, and the weights used in the weighted sum are determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block.

[0102] FIG. 5 is a schematic diagram illustrating a non-isometric bi-directional prediction. As shown in FIG. 5, the non-isometric bi-directional prediction refers to that the current block is a bi-directional encoded block or decoded block, the reference blocks are located on different sides of the current block, and the two reference blocks have different POC differences from the current block, i.e., POC difference 1 is not equal to POC difference 0 in FIG. 5. In such a case, if the optical flow compensation result is still obtained according to the optical flow compensation method under the condition of the isometric bi-directional prediction, by averaging the first optical flow compensation result obtained based on the first reference block and the second optical flow compensation result obtained based on the second reference block, then the error of the optical flow compensation result obtained at this time is large.

[0103] In view of this, the present disclosure proposes to obtain the optical flow compensation result of the current block by fusing the first optical flow compensation result of the current block obtained based on the first reference block and the second optical flow compensation result of the current block obtained based on the second reference block, based on the POC of the current block and the POCs of the first reference block and the second reference block of the current block, so as to reduce the error of the optical flow compensation result.

[0104] For example, according to an embodiment of the present disclosure, the BDOF with the POC is proposed, in which the weights of the first optical flow compensation result and the second optical flow compensation result are determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and the optical flow compensation result of the current block is obtained by performing the weighted sum on the first optical flow compensation result and the second optical flow compensation result. According to an embodiment, the weights used in the weighted sum may be determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and a POC difference between the first reference block and the second reference block. According to an embodiment, the weight of the first optical flow compensation result in the weighted sum is a first weight corresponding to the first reference block, and the weight of the second optical flow compensation result is a second weight corresponding to the second reference block. According to an embodiment, the first weight corresponding to the first reference block and the second weight corresponding to the second reference block may be determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and the POC difference between the first reference block and the second reference block. For example, the first weight and the second weight may be determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and the POC difference between the first reference block and the second reference block. According to an embodiment, alternatively, the first weight and the second weight may be determined based on a ratio of a first value to a second value, wherein the first value is determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and the second value is determined based on the POC difference between the first reference block and the second reference block. For example, the second value may be the POC difference between the first reference block and the second reference block, the first value may be the POC differences themselves between the first reference block as well as the second reference block, respectively, and the current block, may be the maximum value in the POC differences between the first reference block as well as the second reference block, respectively, and the current block, may be the minimum value in the POC differences between the first reference block as well as the second reference block, respectively, and the current block, but is not limited thereto. For example, the first weight and the second weight may have a multiplicative relationship, an exponential relationship, or the like, but the relationship between the first weight and the second weight is not limited thereto.

[0105] In the following, these two optical flow compensation methods aforementioned are described respectively.

[0106] For the current block whose residual motion related information satisfies the first condition (also referred to as the "current block with the reference block consistency"), the optical flow compensation result of the current block is obtained utilizing the traditional BDOF, and the specific computational inference process is as follows:

[0107] For a given target pixel, the brightness value thereof is assumed to remain constant from a moment t to a moment t+ t. The coordinates of the given point in the block at the moment t are (x, y), and the coordinates of the given point at the moment t+ t become (x+ x, y+ y). Thus the relationship between the block corresponding to the moment t and the block corresponding to the moment t+ t may be given by the following expression:

[0108] Equation 1

[0109] The Taylor expansion is performed on the right end of the above equation, it is obtained:

[0110] Equation 2

[0111] It is considered that the assumed premise of the optical flow technology is that the brightness value is constant and the higher order term ε thereof may be ignored. Therefore, the above equation may be directly transformed as:

[0112] Equation 3

[0113] Equation 4 is obtained by dividing both sides of the above equation by dt:

[0114] Equation 4

[0115] wherein , denote optical flow speeds of this target pixel in the horizontal and vertical directions, respectively, which are also referred to as optical flow vectors. From this, the expression of the relationship between the first reference block and the second reference block in the bi-directional prediction process may be derived as:

[0116] Equation 5

[0117] The expression of the relationship between the second reference block and the current block is:

[0118] Equation 6

[0119] wherein is the pixel value of the current block, and and are the pixel values of the first reference block and the second reference block at the same coordinate, respectively. The above equations are transformed into forms in which the pixel values of the current block are represented by and , respectively, and the two equations are subtracted from each other, and the result of the subtraction is set to diff. When the value of the diff is 0, the transformed equation is:

[0120] Equation 7

[0121] At this time, the optical flow vectors , that satisfy Equation 7 are the optical flow vectors which make the optical flow compensation results given by the two reference blocks closest to each other.

[0122] The above equation is solved using the least squares method,

[0123] , wehrein , ;

[0124] , wehrein , ; Equation 8

[0125] The optical flow vectors , may be obtained as:

[0126]

[0127] wherein

[0128]

[0129]

[0130]

[0131]

[0132]

[0133] From this, the optical flow vectors ( , ) of the current block may be obtained in conjunction with Equation 8.

[0134] By adding Equation 5 to Equation 6, the optical flow compensation result of the current block may be obtained as:

[0135] Equation 9

[0136] Since , may be approximated as:

[0137] Equation 10

[0138] That is, Equation 11

[0139] wherein is the first optical flow compensation result based on the first reference block, and is the second optical flow compensation result based on the second reference block, and the optical flow compensation result of the current block may be obtained by averaging the two predicted values.

[0140] For the current block whose residual motion related information does not satisfy the first condition (also referred to as the "current block without the reference block consistency"), the optical flow compensation result of the current block is obtained by the BDOF with the POC, and the specific computational inference process is as follows:

[0141] For the current encoded block without the reference block consistency, when obtaining the optical flow compensation result of the current block through two reference blocks thereof, due to large variation of the temporal domain gradient between the blocks, the derivation manner of the temporal domain gradient needs to accurately consider the POCs of the reference blocks and the current block, in a phase of the optical flow vector derivation, for example, the POC difference is used as a time dimension variable of the temporal domain gradient calculation. According to an embodiment, the optical flow vectors of the current block may be determined using POC differences between the first reference block as well as the second reference block, respectively, and the current block. The optical flow vectors of the current block may include a horizontal direction optical flow vector (denoted as ) and a vertical direction optical flow vector (denoted as ). According to an embodiment, alternatively, the determining of the optical flow vectors of the current block using the POC differences between the first reference block as well as the second reference block, respectively, and the current block, may include: determining the horizontal direction optical flow vector and the vertical direction optical flow vector, respectively, based on gradient values of pixel values of the first reference block and the second reference block corresponding to a pixel value of the current block, in the horizontal direction and the vertical direction, respectively, and the POC differences between the first reference block as well as the second reference block, respectively, and the current block.

[0142] For example, assuming that is the POC difference between the first reference block and the current block, and is the POC difference between the second reference block and the current block, the relationships between the first reference block as well as the second reference block and the current block are expressed by the following equations:

[0143] Equation 12

[0144] Equation 13

[0145] In the process of solving for the optical flow vectors using the least squares method, the expressions of the least squares method is changed to the following:

[0146]

[0147] Equation 14

[0148] Based on Equation 14, the optical flow vectors of the current block, i.e., and , may be derived.

[0149] Subsequently, the first optical flow compensation result and the second optical flow compensation result may be obtained based on the optical flow vectors. According to an embodiment, alternatively, the obtaining of the first optical flow compensation result and the second optical flow compensation result based on the optical flow vectors may include: obtaining the first optical flow compensation result based on a POC difference between the first reference block and the current block, the optical flow vectors, and a pixel value of the first reference block corresponding to the pixel value of the current block; obtaining the second optical flow compensation result based on a POC difference between the second reference block and the current block, the optical flow vectors, and a pixel value of the second reference block corresponding to the pixel value of the current block. Finally, the optical flow compensation result of the current block is obtained by performing the weighted sum on the first optical flow compensation result and the second optical flow compensation result.

[0150] For example, considering that in the whole video sequence, a reference block that is closer to the current block in the POC distance (i.e., a reference block having a smaller POC difference with the current block) has a higher degree of trustworthiness for the current block, and thus the predicted value given by this reference block may be given a larger weight, on the contrary, a reference block that is farther away from the current block in the POC distance (i.e., a reference block having a larger POC difference with the current block) has a lower degree of trustworthiness for the current block, and thus the predicted value given by this reference block may be given a smaller weight, but the manner in which the weight is given is not limited thereto. According to an embodiment, for example, in the weighted sum described above, the optical flow compensation result corresponding to the reference block of the first reference block and the second reference block having a larger POC difference with the current block is given a smaller weight, and the optical flow compensation result corresponding to the reference block having a smaller POC difference with the current block is given a larger weight. For example, alternatively, the weights used in the weighted sum may include: a ratio of the POC difference between the first reference block and the current block to the POC difference between the first reference block and the second reference block; and a ratio of the POC difference between the second reference block and the current block to the POC difference between the first reference block and the second reference block.

[0151] For example, after modifying Equation 5 and Equation 6 to Equation 12 and Equation 13 and modifying Equation 8 to Equation 14, following a similar derivation process as that of the traditional BDOF, it may be obtained that the modified formula for the optical flow compensation result is:

[0152]

[0153]

[0154] That is,

[0155] .

[0156] wherein （ ） is the second optical flow compensation result based on the second reference block, （ ）is the first optical flow compensation result based on the first reference block, is the weight corresponding to the second optical flow compensation result, and is the weight corresponding to the first optical flow compensation result.

[0157] After the optical flow compensation method for the current block is determined based on the residual motion related information, at the step S330, the optical flow compensation result of the current block may be obtained using the determined optical flow compensation method, based on the first reference block and the second reference block. For example, if the determined optical flow compensation method is the first optical flow compensation method, the optical flow compensation result of the current block is obtained by averaging the first optical flow compensation result obtained based on the first reference block and the second optical flow compensation result obtained based on the second reference block; and if the determined optical flow compensation method is the second optical flow compensation method, the optical flow compensation result of the current block is obtained by performing the weighted sum on the first optical flow compensation result and the second optical flow compensation result, after determining the weights corresponding to each of the first optical flow compensation result and the second optical flow compensation result based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block.

[0158] According to an embodiment of the present disclosure, since through determining the residual motion related information of the pixels in the current block, determining the optical flow compensation method for the current block based on the residual motion related information of the pixels in the current block, and obtaining the optical flow compensation result of the current block using the determined optical flow compensation method, based on the first reference block and the second reference block, it enables the optical flow compensation to better improve the residual motion problem within the current block without being constrained by the isometric bi-directional prediction condition.

[0159] The optical flow technology may include a block based optical flow compensation technology and a sample based optical flow compensation technology, for example, the current BDOF technologies may be divided into two categories: the block based bi-directional optical flow technology and the sample based bi-directional optical flow technology. The block based BDOF technology deals with the current block that is not a sub-block, obtains the optical flow vectors of the current block by the optical flow compensation technology, and the optical flow vectors are used to compensate every pixel in the current block. The sample based BDOF technology deals with the current block that is determined to be a sub-block, obtains the optical flow vectors of a specified pixel in the sub-block by the optical flow compensation technology, and the optical flow vectors are used for the motion compensation of the specified pixel.

[0160] For example, FIG. 6 is a schematic diagram illustrating a structure of the current BDOF technology. As shown in FIG. 6, the optical flow compensation is performed using the block based BDOF for all non-subblocks that satisfy the conditions for the optical flow compensation, and then the two bi-directional reference blocks pointed by the corrected MV obtained in the preceding step are analyzed, and when the average brightness difference of the two bi-directional reference blocks < a preset value, it is not necessary to perform any compensation operation for the current block; and when the average brightness difference of the two bi-directional reference blocks >= the preset value, it is necessary to continue the optical flow compensation based on the two reference blocks pointed by the corrected MV using the sample based BDOF technology. For example, the preset value may be 2, but is not limited thereto, and may be other values. The optical flow compensation is performed using the sample based BDOF technology for all sub-blocks that satisfy the condition for the optical flow compensation. According to the above processing method, all non-subblocks need to be processed by the block based BDOF first, for a encoded block with a simple texture, if the MV obtained in the preceding step is not accurate enough, then use of the block based BDOF will not bring a significant error to the compensation result, but for a encoded block with a complex texture, if the MV obtained in the preceding step is not accurate enough, then the use of the block based BDOF technology will lead to a significant error in the optical flow compensation result at this time. In addition, in the above processing method, the sample based BDOF technology is always used for the optical flow compensation for the sub-blocks. However, for the encoded block with the simple texture, the optical flow compensation results obtained by using the block based BDOF technology and the sample based BDOF technology are similar, but the computational complexity of the sample based BDOF technology for a single pixel is much higher than that of the block based BDOF technology. It can be seen that there are two problems with the current optical flow compensation technology: for a complex texture block in non-subblocks, processing methods of the block based BDOF and then the sample based BDOF are prone to introduce the optical flow compensation error, which leads to the inaccurate optical flow compensation result and affects the performance of the codec. For a simple texture block in the sub-blocks, directly using the processing method of the sample based BDOF for all of them will bring additional computational complexity to the optical flow compensation technology.

[0161] In this regard, the present disclosure proposes to balance both the reduction of the error of the optical flow compensation result and the reduction of the computational complexity based on pixel fluctuation related information, so that it is possible to achieve a more accuracy and improvement of the residual motion problem with a lower computational complexity. In the present disclosure, the pixel fluctuation related information represents a texture change situation in a region surrounding one pixel.

[0162] According to an embodiment, the step S330 may include: determining pixel fluctuation related information of the current block, wherein the pixel fluctuation related information represents a texture change situation in a region surrounding one pixel; determining whether a processing object of the optical flow compensation method is the current block or a sub-block of the current block, or a pixel in the current block, based on the pixel fluctuation related information; obtaining, for the determined processing object, the optical flow compensation result of the processing object using the optical flow compensation method.

[0163] In obtaining the optical flow compensation result of the current block using the determined optical flow compensation method, the processing object (or the processing unit) may be the current block or a sub-block of the current block, or may be a pixel in the current block, based on the judgment of the pixel fluctuation related information, i.e., it may be determined whether to perform the optical flow compensation in a unit of a block, or to perform the optical flow compensation in a unit of a pixel, based on the judgment of the pixel fluctuation related information. If the processing object is a block (the current block or a sub-block of the current block), the optical flow compensation result of the block is obtained using the determined optical flow compensation technology, for example, the optical flow vectors of the block are obtained, and the optical flow compensation result of the block is obtained by performing the optical flow compensation for all pixels in the block using the optical flow vectors of the block. If the processing object is a pixel in the current block, for each pixel in the current block, the optical flow compensation result of the pixel is obtained using the determined optical flow compensation technology, for example, the optical flow vectors of the pixel are obtained, and the optical flow compensation result of the pixel is obtained by performing the optical flow compensation for the pixel using the optical flow vectors of the pixel.

[0164] According to an embodiment, the determining of the pixel fluctuation related information of the current block may include: determining the pixel fluctuation related information of the first reference block and the second reference block, respectively; determining the pixel fluctuation related information of the current block based on the pixel fluctuation related information of the first reference block and the second reference block. In other words, the pixel fluctuation related information of the current block may be judged with the help of the pixel fluctuation related information of the reference blocks.

[0165] According to an embodiment, the determining of whether the processing object of the optical flow compensation method is the current block or the sub-block of the current block, or the pixel of the current block, based on the pixel fluctuation related information includes: determining that the processing object of the optical flow compensation method is the pixel in the current block, if the pixel fluctuation related information of at least one of the first reference block and the second reference block exceeds a first threshold; determining that the processing object of the optical flow compensation method is the current block or the sub-block in the current block, if the pixel fluctuation related information of at least one of the first reference block and the second reference block does not exceed the first threshold (e.g. the pixel fluctuation related information of the first reference block and the second reference block neither exceed the first threshold). The processing object being the current block or a sub-block of the current block indicates that the optical flow compensation method determined by the step S320 is the block based optical flow compensation method, which may be, for example, the block based traditional BDOF or the block based BDOF with the POC. The processing object being a pixel in the current block indicates that the optical flow compensation method determined by the step S320 is the sample based optical flow compensation method, which may be, for example the sample based traditional BDOF or the sample based BDOF with the POC. As an example, in the case that the pixel fluctuation related information of at least one of the first reference block and the second reference block exceeds the first threshold, it may be considered that the pixel fluctuation related information of the current block also exceeds the first threshold. As an example, in the case that the pixel fluctuation related information of the first reference block and the second reference block neither exceed the first threshold, it may be considered that the pixel fluctuation related information of the current block also does not exceed the first threshold. For example, a block in which the pixel fluctuation related information exceeds the first threshold may be referred to as a complex texture block, and a block in which the pixel fluctuation related information does not exceed the first threshold may be referred to as a simple texture block. According to an embodiment, the determining of the pixel fluctuation related information of the first reference block and the second reference block, respectively, may include: determining gradients in a plurality of directions of each pixel of a plurality of pixels included in the first reference block and the second reference block, respectively; determining the pixel fluctuation related information of each of the first reference block and the second reference block based on the respectively determined gradients in the plurality of directions of each of the first reference block and the second reference block. For example, the gradients in the plurality of directions may include a gradient in a horizontal direction, a gradient in a vertical direction, and a gradient in at least one inclined direction different from the horizontal direction and vertical direction, e.g., gradients in two inclined directions. For example, the gradients in two inclined directions may be the gradients in two diagonal directions, respectively. However, determining the gradients in the plurality of directions of the pixel is not limited to determining the gradients in only the four directions described above.

[0166] According to an embodiment, the determining of the pixel fluctuation related information of each of the first reference block and the second reference block based on the respectively determined gradients in the plurality of directions of each of the first reference block and the second reference block may include: determining, for the first reference block and the second reference block, the pixel fluctuation related information thereof by performing a sum or a weighted sum on the gradient in the horizontal direction, the gradient in the vertical direction, and the gradient in the at least one inclined direction of each pixel of the plurality of pixels included therein, respectively. For example, for the first reference block and the second reference block, the pixel fluctuation related information thereof is determined by performing a sum or a weighted sum on the gradient in the horizontal direction, the gradient in the vertical direction, and the gradient in the at least one inclined direction of each pixel of the plurality of pixels included therein, respectively. For example, the pixel fluctuation related information may be determined based on the sum of gradients or the weighted sum of the gradients, such as, the sum of gradients or the weighted sum of the gradients may be directly used as the pixel fluctuation related information, or the sum of gradients or the weighted sum of the gradients may be further calculated and transformed to obtain the pixel fluctuation related information.

[0167] FIG. 7 is a schematic diagram illustrating an example of judgment of pixel fluctuation related information according to an embodiment of the present disclosure.

[0168] As shown in FIG. 7, for each optical flow block divided from the reference block 0 and the reference block 1, the pixel fluctuation related information thereof may be determined respectively. Alternatively, if the reference block 0 and the reference 1 are of a suitable size, the division may not be performed, and the reference block may be directly used as the optical flow block. For example, a gradient in the horizontal direction, a gradient in the vertical direction, and gradients in both inclined directions (e.g., the diagonal directions) of each pixel of the plurality of pixels (e.g., the plurality of pixels excluding the edge pixels) in each optical flow block may be determined respectively, and then the sum or the weighted sum may be performed on the gradient in the horizontal direction, the gradient in the vertical direction, and the gradients in both inclined directions of each pixel as the gradient of that pixel. For example, when the optical flow block is in a 4x4 size, a 3x3 region may be used for the gradient calculation, i.e., the pixel fluctuation related information may be calculated without considering the most edging pixel points in the optical flow block. For example, if it is considered that a brightness difference can be easily recognized by the human eyes when the brightness difference between two adjacent pixels is greater than or equal to a threshold S (e.g., S=5), then it may be set that, for a pixel point, the pixel point may be regarded as a complex texture pixel when the sum of the gradients in its four directions is greater than . In one optical flow block, the number of pixel points (i.e., removing the edge pixel points) for which the pixel fluctuation related information may be calculated, among 16 pixels is 9. Therefore, when the gradient sum of one optical flow block is greater than or equal to , the optical flow block is a complex texture block; in the contrary, the optical flow block is a simple texture block. For example, the following formulas may be used to calculate the gradient sum of a pixel:

[0169]

[0170]

[0171] wherein is the gradient in the horizontal direction, is the gradient in the vertical direction, is the gradient in the left diagonal direction, is the gradient in the right diagonal direction, and the weights of and may be .

[0172] If the gradient sum of the corresponding gradient block of the optical flow block is greater than M, e.g., M may be set to 153, but is not limited thereto, the optical flow block may be considered as a complex texture block, on the contrary, may be considered as a simple texture block. If both reference blocks are simple texture blocks, the current block may be considered as a simple texture block as well, in which case the optical flow compensation result of the current block may be obtained using the block based optical flow compensation method, for example, the optical flow compensation result of the current block may be obtained using the block based traditional BDOF or the block based BDOF with the POC. If at least one of the two reference blocks is a complex texture block, the current block may be considered as a complex texture block as well, in which case the optical flow compensation result of each pixel in the current block may be obtained by the sample based optical flow compensation method, for example, the optical flow compensation result of each pixel in the current block may be obtained by the sample based traditional BDOF or the sample based BDOF with the POC. For example, for the complex texture block, a size of a sliding window in obtaining the optical flow compensation result of each pixel in the current block by the sample based optical flow compensation method may be 5x5.

[0173] After using the judgment of the pixel fluctuation related information, on the one hand, the problem of increased complexity due to the sample based optical flow compensation for the simple texture block can be improved, and on the other hand, the problem of compensation error due to the block based optical flow compensation for the complex texture block can be improved.

[0174] FIG. 8 is a schematic diagram illustrating an example of obtaining an optical flow compensation result by an optical flow compensation according to an embodiment of the present disclosure.

[0175] In the example of FIG. 8, judgment of the residual motion related information may first be performed based on a forward reference block and a backward reference block of the current block, i.e., the residual motion related information of pixels in the current block is determined. If the residual motion is determined to be small based on the residual motion related information, it may be determined that the optical flow compensation method for the current block is the traditional BDOF, and if the residual motion is determined to be large based on the residual motion related information, it is determined that the optical flow compensation method for the current block is the BDOF with the POC. Next, regardless of which BDOF is determined based on the judgment of the residual motion related information, judgment of the pixel fluctuation related information is further performed. For example, in the case that it is initially determined that the optical flow compensation method for the current block is the traditional BDOF, if both of the two reference blocks are simple texture blocks, the optical flow compensation result of the current block may be obtained using the block based traditional BDOF, whereas if at least one of the two reference blocks is a complex texture block, the optical flow compensation result of each pixel in the current block may be obtained respectively by using the sample based traditional BDOF. For example, in the case that it is initially determined that the optical flow compensation method for the current block is the BDOF with the POC, if both of the two reference blocks are simple texture blocks, the optical flow compensation result of the current block may be obtained using the block based BDOF with the POC, whereas if at least one of the two reference blocks is a complex texture block, the optical flow compensation result of each pixel in the current block may be obtained respectively by using the sample based BDOF with the POC. Although, in the example of FIG. 8, it is shown that the first reference block and the second reference block are the forward reference block and the backward reference block of the current block, respectively, and the optical flow compensation technology is the BDOF, the first reference block and the second reference block are not limited to being only the forward reference block and the backward reference block of the current block, and the optical flow compensation technology is not limited to being only BDOF, as mentioned above.

[0176] According to an embodiment of the present disclosure, although it is mentioned in the description with respect to FIG. 3 that the optical flow compensation method for the current block is determined based on the residual motion related information, and then the optical flow compensation result of the current block is obtained using the determined optical flow compensation method, however, alternatively, it is also possible to directly fuse the first optical flow compensation result obtained based on the first reference block and the second optical flow compensation result obtained based on the second reference block to obtain the optical flow compensation result of the current block, based on the POC of the current block and the POCs of the first reference block and the second reference block of the current block. Thus, alternatively, according to an embodiment of the present disclosure, a method performed by the electronic apparatus may also be provided.

[0177] FIG. 9 is a flowchart illustrating a method performed by a video decoding apparatus or a video encoding apparatus according to an embodiment of the present disclosure.

[0178] Referring to FIG. 9, at step S910, a POC of a current block and the POCs of a first reference block and a second reference block of the current block are determined. The present disclosure does not limit the manner of determining the POCs of the current block, the first reference block and the second reference block.

[0179] At step S920, a first optical flow compensation result of the current block obtained based on the first reference block and a second optical flow compensation result of the current block obtained based on the second reference block are fused to obtain an optical flow compensation result of the current block, based on the determined POCs.

[0180] According to an embodiment, alternatively, the fusing of the first optical flow compensation result of the current block obtained based on the first reference block and the second optical flow compensation result of the current block obtained based on the second reference block to obtain the optical flow compensation result of the current block, based on the determined POCs, includes: obtaining the optical flow compensation result of the current block by performing a weighted sum on the first optical flow compensation result and the second optical flow compensation result, based on POC differences between the first reference block as well as the second reference block, respectively, and the current block. According to an embodiment, a first weight corresponding to the first reference block and a second weight corresponding to the second reference block may be determined first based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block; the optical flow compensation result of the current block is obtained by performing the weighted sum on the first optical flow compensation result and the second optical flow compensation result, using the first weight and the second weight. For example, the first weight and the second weight may be determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and a POC difference between the first reference block and the second reference block. As an example, the first weight and the second weight may be determined based on a ratio of a first value to a second value, wherein the first value is determined based on the POC differences between the first reference block as well as the second reference block, respectively, and the current block, and the second value is determined based on the POC difference between the first reference block and the second reference block. How to determine the weights has been described above in conjunction with examples, and will not be repeated here, and relevant details may be referred to the description above.

[0181] Alternatively, the method shown in FIG. 9 may further include: determining optical flow vectors of the current block using POC differences between the first reference block as well as the second reference block, respectively, and the current block; obtaining the first optical flow compensation result and the second optical flow compensation result based on the optical flow vectors. For example, the optical flow vectors of the current block may include a horizontal direction optical flow vector and a vertical direction optical flow vector. As an example, the determining of the optical flow vectors of the current block using the POC differences between the first reference block as well as the second reference block, respectively, and the current block, may include: determining the horizontal direction optical flow vector and the vertical direction optical flow vector, respectively, based on gradient values of pixel values of the first reference block and the second reference block corresponding to a pixel value of the current block, in the horizontal direction and the vertical direction, respectively, and the POC differences between the first reference block as well as the second reference block, respectively, and the current block. As an example, the obtaining of the first optical flow compensation result and the second optical flow compensation result based on the optical flow vectors may include: obtaining the first optical flow compensation result based on a POC difference between the first reference block and the current block, the optical flow vectors, and a pixel value of the first reference block corresponding to the pixel value of the current block; obtaining the second optical flow compensation result based on a POC difference between the second reference block and the current block, the optical flow vectors, and a pixel value of the second reference block corresponding to the pixel value of the current block. In the above, how to obtain the first optical flow compensation result, the second optical flow compensation result, and how to obtain the optical flow compensation result of the current block based on the first optical flow compensation result and the second optical flow compensation result have been described in conjunction with examples, and will not be repeated here, and the relevant details may be referred to the description above.

[0182] According to the method shown in FIG. 9, the optical flow compensation result of the current block is obtained by fusing, based on the POC of current block, the POCs of the first reference block and the second reference of the current block, the first optical flow compensation result of the current block obtained based on the first reference block and the second optical flow compensation result of the current block obtained based on the second reference block, which enables to obtain a more accurate optical flow compensation result, so as to better improve the residual motion problem of the current block.

[0183] According to an embodiment of the present disclosure, although it is mentioned above in the description of step S330 of FIG. 3 that determining whether the processing object of the optical flow compensation method is the current block or a sub-block of the current block, or a pixel in the current block, based on the pixel fluctuation related information, and then obtaining, for the determined processing object, the optical flow compensation result of the processing object using the optical flow compensation method, based on the first reference block and second reference block of the current block, however, alternatively, instead of performing the above operations in the framework of FIG. 3, the above operations may also be applied separately. Thus, alternatively, according to an embodiment of the present disclosure, a method performed by an electronic apparatus, which includes the above-described operations, may also be provided.

[0184] FIG. 10 is a flowchart illustrating a method performed by a video decoding apparatus or a video encoding apparatus according to an embodiment of the present disclosure.

[0185] As shown in FIG. 10, at step S1010, pixel fluctuation related information of a current block is determined. According to an embodiment, alternatively, step S1010 may include: determining the pixel fluctuation related information of the first reference block and the second reference block, respectively; determining the pixel fluctuation related information of the current block based on the pixel fluctuation related information of the first reference block and the second reference block. For example, gradients in a plurality of directions of each pixel of a plurality of pixels comprised in the first reference block and the second reference block may be determined respectively; the pixel fluctuation related information of each of the first reference block and the second reference block is determined based on the respectively determined gradients in the plurality of directions of each of the first reference block and the second reference block. As examples, the gradients in the plurality of directions comprises a gradient in a horizontal direction, a gradient in a vertical direction, and a gradient in at least one inclined direction different from the horizontal direction and vertical direction. As an example, for the first reference block and the second reference block, the pixel fluctuation related information thereof may be determined by performing a sum or a weighted sum on the gradient in the horizontal direction, the gradient in the vertical direction, and the gradient in the at least one inclined direction of each pixel of the plurality of pixels comprised therein, respectively. How to determine the pixel fluctuation related information has been exemplarily described above and will not be repeated here.

[0186] At step S1020, whether a processing object of an optical flow compensation method is the current block or a sub-block of the current block, or a pixel in the current block is determined, based on the pixel fluctuation related information.

[0187] At step S1030, for the determined processing object, an optical flow compensation result of the processing object is obtained using the optical flow compensation method, based on the first reference block and second reference block of the current block.

[0188] According to the method of FIG. 10, by determining the pixel fluctuation related information of the current block, determining whether the processing object of the optical flow compensation method is the current block or a sub-block of the current block, or a pixel in the current block, based on the pixel fluctuation related information, and obtaining, for the determined processing object, the optical flow compensation result of the processing object using the optical flow compensation method, based on the first reference block and second reference block of the current block, it enables to balance both the reduction of the error of the optical flow compensation result and the reduction of the computational complexity, so that it is possible to achieve a more accuracy and improvement of the residual motion problem with a lower computational complexity.

[0189] According to an embodiment, the method described above with reference to FIG. 3, the method described above with reference to FIG. 9, and the method described above with reference to FIG. 10 may be applied to both the encoding side and the decoding side.

[0190] The following describes technical solutions in which the method described with reference to FIG. 3, the method described with reference to FIG. 9, and the method described with reference to FIG. 10 are applied to both an encoding side and a decoding side, respectively.

[0191] FIG. 11 is a flowchart illustrating a method performed by a video encoding apparatus according to an embodiment of the present disclosure. The method shown in FIG. 11 corresponds to the method described in FIG. 3, which is a technical solution of applying the method described in FIG. 3 to the encoding side.

[0192] Referring to FIG. 11, at step S1110, residual motion related information of pixels in a first encoded block is determined. Step S1110 corresponds to step S310 above, except that the current block is a first encoded block, therefore, the description of the step S310 above is all adapted to the step S1110, which will not be repeated here.

[0193] At step S1120, an optical flow compensation method for the first encoded block is determined based on the residual motion related information, wherein the optical flow compensation method comprises: fusing a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the first encoded block, based on a picture order count (POC) of the first encoded block and the POCs of the first reference block and the second reference block of the first encoded block. It is determined, based on the residual motion related information, whether to obtain a weighted sum of a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the first encoded block. The weights for the first optical flow compensation result and the second optical flow compensation result are determined based on a difference between the POC of the first encoded block and the POC of the first reference block and a difference between the POC of the first encoded block and the POC of the second reference block.

[0194] The step S1120 corresponds to the step S320 above, except that the current block is the first encoded block, therefore, the description of the step S320 above is all adapted to the step S1120, which will not be repeated here.

[0195] At step S1130, the optical flow compensation result of the first encoded block is obtained using the determined optical flow compensation method, based on the first reference block and the second reference block. Based on determining that the weighted sum is obtained at step S1120, the optical flow compensation result of the first encoded block is obtained using the weighted sum of the first optical flow compensation result and the second optical flow compensation result at step S1130. The step S1130 corresponds to the step S330 above, except that the current block is the first encoded block, therefore, the description of the step S330 above is all adapted to the step S1130, which will not be repeated here.

[0196] At step S1140, encoding on a second encoded block is performed based on the optical flow compensation result. According to an embodiment, the second encoded block may be any encoded block different from the first encoded block, e.g., it may be the encoded block using the first encoded block as a reference block, e.g., the second encoded block may be a next encoded block of the first encoded block, but is not limited thereto. According to the encoding mode, the second encoded block may be different. Alternatively, the performing of the encoding on the second encoded block may include performing steps S1110 to S1130 above, except that the first encoded block is replaced with the second encoded block at this time, and one of the first reference block and the second reference block of the second encoded block may be the first encoded block at this time, wherein the optical flow compensation result of the first encoded block may be used in obtaining the optical flow compensation result of the second encoded block based on the first encoded block. A bitstream is generated to include encoding information of the second encoded block.

[0197] Alternatively, the method shown in FIG. 11 may further comprise: encoding information for indicating a motion vector of the first encoded block into a bitstream. According to an embodiment, the information may include differences between the motion vector of the first encoded block and the motion vector of the first reference block as well as the motion vector of the second reference block, respectively, but not limited thereto. For example, at the encoding end, a motion vector prediction (MVP) candidate list may be constructed for the first encoded block by blocks neighboring the first encoded block in the temporal domain and blocks neighboring the first encoded block in the space domain. When there are more than one reference frame for the first encoded block, it needs to maintain more than one MVP candidate list for the first encoded block. For example, if there are two reference frames for the first encoded block, each reference frame may correspond to one MVP candidate list. Each MVP candidate list may include a plurality of MVP candidates, and each MVP candidate is one MV that points to one candidate reference block. By calculating the sum of absolute difference (SAD) between the first encoded block and the candidate reference block pointed by each MVP candidate in the MVP candidate list, the reference block with the smallest SAD may be determined as the reference block of the first encoded block. For example, assuming that the reference blocks of the first encoded block determined in the manner described above are the first reference block and the second reference block as described above, a more accurate MV of the first encoded block may be searched for with the MVP candidate corresponding to the first reference block and the MVP candidate corresponding to the second reference block, respectively, as the starting points of the motion search. Assuming that the more accurate MV of the first encoded block searched for with the MVP candidate corresponding to the first reference block is MV1 and the more accurate MV of the first encoded block searched for with the MVP candidate corresponding to the second reference block is MV2, the motion vector difference MVD1 which is obtained by subtracting the MVP candidate corresponding to the first reference block from MV1 and the motion vector difference MVD2 which is obtained by subtracting the MVP candidate corresponding to the first reference block from MV2 may be encoded into the bitstream. In addition, a reference frame list, an index of a reference frame selected in the reference frame list, a MVP candidate list corresponding to the index of the selected reference frame, and an index of an MVP candidate selected in the MVP candidate list of the first encoded block may be encoded into the bitstream, for example, the above items together with the motion vector difference are encoded into the bitstream to be transmitted to the decoding end.

[0198] According to the method shown in FIG. 11, by determining the residual motion related information of the pixels in the first encoded block and determining, based on the residual motion related information of the pixels in first encoded block, the optical flow compensation method for the first encoded block, which may include fusing the first optical flow compensation result obtained based on a first reference block and the second optical flow compensation result obtained based on a second reference block to obtain the optical flow compensation result of the first encoded block, based on the POC of the first encoded block and the POCs of the first reference block and the second reference block of the first encoded block, it enables to obtain the optical flow compensation result of the first encoded block without being constrained by the isometric bi-directional prediction condition, therefore, the residual motion problem within the current block may be better improved in the encoding, which improves the performance of the encoding.

[0199] FIG. 12 is a flowchart illustrating a method performed by a video encoding apparatus according to an embodiment of the present disclosure. The method shown in FIG. 12 corresponds to the method described in FIG. 9, which is a technical solution of applying the method described in FIG. 9 to the encoding side.

[0200] Referring to FIG. 12, at a step S1210, a POC of a first encoded block and the POCs of a first reference block and a second reference block of the first encoded block are determined. at a step S1220, a first optical flow compensation result of the first encoded block obtained based on the first reference block and a second optical flow compensation result of the first encoded block obtained based on the second reference block are fused to obtain an optical flow compensation result of the first encoded block, based on the determined POCs. The step S1210 corresponds to the step S910 above, and the step S1220 corresponds to the step S920 above, except that the current block is the first encoded block, and thus the description of the step S910 above is adapted to the step S1210 and the description of the step S920 is adapted to the step S1220, which will not be repeated here.

[0201] At step S1230, encoding on a second encoded block is performed based on the optical flow compensation result. In the above, the performing of encoding on the second encoded block based on the optical flow compensation result has been described with reference to the step S1140, and the step S1230 is the same as the step S1140, which will not be repeated here. In addition, alternatively, the method shown in FIG. 12 may further include: encoding information for indicating a motion vector of the first encoded block into a bitstream. The example of the specific implementation of encoding the information for indicating the motion vector of the first encoded block into the bitstream has been described above and will not be repeated here.

[0202] According to the method shown in FIG. 12, the optical flow compensation result of the first encoded block is obtained by fusing, based the POC of the first encoded block and the POCs of the first reference block and the second reference block of the first encoded block, the first optical flow compensation result of the first encoded block obtained based on the first encoded block and the second optical flow compensation result of the first encoded block obtained based on the second reference block, which enables to obtain more accurate optical flow compensation result during the encoding process, so that it is possible to better improve the residual motion problem of the current block and improve the performance of encoding.

[0203] FIG. 13 is a flowchart illustrating a method performed by a video encoding apparatus according to an embodiment of the present disclosure. The method shown in FIG. 13 corresponds to the method described in FIG. 10, which is a technical solution of applying the method described in FIG. 10 to the encoding side.

[0204] Referring to FIG. 13, at step S1310, pixel fluctuation related information of the first encoded block is determined. At step S1320, whether a processing object of an optical flow compensation method is the first encoded block or a sub-block of the first encoded block, or a pixel in the first encoded block is determined based on the pixel fluctuation related information. At step S1330, for the determined processing object, an optical flow compensation result of the processing object is obtained using the optical flow compensation method, based on a first reference block and a second reference block of the first encoded block. The step S1310 corresponds to the step S1010 above, the step S1320 corresponds to the step S1020 above, and the step S1330 corresponds to the step S1030 above, except that the current block is the first encoded block, and thus the description of the step S1010 above is adapted to the step S1310, the description of the step S1020 is adapted to the step S1320, and the description of the step S1030 is adapted to the step S1330, which will not be repeated here.

[0205] At step S1340, encoding on a second encoded block is performed based on the optical flow compensation result. In the above, the performing of encoding on the second encoded block based on the optical flow compensation result has been described with reference to the step S1140, and the step S1340 is the same as step the S1140, which will not be repeated here. In addition, alternatively, the method shown in FIG. 13 may also include: encoding information for indicating a motion vector of the first encoded block into a bitstream. The example of the specific implementation of encoding the information for indicating the motion vector of the first encoded block into the bitstream has been described above and will not be repeated here.

[0206] According to the method shown in FIG. 13, by determining the pixel fluctuation related information of the first encoded block, determining whether the processing object of the optical flow compensation method is the first encoded block or a sub-block of the first encoded block, or a pixel in the first encoded block, based on the pixel fluctuation related information, and obtaining, for the determined processing object, the optical flow compensation result of the processing object using the optical flow compensation method, based on the first reference block and the second reference block of the first encoded block, it enables to balance both the reduction of the error of the optical flow compensation result and the reduction of the computational complexity in the encoding, so that it is possible to achieve a more accuracy and improvement of the residual motion problem with a lower computational complexity in the encoding, which improves the performance of encoding.

[0207] In the above, the methods performed by the video encoding apparatus according to embodiments of the present disclosure have been described. In the following, methods performed by the video decoding apparatus corresponding to FIGS. 3, 9, and 10, respectively, are briefly described.

[0208] FIG. 14 is a flowchart illustrating a method performed by a video decoding apparatus according to an embodiment of the present disclosure. The method shown in FIG. 14 corresponds to the method described in FIG. 3, and it is a technical solution of applying the method described in FIG. 3 to the decoding side.

[0209] Referring to FIG. 14, at step S1410, a motion vector of a current block is decoded from a bitstream. For example, as mentioned above in the description with reference to FIG. 3, the motion vector difference MVD1 as well as MV2 may be encoded into the bitstream at the encoding side, in addition, a reference frame list, an index of a reference frame selected in the reference frame list, a MVP candidate list corresponding to the index of the selected reference frame, and an index of an MVP candidate selected in the MVP candidate list of the current block may be encoded into the bitstream. The above information is encoded into the bitstream to be transmitted to the decoding end as information for indicating the motion vector of the current block. The decoding end determines the selected MVP candidate based on the reference frame list, the index of the reference frame selected in the reference frame list, the MVP candidate list corresponding to the index of the selected reference frame, and the index of the MVP candidate selected in the MVP candidate list of the current block, and subsequently, the selected MVP candidate may be added with the motion vector difference MVD1 as well as MV2, and the motion vector of the current block may be determined.

[0210] Subsequently, at step S1420, a first reference block and a second reference block of the current block are determined based on the motion vector of the current block.

[0211] At step S 1430, residual motion related information of pixels in the current block is determined.

[0212] At step S 1440, an optical flow compensation method for the current block is determined based on the residual motion related information, wherein the optical flow compensation method comprises: fusing a first optical flow compensation result obtained based on the first reference block and a second optical flow compensation result obtained based on the second reference block to obtain an optical flow compensation result of the current block, based on a picture order count (POC) of the current block and the POCs of the first reference block and the second reference block of the current block.

[0213] At step S1450, the optical flow compensation result of the current block is determined using the determined optical flow compensation method, based on the first reference block and the second reference block.

[0214] Steps S1430 to step S1450 are the same as the operations S310 to S330 in FIG. 3, respectively, and therefore, the operations in steps S1430 to S1450 will not be repeated here, and the relevant details may be referred to the description of the steps of FIG. 3 above.

[0215] According to the method of FIG. 14, by determining the residual motion related information of pixels in the current block, and determining the optical flow compensation method for the current block based on the residual motion related information, which may include fusing the first optical flow compensation result obtained based on the first reference block and the second optical flow compensation result obtained based on the second reference block to obtain the optical flow compensation result of the current block, based on the POCs of the current block, the first reference block and the second reference block, it enables to obtain the optical flow compensation result of the current block without being constrained by the isometric bi-directional prediction condition, so as to facilitate better improvement of the residual motion problem within the current block in the decoding, which improves the performance of the decoding.

[0216] FIG. 15 is a flowchart illustrating a method performed by a video decoding apparatus according to an embodiment of the present disclosure. The method shown in FIG. 15 corresponds to the method described in FIG. 9, which is a technical solution of applying the method described in FIG. 9 to the decoding side.

[0217] Referring to FIG. 15, at step S1510, a motion vector of a current block is decoded from a bitstream. This operation has been described in the step S1410 of FIG. 14 and will not be repeated here.

[0218] Subsequently, at step S1520, a first reference block and a second reference block of the current block are determined based on the motion vector of the current block.

[0219] Next, at step S1530, POCs of the current block, the first reference block and the second reference block are determined. At step S1540, based on the determined POCs, a first optical flow compensation result of the current block obtained based on the first reference block and a second optical flow compensation result of the current block obtained based on the second reference block are fused to obtain an optical flow compensation result of the current block.

[0220] Step S1530 and step S1540 are the same as the operations S910 to S920 in FIG. 9, respectively, and therefore, the operations in the step S1530 and the step S1540 will not be repeated here, and the relevant details may be referred to the description of the steps of FIG. 9 above.

[0221] According to the method of FIG. 15, the optical flow compensation result of the current block is obtained by fusing, based on the determined POCs, the first optical flow compensation result of the current block obtained based on the first reference block and the second optical flow compensation result of the current block obtained based on the second reference block, which enables to obtain a more accurate optical flow compensation result during decoding, so that it is possible to better improve the residual motion problem of the current block, which improves the performance of the decoding.

[0222] FIG. 16 is a flowchart illustrating a method performed by a video decoding apparatus according to an embodiment of the present disclosure. The method shown in FIG. 16 corresponds to the method described in FIG. 10, which is a technical solution of applying the method described in FIG. 10 to the decoding side.

[0223] Referring to FIG. 16, at step S1610, a motion vector of the current block is decoded from a bitstream. This operation has been described in the step S1410 of FIG. 14 and will not be repeated here.

[0224] Subsequently, at step S1620, a first reference block and a second reference block of the current block are determined based on the motion vector of the current block.

[0225] Next, at step S1630, pixel fluctuation related information of the current block is determined. At step S1640, whether a processing object of an optical flow compensation method is the current block or a sub-block of the current block, or a pixel in the current block is determined, based on the pixel fluctuation related information. At step S1650, for the determined processing object, an optical flow compensation result of the processing object is determined using the optical flow compensation method, based on the first reference block and second reference block of the current block.

[0226] Steps S1630 to step S1650 are the same as operations S1010 to S1030 in FIG. 10, respectively, and therefore, the operations in steps S1630 to S1650 are not repeated here, and the relevant details may be referred to the description of the steps of FIG. 10 above.

[0227] According to the method of FIG. 16, by determining the pixel fluctuation related information of the current block, determining, based on the pixel fluctuation related information, whether the processing object of the optical flow compensation method is the current block or a sub-block of the current block, or a pixel in the current block, and obtaining, for the determined processing object, the optical flow compensation result of the processing object using the optical flow compensation method based on the first reference block and second reference block of the current block, it enables to balance both the reduction of the error of the optical flow compensation result and the reduction of the computational complexity in the decoding, so that it is possible to achieve a more accuracy and improvement of the residual motion problem with a lower computational complexity in the decoding, which improves the performance of the decoding.

[0228] In the above, the methods performed by the video decoding apparatus according to embodiments of the present disclosure have been described.

[0229] In the following, the electronic apparatus according to an embodiment of the present disclosure are briefly described. FIG. 17 is a block diagram illustrating a video decoding apparatus according to an embodiment of the present disclosure. Referring to FIG. 17, the video encoding apparatus 1700 may include a memory 1701 and a processor 1702, wherein the processor 1702 is coupled to the memory 1701 and configured to perform any of the methods described in Fig. 3, 14 to 16.

[0230] FIG. 18 is a block diagram illustrating a video encoding apparatus according to an embodiment of the present disclosure. Referring to FIG. 18, the video encoding apparatus 1800 may include a memory 1801 and a processor 1802, wherein the processor 1802 is coupled to the memory 1801 and configured to perform the methods described in Fig. 3, 11 to 13.

[0231] In embodiments of the present disclosure, there is also provided a video encoding apparatus or a video encoding apparatus that includes at least one processor, and alternatively, further includes at least one transceiver and / or at least one memory coupled to the at least one processor, wherein, the at least one processor is configured to perform the steps of the method provided in any alternative embodiment of the present disclosure.

[0232] FIG. 19 illustrates a schematic diagram of a structure of a video decoding apparatus or a video encoding apparatus applicable to an exemplary embodiment of the present application. As shown in FIG. 19, the electronic apparatus 4000 shown in FIG. 19 includes: a processor 4001 and a memory 4003. Wherein the processor 4001 and the memory 4003 are coupled, e.g., through a bus 4002. Alternatively, the electronic apparatus 4000 may further include a transceiver 4004 which may be used for data interaction between the electronic apparatus and other electronic apparatuses, such as transmitting of data and / or receiving of data. It should be noted that, each of the processor 4001, the memory 4003, and the transceiver 4004 is not limited to one in a practice application, and the structure of the electronic apparatus 4000 does not constitute a limitation of the embodiments of the present disclosure. Alternatively, the electronic apparatus may be the first network node, the second network node, or the third network node.

[0233] The processor 4001 may be a Central Processing Unit (CPU), general purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic device, transistor logic device, hardware part, or any combination thereof. It may implement or perform various exemplary logic boxes, modules, and circuits described in conjunction with the disclosed contents of the present disclosure. The processor 4001 may also be a combination that implements computing functions, such as a combination containing one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0234] The bus 4002 may include a pathway to transfer information between the above components. The bus 4002 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, and the like. The bus 4002 may be classed as an address bus, a data bus, a control bus, and the like. For ease of representation, only one bold line is shown in FIG. 19, but it does not mean that there is only one bus or one type of bus.

[0235] The memory 4003 may be a Read Only Memory (ROM) or other types of static storage apparatuses that can store static information and instructions, a Random Access Memory (RAM) or other types of dynamic storage apparatuses that can store information and instructions, may be an Electrically Erasable Programmable Read Only Memory (EEPROM), Compact Disc Read Only Memory (CD-ROM) or other optical disc storages, an optical disc storage (including a compressed disc, laser disc, optical disc, digital universal disc, Blu-ray disc, etc.), a disk storage medium, other magnetic storage apparatuses, or any other medium that can be used to carry or store computer programs and can be read by a computer, it is not limited herein.

[0236] The memory 4003 is used to store computer programs or executable instructions for performing the embodiments of the present disclosure, and is controlled for execution by the processor 4001. The processor 4001 is used to execute the computer programs or executable instructions stored in the memory 4003 to implement the steps shown in the preceding method of the embodiments.

[0237] An embodiment of the present disclosure provides a computer readable storage medium storing computer programs or instructions, the computer programs or instructions, when being executed by at least one processor may perform or implement the steps in the preceding method of the embodiments and corresponding contents.

[0238] An embodiment of the present disclosure provides a computer program product including computer programs, the computer programs, when being executed by a processor, may implement the steps shown in the preceding method of the embodiments and corresponding contents.

[0239] The terms "first", "second", "third", "fourth", "1", "2" and the like (if exists) in the specification and claims of the present disclosure and the above drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequence. It should be understood that, data used as such may be interchanged in appropriate situations, so that the embodiments of the present disclosure described here may be implemented in an order other than the illustration or text description.

[0240] It should be understood that, although each operation step is indicated by an arrow in the flowcharts of the embodiments of the present disclosure, an implementation order of these steps is not limited to an order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of the present disclosure, the implementation steps in the flowcharts may be executed in other orders according to requirements. In addition, some or all of the steps in each flowchart may include a plurality of sub steps or stages, based on an actual implementation scenario. Some or all of these sub steps or stages may be executed at the same time, and each sub step or stage in these sub steps or stages may also be executed at different times. In scenarios with different execution times, an execution order of these sub steps or stages may be flexibly configured according to a requirement, which is not limited by the embodiment of the present disclosure.

[0241] The above text and accompanying drawings are provided as examples only to assist readers in understanding the present disclosure. They are not intended and should not be interpreted as limiting the scope of the present disclosure in any way. Although certain embodiments and examples have been provided, based on the content disclosed herein, it is apparent to those skilled in the art that, changes can be made to the illustrated embodiments and examples without departing from the scope of the present disclosure, and other similar implementation methods based on the technical concepts of the present disclosure also belongs to a protection scope of the embodiments of the present disclosure.

Claims

A method performed by a video decoding apparatus comprising:determining (310) residual motion related information of pixels in a current block;determining (320), based on the residual motion related information, whether to obtain a weighted sum of a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the current block, wherein weights for the first optical flow compensation result and the second optical flow compensation result are determined based on a difference between a picture order count (POC) of the current block and a POC of the first reference block and a difference between the POC of the current block and a POC of the second reference block of the current block;based on determining that the weighted sum is obtained, obtaining (330) the optical flow compensation result of the current block using the weighted sum of the first optical flow compensation result and the second optical flow compensation result.The method of claim 1, wherein the obtaining of the optical flow compensation result of the current block, comprises:determining pixel fluctuation related information of the current block, wherein the pixel fluctuation related information represents a texture change situation in a region surrounding one pixel;determining whether a processing object of optical flow compensation is the current block or a sub-block of the current block, or a pixel in the current block, based on the pixel fluctuation related information;obtaining, for the determined processing object, the optical flow compensation result of the processing object based on the first reference block and the second reference block.The method of claim 1 or 2, wherein the determining of the residual motion related information of the pixels in the current block comprises:determining the residual motion related information of the pixels in the current block based on the first reference block and the second reference block of the current block.The method of claim 3, wherein the determining of the residual motion related information of the pixels in the current block based on the first reference block and the second reference block of the current block comprises:determining the residual motion related information of the pixels in the current block based on a feature related to a pixel gradient of the first reference block and the second reference block.The method of any one of claims 2 to 4, wherein the determining of the pixel fluctuation related information of the current block comprises:determining the pixel fluctuation related information of the first reference block and the second reference block, respectively;determining the pixel fluctuation related information of the current block based on the pixel fluctuation related information of the first reference block and the second reference block.The method of any one of claims 2 to5, wherein the determining of the pixel fluctuation related information of the first reference block and the second reference block, respectively, comprises:determining gradients in a plurality of directions of each pixel of a plurality of pixels comprised in the first reference block and the second reference block, respectively;determining the pixel fluctuation related information of each of the first reference block and the second reference block based on the respectively determined gradients in the plurality of directions of each of the first reference block and the second reference block.A method performed by a video encoding apparatus, the video encoding method comprising:determining (1110) residual motion related information of pixels in a first encoded block;determining (1120), based on the residual motion related information, whether to obtain a weighted sum of a first optical flow compensation result obtained based on a first reference block and a second optical flow compensation result obtained based on a second reference block to obtain an optical flow compensation result of the first encoded block, wherein weights for the first optical flow compensation result and the second optical flow compensation result are determined based on a difference between a picture order count (POC) of the first encoded block and a POC of the first reference block and a difference between the POC of the first encoded block and a POC of the second reference block of the first encoded block;based on determining that the weighted sum is obtained, obtaining (1130) the optical flow compensation result of the first encoded block using the weighted sum of the first optical flow compensation result and the second optical flow compensation result;performing (1140) encoding on a second encoded block based on the optical flow compensation result; andgenerating a bitstream including encoding information of the second encoded block.The method of claim 7, wherein the obtaining of the optical flow compensation result of the first encoded block, comprises:determining pixel fluctuation related information of the first encoded block, wherein the pixel fluctuation related information represents a texture change situation in a region surrounding one pixel;determining whether a processing object of optical flow compensation is the first encoded block or a sub-block of the first encoded block, or a pixel in the first encoded block, based on the pixel fluctuation related information;obtaining, for the determined processing object, the optical flow compensation result of the processing object based on the first reference block and the second reference block.The method of claim 7 or 8, wherein the determining of the residual motion related information of the pixels in the first encoded block comprises:determining the residual motion related information of the pixels in the first encoded block based on the first reference block and the second reference block of the first encoded block.The method of claim 9, wherein the determining of the residual motion related information of the pixels in the first encoded block based on the first reference block and the second reference block of the first encoded block comprises:determining the residual motion related information of the pixels in the first encoded block based on a feature related to a pixel gradient of the first reference block and the second reference block.The method of any one of claims 8 to 10, wherein the determining of the pixel fluctuation related information of the first encoded block comprises:determining the pixel fluctuation related information of the first reference block and the second reference block, respectively;determining the pixel fluctuation related information of the first encoded block based on the pixel fluctuation related information of the first reference block and the second reference block.The method of any one of claims 8 to 11, wherein the determining of the pixel fluctuation related information of the first reference block and the second reference block, respectively, comprises:determining gradients in a plurality of directions of each pixel of a plurality of pixels comprised in the first reference block and the second reference block, respectively;determining the pixel fluctuation related information of each of the first reference block and the second reference block based on the respectively determined gradients in the plurality of directions of each of the first reference block and the second reference block.A video decoding apparatus (1700) comprising:a memory (1701); anda processor (1702) coupled to the memory and configured to perform the method of any one of claims 1 to 6.A video encoding apparatus (1800) comprising:a memory (1801); anda processor (1802) coupled to the memory and configured to perform the method of any one of claims 7 to 12.A method of transmitting a bitstream generated by the method of claim 7.

Images