Moving image processing method, bit stream generation method, and moving image processing apparatus

Abstract

To provide a method and a device for image motion compensation that can increase coding efficiency.SOLUTION: An image processing method includes: acquiring an initial motion vector MV of the current image block including a first initial MV directing a first reference image and a second initial MV directing a second reference image; correcting the first initial MV and the second initial MV to obtain a first MV after correction and a second MV after correction; and executing motion prediction for the current image block on the basis of the first MV after correction and the second MV after correction. The first reference image and the second reference image are a short-term reference image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] Copyright Declaration The subject matter disclosed by this patent document contains material protected by copyright. Such copyright is owned by the Copyright Owner. The Copyright Owner has no objection to anyone copying the patent document or the patent disclosure as it appears in the official records and archives of the Patent and Trademark Office.

[0002] The present application relates to the field of image processing, and in particular to a method and apparatus for image motion compensation. [Background technology]

[0003] In recent years, with the popularity of portable devices, handheld devices, and wearable devices, the capacity of video is constantly increasing. As video formats become more complex, video storage and transmission also becomes more difficult. In order to reduce the bandwidth occupied by video storage and transmission, video data is usually encoded and compressed at the encoding end, and decoded at the decoding end.

[0004] The encoding compression process includes processes such as prediction, transformation, quantification, and entropy coding. Here, prediction includes two types, intraframe prediction and interframe prediction, which aims to remove redundant information of the image block currently being encoded by using predicted block data. Intraframe prediction uses information of the current frame image to obtain predicted block data. Interframe prediction uses information of a reference image to obtain predicted block data, and the process is as follows: the current image to be encoded is divided into a number of image blocks to be encoded, and the image block to be encoded is divided into a number of sub-image blocks. Then, for each sub-image block, an image block in the reference image that is most matched with the current sub-image block is searched for as a predicted image block, and the relative displacement between the predicted image block and the current sub-image block is the motion vector. Then, the corresponding pixel values ​​of the sub-image block and the predicted image block are subtracted to obtain a residual. The obtained residuals corresponding to each sub-image block are combined together to obtain the residual of the image block to be encoded. After the residual undergoes processes such as transformation, quantification, entropy coding, etc., an entropy-coded bitstream is obtained, and the entropy-coded bitstream and post-encoding coding mode information, such as information on intra-frame prediction mode, motion vector (or motion vector residual), etc., are stored or transmitted to the decoding end.

[0005] At the image decoding end, an entropy-encoded bit stream is obtained, and then entropy-decoded to obtain the corresponding residual. Based on information such as the motion vector and intra-frame prediction obtained by decoding, a predicted image block corresponding to the image block to be decoded is searched for, and the value of each pixel point in the image block to be decoded is obtained based on the predicted image block and the residual.

[0006] From the above description, it can be seen that when performing inter-frame prediction, the more similar the selected reference image is to the current image to be encoded, the smaller the residual error caused by inter-frame prediction, thereby improving the coding efficiency of inter-frame prediction. Specifically, some existing technologies can use each image in a video to construct a high-quality specific reference image of background content including a scene. When performing inter-frame prediction, the background part of the current image to be encoded or the current image to be decoded can be used to reduce the residual information of inter-frame prediction by referring to the high-quality specific reference image, thereby improving the coding efficiency. That is, the specific reference image is a reference image for inter-frame prediction. A long-term reference image is not a decoded image, but an artificially constructed image. A long-term reference image includes multiple image blocks, and any image block is extracted from a decoded image, and different image blocks in a long-term reference image may be from different decoded images.

[0007] In order to improve the coding efficiency and reduce the amount of information transmitted by the coding end, some existing technologies can directly derive motion vectors at the decoding end, so that the coding end does not need to transmit motion vector information or motion vector residual information, and the decoding end can obtain the true motion vector without decoding the motion vector information or motion vector residual information. Summary of the Invention [Problem to be solved by the invention]

[0008] Some existing techniques using motion vector derivation and techniques using bidirectional motion prediction do not consider the specificity of long-term reference picture. Some techniques using motion vector derivation do not consider whether the reference picture pointed to by the motion vector is a long-term reference picture, and when correcting the motion vector, motion search may be performed in the long-term reference picture, which may actually reduce search efficiency and coding efficiency. Techniques using bidirectional motion prediction operate motion vectors based on the time correlation of pictures, and when the reference picture pointed to by the related motion vector is a long-term reference picture, these operations may be invalid because the definition of the time distance between the current picture to be coded or the current picture to be decoded and the long-term reference picture is unclear. [Means for solving the problem]

[0009] The present application provides a method for image motion compensation and an apparatus for image motion compensation, which can improve encoding / decoding efficiency.

[0010] In a first aspect, there is provided a method for image motion compensation, the method comprising: Obtaining an initial motion vector MV of a current image block; If the reference image indicated by the initial MV is a specific reference image, performing motion compensation on the current image block according to the initial MV; If the reference picture to which the initial MV is directed is a non-specific reference picture, the method includes modifying the initial MV to obtain a modified MV, and performing motion compensation on the current image block based on the modified MV.

[0011] In a second aspect, there is provided an apparatus for image motion compensation, the apparatus comprising: at least one memory for storing computer executable instructions; and at least one processor adapted, either alone or in combination, to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining an initial motion vector MV of a current image block; If the reference image indicated by the initial MV is a specific reference image, performing motion compensation on the current image block according to the initial MV; If the reference picture to which the initial MV is directed is a non-specific reference picture, modify the initial MV to obtain a modified MV, and perform motion compensation on the current image block based on the modified MV.

[0012] In a third aspect, there is provided a computer readable storage medium having stored thereon instructions which, when operated on a computer, cause the computer to perform a method for image motion compensation according to the first aspect.

[0013] In a fourth aspect, there is provided an encoding device including an apparatus for image motion compensation according to the second aspect.

[0014] In a fifth aspect, there is provided a decoding device including an apparatus for image motion compensation according to the second aspect.

[0015] In a sixth aspect, there is provided a method for image motion compensation, the method comprising: Obtaining an initial motion vector MV of a current image block; Determining a scaling ratio of the initial MV, where when the initial MV is directed to a specific reference image, the scaling 1 ratio of the initial MV is 1; Scaling the initial MV based on a scaling ratio of the initial MV; and performing motion compensation on the current image block based on the first scaled MV.

[0016] In a seventh aspect, there is provided an apparatus for image motion compensation, the apparatus comprising: at least one memory for storing computer executable instructions; and at least one processor adapted, either alone or in combination, to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining an initial motion vector MV of a current image block; Determining a scaling ratio of the initial MV, where when the initial MV is directed to a specific reference image, the scaling 1 ratio of the initial MV is 1; Scaling the initial MV based on a scaling ratio of the initial MV; performing motion compensation on the current image block based on the scaled MV.

[0017] In an eighth aspect, there is provided a computer readable storage medium having stored thereon instructions which, when operated on a computer, cause the computer to perform a method for image motion compensation according to the sixth aspect.

[0018] In a ninth aspect, there is provided an encoding device including an apparatus for image motion compensation according to the seventh aspect.

[0019] In a tenth aspect, there is provided a decoding device including an apparatus for image motion compensation according to the seventh aspect.

[0020] In an eleventh aspect, there is provided a method for image processing, the method comprising: Obtaining a first initial motion vector MV and a second initial MV, the first initial MV pointing to a first reference image and the second initial MV pointing to a second reference image; When at least one of the first reference image and the second reference image is a specific reference image, calculating and obtaining a predicted image block of a current image block according to the first initial MV and the second initial MV; When the first reference image and the second reference image are both non-specific reference images, the method includes calculating and obtaining an MV of a current image block based on gradient values ​​of pixel points to which the first initial MV and the second initial MV point, and calculating and obtaining a predicted image block of the current image block based on the MV of the current image block.

[0021] In a twelfth aspect, there is provided an apparatus for image processing, the apparatus comprising: at least one memory for storing computer executable instructions; and at least one processor adapted, either alone or in combination, to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining a first initial motion vector MV and a second initial MV, the first initial MV pointing to a first reference image and the second initial MV pointing to a second reference image; When at least one of the first reference image and the second reference image is a specific reference image, calculating and obtaining a predicted image block of a current image block according to the first initial MV and the second initial MV; When the first reference image and the second reference image are both non-specific reference images, the MV of the current image block is calculated based on the gradient values ​​of the pixel points to which the first initial MV and the second initial MV point, and a predicted image block of the current image block is calculated based on the MV of the current image block.

[0022] In a thirteenth aspect, there is provided a computer readable storage medium having stored thereon instructions which, when executed on a computer, cause the computer to perform a method for image processing according to the eleventh aspect.

[0023] In a fourteenth aspect, there is provided an encoding device comprising an apparatus for image processing according to the twelfth aspect.

[0024] In a fifteenth aspect, there is provided a decoding device comprising an apparatus for image processing according to the twelfth aspect. [Brief description of the drawings]

[0025] [Figure 1] 1 is a schematic flowchart of a method for image motion compensation in one embodiment of the present application. [Diagram 2] FIG. 1 is a schematic diagram illustrating the principle of a two-way matching method according to one embodiment of the present application. [Diagram 3] FIG. 2 is a schematic diagram illustrating the principle of a template matching method according to one embodiment of the present application. [Figure 4] FIG. 1 is a schematic diagram illustrating the principle of DMVR technology in one embodiment of the present application. [Diagram 5] 4 is a schematic flowchart of a method for image processing according to another embodiment of the present application; [Figure 6] FIG. 1 is a schematic diagram of the principle of the BIO technology according to one embodiment of the present application. [Figure 7] FIG. 2 is a schematic frame diagram of an apparatus for image motion compensation according to one embodiment of the present application; [Figure 8] FIG. 1 is a schematic frame diagram of an apparatus for image processing according to one embodiment of the present application. [Figure 9] 4 is a schematic flowchart of a method for image motion compensation according to another embodiment of the present application; [Figure 10] FIG. 11 is a schematic frame diagram of an apparatus for image motion compensation according to another embodiment of the present application; [Figure 11] 11 is a schematic flowchart of a video processing method 1100 according to another embodiment of the present application. [Figure 12] 12 is a schematic flowchart of a video processing method 1200 according to another embodiment of the present application. [Figure 13] 12 is a schematic flowchart of a video processing method 1200 according to another embodiment of the present application. [Figure 14] This shows a current image block A divided into four equal-sized sub-image blocks. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The technical solutions in the embodiments of the present application are described below with reference to the drawings.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art of the present application. In this specification, the technical terms used in the present application are only for the purpose of describing specific examples, and are not intended to limit the present application.

[0028] First, the relevant techniques and concepts related to the embodiments of the present application are described.

[0029] A moving image is composed of a number of images. When encoding / decoding a moving image, different images in the moving image can adopt different prediction methods. The prediction method adopted based on an image can divide the image into an intraframe predicted image and an interframe predicted image, where the interframe predicted image includes a forward predicted image and a bidirectional predicted image. An I image is an intraframe predicted image, also called a key frame, and a P image is a forward predicted image, and a P image or an I image that has been previously encoded / decoded is used as a reference image. A B image is a bidirectional predicted image, and an image before and after is used as a reference image. In one embodiment, the encoding / decoding end generates a group of pictures (GOP) for each row after encoding / decoding a number of images, and the GOP is a group of pictures consisting of at least one of an I image and a number of B images (or bidirectional predicted images) and P images (or forward predicted images). The decoding end reads and decodes the GOP for each row during playback, and then reads the screen and renders and displays it.

[0030] In modern video coding / decoding standards, an image can be divided into multiple small blocks and coded / decoded for images of different resolutions, i.e., the image can be divided into multiple image blocks. The image can be divided into any number of image blocks. For example, the image can be divided into an m×n image block matrix. The image blocks can have a rectangular shape, a square shape, a circular shape, or any other shape. The image blocks can have any size, for example, p×q pixels. Each image block can have at least one of the same size and shape. Alternatively, two or more image blocks can have at least one of different sizes and shapes. The image blocks can have any overlapping portion or no overlapping portion. In some embodiments, the image block is called a macroblock or a Largest Coding Unit (LCU) in some coding / decoding standards. For the H.264 standard, the image block is called a macroblock and its size can be 16×16 pixels. In the High Efficiency Video Coding (HEVC) standard, an image block is called a Coding Tree Unit (CTU), and its size may be 8×8 pixels to 64×64 pixels.

[0031] In some other embodiments, an image block may not be a macroblock or coding tree unit, but may include a portion of a macroblock or coding tree unit, or may include at least two complete macroblocks (or coding tree units), or may include at least one complete macroblock (or coding tree unit) and a portion of one macroblock (or coding tree unit), or may include at least two complete macroblocks (or coding tree units) and portions of some macroblocks (or coding tree units). Thus, after an image is divided into a number of image blocks, these image blocks in the image data can be coded / decoded respectively.

[0032] In some other embodiments, the image block may be a macroblock or a part of a coding tree unit, for example, the image block may be a coding unit (CU) in some video encoding / decoding standards.

[0033] The encoding process includes processes such as prediction, transformation, quantification, and entropy coding. Here, prediction includes two types, intraframe prediction and interframe prediction, which aims to remove redundant information of the image block currently being encoded by using predicted block data. Intraframe prediction uses information of the current frame image to obtain predicted block data. Interframe prediction uses information of a reference image to obtain predicted block data, and the process divides the currently being encoded image into a plurality of image blocks to be encoded, and divides the image blocks to be encoded at least once (for example, using the quadtree division method in the HEVC standard, or using the quadtree binary tree division method) into a plurality of image blocks. Then, for each image block, an image block that best matches the current image block in the reference image is searched for as a predicted image block, and the relative displacement between the predicted image block and the current image block is the motion vector. Then, the corresponding pixel values ​​of this image block and the predicted image block are subtracted to obtain a residual. The obtained residuals corresponding to each image block are combined together to obtain the residual of the image block to be encoded.

[0034] In the embodiments of the present application, the residual correlation of the image block can be eliminated by the transform matrix, i.e., the redundant information of the image block is eliminated, so as to improve the coding efficiency. The transformation of the data block in the image block is usually adopted as a two-dimensional transform, i.e., the coding end multiplies the residual information of the data block with an N×M transform matrix and its transpose matrix, respectively, and the obtained transform coefficients are obtained after multiplication. The transform coefficients are quantified, and then the quantified coefficients are entropy coded to obtain an entropy coded bitstream. The entropy coded bitstream and coding mode information after coding, such as information on intraframe prediction mode, motion vector (or motion vector residual), etc., are stored or sent to the decoding end.

[0035] At the image decoding end, the entropy-encoded bit stream is obtained, and then the corresponding residual is obtained by entropy encoding. A predicted image block corresponding to the image block is searched for according to the motion vector, intraframe prediction and other information obtained by decoding. The value of each pixel point in the current sub-image block is obtained according to the predicted image block and the residual.

[0036] It has been mentioned in the previous paragraph that a picture that has already been coded / decoded is used as a reference picture for the current coding / decoding target picture. In some embodiments, the reference picture can be further constructed to increase the similarity between the reference picture and the picture that is currently being coded / decoded.

[0037] For example, there is a specific encoding / decoding scene in video content, and when the background in this scene does not change basically, only the foreground in the video changes or moves. For example, video monitoring belongs to this type of scene. In a video monitoring scene, it can be considered that the monitoring camera is usually fixed and does not move, or only moves slowly, and the background does not change basically. In contrast, it can be considered that objects such as people or vehicles captured by the video monitoring camera always move or change, and the foreground always changes. In this type of scene, a specific reference image can be constructed, and this specific reference image only contains high-quality background information. This specific reference image can include multiple image blocks, and any image block is extracted from a decoded image, and different image blocks in this specific reference image may be from different decoded images. When performing inter-frame prediction, the background part of the current encoding / decoding target image can refer to this specific reference image, which can be used to reduce the residual information of inter-frame prediction, thereby improving the encoding / decoding efficiency.

[0038] The above is a specific example of the specific reference image. In some embodiments, the specific reference image has at least one of the following characteristics: a composite frame, a long-term reference image, and a non-output image. Here, the non-output image is an image that is not output and displayed. Usually, the non-output image exists as a reference image for other images. For example, the specific reference image may be a composite long-term reference image, a non-output composite frame, or a non-output long-term reference image, etc. In some embodiments, the composite frame is called a composite reference frame.

[0039] In some embodiments, a non-specific reference image may be a reference image that does not have at least one of the following properties: a synthesized frame, a long-term reference image, an image that is not output. For example, the specific reference image may include a reference image other than a synthesized frame, or may include a reference image other than a long-term reference image, or may include a reference image other than an image that is not output, or may include a reference image other than a synthesized long-term reference image, or may include a reference image other than a synthesized frame that is not output, or may include a reference image other than a long-term reference image that is not output, etc.

[0040] In some embodiments, when an image in a video can be used as a reference image, it can be divided into a long-term reference image and a short-term reference image. Here, the short-term reference image is a concept corresponding to the long-term reference image. The short-term reference image exists in a certain time period in the buffer area of ​​the reference image, and after the decoded reference image after the short-term reference image has been input and output in the buffer area of ​​the reference image, the short-term reference image is moved out of the buffer area of ​​the reference image. The buffer area of ​​the reference image can be called a reference image list buffer memory, a reference image list, a reference frame list buffer memory, or a reference frame list, and is unified as the buffer area of ​​the reference image in this text.

[0041] The long-term reference picture (or a portion of the data in the long-term reference picture) can continue to exist in the buffer area of ​​the reference picture, and this long-term reference picture (or a portion of the data in the long-term reference picture) is not affected by the input / output operation of the decoded reference picture in the buffer area of ​​the reference picture, and only when the decoding end sends an update command operation, this long-term reference picture (or a portion of the data in the long-term reference picture) is moved from the buffer area of ​​the reference picture.

[0042] The short-term reference picture and the long-term reference picture may be called differently in different standards. For example, in standards such as H.264 / advanced video coding (AVC) or H.265 / HEVC, the short-term reference picture is called a short-term reference frame, and the long-term reference picture is called a long-term reference frame. In standards such as AVS coding standard (audio video coding standard, AVS), 1-P2, AVS2-P2, IEEE (institute of electrical and electronics engineers, IEEE) 1857.9-P4, the long-term reference picture is called a background picture. In standards such as VP8 and VP9, ​​the long-term reference picture is called a golden frame.

[0043] It should be understood that the adoption of specific technical terms in the embodiments of the present application does not indicate that they must be applied to a specific scene, for example, the term "long-term reference image" as "long-term reference frame" does not indicate that they must be used in a technology corresponding to a standard such as H.264 / AVC or H.265 / HEVC.

[0044] The long-term reference picture mentioned above may be constructed from image blocks extracted from multiple decoded pictures, or may be obtained by updating an existing reference frame (e.g., a pre-stored reference frame) using multiple decoded pictures. Of course, the specific constructed reference picture may also be a short-term reference picture. Or, the long-term reference picture may not be a constructed reference picture.

[0045] In the above-described embodiments, the specific reference image may include a long-term reference image, and the non-specific reference image may include a short-term reference image.

[0046] Optionally, the type of reference frame can be identified from a special field in the stream structure.

[0047] Optionally, if the reference image is determined to be a long-term reference image, the reference image is determined to be the specific reference image; if the reference image is determined to be a frame that is not output, the reference image is determined to be the specific reference image; if the reference image is determined to be a composite frame, the reference image is determined to be the specific reference image; if the reference image is determined to be a frame that is not output and the reference image is determined to be a composite frame, the reference image is determined to be the specific reference image.

[0048] Optionally, each of various types of reference pictures can have a corresponding mark, and the decoding end can determine whether the reference picture is a specific reference picture according to the mark carried by the reference picture.

[0049] In some embodiments, if a reference image is determined to have a mark of a long-term reference image, the reference image is determined to be a specific reference image.

[0050] In some embodiments, if a reference image is determined to have marks that are not output, then the reference image is determined to be a specific reference image.

[0051] In some embodiments, if a reference image is determined to have a mark of a synthesis frame, the reference image is determined to be a specific reference image.

[0052] In some embodiments, if the reference image has at least two of the following three marks, the reference image is determined to be a specific reference image: a long-term reference image mark, a non-output mark, and a synthetic frame or synthetic reference frame mark. For example, if the reference image is determined to have a non-output mark and the reference image is determined to have a synthetic frame mark, the reference image is determined to be a specific reference image.

[0053] Specifically, an image can have a mark indicating whether it is a frame to be output, and when an image is indicated not to be output, this frame is indicated to be a reference image, and further, whether this frame has a mark of a composite frame is judged, and if so, the reference image is determined to be a specific reference image.When an image is indicated to be output, it is not necessary to judge whether it is a composite frame, and this frame is directly determined to be not a specific reference image.Or, when an image is indicated not to be output, but has a mark of not a composite frame, this frame can be determined to be not a specific reference image.

[0054] Optionally, the reference picture is determined to be a specific reference picture if it is determined that the reference picture satisfies one of the following conditions by analyzing parameters from the picture header, picture parameter set (PPS), and slice header:

[0055] The reference image is a long-term reference image.

[0056] The reference image is a synthetic reference image.

[0057] The reference image is an image that is not output.

[0058] If the reference image is a non-output image, it is further determined whether the reference image is a composite reference image.

[0059] In the above paragraph, the technique of using the derivation of motion vector is mentioned, but when the motion vector is modified, if the motion search is performed on the specific reference picture, the search efficiency and the encoding / decoding efficiency are rather reduced.The specific reference picture is artificially constructed or is from a specific reference picture that is earlier in time order, and there is not necessarily a spatial relationship between the image blocks in the specific reference picture, and there is obvious shaking at the image block edge, so there is no great significance in searching the motion vector based on such a specific reference picture.

[0060] Pattern Matching Motion Vector Derivation (PMMVD) technology and Decode Motion Vector Refinement (DMVR) technology are both technologies that use motion vector derivation.

[0061] The above paragraph further refers to a technique using bidirectional motion prediction, which operates motion vectors based on the temporal correlation of images, and when the reference image to which the associated motion vector points is a specific reference image, these operations may be invalid because the definition of the temporal distance between the currently encoded image or the currently decoded image and the specific reference image is unclear. Bidirectional Optical flow (BIO) prediction technology is a technique using bidirectional motion prediction.

[0062] Based on PMMVD, DMVR and BIO, the method for image motion compensation of the present invention will be described below by taking an example. It should be mentioned that the method for image motion compensation of the present invention is not limited to these three techniques.

[0063] Before describing the method for image motion compensation according to the embodiment of the present application, a video encoding / decoding process in the HEVC standard will be briefly described.

[0064] The HEVC standard defines three interframe prediction modes: Inter mode, Merge mode, and Skip mode. The purpose of interframe prediction is to obtain a motion vector (MV), and then determine the position of the predicted image block in the reference image based on this motion vector. Adjacent image blocks have similarity in motion manner, for example, an image block (e.g., at least one of the image block to be coded and the image block to be decoded) and the adjacent image block are both part of the same object, and when the camera moves, their moving distances and directions are naturally similar or the same, so there is often no need to calculate a motion vector, and the motion vector of the adjacent image block is directly taken as the motion vector of the current image block. Here, in Merge mode and Skip mode, the residual of the motion vector (Motion Vector Difference, MVD) is 0, that is, the motion vector is directly obtained based on the adjacent coded image block or the decoded image block.

[0065] When at least one of the modes of the image block to be coded and the image block to be decoded is a merge mode, the operation principle is as follows: A motion vector prediction (MVP) candidate list is constructed by the neighboring image blocks, and the most suitable MVP is selected from the MVP candidate list as the motion vector of the current image block, and then the position of the predicted image block is determined based on the motion vector, and the residual is calculated after the predicted image block is determined. In the merge mode, the motion vector is selected from the MVP candidate list, so there is no MVD. The encoding end only needs to code the residual and the index of the selected motion vector in the MVP candidate list, and does not need to code the MVD. The decoding end can construct the MVP candidate list in a similar manner, and then obtain the motion vector based on the index transmitted by the encoding end. The decoding end determines the predicted image block based on the motion vector, and takes and decodes the residual to obtain the current image block.

[0066] The specific operation process of the encoding end in the merge mode is as follows: 1. Get a list of MVP candidates. 2. Select the best MVP from the MVP candidate list and obtain the index of this MVP in the MVP candidate list. 3. The selected MVP is taken as the motion vector of the current image block. 4. Based on the motion vector, a predicted image block is determined from the reference image. 5. Subtract the predicted image block from the current image block to get the residual. 6. Since the motion vector is selected from the MVP candidate list, there is no MVD, and it is only necessary to send the residual and the index of the selected MVP in the MVP candidate list to the decoding end.

[0067] The specific operation process of the decoding end in the merge mode is as follows: 1. Receive the residual and the index in the MVP candidate list of the motion vector. 2. Get a list of MVP candidates. 3. Based on the index, search for a motion vector in the MVP candidate list and take it as the motion vector of the current image block. 4. Based on the motion vector, a predicted image block is determined, and the residual is taken and decoded to obtain the current image block.

[0068] The above is the normal processing process in Merge mode.

[0069] Skip mode is a special case of Merge mode. After obtaining the motion vector by Merge mode, if the encoder determines that the current image block and the predicted image block are basically the same according to a certain method, it does not need to transmit the residual data, and only needs to transmit the index in the MVP candidate list of the motion vector, and the current image block can directly obtain the mark from the predicted image block.

[0070] In Inter mode, first determine the MVP, then modify the MVP to obtain the MVD. The encoding end not only needs to transmit the index and residual to the decoding end, but also needs to transmit the MVD to the decoding end. Advanced Motion Vector Prediction (AMVP) is a tool that realizes motion vector prediction by a competitive mechanism.

[0071] In the AMVP mode, there is also an MVP candidate list, and the motion vector in this MVP candidate list is obtained from the neighboring blocks in the spatial domain or the temporal domain of the current image block. The MVP candidate list in the AMVP mode may be different from the MVP candidate list in the Merge mode. The encoding end or the decoding end selects the most suitable MVP from the MVP candidate list. Using this MVP as the search starting point, it searches its vicinity to obtain the most suitable motion vector, which is the motion vector of the current image block. Based on this motion vector, the position of the predicted image block is determined, and after determining the predicted image block, the residual is calculated to obtain it. At the same time, the MV and MVP are subtracted from each other to obtain the MVD. The encoding end encodes the residual, the index of the MVP in the MVP candidate list, and the MVD, and transmits them to the decoding end. The decoding end can construct the MVP candidate list in a similar manner, and then obtains the MVP based on the index transmitted by the encoding end. The decoding end determines the MV according to the MVP and the MVD, determines a predicted image block according to the MV, and takes and decodes the residual to obtain a current image block.

[0072] The specific operation process of the encoding end in AMVP mode is as follows: 1. Get a list of MVP candidates. 2. Select the best MVP from the MVP candidate list and obtain the index of this MVP in the MVP candidate list. 3. Based on the MVP, determine the starting point for your search. 4. Search near the starting point to get the best motion vector. 5. Based on the motion vector, a predicted image block is determined from the reference image. 6. Subtract the predicted image block from the current image block to get the residual. 7. Subtract the MVP from the motion vector to get the MVD. 8. Send the residual, the index of the selected MVP in the MVP candidate list, and the MVD to the decoding end.

[0073] The specific operation process of the decoding end in AMVP mode will not be described further.

[0074] An embodiment of the present application provides a method 100 for image motion compensation. Figure 1 is a schematic flowchart of the method 100 for image motion compensation in one embodiment of the present application. As shown in Figure 1, the method 100 includes: S110 obtaining an initial motion vector for a current image block; S120, if the reference picture to which the initial motion vector points is a specific reference picture, perform motion compensation on the current image block according to the initial motion vector; and if the reference picture to which the initial motion vector points is a non-specific reference picture, modifying the initial motion vector to obtain a modified motion vector, and performing motion compensation on the current image block based on the modified motion vector (S130).

[0075] In some embodiments, when the initial motion vector points to a specific reference picture, motion compensation is performed directly; when the initial motion vector points to a non-specific reference picture, the initial motion vector is modified, and motion compensation is performed based on the modified motion vector, thereby avoiding meaningless searches due to obvious shaking of image block edges of a specific reference picture, and improving encoding / decoding efficiency.

[0076] In a possible embodiment, the method 100 for image motion compensation in the embodiment of the present application can be applied to PMMVD technology.

[0077] The PMMVD technique is a special merge mode based on the Frame Rate Up Conversion (FRUC) technique, in which the motion information (e.g., MV and MVD) of the current image block is not coded in the stream but is directly generated at the decoding end.

[0078] Optionally, in this possible embodiment, S130, when the reference picture to which the initial motion vector is directed is a non-specific reference picture, modifying the initial motion vector, obtaining a modified motion vector, and performing motion compensation for the current image block based on the modified motion vector may include obtaining a motion vector candidate list for the current image block, when a reference picture to which any motion vector candidate in the motion vector candidate list is directed is a non-specific reference picture, determining the initial motion vector based on the motion vector candidate list, modifying the initial motion vector and obtaining a modified motion vector, and performing motion compensation for the current image block based on the modified motion vector.

[0079] In this embodiment, an initial motion vector is determined from a motion vector candidate list. When the reference image to which the candidate vector in this motion vector candidate list points is a non-specific reference image, the initial motion vector is modified to obtain a modified motion vector, and motion compensation is performed on the current image block based on the modified motion vector, thereby avoiding meaningless searches due to obvious shaking of the image block edge of the specific reference image, and improving the encoding and decoding efficiency.

[0080] Optionally, in a possible embodiment, obtaining a motion vector candidate list for the current image block may include determining a motion vector candidate for adding to the motion vector candidate list, and adding the motion vector candidate to the motion vector candidate list if the reference image to which the motion vector candidate is directed is a non-specific reference image.

[0081] Specifically, in one embodiment, the method 100 may further include: determining, based on a frame mark of the reference image to which the motion vector candidate is directed, that the reference image to which the motion vector candidate is directed is a non-specific reference image. Specifically, based on a frame mark of the reference image to which the motion vector candidate is directed, it may be determined whether the reference image to which the motion vector candidate is directed is a specific reference image. If the reference image to which the motion vector candidate is directed is a specific reference image, the motion vector candidate corresponding to the specific reference image does not need to be added to a motion vector candidate list. In this way, when selecting an initial motion vector from the motion vector candidate list, the motion vector corresponding to the specific reference image is not selected.

[0082] It should be understood that the motion vector in each embodiment of the present application includes three parameters: the horizontal component v x , vertical component v y and a frame mark of the reference picture to which the frame mark is directed. For example, the frame mark may be a Picture Order Count (POC) or other types of marks. The encoding end and the decoding end can determine the attribute of the reference picture by the frame mark and judge whether the reference picture is a specific reference picture or a non-specific reference picture.

[0083] Optionally, in one embodiment, determining an initial motion vector based on the motion vector candidate list may include determining whether the selected initial motion vector is a motion vector pointing to a specific reference image, and if at least one of the selected initial motion vectors points to a specific reference image, reselecting an initial motion vector until the reference image to be pointed to selects an initial motion vector of a reference image other than the specific reference image.

[0084] Specifically, if the determined initial motion vector is directed to a specific reference image based on the motion vector candidate list, a motion vector candidate can be reselected from the motion vector candidate list based on a preset rule to be used as the initial motion vector. For example, the preset rule may be to select a motion vector candidate with the next smallest matching cost, and if the motion vector candidate is not directed to a specific reference image, use the motion vector candidate as the initial motion vector. This embodiment is merely illustrative and does not limit the present application.

[0085] Taking the encoding end as an example, the motion vector derivation process in FRUC merge mode is divided into two steps. The first step is a coding unit (CU) level motion search, and the second step is a sub-CU level motion refinement process. Similarly, the decoding end can also realize functions similar to those of the encoding end.

[0086] In the CU level motion search, a CU level motion vector candidate list is generated. Based on the bidirectional matching method, a motion vector with the smallest matching cost is searched from the CU level motion vector candidate list, for example, MV-A. Based on the template matching method, a motion vector with the smallest matching cost is also searched from the CU level motion vector candidate list, for example, MV-B. Then, based on the rate distortion cost (RD-Cost) policy used in the merge mode policy, it is determined whether the current CU uses the FRUC merge mode. That is, the results of the two matching methods (bidirectional matching method and template matching method) are checked using the RD-Cost policy. The result obtained by the matching method with the smallest matching cost is further compared with the results of the other CU modes. If the matching method with the smallest matching cost among the two matching methods is the one with the smallest final matching cost, the FRUC flag of the current CU is set to TRUE, and the decoding end is instructed to use the corresponding matching method.

[0087] In this specific embodiment, the CU level motion vector candidate list can correspond to the motion vector candidate list in the method 100 of the embodiment of the present application. When generating the motion vector candidate list, the motion vector to be added to the list is scanned, and if the motion vector is directed to a non-specific reference image, the motion vector can be added to the motion vector candidate list, and if the motion vector is directed to a specific reference image, the motion vector is rejected and not added to the motion vector candidate list. In this embodiment, when the current image block is in an advanced motion vector prediction AMVP mode, the motion vector candidate list can include at least one motion vector candidate among the original AMVP motion vector candidate obtained for the non-specific reference image, the merge motion vector candidate obtained for the non-specific reference image, the motion vector obtained by interpolation from the non-specific reference image, and the upper adjacent motion vector and the left adjacent motion vector for the non-specific reference image of the current block. This motion vector candidate list may be a CU level motion vector candidate list. It should be understood that in this embodiment, the non-specific reference image may specifically be a short-term reference image or a short-term reference frame.

[0088] Determining the initial motion vector based on the motion vector candidate list may include determining distortion costs corresponding to motion vector candidates in the motion vector candidate list based on at least one of a bidirectional matching method and a template matching method, and selecting a motion vector in the motion vector candidate list that has the smallest distortion cost as the initial motion vector.

[0089] Specifically, taking the encoding end as an example, determining the initial motion vector based on the motion vector candidate list can correspond to CU level motion search. In the CU level motion search, based on the bidirectional matching method, the motion vector with the smallest matching cost is searched from the CU level motion vector candidate list, for example, MV-A. In the template matching method, the motion vector with the smallest matching cost is also searched from the CU level motion vector candidate list, for example, MV-B. Then, based on the RD-Cost policy used in the Merge mode policy, it is determined whether the current CU uses the FRUC merge mode. That is, the RD-Cost policy is used to check the results of the two matching methods (bidirectional matching method and template matching method). The result obtained by the matching method with the smallest matching cost is further compared with the results of other CU modes. If the matching method with the smallest matching cost among the two matching methods is the one with the smallest final matching cost, the flag of the FRUC of the current CU is set to TRUE, and the decoding end is instructed to use the corresponding matching method. Similarly, the decoding end can also realize functions similar to the encoding end, but the decoding end is not as complex as the encoding end. For example, the decoding end directly receives a flag to indicate the matching method of the encoding end and does not need to implement the RD-Cost policy.

[0090] Optionally, modifying the initial motion vector and obtaining a modified motion vector may include generating a sub-motion vector candidate list for the current image block, the sub-motion vector candidate list including the initial motion vector, and determining a motion vector having the smallest distortion cost from the sub-motion vector candidate list as the modified motion vector.

[0091] Specifically, modifying an initial motion vector and obtaining a modified motion vector can correspond to sub-CU level motion refinement.

[0092] In the CU level motion search, the motion vector with the smallest matching cost is the initial motion vector, which is taken as the starting point of the CU level motion refinement. A local search is performed around the starting point based on the matching method (bidirectional matching or template matching method) determined by the CU level. Specifically, in the sub-CU level motion refinement, a sub-CU level motion vector candidate list can be generated. In the sub-CU level motion vector candidate list, a motion vector with the smallest matching cost is searched for and taken as the motion vector of the current CU.

[0093] The sub-CU level motion vector candidate list may include a motion vector determined from a CU-level motion search (an MV determined from a CU-level search), the top, left, top-left and top-right neighbouring motion vectors of the current image block, scaled versions of collocated MVs from reference pictures (scaled versions of collocated MVs from reference pictures), and motion vector candidates derived from other time domains (up to 4 ATMVP candidates and up to 4 STMVP candidates).

[0094] In the above-mentioned specific embodiment, when the reference picture to which the motion vector candidate points is a specific reference picture, the motion vector candidate corresponding to the specific reference picture is not added to the CU level motion vector candidate list, so that the motion vector candidate corresponding to the specific reference picture has no chance of becoming an initial motion vector.

[0095] In another specific embodiment, when the initial motion vector determined based on the CU level motion vector candidate list is directed to a specific reference image, a motion vector candidate can be reselected from the CU level motion vector candidate list according to a preset rule as the initial motion vector, for example, the preset rule can be to select the motion vector candidate with the next smallest matching cost, and if the motion vector candidate is not directed to the specific reference image, the motion vector candidate can be selected as the initial motion vector.

[0096] FIG. 2 is a principle schematic diagram of the bidirectional matching method in one embodiment of the present application. As shown in FIG. 2, the bidirectional matching method searches for the closest matching between two predicted image blocks in different reference images in the motion trajectory of the current CU, and derives the motion vector of the current CU. The bidirectional matching method is based on the assumption that the motion trajectory of the current image block is continuous. The motion vectors MV0 and MV1 of the two predicted image blocks are assumed to be proportional to the time domain distance (TD0 and TD1) between the current image and the two reference images (reference image 0 and reference image 1). The motion vector candidate list is scanned, and for MV0, for example, a motion vector pair MV0 and MV1 can be generated. Here, MV1 can be generated based on MV0, TD0, and TD1. If the distortion between the two predicted image blocks oriented by the motion vector pair corresponding to MV0 is the smallest, this motion vector (i.e., MV0) is the motion vector of the current CU. If TD0=TD1, the bidirectional matching is changed to bidirectional matching based on the mirror image.

[0097] It should be understood that if one of the two reference pictures is a specific reference picture, the definition of the time distance between the current picture (the picture currently being coded or the picture currently being decoded) and the specific reference picture is unclear, so bidirectional matching cannot be performed. Moreover, the specific reference picture is artificially constructed or from a specific reference picture that is earlier in time order, and there is not necessarily a spatial relationship between the image blocks in the specific reference picture, and there is obvious shaking at the image block edges, so bidirectional matching is not very meaningful to search for a motion vector based on such a specific reference picture. In a possible embodiment of the present application, the motion vector candidate corresponding to the specific reference picture is not added to the CU level motion vector candidate list, thereby avoiding the above-mentioned problem.

[0098] Fig. 3 is a principle schematic diagram of the template matching method of one embodiment of the present application. As shown in Fig. 3, the template matching method searches for the closest match between the template of the current image (at least one of the upper image block and the left image block of the current CU) and a block in the reference image (same size as the template), and derives the motion vector of the current CU. After obtaining the template, if the distortion between the template and the predicted image block to which the motion vector of the candidate is directed is the smallest, the motion vector of the candidate is the motion vector of the current CU.

[0099] When the reference image is a specific reference image, the specific reference image is artificially constructed or from a specific reference image that is earlier in time order, and the image blocks in the specific reference image do not necessarily have spatial correlation, and the image block edges have obvious shaking, so there is little significance in searching for a motion vector based on such a specific reference image, that is, the result of template matching method is inaccurate, and even meaningless. In a possible embodiment of the present application, the motion vector candidate corresponding to the specific reference image is not added to the CU level motion vector candidate list, thereby avoiding the above-mentioned problem.

[0100] In a specific embodiment, the method 100 for image motion compensation in the embodiment of the present application can be applied to DMVR technology.

[0101] The DMVR technique is a refinement technique used to make a more accurate prediction for a current image block when performing bidirectional prediction. The DMVR technique will be described in detail below by taking the decoding end as an example. The DMVR technique mainly includes two major steps: the first step is to construct a template based on a decoded image block corresponding to a number of initial motion vectors; the second step is to modify the number of initial motion vectors based on the template.

[0102] Specifically, the decoding end can generate a motion vector candidate list. For example, the motion vector candidate list can be the motion vector candidate list of the AMVP mode described above, or the motion vector candidate list of the Merge template. The decoding end can receive a number of indexes for indicating initial motion vectors sent from the encoding end. The decoding end obtains a number of initial motion vectors from the motion vector candidate list according to the indexes. The decoding end generates a template (for example, a method of weighted summing pixels) according to the decoded image blocks corresponding to the number of initial motion vectors, and modifies the number of initial motion vectors using the generated template, respectively. Finally, the current image block is motion compensated according to the modified motion vectors.

[0103] In this specific embodiment, the initial motion vector can include a first initial motion vector and a second initial motion vector. When a specific reference image exists in the initial motion vector, the following processing can be adopted. S120, when the reference image to which the current initial motion vector is directed is a specific reference image, performing motion compensation on the current image block based on the initial motion vector can include: when the reference image to which at least one of the first initial motion vector and the second initial motion vector is directed is a specific reference image, performing motion compensation on the current image block based on the first initial motion vector and the second initial motion vector; when the reference images to which the first initial motion vector and the second initial motion vector are directed are both non-specific reference images, modifying the first initial motion vector and the second initial motion vector to obtain a modified first motion vector and a modified second motion vector, and performing motion compensation on the current image block based on the modified first motion vector and the modified second motion vector. When there is no specific reference image in the initial motion vector, i.e., when both the initial motion vectors are non-specific reference images, a processing method of an existing DMVR can be adopted.

[0104] Fig. 4 is a principle schematic diagram of the DMVR technology of one embodiment of the present application. Specifically, assuming that the initial motion vector includes a first initial motion vector (for example, may be MV0) and a second initial motion vector (for example, may be MV1), and the decoded image block corresponding to the first initial motion vector belongs to the first decoded image block of the first frame, the first frame may be the first reference image, and the first decoded image block may be the first reference image block. Assuming that the decoded image block corresponding to the second motion vector belongs to the second decoded image block of the second frame, the second frame may be the second reference image, and the second decoded image block may be the second reference image block. The first reference image block and the second reference image block can be weighted and summed to obtain a template. Here, the template can be called a bidirectional template.

[0105] Optionally, in one embodiment, the initial motion vector includes a first initial motion vector and a second initial motion vector. Modifying the first initial motion vector and the second initial motion vector to obtain a modified first motion vector and a modified second motion vector can include generating a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial motion vector and belongs to a first reference image, and the second reference image block corresponds to the second initial motion vector and belongs to a second reference image, and modifying the first initial motion vector and the second initial motion vector based on the template to obtain a modified first motion vector and a modified second motion vector.

[0106] Specifically, modifying the first initial motion vector and the second initial motion vector based on the template to obtain the modified first motion vector and the modified second motion vector may include: using N third reference image blocks to match with the template, respectively, where the N third reference image blocks correspond to N third initial motion vectors and belong to the first reference image; using M fourth reference image blocks to match with the template, respectively, where the M fourth reference image blocks correspond to M fourth initial motion vectors and belong to the second reference image; selecting one third initial motion vector from the N third initial motion vectors, and selecting one fourth initial motion vector from the M fourth initial motion vectors based on the matching result, and using the one third initial motion vector and the one fourth initial motion vector as the motion vector of the current image block (i.e., the modified first motion vector and the modified second motion vector) or using them to determine the motion vector of the current image block.

[0107] Optionally, the selected third initial motion vector may be the motion vector corresponding to the smallest distortion cost, or the selected third initial motion vector may be the motion vector corresponding to a distortion cost less than a certain value.

[0108] Optionally, the selected fourth initial motion vector may be the motion vector corresponding to the smallest distortion cost, or the selected fourth initial motion vector may be the motion vector corresponding to a distortion cost less than a certain value.

[0109] Here, the one third initial motion vector and the one fourth initial motion vector are set as the motion vectors of the current image block, and in this case, a predicted image block can be obtained by weighted summing a known image block (i.e., an initial predicted image block) corresponding to the one third initial motion vector and the one fourth initial motion vector.

[0110] Or, the one third initial motion vector and the one fourth initial motion vector can be used to determine the motion vector of the current image block, that is, the one third initial motion vector and the one fourth initial motion vector can be MVP respectively. In this case, the third MVP can be used as a starting point to perform search optimization to obtain an optimized motion vector, and the fourth MVP can be used as a starting point to perform search optimization to obtain another optimized motion vector. A predicted image block can be obtained by performing weighted sum of the known image blocks (i.e., initial predicted image blocks) corresponding to the two optimized motion vectors.

[0111] Alternatively, N and M may be equal.

[0112] Optionally, the third initial motion vector includes the first initial motion vector, and the fourth initial motion vector includes the second initial motion vector, i.e., the reference image block corresponding to the first initial motion vector to generate a template and the reference image block corresponding to the second motion vector also need to be matched with the template, respectively.

[0113] Optionally, in an embodiment of the present application, at least some of the N third initial motion vectors are obtained by shifting based on the first initial motion vector, and at least some of the M fourth initial motion vectors are obtained by shifting based on the second initial motion vector.

[0114] For example, the initial motion vectors other than the first initial motion vector in the N third initial motion vectors are obtained by shifting based on the first initial motion vector, for example, N may be equal to 9, and eight of the initial motion vectors are obtained by shifting based on the first initial motion vector, for example, by shifting in eight directions, or by shifting different pixels in the vertical direction or horizontal direction.

[0115] Also, for example, the initial motion vectors other than the second initial motion vector among the N fourth initial motion vectors are obtained by shifting based on the second initial motion vector, for example, N may be equal to 9, and eight of the initial motion vectors are obtained by shifting based on the second initial motion vector, for example, by shifting in eight directions, or by shifting different pixels in the vertical or horizontal direction.

[0116] Optionally, in the embodiments of the present application, the first reference image may be a forward frame of the current image block and the second reference image may be a backward frame of the current image block, or the first reference image may be a forward frame of the current image block and the second reference image may be a forward frame of the current image block.

[0117] In a specific embodiment, the decoding end generates a motion vector candidate list and receives two indexes for indicating the initial motion vector sent from the encoding end. The decoding end judges the DMVP condition, which requires that neither of the two initial motion vectors (for example, MV0 and MV1) points to a specific reference picture, and the prediction directions of the two initial motion vectors are opposite, that is, one is forward and the other is backward. If the DMVR condition is satisfied, the image block corresponding to MV0 and the image block corresponding to MV1 are weighted-summed to generate a bidirectional template.

[0118] In a possible embodiment of the present application, the motion vector candidate corresponding to the specific reference picture is not added to the motion vector candidate list, thereby avoiding that the reference picture pointed to by the initial motion vector indicated by the two indexes is the specific reference picture. In another possible embodiment, if the reference picture pointed to by the initial motion vector indicated by the two indexes is the specific reference picture, the decoding end can request the encoding end to re-indicate the motion vector that does not point to the specific reference picture as the initial motion vector, or the decoding end returns or marks the expiration of the DMVR algorithm.

[0119] In the first reference image, for an image block corresponding to MV0, one luminance pixel is shifted in at least one of the horizontal direction and the vertical direction, and the motion vectors of the searched neighboring eight pixel blocks can form a reference list list0 together with a total of nine motion vectors of MV0. In the second reference image, for an image block corresponding to MV1, one luminance pixel is shifted in at least one of the horizontal direction and the vertical direction, and the motion vectors of the searched neighboring eight pixel blocks can form a reference list list1 together with a total of nine motion vectors of MV1.

[0120] The bidirectional template matching used by the decoding end performs distortion-based search between the bidirectional template and the reconstructed block in the reference image, so as to finally obtain a refined motion vector without additional motion information. For the motion vectors in the two reference images (the motion vectors in List0 and the motion vectors in list1), the motion vector with the smallest matching cost replaces the original motion vector as the updated motion vector. Finally, two new motion vectors (MV0' and MV1' shown in FIG. 3) replace the original MV0 and MV1. Based on the predicted image block corresponding to MV0' and the predicted image block corresponding to MV1', a final bidirectional prediction result is generated.

[0121] In DMVR technology, when the reference image is a specific reference image, the specific reference image is artificially constructed or from a specific reference image that is earlier in time order, and the image blocks in the specific reference image do not necessarily have spatial correlation, and the image block edges have obvious shaking, so it is not very meaningful to search for a motion vector based on such a specific reference image, that is, the DMVR technology is inaccurate and even meaningless.In the embodiment of the present application, the motion vector corresponding to the specific reference image is not refined and is directly used for motion compensation, thereby avoiding the above-mentioned problems.

[0122] An embodiment of the present application provides a method 200 for image processing. Figure 5 is a schematic flow chart of a method 200 for image processing in another embodiment of the present application. As shown in Figure 5, the method 200 includes: Obtaining a first initial motion vector and a second initial motion vector, the first initial motion vector pointing to a first reference image and the second initial motion vector pointing to a second reference image (S210); S220, if at least one of the first reference image and the second reference image is a specific reference image, calculating a predicted image block of the current image block according to the first initial motion vector and the second initial motion vector; If the first reference image and the second reference image are both non-specific reference images, the method includes calculating and obtaining a motion vector of a current image block based on gradient values ​​of pixel points to which the first initial motion vector and the second initial motion vector point, and calculating and obtaining a predicted image block of the current image block based on the motion vector of the current image block (S230).

[0123] In the image processing method of the embodiment of the present application, when the reference images pointed to by two initial motion vectors are both non-specific reference images, the motion vector of the current image block is calculated and obtained by using the gradient value of pixel point and the optimization principle, and the predicted image block of the current image block is calculated and obtained directly according to the two initial motion vectors when a specific reference image exists in the reference images pointed to by two initial motion vectors, so as to avoid prediction failure due to unclear definition of the time distance between the current image and the specific reference image, and to improve the efficiency of encoding and decoding.

[0124] Optionally, in S230, calculating and obtaining an MV of the current image block based on gradient values ​​of pixel points to which the first initial MV and the second initial MV are directed may include calculating and obtaining an MV of the current image block based on gradient values ​​of pixel points to which the first initial MV and the second initial MV are directed, and an optimization principle.

[0125] In a specific embodiment, the method 200 for image processing of the embodiment of the present application can be applied to the improvement of bidirectional motion prediction mode. Bidirectional motion prediction mode refers to the fact that, when encoding a current image block, two initial predicted image blocks from two different reference images are used simultaneously to predict the current image block, and the two initial predicted image blocks are combined into one image block as the predicted image block of the current image block. BIO prediction technology is a technology in bidirectional motion prediction mode. In BIO prediction technology, the motion vector has more coding marks than the normal bidirectional motion prediction mode, but the derivation process of the predicted image block is different. BIO prediction technology is a motion optimization based on block motion compensation, calculates the motion vector by optical flow mode, and is a motion optimization at the sampling point level.

[0126] The BIO prediction technique of the embodiment of the present application may include two steps. The first step is to calculate and obtain the MV of the current image block based on two initial MVs (for example, a first initial MV and a second initial MV). Specifically, the MV of the current image block is calculated and obtained based on the gradient value of the pixel point to which the first initial MV and the second initial MV are directed, and an optimization principle. The second step is to calculate and obtain the predicted image block of the current image block based on the MV of the current image block, the first reference image block to which the first initial MV is directed, and the second reference image block to which the second initial MV is directed. Specifically, the BIO prediction technique is to interpolate two reference image blocks to which two initial MVs are directed to obtain two initial predicted image blocks of the same size as the current image block, and then weighted-sum the two initial predicted image blocks and combine them as the predicted image block of the current image block.

[0127] FIG. 6 is a schematic diagram of the principle of the BIO technology according to one embodiment of the present application. The BIO technology assumes that the motion of an object is uniform in both the horizontal and vertical directions within a short period of time, and conforms to the analytical conditions of the optical flow method. As shown in FIG. 6, the motion vector (v x , v y ), and the brightness value I (k) can be assumed to conform to the optical flow formula: ∂I (k) / ∂t+v x ∂I (k) / ∂x+v y ∂I (k) / ∂y=0 Formula 1 In the formula, ∂I (k) / ∂x and ∂I (k) / ∂y denotes the horizontal and vertical components of the gradient.

[0128] The optical flow formula and Hermite are simultaneously interpolated to obtain the following polynomial, which is the BIO predicted value when t=0. pred BIO =1 / 2·(I(0) +I (1) +v x / 2·(τ1∂I (1) / ∂x-τ0∂I (0) / ∂x)+v y / 2·(τ1∂I (1) / ∂y-τ0∂I (0) / ∂y)) Formula 2 In the formula, τ0 and τ1 respectively indicate the distance from the current image to reference image 0 and reference image 1, and both can be calculated using the POC of the current image and the two reference images. τ0=POC(current)-POC(Ref0) Formula 3 τ1=POC(Ref1)-POC(current) Formula 4 where POC(current) is the picture order count of the current picture, POC(Ref0) is the picture order count of reference picture 0, and POC(Ref1) is the picture order count of reference picture 1.

[0129] The reference images can be of different orientations, i.e. one from the past and one from the future. The two reference images can also be of the same orientation, i.e. both from the past or both from the future. If the two reference images are of the same orientation, τ0 and τ1 have opposite signs. In such a situation, the two reference images cannot be the same, i.e. τ0 ≠ τ1, and the reference region must have a non-zero motion (MV x0 ,MV y0 ,MV x1 ,MV y1 ≠0), and the motion vector is proportional to the distance in the time domain (MV x0 / MV x1 =MV y0 / MV y1 =-τ0 / τ1).

[0130] Assuming that the motion within a small region is the same, the BIO motion vector can satisfy the first-order Taylor expansion formula as follows: Δ=(I (0) -I (1) +v x (τ1∂I(1) / ∂x+τ0∂I (0) / ∂x)+v y (τ1∂I (1) / ∂y+τ0∂I (0) / ∂y)) Formula 5 In the formula, Δ is the Taylor linear expansion of the pixel difference between two corresponding reference points (e.g., points A and B in FIG. 4) in two reference images. By analyzing the gradient value and the optimization method, it is determined that the optimal motion vector of the current image block satisfies the square of Δ in the entire region and is the minimum, so that the optimal motion vector (v x , v y ) can be calculated, and considering the robustness of the algorithm, v x and v y is assumed to be within a certain threshold range.

[0131] Based on the above formula, the process of BIO prediction technology is as follows: For the current image block, the pixel values ​​of the corresponding positions of the two reference images have been obtained. (0) and I (1) indicate the pixel values ​​in the two reference images, respectively, and the known values ​​in the above formula are I (0) , I (1) and τ0 and τ1, the horizontal and vertical components of the gradient are calculated in the reference image, and the unknowns are v x , v y For every pixel point in a region, we can calculate a Δ, and then use the optimization method to find the v x and v y i.e. the optimal motion vector required. Here, v x and v y When calculating v x and v yAn interval is given to the current image, and the threshold of this interval is determined according to the direction of two reference images to the current image. After obtaining the optimal motion vector, the image block + residual that this optimal motion vector points to is not directly taken as the current image block, but each pixel of the current image block is calculated according to formula 2, which is also called BIO prediction value, and the prediction value of each pixel is combined together to form a prediction image block.

[0132] Optionally, in an embodiment of the present application, calculating and obtaining a predicted image block of a current image block based on the first initial motion vector and the second initial motion vector includes weighted-summing a first reference image block pointed to by the first initial motion vector and a second reference image block pointed to by the second initial motion vector to obtain a predicted image block of the current image block.

[0133] Specifically, before starting to execute BIO prediction, a judgment condition is added, or a necessary condition is added to the BIO condition. If the first reference image pointed to by the first initial motion vector and the second reference image pointed to by the second initial motion vector are not specific reference images and meet the original BIO condition, BIO prediction can be performed. Otherwise, directly calculate or return the predicted image block of the current image block according to the two initial motion vectors, or mark the BIO prediction algorithm as invalid.

[0134] The BIO condition may further include: the prediction directions of the first initial motion vector and the second initial motion vector are different. Or alternatively, the BIO condition may further include: the prediction directions of the first initial motion vector and the second initial motion vector are the same, the first initial motion vector and the second initial motion vector are not zero, and the first reference image and the second reference image are different. At the same time, the ratios of the directional components of the first initial motion vector and the second initial motion vector are the same, and are both equal to the ratio of the distance between the first reference image and the current image to which the motion vector is directed, and the distance between the second reference image and the current image.

[0135] It will be understood that the motion vector in each embodiment of the present application includes three parameters: horizontal component v x , vertical component v y , and a frame mark of the reference image to which the frame mark is directed. For example, the frame mark may be a POC or other types of marks. The encoding end and the decoding end can determine the attribute of the reference image through the frame mark, and judge whether the reference image is a specific reference image.

[0136] Optionally, in an embodiment of the present application, determining whether the reference picture to which the initial motion vector points is a specific reference picture may include determining whether the reference picture to which the initial motion vector points is a specific reference picture based on a frame mark of the reference picture to which the initial motion vector points.

[0137] Optionally, in the embodiment of the present application, calculating and obtaining a predicted image block of the current image block according to the motion vector of the current image block in S230 may include calculating and obtaining a predicted image block of the current image block according to the motion vector of the current image block, a first reference image block to which the first initial motion vector points, and a second reference image block to which the second initial motion vector points. A specific calculation formula is described in detail in the preceding paragraph, and will not be described further here.

[0138] A specific embodiment of the method 200 may include the following steps.

[0139] 1. Determine whether the current two initial motion vectors meet the BIO conditions. The determination conditions are as follows:

[0140] a) The reference pictures to which the two initial motion vectors point are both non-specific reference pictures, i.e., the first reference picture to which the first initial motion vector points and the second reference picture to which the second initial motion vector points are both non-specific reference pictures.

[0141] b) One of the following two conditions is met: i) The two initial motion vectors are from different prediction directions (forward and backward, respectively). ii) Two initial motion vectors are from different reference pictures with the same prediction direction, and neither of the two initial motion vectors is zero, and the ratios of the directional components of the two initial motion vectors are the same, and both are equal to the ratio of the distance between the reference picture to which the motion vector points and the current picture.

[0142] An initial motion vector that simultaneously satisfies a) and b) complies with the BIO condition and is capable of performing BIO prediction.

[0143] 2. Determine a calculation threshold value according to the prediction directions of the two initial motion vectors, i.e., v x and v y is the threshold of the interval.

[0144] 3. Calculate the gradient values ​​of the pixel points to which the two initial motion vectors point.

[0145] 4. Based on the gradient value and the optimization principle, calculate the optimal motion vector, which is the motion vector of the current image block.

[0146] 5. Obtain a BIO prediction value according to the motion vector of the current image block and the reference image blocks pointed to by the two initial motion vectors.

[0147] b) if only i) is satisfied, a first reference image block pointed to by the first initial motion vector and a second reference image block pointed to by the second initial motion vector are weighted-summed to obtain a predicted image block of the current image block.

[0148] The principles, formulas and steps referred to in the BIO predictions in the embodiments of the present application may be referred to in the above descriptions and will not be further described here.

[0149] It should be understood that in BIO prediction technology, when reference image is specific reference image, the definition of the time distance between current image (current encoding target image or current decoding target image) and specific reference image is unclear, so BIO prediction technology cannot be implemented.In the embodiment of the present application, when specific reference image exists in the reference image that two initial motion vectors point to, directly calculate and obtain the predicted image block of current image block according to two initial motion vectors, thereby avoiding the above-mentioned problem.

[0150] It should be understood that the methods in the embodiments of the present application can be applied to both the encoding end and the decoding end. The current image block in the embodiments of the present application can be the image block to be encoded or the image block to be decoded.

[0151] It should be understood that the embodiments of the present application merely exemplarily present methods for image processing applied to PMMVD technology, DMVR technology, and BIO prediction technology, and that the methods of the embodiments of the present application may also be applied to other existing or future video encoding / decoding technologies, and that the embodiments of the present application are not limited thereto.

[0152] In the embodiment of the present application, it is mentioned that in some embodiments, in the process of determining the motion vector of an image block, the motion vector of the image block must be determined using the motion vector of an image block of another image. For ease of explanation, this image block is hereinafter referred to as a first image block, and the image block of another image that must be used is hereinafter referred to as the time-domain reference block of the first image block. It can be understood that the first image block and the time-domain reference block of the first image block are located in different images. Thus, in the process of determining the motion vector of the first image block using the motion vector of the time-domain reference block, the motion vector of the time-domain reference block must be scaled.

[0153] Typically, the scaling ratio of the motion vector of the time domain reference block is determined according to the time distance between the reference picture to which the motion vector of the time domain reference block points and the picture in which the time domain reference block is located, and the time distance between the reference picture of the first image block and the picture in which the first image block is located. Specifically, the scaling ratio of the motion vector of the time domain reference block is determined according to the time distance between the reference picture to which the motion vector of the time domain reference block points and the picture in which the time domain reference block is located, and the time distance between the reference picture of the first image block and the picture in which the first image block is located.

[0154] In the example, the motion vector of the time-domain reference block is MV2, and the reference frame index value of the reference image to which the motion vector MV2 points is x, where the reference frame index value x is the difference between the ordinal number of the reference image to which the MV2 points (e.g., POC) and the ordinal number of the image in which the time-domain reference block is located. The reference frame index value of the reference image of the first image block is y, where the reference frame index value y is the difference between the ordinal number of the reference image of the first image block and the ordinal number of the image in which the first image block is located. Then, the scaling ratio of the motion vector MV2 is y / x. Optionally, the product of the motion vector MV2 and y / x may be taken as the motion vector of the first image block.

[0155] However, when the motion vector MV2 of the time domain reference block points to a specific reference picture, or when the reference picture of the first image block is a specific reference picture, the definition of the temporal distance between the specific reference picture and the picture in which the first image block is located is unclear, so scaling the motion vector MV2 of the time domain reference block becomes meaningless.

[0156] An embodiment of the present application provides a video processing method 1100. FIG. 11 is a schematic flowchart of a video processing method 1100 according to another embodiment of the present application. As shown in FIG. 11, the method 1100 includes: S1101, determining a motion vector of a time-domain reference block of a first image block, where the time-domain reference block and the first image block are located in different images; and S1102, when the motion vector of the time domain reference block points to a specific reference picture or when the reference picture of the first image block is a specific reference picture, determining a motion vector of the first image block based on the motion vector of the time domain reference block after processing, wherein the motion vector of the time domain reference block after processing and the motion vector of the time domain reference block before processing are the same.

[0157] 12 is a schematic flowchart of a video processing method 1200 according to another embodiment of the present application. As shown in FIG. 12, the method 1200 includes: determining a motion vector of a time-domain reference block of the first image block, where the time-domain reference block and the first image block are located in different images; and S1202, when the motion vector of the time domain reference block points to a specific reference picture, or when the reference picture of the first image block is a specific reference picture, obtaining the motion vector of the first image block by abandoning reference to the motion vector of the time domain reference block.

[0158] 13 is a schematic flow chart of a video processing method 1200 according to another embodiment of the present application. As shown in FIG. 13, the method 1300 includes: S1301, determining a motion vector of a time-domain reference block of a first image block, where the time-domain reference block and the first image block are located in different images; and and S1302, when the motion vector of the time domain reference block points to a specific reference picture or when the reference picture of the first image block is a specific reference picture, setting the motion vector of the time domain reference block to the motion vector of the first image block.

[0159] In the example, when the motion vector MV2 of the time domain reference block points to a specific reference image, or when the reference image of the first image block is a specific reference image, the motion vector of the first image block is determined according to the motion vector of the time domain reference block after processing. The motion vector of the time domain reference block after processing described here can be said to be different from the operation of this motion vector under normal circumstances. Here, the operation of the motion vector under normal circumstances here refers to the operation of the motion vector under at least one of the following cases: when the motion vector MV2 of the time domain reference block does not point to a specific reference image, and when the reference image of the first image block is not a specific reference image.

[0160] In the example, if it is determined that the motion vector MV2 of the time-domain reference block points to a specific reference picture, or if it is determined that the reference picture of the first image block is a specific reference picture, the scaling ratio of the motion vector MV2 of the time-domain reference block is defaulted to 1. The motion vector of the time-domain reference block after the processing described herein may be the motion vector MV2 after being scaled by the default scaling ratio 1.

[0161] In the example, if it is determined that the motion vector MV2 of the time domain reference block points to a specific reference image, or if it is determined that the reference image of the first image block is a specific reference image, skip the scaling step for this motion vector MV2. The motion vector of the time domain reference block after the processing described herein may be the motion vector MV2 after skipping this scaling step.

[0162] In the example, if it is determined that the motion vector MV2 of the time domain reference block points to a specific reference picture, or if it is determined that the reference picture of the first image block is a specific reference picture, the motion vector of the first image block is obtained by abandoning reference to the motion vector of the time domain reference block.

[0163] In the example, the first image block may be a coding unit (CU), or may be a sub-image block in the coding unit, such as a sub-CU, which is not limited here.

[0164] Here, the method for determining the time-domain reference block of the first image block may refer to the method for determining the associated reference block of the current image block described below, which will not be described here.

[0165] Next, the embodiment of the present application will be described with specific examples based on specific scenes.

[0166] In some embodiments where the AMVP mode, Merge mode, etc. derives the motion vector, when constructing the motion information candidate list (e.g., MVP candidate list, reference frame index value candidate list, or other motion information candidate list) for the current image block, the candidate list is not only obtained based on at least one of the motion vectors of adjacent image blocks in the spatial domain and the reference frame index value of the current image block, but also obtained based on at least one of the motion vectors of image blocks adjacent to the current image block in the image in which the current image block is located and the reference frame index value of the current image block, and further obtained based on the motion vectors of adjacent image blocks in the temporal domain (corresponding to the temporal domain reference blocks described above). For example, the temporal domain reference block of the current image block may be an image block related to the position of the current image block in another image.

[0167] In some embodiments, the "image block related to the position of the current image block in another image" refers to a co-located block (hereinafter referred to as a co-located reference block) with which the position of the current image block is related, and a reference image including the co-located reference block is referred to as a co-located reference picture (hereinafter referred to as a co-located reference picture).

[0168] The related reference block of the current image block may be an image block having the same position as the current image block in the related reference image, or may be an image block having other relationship with the position of the current image block in the related reference image, and is not limited thereto.

[0169] There are several ways to identify the relevant reference blocks of a current image block, some of which are described here with examples.

[0170] In the example, a reference image of a current image block is first determined. For ease of explanation, this reference image is then referred to as a first reference image. This first reference image may be an image in the reference image set of the current image block that is closest to the image in which the current image block is located (i.e., the current image). Here, the closest distance described here may refer to the closest distance in coding order or the closest distance in display order. Here, the reference image set of the current image block may refer to the reference image set of the slice in which the current image block is located, or may refer to the reference image set of the frame in which the current image block is located.

[0171] Alternatively, the relevant reference block of the current image block may be the same image block as the position of the current image block in the first reference image, or may be an image block to which the motion vector of a specific adjacent image block of the current image block is directed, or may be an image block corresponding to a numerically cut-off position of the position to which the motion vector of a specific adjacent image block of the current image block is directed, or may be an image block to which the motion vector of the same image block as the position of the current image block is directed in the first reference image, or may be a specific adjacent image block of the same image block as the position of the current image block in the first reference image, or may be an image block to which the motion vector of a specific adjacent image block of the same image block as the position of the current image block is directed in the first reference image.

[0172] In the example, the first reference image may be a reference frame of a specific neighboring image block in the current image of the current image block. For example, the first reference image may be an image to which the motion vector of the specific neighboring image block in the current image of the current image block is directed. Alternatively, the relevant reference block of the current image block may be the image block to which the motion vector of the specific neighboring image block is directed. Or may be the same image block as the position of the current image block in the first reference image. Or may be the specific neighboring image block of the image block that is the same as the position of the current image block in the first reference image. Or may be the image block to which the motion vector of the specific neighboring image block of the image block that is the same as the position of the current image block in the first reference image is directed.

[0173] For example, the first reference image may be an image that is closest to the current image in the reference image set of a specific adjacent image block in the current image of the current image block. Alternatively, the related reference block of the current image block may be an image block that is the same as the current image block or the specific adjacent image block in the first reference image. Or it may be an image block to which the motion vector of the image block adjacent to the current image block is directed. Or it may be an image block that corresponds to a position that is a numerical cutoff of the position to which the motion vector of the image block adjacent to the current image block is directed. Or it may be an image block to which the motion vector of the image block to which the motion vector of the image block adjacent to the current image block is directed in the first reference image. Or it may be a specific adjacent image block of the image block to which the motion vector of the image block to which the position of the current image block is directed in the first reference image. Or it may be an image block to which the motion vector of the specific adjacent image block of the image block to which the position of the current image block is directed in the first reference image.

[0174] In the example, the first reference image may be a reference image defaulted by the encoding / decoding end, for example, a reference image defaulted according to the rule described in the above example, or the encoding end adds index information of the first reference image of the current image block in the stream, and the decoding end determines the associated reference image of the current image block according to the index information.

[0175] Here, the default reference picture may be determined by the encoding / decoding system, and may also be determined in at least one of a sequence header, a picture header, a video parameter set, a sequence parameter set (SPS), a picture parameter set (PPS), and a slice header.

[0176] Here, the specific neighboring image block in the current image of the current image block mentioned in each of the above examples may have a size equal to or different from that of the current image block. There are multiple ways to determine the specific neighboring image block. In the example, the method of determining the specific neighboring image block may be to obtain a first neighboring image block in the current image by a specific scanning order. In the example, the method of determining the specific neighboring image block may be to obtain a first neighboring image block having the same reference frame index value as the current image block by a specific scanning order in the current image. In the example, the method of determining the specific neighboring image block may be to obtain a first image block including motion information by a specific scanning order. In the example, the method of determining the specific neighboring image block may be to obtain a first neighboring image block having the same reference frame index value as the current image block by a specific scanning order.

[0177] Here, the specific scanning order may be a scanning order along "bottom to top, left to right" starting from the current image block, or a scanning order along "right to left, top to bottom" starting from the current image block, or other scanning orders. In the example, the image block obtained by scanning according to the specific scanning order may include continuous image blocks or may include a plurality of discrete image blocks. The specific scanning order is determined by the encoding / decoding terminal.

[0178] According to the motion vectors of the image blocks adjacent in the time domain, a candidate list is obtained, for example, according to the motion vector of the related reference block of the current image block, the motion information candidate list of the current image block is obtained, the motion vector of the related reference block is scaled, for example, according to the distance between the image to which the motion vector of the related reference block points and the image in which the related reference block is located, and the distance between the reference frame of the current image block and the current image, the motion vector of the related reference block after scaling is then added to the candidate list.

[0179] If the motion vector of this associated reference block points to a specific reference picture, there is no point in scaling said motion vector, since the definition of the time distance between the current picture and the specific reference picture is unclear.

[0180] In one embodiment, if the obtained motion vector of the related reference block points to a specific reference image, the motion vector of the related reference block is not scaled and the motion vector is directly added to the candidate list, or the scaling ratio of the motion vector of the related reference block is defaulted to 1, and then the scaled motion vector is added to the candidate list based on the default scaling ratio, or the motion information of the related reference block is directly taken as the motion information of the current image block.

[0181] In one embodiment, when the reference frame of the current image block is a specific reference image, the motion vector of the associated reference block is not scaled, and the motion vector is directly added to the candidate list, or the scaling ratio of the motion vector of the associated reference block is defaulted to 1, and then the scaled motion vector is added to the candidate list based on the default scaling ratio, or the motion information of the associated reference image block is directly taken as the motion information of the current image block.

[0182] In one embodiment, if the obtained motion vector of the relevant reference block points to a specific reference picture, the motion vector of the relevant reference block is not added to the candidate list.

[0183] In one embodiment, if the reference frame of the current image block is a particular reference image, the motion vector of this associated reference block is not added to the candidate list.

[0184] To improve the accuracy of the motion vector of the current image block, the current image block is optionally further divided into a number of sub-image blocks, which in some embodiments are referred to as dividing a current coding unit (CU) into a number of sub-CUs. For each sub-image block in the current image block, obtain an image block to which the position of this sub-image block is related. Then, obtain the motion vector of this sub-image block based on the motion vector of the reference block to which the position is related.

[0185] That is, the current image block may be a specific example of the first image block described above, or a sub-image block of the current image block may be a specific example of the first image block described above.

[0186] In one embodiment, in the situation of dividing the current image block into multiple sub image blocks, selectively construct one motion information candidate list for each sub image block (for example, MVP candidate list, reference frame index value candidate list, or other motion information candidate list). Specifically, when constructing the motion information candidate list for each sub image block, the candidate list can be obtained based on at least one of the motion vector of the image block adjacent to the current image block in the spatial domain and the reference frame index value of the current sub image block, for example, the motion information candidate list for the current sub image block is constructed by obtaining the candidate list based on at least one of the motion vector of the image block adjacent to the current image block in the image in which the current image block is located and the reference frame index value of the current sub image block. Here, the reference frame index value of each sub image block is the same as the reference frame index value of the current image block. Or the reference frame index value of each sub image block is determined based on the reference frame index value of the current image block.

[0187] The motion vector of the current image block is determined based on the motion information candidate list of each sub-image block of the current image block.

[0188] Take a current image block as an example, where the current image block is divided into four sub image blocks, each sub image block in the current image block has one motion information candidate list, and the current image block has a total of four motion information candidate lists, and the motion vector of the current image block is obtained according to the four motion information candidate lists.

[0189] In one embodiment, the above-described "obtaining a candidate list based on the motion vectors of adjacent image blocks in the time domain" includes obtaining a candidate list based on the motion vectors of adjacent image blocks in the time domain of at least some of the sub-image blocks in the current image block. In an example, the candidate list can be obtained based on the motion vectors of adjacent image blocks in the time domain of each sub-image block in the current image block, for example, adding the motion vectors of adjacent image blocks in the time domain of each sub-image block in the current image block to the candidate list. In an example, the candidate list can be obtained based on the motion vectors of adjacent image blocks in the time domain of some of the sub-image blocks in the current image block, for example, adding the motion vectors of adjacent image blocks in the time domain of some of the sub-image blocks in the current image block to the candidate list.

[0190] Here, there are several methods for determining the time-domain neighboring image blocks of at least some sub-image blocks of the current image block, for example, the above describes the method for determining the relevant reference blocks of the current image block.

[0191] In the example, the associated reference block of the current image block is divided into multiple sub-image blocks in the same manner as "dividing the current image block into multiple sub-image blocks", so that each sub-image block in the current image block corresponds one by one to each sub-image block in the associated reference block. The time-domain neighboring image blocks of each sub-image block in the current image block may be the sub-image blocks in the associated reference block of the current image block that correspond to each sub-image block, respectively.

[0192] In the example, similar to the methods of determining all "related reference images and related reference blocks of a current image block" described above, all current image blocks in each method are replaced with the current sub-image block, and this method can be adopted to determine the related reference images and related reference blocks of the current sub-image block.

[0193] After determining the associated reference blocks of at least some of the sub-image blocks in the current image block, when obtaining the candidate list based on the motion vectors of the associated reference blocks of the at least some of the sub-image blocks, the method relates to scaling the motion vectors of the associated reference blocks of the at least some of the sub-image blocks, for example, scaling the motion vectors of the associated reference blocks of the sub-image blocks based on the distance between the image to which the motion vector of the associated reference block points and the image in which the associated reference block is located and the distance between the reference frame of the current image block and the current image, and then adding the scaled motion vector to the candidate list.

[0194] In one embodiment, if the motion vector of the associated reference block of this sub-image block obtained points to a specific reference image, the motion vector is not scaled and is directly added to the candidate list, or the scaling ratio of this motion vector is defaulted to 1, and then the scaled motion vector is added to the candidate list based on the default scaling ratio, or the motion vector of the associated reference block of this sub-image block is directly taken as the motion vector of this sub-image block.

[0195] In one embodiment, when the reference frame of the current image block is a specific reference image, the motion vector of the associated reference block is not scaled, and the motion vector is directly added to the candidate list, or the scaling ratio of the motion vector of the associated reference block is defaulted to 1, and then the scaled motion vector is added to the candidate list based on the default scaling ratio, or the motion vector of the associated reference block of the sub-image block is directly taken as the motion vector of the sub-image block.

[0196] In one embodiment, if the obtained motion information of the associated reference block of this sub-image block points to a particular reference image, then this motion vector is not added to the candidate list.

[0197] In one embodiment, if the reference frame of the current image block is a particular reference image, the motion vector of this associated reference block is not added to the candidate list.

[0198] Next, a specific example will be given for explanation.

[0199] As shown in FIG. 14, the current image block A is divided into four sub-image blocks of the same size. A related reference block Col A (co-located A) is obtained in the related reference image of the current image block A. The related reference image and the related reference image are obtained by the above-mentioned method. Specifically, in the example, the related reference block Col A is divided into four sub-image blocks of the same size as the four sub-image blocks of the current image block A. The four sub-image blocks in the related reference block Col A and the four sub-image blocks in the current image block A correspond to each other. Based on the motion vector of each sub-image block of the related reference block Col A, the motion vector of the corresponding sub-image block in the current image block A is obtained.

[0200] After the current image block obtains the motion vector of the position-related sub-image block from the related reference block in units of sub-image blocks, the motion vector of the sub-image block of the current image block is calculated as follows.

[0201] The motion vector of a sub-image block whose position in the relevant reference image block is related to the MV ColSubblock , and the reference frame index value of this sub-image block is Ref ColSubblock This reference frame index value Ref ColSubblock is to indicate the distance between the reference frame of the sub-image block whose position in this associated reference block is associated with and the image in which this associated reference block resides.

[0202] The reference frame index value of the current image block is RefCurrSubblock This reference frame index value Ref CurrSubblock is intended to indicate the distance between the reference frame of the current reference block and the image in which this current reference block resides.

[0203] Based on the object's motion mode, Ref ColSubblock and Ref CurrSubblock If the value is not equal to MV ColSubblock must be scaled to fit the assumption of the object motion mode. The motion vectors are scaled to obtain the motion vectors MV CurrSubblock The method for obtaining it is as follows: Music Video CurrSubblock =(Ref CurrSubblock / Ref ColSubblock )×MV ColSubblock

[0204] Specifically, considering that floating-point numbers and decimals are not usually used in video encoding / decoding systems, the above motion vector scaling formula merely reflects the basic principle of motion vector scaling, and when specifically implemented, it is necessary to convert it into an integer for calculation.

[0205] Reference frame index value Ref ColSubblock If points to a specific reference image, the motion vector MV ColSubblock If no scaling operation is performed on the current image block and the reference frame index of the current image block points to a specific reference image, the motion vector M ColSubblock No scaling operations are performed on the

[0206] In the example, the video processing device includes: at least one memory for storing computer executable instructions; and at least one processor adapted, either alone or in combination, to access said at least one memory and execute said computer-executable commands to perform the following operations: determining a motion vector of a time-domain reference block of a first image block, where the time-domain reference block and the first image block are located in different images; if the motion vector of the time domain reference block points to a specific reference image or the reference image of the first image block is a specific reference image, determining a motion vector of the first image block according to the motion vector of the time domain reference block after processing, and the motion vector of the time domain reference block after processing and the motion vector of the time domain reference block before processing are identical; or Obtaining the motion vector of the first image block by abandoning reference to the motion vector of the time domain reference block when the motion vector of the time domain reference block points to a specific reference picture or when the reference picture of the first image block is a specific reference picture; or If the motion vector of the time domain reference block points to a specific reference picture, or if the reference picture of the first image block is a specific reference picture, the motion vector of the time domain reference block is the motion vector of the first image block.

[0207] Optionally, the motion vector of the time domain reference block after said processing is a motion vector obtained after scaling the motion vector of said time-domain reference block according to a scaling ratio equal to 1; or It includes the motion vector of the time-domain reference block with the scaling step skipped.

[0208] Optionally, the processor further comprises: When the motion vector of the time domain reference block points to a non-specific reference picture and the reference picture of the first image block is a non-specific reference picture, the motion vector of the first image block is determined based on the time distance between the reference picture to which the motion vector of the time domain reference block points and the picture in which the time domain reference block is located, and the time distance between the reference picture of the first image block and the picture in which the first image block is located.

[0209] Optionally, the processor: determining a scaling ratio of the motion vector of the time domain reference block according to a time distance between a reference image to which the motion vector of the time domain reference block points and an image in which the time domain reference block is located, and a time distance between a reference image of the first image block and an image in which the first image block is located; and determining the motion vector of the first image block based on the motion vector of the time-domain reference block after scaling by the scaling ratio.

[0210] Optionally, the processor further comprises: constructing a motion information candidate list for the first image block, the motion information candidate list including at least one of a motion vector of a spatial domain reference block of the first image block and a reference frame index value of the first image block, where the spatial domain reference block and the first image block are located in the same image; and determining a motion vector for the first image block based on the motion information candidate list for the first image block.

[0211] Optionally, determining the motion vector of the first image block based on the motion vector of the processed time-domain reference block comprises: adding the motion vector of the processed time domain reference block to the motion vector candidate list; Or, obtaining the motion vector of the first image block by abandoning reference to the motion vector of the time domain reference block includes: abandoning adding the motion vector of the time-domain reference block to the motion vector candidate list.

[0212] Optionally, the first image block is a coding unit CU.

[0213] Optionally, the first image block is a portion of a second image block, and the processor further comprises: constructing a motion information candidate list for the first image block, the motion information candidate list including at least one of a motion vector of a spatial domain reference block of the second image block and a reference frame index value of the first image block, where the spatial domain reference block and the second image block are located in the same image; and determining a motion vector for the second image block based on the motion information candidate list for the first image block.

[0214] Optionally, the reference frame index value of the first image block and the reference frame index value of the second image block are the same; or A reference frame index value of the first image block is determined based on a reference frame index value of the second image block.

[0215] Optionally, determining the motion vector of the first image block based on the motion vector of the processed time-domain reference block comprises: adding the motion vector of the processed time-domain reference block to a motion information candidate list of the second image block; Or, obtaining the motion vector of the first image block by abandoning reference to the motion vector of the time domain reference block includes: The method includes abandoning adding the motion vector of the time-domain reference block to the motion information candidate list of the second image block.

[0216] Optionally, the second image block is a coding unit CU, the first image block is a sub-image block in the coding unit, and the sub-image block is obtained by splitting the coding unit.

[0217] Optionally, determining a motion vector of a time-domain reference block for the first image block comprises: determining an image in a reference image set for the first image block that is closest to the image in which the first image block is located; determining a time-domain reference block of the first image block and a motion vector of the time-domain reference block based on the determined image.

[0218] Optionally, the time-domain reference block is an image block located in the determined image at the same position as the first image block; or The time-domain reference block is an image block to which a motion vector of an image block adjacent to the first image block points in the determined image; or The time-domain reference block is an image block corresponding to a numerical cut-off position of a position to which a motion vector of an image block adjacent to the first image block points in the determined image; or The time-domain reference block is an image block to which a motion vector of the same image block as the position of the first image block in the determined image points, or The time-domain reference block is a specific neighboring image block of the same image block as the location of the first image block in the determined image; or The time-domain reference block is an image block to which a motion vector of a specific adjacent image block of the same image block as the position of the first image block in the determined image points.

[0219] Optionally, determining a motion vector of a time-domain reference block for the first image block comprises: determining a particular neighboring image block of the first image block in an image in which the first image block resides; determining a time-domain reference block of the first image block and a motion vector of the time-domain reference block according to a reference image of the specific neighboring image block.

[0220] Optionally, the time-domain reference block is an image block that is located at the same position as the first image block in the reference image of the specific neighboring image block; or The time domain reference block is an image block in the reference image of the specific neighboring image block that is located at the same position as the specific neighboring image block; or The time-domain reference block is the image block to which the motion vector of the particular neighboring image block points.

[0221] Optionally, determining the particular neighboring image block of the first image block includes: In an image in which the first image block is located, obtaining a first adjacent image block according to a specific scanning order; or Obtaining a first neighboring image block in an image in which the first image block is located, the first neighboring image block having the same reference frame index value as the first image block according to a specific scanning order; or Obtaining a first image block having motion information according to a particular scanning order; or Obtaining a first adjacent image block that is the same as a default reference image of the first image according to a specific scanning order.

[0222] Optionally, the particular scanning order is a scanning order starting from the first block along one of the following directions: Bottom to top, left to right, right to left, top to bottom.

[0223] Optionally, the specific reference image includes at least one of a long-term reference image, a synthesis frame, and a non-output frame.

[0224] Optionally, the processor further comprises: To determine that the reference image is a frame that is not to be output, and to determine that the reference image is the specific reference image when the reference image is a composite frame.

[0225] Optionally, the non-specific reference image includes a short-term reference image.

[0226] 7 is a schematic frame diagram of an apparatus 70 for image motion compensation in one embodiment of the present application. As shown in FIG. 7, the apparatus 70 for image motion compensation includes: at least one memory 701 for storing computer executable commands; and at least one processor 702 adapted to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining an initial motion vector MV of a current image block; If the reference image indicated by the initial MV is a specific reference image, performing motion compensation on the current image block according to the initial MV; If the reference picture to which the initial MV is directed is a non-specific reference picture, modify the initial MV to obtain a modified MV, and perform motion compensation on the current image block based on the modified MV.

[0227] In some embodiments, the processor 702 specifically: Obtain an MV candidate list for the current image block, and a reference image to which any candidate MV in the MV candidate list points is a non-specific reference image; determining the initial MV based on the MV candidate list; The initial MV is modified to obtain a modified MV, and performing motion compensation on the current image block based on the modified MV.

[0228] In some embodiments, the processor 702 further comprises: determining a candidate MV for adding to the MV candidate list, and adding the candidate MV to the MV candidate list if a reference picture to which the candidate MV points is a non-specific reference picture;

[0229] In some embodiments, the processor 702 further comprises: According to the frame mark of the reference picture to which the candidate MV is directed, it is determined that the reference picture to which the candidate MV is directed is a non-specific reference picture.

[0230] In some embodiments, the MV candidate list includes at least one of the following candidate MVs: If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for a non-specific reference image; The merge candidate MV obtained for the non-specific reference image, MVs obtained by interpolation from non-specific reference images, and The upper neighboring MV and the left neighboring MV for the non-specific reference image of the current block.

[0231] In some embodiments, the processor 702 specifically: determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method; The MV with the smallest distortion cost in the MV candidate list is set as the initial MV.

[0232] In some embodiments, the processor 702 specifically: Generate a sub-MV candidate list for the current image block, the sub-MV candidate list including the initial MV; The purpose is to determine the MV with the smallest distortion cost from the sub-MV candidate list as the modified MV.

[0233] In some embodiments, the initial MV includes a first initial MV and a second initial MV; Specifically, the processor 702: performing motion compensation on the current image block based on the first initial MV and the second initial MV when a reference image indicated by at least one of the first initial MV and the second initial MV is a specific reference image; When the reference images to which the first initial MV and the second initial MV are directed are non-specific reference images, the first initial MV and the second initial MV are modified to obtain a modified first MV and a modified second MV, and motion compensation is performed on the current image block based on the modified first MV and the modified second MV.

[0234] In some embodiments, the processor 702 specifically: Generate a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial MV and belongs to a first reference image, and the second reference image block corresponds to the second initial MV and belongs to a second reference image; The first initial MV and the second initial MV are modified based on the template to obtain a modified first MV and a modified second MV.

[0235] In some embodiments, the processor 702 specifically: Using N third reference image blocks to match the template respectively, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; Using M fourth reference image blocks to match the template respectively, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; Based on the matching result, a third initial MV is selected from the N third initial MVs, and a fourth initial MV is selected from the M fourth initial MVs, and the third initial MV and the fourth initial MV are used as the MV of the current image block, or for determining the MV of the current image block.

[0236] In some embodiments, the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV.

[0237] In some embodiments, at least some of the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MV.

[0238] In some embodiments, N is equal to M.

[0239] In some embodiments, the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block; or The first reference image is a forward frame of the current image block, and the second reference image is a forward frame of the current image block.

[0240] It should be understood that the apparatus 70 for image motion compensation can further be realized by corresponding software modules, which will not be described further herein.

[0241] 8 is a schematic frame diagram of an apparatus 80 for image processing according to another embodiment of the present application. As shown in FIG. 8, the apparatus 80 for image processing includes: At least one memory 801 for storing computer executable commands; and at least one processor 802 adapted to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining a first initial motion vector MV and a second initial MV, the first initial MV pointing to a first reference image and the second initial MV pointing to a second reference image; When at least one of the first reference image and the second reference image is a specific reference image, calculating and obtaining a predicted image block of a current image block according to the first initial MV and the second initial MV; When the first reference image and the second reference image are both non-specific reference images, the MV of the current image block is calculated based on the gradient values ​​of the pixel points to which the first initial MV and the second initial MV point, and a predicted image block of the current image block is calculated based on the MV of the current image block.

[0242] In some embodiments, the processor 802 specifically: Calculate and obtain the MV of the current image block according to the gradient value of the pixel point to which the first initial MV and the second initial MV are directed, and an optimization principle.

[0243] In some embodiments, the processor 802 specifically: A first reference image block pointed to by the first initial MV and a second reference image block pointed to by the second initial MV are weighted-summed to obtain a predicted image block of the current image block.

[0244] In some embodiments, the prediction directions of the first initial MV and the second initial MV are different.

[0245] In some embodiments, the prediction direction of the first initial MV and the second initial MV is the same, neither of the first initial MV and the second initial MV is 0, and the first reference image and the second reference image are different.

[0246] In some embodiments, the processor 802 specifically: It is for calculating and obtaining a predicted image block of the current image block based on the MV of the current image block, the first reference image block to which the first initial MV points, and the second reference image block to which the second initial MV points.

[0247] It is to be understood that the device 80 for image processing can further be realized by corresponding software modules, which will not be described further here.

[0248] In some techniques using derivation of motion vector, when the motion vector is directed to a specific reference image, scaling for this motion vector is meaningless, which reduces search efficiency and encoding / decoding efficiency. This is because, when scaling a motion vector, the motion vector is scaled based on the distance between images, while the specific reference image may be artificially constructed and does not have a distance from other reference images, so scaling the motion vector based on such a specific reference image is not very meaningful.

[0249] The present application further provides a method for image motion compensation. Figure 9 is a schematic flowchart of a method 900 for image motion compensation according to another embodiment of the present application. As shown in Figure 9, the method 900 includes: S910, obtaining an initial motion vector MV of a current image block; S920, determining a scaling ratio of the initial MV, where if the initial MV is directed to a specific reference image, the scaling ratio of the initial MV is 1; S930, scaling the initial MV based on a scaling ratio of the initial MV; S940, performing motion compensation on the current image block based on the scaled MV.

[0250] Here, the method of obtaining the initial MV may be the same as the method of obtaining the initial MV described above, and will not be described further here.

[0251] In the method for image motion compensation in the embodiment of the present application, when the initial MV points to a specific reference image, its scaling ratio is 1. When the initial MV points to a non-specific reference image, its scaling ratio is not limited, and the motion vector is scaled according to the distance between the current image and the non-specific reference image and the scaling policy, thus performing motion compensation, and improving the efficiency of encoding and decoding.

[0252] In some embodiments, motion compensation is performed directly on the current image block based on the scaled MV.

[0253] In some embodiments, for the scaled MV, the scaled MV can be further modified to obtain a modified MV, and perform motion compensation on the current image block according to the modified MV, where the method of modifying the scaled MV can be the same as the method of modifying the initial MV described above, and will not be further described here.

[0254] For example, as shown in FIG. 5, the first initial MV points to a first reference image, and the second initial MV points to a second reference image. When the first reference image and the second reference image both point to a non-specific reference image, the first initial MV and the second initial MV are scaled according to the distance between the first reference image and the second reference image. Then, the current image block is motion compensated according to the first initial MV after scaling and the second initial MV after scaling. In one embodiment, a first reference image block is determined, the first reference image block corresponds to the first initial MV after scaling and belongs to the second reference image, and a second reference image block is determined, the second reference image block corresponds to the second initial MV after scaling and belongs to the first reference image. A template is generated according to the first reference image block and the second reference image block, and the first initial MV and the second initial MV are modified according to the template to obtain the modified first MV and the modified second MV. The modified first MV and the modified second MV are used to calculate the motion vector of the current image block.

[0255] Here, the method of modifying the first initial MV and the second initial MV based on this template may be the same as the method of modifying the first initial MV and the second initial MV based on the template described above, and will not be described further here.

[0256] This application further provides another method for image motion compensation, in which an initial motion vector MV of a current image block is obtained, and when the initial MV points to a specific reference image and when the initial MV points to a non-specific reference image, different operations are adopted to perform motion compensation on the current image block according to the initial MV.

[0257] Here, the method of obtaining the initial MV may be the same as the method of obtaining the initial MV described above, and will not be described further here.

[0258] In one embodiment, when the initial MV points to a specific reference image, the initial MV is used to perform motion compensation on the current image block. When the initial MV points to a non-specific reference image, there are four embodiments: 1, scale the initial MV, and perform motion compensation on the current image block based on the scaled initial MV; 2, modify the initial MV, and perform motion compensation on the current image block based on the modified initial MV; 3, scale the initial MV, modify the scaled MV, obtain a modified MV, and perform motion compensation on the current image block based on the modified MV; 4, modify the initial MV, scale the modified MV, obtain a scaled MV, and perform motion compensation on the current image block based on the scaled MV.

[0259] Here, the method of modifying the scaled MV or the method of modifying the initial MV may be the same as the method of modifying the initial MV described above, and will not be described further here.

[0260] In one embodiment, when the initial MV points to a specific reference image, the initial MV can be modified, and motion compensation can be performed based on the modified MV. When the initial MV points to a non-specific reference image, the initial MV can be scaled, or the initial MV can be scaled and modified, and then motion compensation can be performed on the current image block. That is, when the initial MV points to a specific reference image, the scaling step is skipped and the initial MV is directly modified, or the initial MV is directly scaled and modified after the scaling ratio is set to 1. When the initial MV points to a non-specific reference image, the initial MV can be scaled, or the initial MV can be scaled and modified, to improve the efficiency of encoding and decoding.

[0261] In other embodiments of the present application, when the initial MV is directed to a specific reference image, the initial MV is directly used for motion compensation, or the initial MV is modified, and motion compensation is performed based on the modified MV; and when the initial MV is directed to a non-specific reference image, the initial MV is scaled and / or modified for motion compensation; various embodiments may be arbitrarily combined, and the embodiments of the present application are not limited thereto.

[0262] In some embodiments, obtaining an initial motion vector MV of the current image block includes obtaining an MV candidate list of the current image block, and determining the initial MV according to the MV candidate list. That is, after scaling the initial MV (including scaling with a scaling ratio of 1), an MV that points to a non-specific reference picture is selected and modified, and regardless of which type of reference picture the initial MV points to, any of them may be modified.

[0263] Optionally, the reference picture to which any candidate MV in the MV candidate list points is a non-specific reference picture.

[0264] Optionally, obtaining an MV candidate list for the current image block may include determining a candidate MV for adding to the MV candidate list, and adding the candidate MV to the MV candidate list if the reference image to which the candidate MV points is a non-specific reference image.

[0265] Optionally, the method may further include determining, based on a frame mark of the reference picture to which the candidate MV is pointed, that the reference picture to which the candidate MV is pointed is a non-specific reference picture.

[0266] Optionally, the MV candidate list may include at least one of the following candidate MVs: an original AMVP candidate MV obtained for a non-specific reference picture when the current image block is in advanced motion vector prediction AMVP mode, a merge candidate MV obtained for a non-specific reference picture, an MV obtained by interpolation from a non-specific reference picture, and an upper adjacent MV and a left adjacent MV for the non-specific reference picture of the current block.

[0267] Optionally, determining the initial MV based on the MV candidate list may include determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a two-way matching method and a template matching method, and selecting the MV in the MV candidate list with the smallest distortion cost as the initial MV.

[0268] Optionally, modifying the scaled MV and obtaining a modified MV may include generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the scaled MV, and determining an MV with the smallest distortion cost from the sub-MV candidate list as the modified MV.

[0269] Optionally, the initial MV includes a first initial MV and a second initial MV, and the method includes at least one of: when a reference image to which at least one of the first initial MV and the second initial MV is directed is a specific reference image, setting scaling ratios of the first initial MV and the second initial MV to 1, and performing motion compensation on the current image block based on the first initial MV and the second initial MV; and when a reference image to which the first initial MV and the second initial MV are directed is a non-specific reference image, determining a scaling ratio of the first initial MV and a scaling ratio of the second initial MV, scaling the first initial MV based on the scaling ratio of the first initial MV and scaling the second initial MV based on the scaling ratio of the second initial MV, and performing motion compensation on the current image block based on the first initial MV after the scaling and the second initial MV after the scaling.

[0270] Optionally, performing motion compensation on the current image block based on the first initial MV after scaling and the second initial MV after scaling includes generating a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial MV after scaling and belongs to a second reference image, the second reference image block corresponds to the second initial MV after scaling and belongs to the first reference image, the first reference image is a reference image to which the first initial MV is directed, and the second reference image is a reference image to which the second initial MV is directed; and modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV.

[0271] Optionally, modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV includes: using N third reference image blocks to match the template, respectively, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; using M fourth reference image blocks to match the template, respectively, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; and selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs based on the matching result, and using the one third initial MV and the one fourth initial MV as the MV of the current image block or to determine the MV of the current image block.

[0272] Optionally, the third initial MV may include the first initial MV, and the fourth initial MV may include the second initial MV.

[0273] Optionally, at least some of the N third initial MVs may be obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs may be obtained by shifting based on the second initial MV.

[0274] Optionally, said N may be equal to said M.

[0275] Optionally, the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block, or the first reference image is a forward frame of the current image block and the second reference image is a forward frame of the current image block.

[0276] It is understood that the above alternative embodiments can be implemented similarly to the details of method 100, and will not be described further herein.

[0277] 10 is a schematic frame diagram of an apparatus 1000 for image motion compensation in one embodiment of the present application. As shown in FIG. 10, the apparatus 1000 for image motion compensation includes: at least one memory 1001 for storing computer executable commands; and at least one processor 1002 adapted to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining an initial motion vector MV of a current image block; Determining a scaling ratio of the initial MV, where when the initial MV is directed to a specific reference image, the scaling 1 ratio of the initial MV is 1; Scaling the initial MV based on a scaling ratio of the initial MV; performing motion compensation on the current image block based on the scaled MV.

[0278] In some embodiments, the processor 1002 performing motion compensation on the current image block based on the scaled MV comprises: modifying the scaled MV to obtain a modified MV; and performing motion compensation on the current image block based on the modified MV.

[0279] In some embodiments, the processor 1002 obtaining the initial motion vector MV of the current image block comprises: Obtaining a MV candidate list for the current image block; determining the initial MV based on the MV candidate list.

[0280] In some embodiments, the reference picture to which any candidate MV in the MV candidate list points is a non-specific reference picture.

[0281] In some embodiments, the processor 1002 obtaining the MV candidate list for the current image block comprises: determining a candidate MV for adding to the MV candidate list; and adding the candidate MV to the MV candidate list if the reference picture to which the candidate MV points is a non-specific reference picture.

[0282] In some embodiments, the processor 1002 further comprises: According to the frame mark of the reference picture to which the candidate MV is directed, it is determined that the reference picture to which the candidate MV is directed is a non-specific reference picture.

[0283] In some embodiments, the MV candidate list includes at least one of the following candidate MVs: If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for a non-specific reference image; The merge candidate MV obtained for the non-specific reference image, MVs obtained by interpolation from non-specific reference images, and The upper neighboring MV and the left neighboring MV for the non-specific reference image of the current block.

[0284] In some embodiments, the processor 1002 determining the initial MV based on the MV candidate list includes: determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method; and selecting the MV in the MV candidate list with the smallest distortion cost as the initial MV.

[0285] In some embodiments, the processor 1002 modifying the scaled MV to obtain a modified MV includes: generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the scaled MV; determining a MV with the smallest distortion cost from the sub-MV candidate list as the modified MV.

[0286] In some embodiments, the initial MV includes a first initial MV and a second initial MV, and the processor 1002 specifically: When a reference image to which at least one of the first initial MV and the second initial MV is directed is a specific reference image, setting the scaling ratios of the first initial MV and the second initial MV to 1, and performing motion compensation on the current image block based on the first initial MV and the second initial MV; When the reference images to which the first initial MV and the second initial MV are directed are both non-specific reference images, the method is for at least one of determining a scaling ratio of the first initial MV and a scaling ratio of the second initial MV, scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and performing motion compensation on the current image block based on the first initial MV after the scaling and the second initial MV after the scaling.

[0287] In some embodiments, the processor 1002 performing motion compensation on the current image block based on the first initial MV after scaling and the second initial MV after scaling includes: Generate a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to a first initial MV after the scaling and belongs to a second reference image, the second reference image block corresponds to a second initial MV after the scaling and belongs to the first reference image, the first reference image is a reference image to which the first initial MV is directed, and the second reference image is a reference image to which the second initial MV is directed; and modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV.

[0288] In some embodiments, the processor 1002 modifies the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV, Using N third reference image blocks to match the template respectively, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; Using M fourth reference image blocks to match the template respectively, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; Based on the matching result, selecting one third initial MV from the N third initial MVs and selecting one fourth initial MV from the M fourth initial MVs, and using the one third initial MV and the one fourth initial MV as the MV of the current image block or to determine the MV of the current image block.

[0289] In some embodiments, the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV.

[0290] In some embodiments, at least some of the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MV.

[0291] In some embodiments, N is equal to M.

[0292] In some embodiments, the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block; or The first reference image is a forward frame of the current image block, and the second reference image is a forward frame of the current image block.

[0293] In some embodiments, the specific reference image comprises at least one of a long-term reference image, a synthesis frame, and a non-output frame.

[0294] In some embodiments, the processor 1002 further comprises: and determining, when determining that the reference image is a frame that is not to be output, and further determining that the reference image is a composite frame, that the reference image is the specific reference image.

[0295] In some embodiments, the non-specific reference image comprises a short-term reference image.

[0296] It should be understood that the apparatus 1000 for image motion compensation can further be realized by corresponding software modules, which will not be described further herein.

[0297] It will be understood that the apparatus of each embodiment of the present application can be realized by a memory and a processor, each memory is for storing commands for performing the method of each embodiment of the present application, and the processor executes the commands, causing the apparatus to perform the method of each embodiment of the present application.

[0298] It is understood that the processor referred to in the embodiments of this application may be a Central Processing Unit (CPU), other general purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), off-the-shelf Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The general purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0299] It is further understood that the memory referred to in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Here, the non-volatile memory may be a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), or a Flash memory. The volatile memory may be a Random Access Memory (RAM), and is used as an external high-speed buffer memory. By way of example and not limitation, many forms of RAM may be used, such as Static RAM (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0300] It should be noted that if the processor is a general purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, the memory (storage module) may be integrated into the processor.

[0301] Note that memory as described herein may include, without being limited to, these and any other suitable types of memory.

[0302] An embodiment of the present application further provides a computer-readable storage medium having stored thereon instructions that, when executed by a computer, cause the computer to perform the method of each of the method embodiments described above.

[0303] An embodiment of the present application further provides a computer device, which includes the computer-readable storage medium described above.

[0304] An embodiment of the present application further provides an encoding device, which includes at least one of an apparatus 70 for image motion compensation, an apparatus 80 for image processing, and an apparatus 1000 for image motion compensation.

[0305] An embodiment of the present application further provides a decoding device, which includes at least one of an apparatus 70 for image motion compensation, an apparatus 80 for image processing, and an apparatus 1000 for image motion compensation.

[0306] Embodiments of the present application have application in aviation, particularly in the field of unmanned aerial vehicles.

[0307] It should be understood that the division of the electric circuits, sub-electric circuits, and sub-units of each embodiment of the present application is only schematic, and those skilled in the art can recognize that each exemplary electric circuit, sub-electric circuit, and sub-unit described by the embodiments disclosed herein can be separated or combined.

[0308] The above-described embodiments may be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, the embodiment may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer commands. When the computer commands are loaded or executed by a computer, the computer generates a process or function according to the embodiments of the present application in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer commands may be stored in a computer-readable storage medium or may be transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer commands may be transmitted from a site, computer, server, or data center to another site, computer, server, or data center by wire (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, radio, microwave, etc.) methods. The computer-readable storage medium may be any available medium accessible by a computer, or may be a data storage device including a server, data center, etc., in which one or more available media are integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a Digital Video Disc (DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)).

[0309] It should be understood that references throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic associated with an embodiment is included in at least one embodiment of the present application. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Moreover, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0310] In each embodiment of the present application, the magnitude of the numbers of each process described above does not indicate the order of execution, and the execution order of each process is determined by its function and inherent logic, and it should be understood that this does not impose any limitations on the implementation process of the embodiments of the present application.

[0311] It is to be understood that "B corresponding to A" in the embodiments of the present application indicates that B is related to A and B can be determined based on A. However, it is to be understood that determining B based on A does not mean determining B only by A, and further, that B can be determined based on at least one of A and other information.

[0312] The term "at least one" in this document merely describes the relation between related objects, and it is understood that there can be three relations, for example, at least one of A and B can have three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the symbol " / " in this document usually indicates that the related objects before and after it are in an "or" relationship.

[0313] Those skilled in the art can recognize that each exemplary unit and algorithm step described in the embodiments disclosed herein can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are finally implemented in a hardware or software manner is determined by the specific application of the technical solution and design constraints. Those skilled in the art can realize the described functions by using different methods for each specific application, but such realization should not be recognized as going beyond the scope of this application.

[0314] Those skilled in the art can clearly understand that, for ease of explanation and concise description, the specific operating processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be further described here.

[0315] In the multiple embodiments provided by the present application, it is to be understood that the disclosed system, device, and method can be realized in other ways. For example, the device embodiment described above is merely exemplary. For example, the division of the units is merely a division of logic functions, and in actual implementation, there may be other division methods, for example, multiple units or components can be combined or integrated into other systems, or some features can be ignored or not implemented. In addition, the couplings or direct couplings or communication connections between each other shown or discussed may be through some interfaces, and the indirect couplings or communication connections of the devices or units may be electrical, mechanical, or other forms.

[0316] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed among multiple network units, among which some or all of the units may be selected according to actual needs to achieve the objective of the solution of the present embodiment.

[0317] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically alone, or two or more units may be integrated into one unit.

[0318] The above are only specific embodiments of the present application, and the scope of protection of the present application is not limited thereto, and any person skilled in the art can easily think of modifications or replacements within the technical scope disclosed by the present application, which are all within the scope of protection of the present application. Therefore, the scope of protection of the present application is based on the scope of protection of the claims described herein. [Item 1] 1. A moving image processing method, the method comprising: determining a motion vector of a time-domain reference block of a first image block, where the time-domain reference block and the first image block are located in different images; if the motion vector of the time domain reference block points to a specific reference image or the reference image of the first image block is a specific reference image, determining a motion vector of the first image block according to the motion vector of the processed time domain reference block, and the motion vector of the processed time domain reference block and the motion vector of the pre-processed time domain reference block are identical; or Obtaining the motion vector of the first image block by abandoning reference to the motion vector of the time domain reference block when the motion vector of the time domain reference block points to a specific reference picture or when the reference picture of the first image block is a specific reference picture; or and when the motion vector of the time domain reference block points to a specific reference picture or when the reference picture of the first image block is a specific reference picture, setting the motion vector of the time domain reference block to the motion vector of the first image block. [Item 2] The motion vector of the time domain reference block after the processing is a motion vector obtained after scaling the motion vector of said time-domain reference block according to a scaling ratio equal to 1; or 2. The video processing method according to item 1, comprising the motion vector of said time domain reference block skipping the scaling step. [Item 3] The above method is The video processing method according to item 1 or 2, further comprising, when the motion vector of the time domain reference block points to a non-specific reference picture and the reference picture of the first image block is a non-specific reference picture, determining the motion vector of the first image block based on the time distance between the reference picture to which the motion vector of the time domain reference block points and the image in which the time domain reference block is located, and the time distance between the reference picture of the first image block and the image in which the first image block is located. [Item 4] The above method is determining a scaling ratio of the motion vector of the time domain reference block according to a time distance between a reference image to which the motion vector of the time domain reference block points and an image in which the time domain reference block is located, and a time distance between a reference image of the first image block and an image in which the first image block is located; 4. The moving image processing method according to item 3, further comprising: determining a motion vector of the first image block based on the motion vector of the time domain reference block after scaling with the scaling ratio. [Item 5] The above method is constructing a motion information candidate list for the first image block, the motion information candidate list including at least one of a motion vector of a spatial domain reference block of the first image block and a reference frame index value of the first image block, where the spatial domain reference block and the first image block are located in the same image; 5. The moving image processing method according to any one of items 1 to 4, further comprising determining a motion vector of the first image block based on the motion information candidate list of the first image block. [Item 6] determining a motion vector of the first image block based on the motion vector of the processed time-domain reference block; adding the motion vector of the time domain reference block after the processing to the motion vector candidate list; Or, obtaining the motion vector of the first image block by abandoning reference to the motion vector of the time domain reference block includes: 6. The video processing method according to item 5, further comprising abandoning adding the motion vector of the time domain reference block to the motion vector candidate list. [Item 7] 7. The moving image processing method according to item 5 or 6, wherein the first image block is a coding unit CU. [Item 8] The first image block is a portion of a second image block, and the method comprises: Construct a motion information candidate list for the first image block, the motion information candidate list including at least one of a motion vector of a spatial domain reference block of the second image block and a reference frame index value of the first image block, where the spatial domain reference block and the second image block are located in the same image; 4. The moving image processing method according to any one of items 1 to 3, further comprising determining a motion vector for the second image block based on the motion information candidate list for the first image block. [Item 9] The reference frame index value of the first image block and the reference frame index value of the second image block are the same; or 9. The moving image processing method according to item 8, wherein the reference frame index value of the first image block is determined based on the reference frame index value of the second image block. [Item 10] determining a motion vector of the first image block based on the motion vector of the processed time-domain reference block; adding the motion vector of the time-domain reference block after the processing to a motion information candidate list of the second image block; Or, obtaining the motion vector of the first image block by abandoning reference to the motion vector of the time domain reference block includes: 10. The moving image processing method according to item 8 or 9, comprising abandoning adding the motion vector of the time domain reference block to a motion information candidate list of the second image block. [Item 11] The video processing method according to item 1, 8 or 9, wherein the second image block is a coding unit CU, the first image block is a sub-image block in the coding unit, and the sub-image block is obtained by dividing the coding unit. [Item 12] Determining the motion vector of the time-domain reference block of the first image block includes: determining an image in a reference image set for the first image block that is closest to the image in which the first image block is located; 12. The moving image processing method according to any one of items 1 to 11, further comprising: determining a temporal domain reference block for the first image block and a motion vector for the temporal domain reference block based on the determined image. [Item 13] The time-domain reference block is an image block located at the same position as the first image block in the determined image; or The time-domain reference block is an image block to which a motion vector of an image block adjacent to the first image block points in the determined image; or The time domain reference block is an image block corresponding to a numerical cut-off position of a position to which a motion vector of an image block adjacent to the first image block points in the determined image; or The time-domain reference block is an image block to which the motion vector of the image block in the determined image points at the same position as the first image block; or The time-domain reference block is a specific neighboring image block of the image block that is located at the same position as the first image block in the determined image; or Item 13. The moving image processing method according to item 12, wherein the time domain reference block is an image block to which a motion vector of a specific adjacent image block of the same image block as the position of the first image block is directed in the determined image. [Item 14] Determining the motion vector of the time-domain reference block of the first image block includes: determining a particular neighboring image block of the first image block in an image in which the first image block resides; 12. The moving image processing method according to any one of claims 1 to 11, further comprising: determining a time domain reference block of the first image block and a motion vector of the time domain reference block based on a reference image of the specific adjacent image block. [Item 15] The time-domain reference block is an image block located at the same position as the first image block in the reference image of the specific neighboring image block; or The time domain reference block is an image block in the reference image of the specific neighboring image block that is the same as the position of the specific neighboring image block; or Item 15. The moving image processing method according to item 14, wherein the time domain reference block is an image block to which the motion vector of the specific adjacent image block points. [Item 16] Determining the specific neighboring image block of the first image block includes: In an image in which the first image block exists, obtaining a first adjacent image block according to a specific scanning order; or Obtaining a first neighboring image block in an image in which the first image block is located, the first neighboring image block having the same reference frame index value as the first image block according to a specific scanning order; or Obtaining a first image block having motion information according to a particular scanning order; or Item 15. The moving image processing method according to item 14, further comprising: obtaining a first adjacent image block that is the same as a default reference image of the first image according to a specific scanning order. [Item 17] The specific scanning order starts from the first block, 17. The moving image processing method according to item 16, wherein the scanning order is along one of the following directions: bottom-to-top, left-to-right, right-to-left, top-to-bottom. [Item 18] 15. The method according to any one of items 1 to 14, wherein the specific reference image includes at least one of a long-term reference image, a synthesis frame, and a non-output frame. [Item 19] The above method is 16. The method according to any one of items 1 to 15, further comprising determining that the reference image is the specific reference image when the reference image is determined to be a frame that will not be output and the reference image is determined to be a composite frame. [Item 20] 17. The method according to any one of items 1 to 16, wherein the non-specific reference image comprises a short-term reference image. [Item 21] A video processing device, comprising: at least one memory for storing computer executable instructions; and at least one processor adapted, either alone or in combination, to access said at least one memory and execute said computer-executable commands to perform the following operations: determining a motion vector of a time-domain reference block of a first image block, where the time-domain reference block and the first image block are located in different images; if the motion vector of the time domain reference block points to a specific reference image or the reference image of the first image block is a specific reference image, determining a motion vector of the first image block according to the motion vector of the processed time domain reference block, and the motion vector of the processed time domain reference block and the motion vector of the pre-processed time domain reference block are identical; or Obtaining the motion vector of the first image block by abandoning reference to the motion vector of the time domain reference block when the motion vector of the time domain reference block points to a specific reference picture or when the reference picture of the first image block is a specific reference picture; or When the motion vector of the time domain reference block points to a specific reference picture, or when the reference picture of the first image block is a specific reference picture, the motion vector of the time domain reference block is the motion vector of the first image block. [Item 22] The motion vector of the time domain reference block after the processing is a motion vector obtained after scaling the motion vector of said time-domain reference block according to a scaling ratio equal to 1; or 22. The video processing device according to item 21, comprising a motion vector of the time domain reference block for which a scaling step has been skipped. [Item 23] The processor further comprises: 23. The video processing device according to item 21 or 22, wherein when the motion vector of the time domain reference block points to a non-specific reference picture and the reference picture of the first image block is a non-specific reference picture, the video processing device is configured to determine the motion vector of the first image block based on the time distance between the reference picture to which the motion vector of the time domain reference block points and the image in which the time domain reference block is located, and the time distance between the reference picture of the first image block and the image in which the first image block is located. [Item 24] The processor is determining a scaling ratio of the motion vector of the time domain reference block according to a time distance between a reference image to which the motion vector of the time domain reference block is directed and an image in which the time domain reference block is located, and a time distance between a reference image of the first image block and an image in which the first image block is located; 24. The moving image processing device according to item 23, which is used to determine the motion vector of the first image block based on the motion vector of the time domain reference block after scaling at the scaling ratio. [Item 25] The processor further comprises: constructing a motion information candidate list for the first image block, the motion information candidate list including at least one of a motion vector of a spatial domain reference block of the first image block and a reference frame index value of the first image block, where the spatial domain reference block and the first image block are located in the same image; 25. The moving image processing device according to any one of items 21 to 24, wherein the moving image processing device is configured to determine a motion vector of the first image block based on the motion information candidate list of the first image block. [Item 26] determining a motion vector of the first image block based on the motion vector of the processed time-domain reference block; adding the motion vector of the time domain reference block after the processing to the motion vector candidate list; Or, obtaining the motion vector of the first image block by abandoning reference to the motion vector of the time domain reference block includes: 26. The video processing device according to item 25, further comprising discarding adding the motion vector of the time domain reference block to the motion vector candidate list. [Item 27] 27. The video processing device according to item 25 or 26, wherein the first image block is a coding unit CU. [Item 28] The first image block is a portion of a second image block, and the processor further comprises: Construct a motion information candidate list for the first image block, the motion information candidate list including at least one of a motion vector of a spatial domain reference block of the second image block and a reference frame index value of the first image block, where the spatial domain reference block and the second image block are located in the same image; 24. The moving image processing device according to any one of items 21 to 23, wherein the moving image processing device is configured to determine a motion vector for the second image block based on the motion information candidate list for the first image block. [Item 29] The reference frame index value of the first image block and the reference frame index value of the second image block are the same; or 29. The moving image processing device according to item 28, wherein the reference frame index value of the first image block is determined based on the reference frame index value of the second image block. [Item 30] determining a motion vector of the first image block based on the motion vector of the processed time-domain reference block; adding the motion vector of the time-domain reference block after the processing to a motion information candidate list of the second image block; Or, obtaining the motion vector of the first image block by abandoning reference to the motion vector of the time domain reference block includes: 30. The video processing device according to item 28 or 29, comprising abandoning adding the motion vector of the time domain reference block to a motion information candidate list of the second image block. [Item 31] A video processing device according to item 21, 28 or 29, wherein the second image block is a coding unit CU, the first image block is a sub-image block in the coding unit, and the sub-image block is obtained by dividing the coding unit. [Item 32] Determining the motion vector of the time-domain reference block of the first image block includes: determining an image in a reference image set for the first image block that is closest to the image in which the first image block is located; 32. The moving image processing device according to any one of items 21 to 31, further comprising: determining a time domain reference block for the first image block and a motion vector for the time domain reference block based on the determined image. [Item 33] The time-domain reference block is an image block located at the same position as the first image block in the determined image; or The time-domain reference block is an image block to which a motion vector of an image block adjacent to the first image block points in the determined image; or The time domain reference block is an image block corresponding to a numerical cut-off position of a position to which a motion vector of an image block adjacent to the first image block points in the determined image; or The time-domain reference block is an image block to which the motion vector of the image block in the determined image points at the same position as the first image block; or The time-domain reference block is a specific neighboring image block of the image block that is located at the same position as the first image block in the determined image; or Item 33. The video processing device according to item 32, wherein the time domain reference block is an image block to which a motion vector of a specific adjacent image block of the same image block as the position of the first image block is directed in the determined image. [Item 34] Determining the motion vector of the time-domain reference block of the first image block includes: determining a particular neighboring image block of the first image block in an image in which the first image block resides; 32. The moving image processing device according to any one of items 21 to 31, further comprising: determining a time domain reference block for the first image block and a motion vector for the time domain reference block based on a reference image of the specific adjacent image block. [Item 35] The time-domain reference block is an image block located at the same position as the first image block in the reference image of the specific neighboring image block; or The time domain reference block is an image block in the reference image of the specific neighboring image block that is the same as the position of the specific neighboring image block; or Item 35. The video processing device according to item 34, wherein the time domain reference block is an image block to which the motion vector of the specific adjacent image block points. [Item 36] Determining the specific neighboring image block of the first image block includes: obtaining a first adjacent image block in an image having the first image block according to a specific scanning order; or Obtaining a first neighboring image block in an image in which the first image block is located, the first neighboring image block having the same reference frame index value as the first image block according to a specific scanning order; or Obtaining a first image block having motion information according to a particular scanning order; or Item 35. The video processing device according to item 34, further comprising: obtaining a first adjacent image block that is the same as a default reference image of the first image according to a specific scanning order. [Item 37] Item 37. The video processing device according to item 36, wherein the specific scanning order is set to a search starting point from the first block and to one of the following directions: bottom to top, left to right, right to left, and top to bottom. [Item 38] The device according to any one of items 21 to 34, wherein the specific reference image includes at least one of a long-term reference image, a synthesis frame, and a frame that is not output. [Item 39] The processor further comprises: The device according to any one of items 21 to 35, for determining that the reference image is a frame that will not be output, and for determining that the reference image is the specific reference image when the reference image is determined to be a composite frame. [Item 40] 37. The apparatus according to any one of items 21 to 36, wherein the non-specific reference image comprises a short-term reference image. [Item 41] A computer-readable storage medium, comprising: A computer-readable storage medium having stored therein commands that, when executed by a computer, cause the computer to execute the image processing method according to any one of items 1 to 20. [Item 42] An encoding device including the moving image processing device according to any one of items 21 to 39. [Item 43] A decoding device including the video processing device according to any one of items 21 to 39.

Claims

1. A video processing method, comprising: Obtaining an initial motion vector MV of a current image block, the initial motion vector MV including a first initial motion vector pointing to a first reference image and a second initial motion vector pointing to a second reference image; and modifying the first initial MV and the second initial MV to obtain a modified first MV and a modified second MV, and performing motion prediction for the current image block based on the modified first MV and the modified second MV; wherein the first reference image and the second reference image are short-term reference images.

2. The method of claim 1 , wherein the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block.

3. The step of modifying the first initial MV and the second initial MV to obtain a modified first MV and a modified second MV includes the steps of: generating a template based on a first reference image block corresponding to the first initial motion vector and belonging to the first reference image, and a second reference image block corresponding to the second initial motion vector and belonging to the second reference image; Modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV. The method of claim 1 , comprising:

4. modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV; Using N third reference image blocks to match the template respectively, where the N third reference image blocks correspond to N third initial motion vectors and belong to the first reference image; Using M fourth reference image blocks to match the template respectively, where the M fourth reference image blocks correspond to M fourth initial motion vectors and belong to the second reference image; selecting a third initial MV from the N third initial MVs and selecting a fourth initial MV from the M fourth initial MVs based on a result of the matching; The method of claim 3 , wherein the one third initial MV and the one fourth initial MV are for use as the MV of the current image block or for use in determining the MV of the current image block.

5. 5. The method of claim 4, wherein at least some of the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MV.

6. A video processing device, comprising: at least one memory for storing computer executable instructions; and at least one processor adapted, either alone or in combination, to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining an initial motion vector MV of a current image block, the initial motion vector MV including a first initial motion vector pointing to a first reference image and a second initial motion vector pointing to a second reference image; and modifying the first initial MV and the second initial MV to obtain a modified first MV and a modified second MV, and performing motion prediction for the current image block based on the modified first MV and the modified second MV; wherein the first reference image and the second reference image are short-term reference images.

7. The apparatus of claim 6 , wherein the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block.

8. The at least one processor: generating a template based on a first reference image block corresponding to the first initial motion vector and belonging to the first reference image, and a second reference image block corresponding to the second initial motion vector and belonging to the second reference image; Modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV. The device according to claim 6 .

9. The at least one processor: Using N third reference image blocks to match the template respectively, where the N third reference image blocks correspond to N third initial motion vectors and belong to the first reference image; Using M fourth reference image blocks to match the template respectively, where the M fourth reference image blocks correspond to M fourth initial motion vectors and belong to the second reference image; Selecting a third initial MV from the N third initial MVs and selecting a fourth initial MV from the M fourth initial MVs based on the matching result; The apparatus of claim 8 , wherein the one third initial MV and the one fourth initial MV are used as the MV of the current image block or for determining the MV of the current image block.

10. 10. The apparatus of claim 9, wherein at least some of the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MV.

11. 1. A method for generating a bitstream, comprising the steps of: Obtaining an initial motion vector MV of a current image block, the initial motion vector MV including a first initial motion vector pointing to a first reference image and a second initial motion vector pointing to a second reference image; modifying the first initial MV and the second initial MV to obtain a modified first MV and a modified second MV, and performing motion prediction for the current image block based on the modified first MV and the modified second MV; encoding the current image block based on the motion prediction to generate a bitstream including motion information for the current image block; wherein the first reference image and the second reference image are short-term reference images.

Images