Image processing method, image processing device, and bitstream generation method

Abstract

To provide a method and an apparatus for image motion compensation in a technique using bidirectional motion prediction, taking into account whether a reference picture to which an associated motion vector points is a long-term reference picture or not.SOLUTION: An image processing method includes the steps of obtaining a first initial motion vector pointing to a first reference image and a second initial motion vector pointing to a second reference image, and calculating motion compensation for a current image block on the basis of gradient values of pixel points pointed to by the first initial motion vector and the second initial motion vector when the first reference image and the second reference image are both non-specific reference images that are short-term reference images, and calculating a predicted image block of the current image block on the basis of the motion compensation of the current image block.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] Copyright Declaration The subject matter disclosed by this patent document contains material protected by copyright. Such copyright is owned by the Copyright Owner. The Copyright Owner does not object to anyone reproducing this patent document or the patent disclosure as it appears in the official records and archives of the Patent and Trademark Office.

[0002] The present application relates to the field of image processing, and in particular to an image motion compensation method and apparatus. [Background technology]

[0003] In recent years, with the popularity of portable devices, handheld devices, and wearable devices, the capacity of video data has been increasing. As video formats become more complex, video storage and transmission has become more difficult. To reduce the bandwidth required for video storage and transmission, video data is usually encoded and compressed at the encoding end and decoded at the decoding end.

[0004] The encoding compression process includes processes such as prediction, transformation, quantification, and entropy coding. Prediction includes two types: intraframe prediction and interframe prediction, which use predictive block data to remove redundant information from the currently encoded image block. Intraframe prediction uses information from the current frame image to obtain predictive block data. Interframe prediction uses information from a reference image to obtain predictive block data. The process involves dividing the currently encoded image into multiple image blocks, dividing the currently encoded image block into multiple sub-image blocks, and then searching for an image block in the reference image that best matches the current sub-image block to determine the predicted image block. The relative displacement between the predicted image block and the current sub-image block is the motion vector. The corresponding pixel values ​​of the sub-image block and the predicted image block are then subtracted to obtain a residual. The resulting residuals corresponding to each sub-image block are combined to obtain the residual of the currently encoded image block. After the residual undergoes processing such as transformation, quantification, and entropy coding, an entropy-coded bitstream is obtained, and the entropy-coded bitstream and coding mode information after coding, such as information on intra-frame prediction mode, motion vector (or motion vector residual), etc., are stored or transmitted to the decoding end.

[0005] At the image decoding end, an entropy-encoded bitstream is obtained, and then entropy decoding is performed to obtain the corresponding residual. Based on the information obtained by decoding, such as the motion vector and intra-frame prediction, a predicted image block corresponding to the image block to be decoded is searched for, and the value of each pixel point in the image block to be decoded is obtained based on the predicted image block and the residual.

[0006] From the above description, it can be seen that when performing inter-frame prediction, the more similar the selected reference image is to the current image to be coded, the smaller the residual error generated by inter-frame prediction, thereby improving the coding efficiency of inter-frame prediction. Specifically, some existing technologies use each image in a video to construct a high-quality specific reference image for background content including a scene. When performing inter-frame prediction, the background part of the current image to be coded or the current image to be decoded can be used to reduce the residual information of inter-frame prediction by referring to the high-quality specific reference image, thereby improving coding efficiency. In other words, this specific reference image is a reference image for inter-frame prediction. A long-term reference image is not a decoded image, but an artificially constructed image. A long-term reference image includes multiple image blocks, each of which is extracted from a previously decoded image, and different image blocks in the long-term reference image may be from different previously decoded images.

[0007] In order to improve coding efficiency and reduce the amount of information transmitted by the encoding end, some existing technologies allow the decoding end to directly derive motion vectors, so that the encoding end does not need to transmit motion vector information or motion vector residual information, and the decoding end can also obtain true motion vectors without decoding the motion vector information or motion vector residual information. Summary of the Invention [Problem to be solved by the invention]

[0008] Some existing techniques using motion vector derivation and bidirectional motion prediction do not consider the specificity of long-term reference pictures. Some techniques using motion vector derivation do not consider whether the reference picture pointed to by the motion vector is a long-term reference picture, and when correcting a motion vector, a motion search may be performed in the long-term reference picture, which may actually reduce search efficiency and coding efficiency. Techniques using bidirectional motion prediction operate motion vectors based on the temporal correlation of images, and when the reference picture pointed to by the associated motion vector is a long-term reference picture, these operations may be invalid because the definition of the temporal distance between the current picture to be coded or the current picture to be decoded and the long-term reference picture is unclear. [Means for solving the problem]

[0009] The present application provides an image motion compensation method and an image motion compensation device, which can improve encoding / decoding efficiency.

[0010] In a first aspect, there is provided a method of image motion compensation, the method comprising: Obtaining an initial motion vector MV of a current image block; If the reference image indicated by the initial motion vector is a specific reference image, performing motion compensation on the current image block based on the initial motion vector; If the reference picture to which the initial MV points is a non-specific reference picture, modifying the initial MV to obtain a modified MV, and performing motion compensation on the current image block based on the modified MV.

[0011] In a second aspect, there is provided an image motion compensation apparatus, the apparatus comprising: at least one memory for storing computer executable commands; and at least one processor adapted, alone or in combination, to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining an initial motion vector MV of a current image block; If the reference image indicated by the initial motion vector is a specific reference image, performing motion compensation on the current image block based on the initial motion vector; If the reference picture to which the initial MV points is a non-specific reference picture, modify the initial MV to obtain a modified MV, and perform motion compensation on the current image block based on the modified MV.

[0012] In a third aspect, there is provided a computer readable storage medium having stored thereon instructions which, when executed on a computer, cause the computer to perform the image motion compensation method according to the first aspect.

[0013] In a fourth aspect, there is provided an encoding device including an image motion compensation apparatus according to the second aspect.

[0014] In a fifth aspect, there is provided a decoding device including an image motion compensation apparatus according to the second aspect.

[0015] In a sixth aspect, there is provided a method of image motion compensation, the method comprising: Obtaining an initial motion vector MV of a current image block; Determining a scaling ratio of the initial MV, where if the initial MV is directed to a specific reference image, the scaling ratio of the initial MV is 1; Scaling the initial MV based on a scaling ratio of the initial MV; performing motion compensation on the current image block based on the scaled MV.

[0016] In a seventh aspect, there is provided an image motion compensation apparatus, the apparatus comprising: at least one memory for storing computer executable commands; and at least one processor adapted, alone or in combination, to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining an initial motion vector MV of a current image block; Determining a scaling ratio of the initial MV, where if the initial MV is directed to a specific reference image, the scaling ratio of the initial MV is 1; Scaling the initial MV based on a scaling ratio of the initial MV; performing motion compensation on the current image block based on the scaled MV.

[0017] In an eighth aspect, there is provided a computer readable storage medium having stored thereon instructions which, when executed on a computer, cause the computer to perform the image motion compensation method according to the sixth aspect.

[0018] In a ninth aspect, there is provided an encoding device including an image motion compensation apparatus according to the seventh aspect.

[0019] In a tenth aspect, there is provided a decoding device including an image motion compensation apparatus according to the seventh aspect.

[0020] In an eleventh aspect, there is provided a method of image processing, the method comprising: Obtaining a first initial motion vector MV and a second initial motion vector MV, the first initial motion vector MV pointing to a first reference image, and the second initial motion vector MV pointing to a second reference image; When at least one of the first reference image and the second reference image is a specific reference image, calculating and obtaining a predicted image block of the current image block based on the first initial MV and the second initial MV; When the first reference image and the second reference image are both non-specific reference images, the method includes calculating and obtaining an MV of a current image block based on the gradient values ​​of pixel points pointed to by the first initial MV and the second initial MV, and calculating and obtaining a predicted image block of the current image block based on the MV of the current image block.

[0021] In a twelfth aspect, there is provided an image processing apparatus, the apparatus comprising: at least one memory for storing computer executable commands; and at least one processor adapted, alone or in combination, to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining a first initial motion vector MV and a second initial motion vector MV, the first initial motion vector MV pointing to a first reference image, and the second initial motion vector MV pointing to a second reference image; When at least one of the first reference image and the second reference image is a specific reference image, calculating and obtaining a predicted image block of the current image block based on the first initial MV and the second initial MV; If the first reference image and the second reference image are both non-specific reference images, the MV of the current image block is calculated based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and the predicted image block of the current image block is calculated based on the MV of the current image block.

[0022] In a thirteenth aspect, there is provided a computer readable storage medium having stored thereon instructions which, when executed on a computer, cause the computer to perform an image processing method according to the eleventh aspect.

[0023] In a fourteenth aspect, there is provided an encoding device including an image processing apparatus according to the twelfth aspect.

[0024] In a fifteenth aspect, there is provided a decoding device including an image processing apparatus according to the twelfth aspect. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a schematic flowchart of an image motion compensation method in one embodiment of the present application; [Figure 2] 1 is a schematic diagram illustrating the principle of a two-way matching method according to one embodiment of the present application; [Figure 3] 1 is a schematic diagram illustrating the principle of a template matching method according to one embodiment of the present application; [Figure 4] 1 is a schematic diagram illustrating the principle of DMVR technology in one embodiment of the present application. [Figure 5] 10 is a schematic flowchart of an image processing method according to another embodiment of the present application; [Figure 6] 1 is a schematic diagram illustrating the principle of BIO technology according to one embodiment of the present application. [Figure 7] FIG. 1 is a schematic frame diagram of an image motion compensation device according to one embodiment of the present application; [Figure 8] 1 is a schematic frame diagram of an image processing device according to one embodiment of the present application; [Figure 9] 4 is a schematic flowchart of an image motion compensation method according to another embodiment of the present application; [Figure 10] FIG. 10 is a schematic frame diagram of an image motion compensation apparatus according to another embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0026] The technical solutions in the embodiments of the present application will be described below with reference to the drawings.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art of the present application. In this specification, the technical terms used in the present application are for the purpose of describing specific examples only and are not intended to limit the present application.

[0028] First, the relevant techniques and concepts related to the embodiments of the present application will be described.

[0029] A moving image is composed of multiple images. When encoding / decoding a moving image, different prediction methods can be used for different images in the moving image. The prediction methods used for different images can be divided into intra-frame predicted images and inter-frame predicted images, where inter-frame predicted images include forward predicted images and bidirectional predicted images. An I-image is an intra-frame predicted image, also known as a key frame. A P-image is a forward predicted image, which uses a previously encoded / decoded P-image or I-image as a reference image. A B-image is a bidirectional predicted image, which uses previous and subsequent images as reference images. In one embodiment, the encoding / decoding terminal generates a group of pictures (GOP) for each row after encoding / decoding multiple images. This GOP is a group of pictures consisting of one I-image and at least one of multiple B-images (or bidirectional predicted images) and P-images (or forward predicted images). The decoding terminal reads and decodes each GOP for each row during broadcasting, and then reads and renders the image.

[0030] In modern video encoding / decoding standards, an image can be divided into multiple small blocks and encoded / decoded with different resolutions, i.e., the image can be divided into multiple image blocks. An image can be divided into any number of image blocks. For example, the image can be divided into an m×n image block matrix. The image blocks can have a rectangular, square, circular, or any other shape. The image blocks can have any size, for example, p×q pixels. Each image block can have the same size and / or shape. Alternatively, two or more image blocks can have different sizes and / or shapes. The image blocks may or may not have overlapping portions. In some embodiments, the image block is called a macroblock or a Largest Coding Unit (LCU) in some encoding / decoding standards. For the H.264 standard, the image block is called a macroblock, and its size may be 16×16 pixels. For the High Efficiency Video Coding (HEVC) standard, an image block is called the largest coding unit, and its size may be 64×64 pixels.

[0031] In some other embodiments, an image block may not be a macroblock or a largest coding unit, but may include a portion of a macroblock or a largest coding unit, or may include at least two complete macroblocks (or largest coding units), or may include at least one complete macroblock (or largest coding unit) and a portion of one macroblock (or largest coding unit), or may include at least two complete macroblocks (or largest coding units) and portions of several macroblocks (or largest coding units). In this way, after an image is divided into multiple image blocks, these image blocks can be coded / decoded in the image data respectively.

[0032] The encoding process includes processes such as prediction, transformation, quantification, and entropy coding. Here, prediction includes two types: intraframe prediction and interframe prediction, which use predictive block data to remove redundant information from the image block currently being encoded. Intraframe prediction uses information from the current frame image to obtain predictive block data. Interframe prediction uses information from a reference image to obtain predictive block data. The process involves dividing the current image to be encoded into multiple image blocks, dividing the current image block into multiple sub-image blocks, and then searching for an image block in the reference image that best matches the current sub-image block to determine the predicted image block. The relative displacement between the predicted image block and the current sub-image block is the motion vector. The corresponding pixel values ​​of this sub-image block and the predicted image block are then subtracted to obtain a residual. The obtained residuals corresponding to each sub-image block are combined to obtain the residual of the image block currently being encoded.

[0033] In the embodiments of the present application, the residual correlation of an image block can be eliminated by a transform matrix, i.e., the redundant information of the image block can be eliminated, thereby improving coding efficiency. The transformation of a data block in an image block typically employs a two-dimensional transform, i.e., the encoding end multiplies the residual information of the data block by an N×M transform matrix and its transpose, respectively, to obtain transform coefficients. The transform coefficients are quantified to obtain the quantified coefficients, and finally, the quantified coefficients are entropy coded to obtain an entropy-coded bitstream. The entropy-coded bitstream and post-coding coding mode information, such as intra-frame prediction mode, motion vector (or motion vector residual), etc., are stored or transmitted to the decoding end.

[0034] At the image decoding end, after obtaining the entropy-encoded bitstream, the corresponding residual is obtained by entropy encoding. Based on the motion vector, intra-frame prediction, and other information obtained by decoding, a predicted image block corresponding to the image block is searched for. Based on the predicted image block and the residual, the value of each pixel point in the current sub-image block is obtained.

[0035] In the previous paragraph, it was mentioned that a previously coded / decoded image is used as a reference image for the current coding / decoding target. In some embodiments, the reference image can be further constructed to increase the similarity between the reference image and the current coding / decoding target image.

[0036] For example, there is a specific encoding / decoding scene in video content, where the background in this scene remains essentially unchanged, but only the foreground in the video changes or moves. For example, video monitoring belongs to this type of scene. In a video monitoring scene, the monitoring camera is usually fixed and motionless or moves slowly, while the background remains essentially unchanged. In contrast, objects captured by the video monitoring camera, such as people or vehicles, are constantly moving or changing, and the foreground is constantly changing. In this type of scene, a specific reference image can be constructed, which contains only high-quality background information. This specific reference image can include multiple image blocks, each of which is extracted from a previously decoded image, and different image blocks in this specific reference image may be from different previously decoded images. When performing inter-frame prediction, the background portion of the current image to be encoded / decoded can refer to this specific reference image, which can be used to reduce residual information in inter-frame prediction, thereby improving encoding / decoding efficiency.

[0037] The above is a specific example of a specific reference image. In some embodiments, the specific reference image has at least one of the following characteristics: a composite frame, a long-term reference image, or a non-output image. For example, the specific reference image may be a composite long-term reference image, a non-output composite frame, or a non-output long-term reference image. In some embodiments, the composite frame is called a composite reference frame.

[0038] In some embodiments, a non-specific reference image may be a reference image that does not have at least one of the following properties: a synthesized frame, a long-term reference image, a non-output reference image. For example, the specific reference image may include a reference image other than a synthesized frame, or may include a reference image other than a long-term reference image, or may include a reference image other than a non-output reference image, or may include a reference image other than a synthesized long-term reference image, or may include a reference image other than a non-output synthesized frame, or may include a reference image other than a non-output long-term reference image, etc.

[0039] In some embodiments, when images in a video can be used as reference images, they can be divided into long-term reference images and short-term reference images. Here, the short-term reference images are a concept corresponding to the long-term reference images. The short-term reference images exist in a reference image buffer area for a certain period of time, and after the decoded reference images have passed through the short-term reference images and undergo multiple input / output operations in the reference image buffer area, the short-term reference images are moved out of the reference image buffer area. The reference image buffer area can be referred to as a reference image list buffer memory, a reference image list, a reference frame list buffer memory, or a reference frame list, and in this document, they are all referred to as the reference image buffer area.

[0040] The long-term reference picture (or some data in the long-term reference picture) can continue to exist in the buffer area of ​​the reference picture, and this long-term reference picture (or some data in the long-term reference picture) is not affected by the input / output operation of the decoded reference picture in the buffer area of ​​the reference picture, and only when the decoding end sends an update command operation, this long-term reference picture (or some data in the long-term reference picture) moves out of the buffer area of ​​the reference picture.

[0041] The short-term reference picture and the long-term reference picture may be called differently in different standards. For example, in standards such as H.264 / advanced video coding (AVC) or H.265 / HEVC, the short-term reference picture is called a short-term reference frame, and the long-term reference picture is called a long-term reference frame. In standards such as AVS coding standard (AVS), 1-P2, AVS2-P2, and IEEE (Institute of Electrical and Electronics Engineers) 1857.9-P4, the long-term reference picture is called a background picture. In standards such as VP8 and VP9, ​​the long-term reference picture is called a golden frame.

[0042] It should be understood that the adoption of specific technical terms in the embodiments of the present application does not indicate that they must be applied to a specific scene, for example, the term "long-term reference image" or "long-term reference frame" does not indicate that they must be used in a technology corresponding to a standard such as H.264 / AVC or H.265 / HEVC.

[0043] The long-term reference picture mentioned above may be constructed from image blocks extracted from multiple decoded pictures, or may be obtained by updating an existing reference frame (e.g., a pre-stored reference frame) using multiple decoded pictures. Of course, this specific constructed reference picture may also be a short-term reference picture. Alternatively, the long-term reference picture does not have to be a constructed reference picture.

[0044] In the above-described embodiments, the specific reference image may include a long-term reference image, and the non-specific reference image may include a short-term reference image.

[0045] Optionally, the type of reference frame can be identified from a special field in the stream structure.

[0046] Optionally, if the reference image is determined to be a long-term reference image, the reference image is determined to be the specific reference image; or if the reference image is determined to be a frame that will not be output, the reference image is determined to be the specific reference image; or if the reference image is determined to be a composite frame, the reference image is determined to be the specific reference image; or if the reference image is determined to be a frame that will not be output and the reference image is determined to be a composite frame, the reference image is determined to be the specific reference image.

[0047] Alternatively, each of the various types of reference images can have a corresponding mark, and the decoding end can determine whether the reference image is a specific reference image according to the mark that the reference image has.

[0048] In some embodiments, if a reference image is determined to have the mark of a long-term reference image, the reference image is determined to be a specific reference image.

[0049] In some embodiments, if a reference image is determined to have marks that are not output, the reference image is determined to be a specific reference image.

[0050] In some embodiments, if a reference image is determined to have a mark of a composite frame, the reference image is determined to be a specific reference image.

[0051] In some embodiments, if the reference image has at least two of the following three marks, the reference image is determined to be a specific reference image: a long-term reference image mark, a not-output mark, and a synthetic frame or synthetic reference frame mark. For example, if the reference image is determined to have a not-output mark and the reference image is determined to have a synthetic frame mark, the reference image is determined to be a specific reference image.

[0052] Specifically, an image can have a mark indicating whether it is a frame to be output, and if an image is instructed not to be output, it indicates that this frame is a reference image, and further determines whether this frame has a mark indicating a composite frame, and if so, determines that the reference image is a specific reference image.If an image is instructed to be output, there is no need to determine whether it is a composite frame, and directly determines that this frame is not a specific reference image.Alternatively, if an image is instructed not to be output but has a mark indicating that it is not a composite frame, it can be determined that this frame is not a specific reference image.

[0053] Optionally, the reference picture is determined to be a specific reference picture if it is determined that the reference picture satisfies one of the following conditions by analyzing parameters from the picture header, picture parameter set (PPS), and slice header:

[0054] The reference image is a long-term reference image.

[0055] The reference image is a synthetic reference image.

[0056] The reference image is an image that is not output.

[0057] If the reference image is an image that is not to be output, it is further determined whether the reference image is a composite reference image.

[0058] Although the technique of using motion vector derivation has been mentioned above, when modifying a motion vector, performing motion search on a specific reference picture will actually reduce search efficiency and encoding / decoding efficiency.The specific reference picture is artificially constructed or is derived from a specific reference picture that is earlier in time order, and there is not necessarily a spatial relationship between image blocks in the specific reference picture, and there is obvious shaking at the image block edges, so there is not much significance in searching for a motion vector based on such a specific reference picture.

[0059] Pattern Matching Motion Vector Derivation (PMMVD) technology and Decode Motion Vector Refinement (DMVR) technology are both technologies that use motion vector derivation.

[0060] The preceding paragraph further mentions a technique using bidirectional motion prediction, which operates motion vectors based on the temporal correlation of images, and when the reference image to which the associated motion vector points is a specific reference image, these operations may be invalid because the temporal distance between the current image to be coded or decoded and the specific reference image is unclear. Bidirectional optical flow (BIO) prediction technology is a technique that uses bidirectional motion prediction.

[0061] Next, based on PMMVD, DMVR, and BIO, the image motion compensation method of the present invention will be described with examples. It should be noted that the image motion compensation method of the present invention is not limited to these three techniques.

[0062] Before describing the image motion compensation method according to the embodiment of the present application, a brief description will be given of the video encoding / decoding process according to the HEVC standard.

[0063] The HEVC standard defines three inter-frame prediction modes: Inter mode, Merge mode, and Skip mode. The purpose of inter-frame prediction is to obtain a motion vector (MV) and then determine the position of a predicted image block in a reference image based on the motion vector. Neighboring image blocks have similar motion patterns. For example, an image block (e.g., at least one of an image block to be coded and an image block to be decoded) and its neighboring image blocks belong to the same object. When a camera moves, their movement distances and directions are naturally similar or the same. Therefore, there is often no need to calculate a motion vector; the motion vector of the neighboring image block is directly used as the motion vector of the current image block. Here, in Merge mode and Skip mode, the residual motion vector difference (MVD) of the motion vector is 0. That is, the motion vector is directly obtained based on the neighboring coded or decoded image blocks.

[0064] If at least one of the image blocks to be coded and the image block to be decoded is in Merge mode, the operation principle is as follows: a motion vector prediction (MVP) candidate list is constructed based on neighboring image blocks, and the most suitable MVP from the MVP candidate list is selected as the motion vector for the current image block. The location of the predicted image block is then determined based on this motion vector. After determining the predicted image block, a residual is calculated to obtain the predicted image block. In Merge mode, the motion vector is selected from the MVP candidate list, so there is no MVD. The encoding end only needs to code the residual and the index of the selected motion vector in the MVP candidate list; there is no need to code the MVD. The decoding end can construct an MVP candidate list in a similar manner, and then obtain the motion vector based on the index transmitted by the encoding end. The decoding end determines the predicted image block based on the motion vector, and obtains the current image block by taking and decoding the residual.

[0065] The specific operation process of the encoding end in merge mode is as follows: 1. Get a list of MVP candidates. 2. Select the best MVP from the MVP candidate list and obtain the index of this MVP in the MVP candidate list. 3. The selected MVP is taken as the motion vector of the current image block. 4. Determine a predicted image block from the reference image based on the motion vector. 5. Subtract the predicted image block from the current image block to get the residual. 6. Since the motion vector is selected from the MVP candidate list, there is no MVD, and it is only necessary to transmit the residual and the index of the selected MVP in the MVP candidate list to the decoding end.

[0066] The specific operation process of the decoding end in merge mode is as follows: 1. Receive the residual and the index of the motion vector in the MVP candidate list. 2. Get a list of MVP candidates. 3. Based on the index, search for the motion vector in the MVP candidate list and take it as the motion vector of the current image block. 4. Based on the motion vector, a predicted image block is determined, and the residual is taken and decoded to obtain the current image block.

[0067] The above is the normal Merge mode processing process.

[0068] Skip mode is a special case of Merge mode. After obtaining a motion vector through Merge mode, if the encoder determines that the current image block and the predicted image block are essentially the same according to a certain method, it does not need to transmit the residual data, and only needs to transmit the index of the motion vector in the MVP candidate list, and the current image block can directly obtain the mark from the predicted image block.

[0069] In Inter mode, the MVP is determined first, and then the MVP is modified to obtain the MVD. The encoding end not only needs to transmit the index and residual to the decoding end, but also the MVD to the decoding end. Advanced Motion Vector Prediction (AMVP) is a tool that realizes motion vector prediction through a competitive mechanism.

[0070] AMVP mode also has an MVP candidate list, and the motion vectors in this MVP candidate list are obtained from blocks neighboring the current image block in the spatial or temporal domain. The MVP candidate list in AMVP mode may differ from the MVP candidate list in Merge mode. The encoding or decoding end selects the most suitable MVP from the MVP candidate list. Using this MVP as the search starting point, the neighborhood is searched to obtain the most suitable motion vector, which is the motion vector of the current image block. Based on this motion vector, the position of the predicted image block is determined, and after determining the predicted image block, the residual is calculated to obtain it. At the same time, the MV and MVP are subtracted from each other to obtain the MVD. The encoding end encodes the residual, the index of the MVP in the MVP candidate list, and the MVD, and transmits them to the decoding end. The decoding end can construct an MVP candidate list in a similar manner, and then obtain the MVP based on the index transmitted by the encoding end. The decoding end determines the MV according to the MVP and MVD, determines a predicted image block according to the MV, and takes and decodes the residual to obtain the current image block.

[0071] The specific operation process of the encoding end in AMVP mode is as follows: 1. Get a list of MVP candidates. 2. Select the best MVP from the MVP candidate list and obtain the index of this MVP in the MVP candidate list. 3. Determine the starting point for your search based on the MVP. 4. Search near the starting point to get the best motion vector. 5. Based on the motion vector, a predicted image block is determined from the reference image. 6. Subtract the predicted image block from the current image block to get the residual. 7. Subtract MVP from the motion vector to get MVD. 8. Send the residual, the index of the selected MVP in the MVP candidate list, and the MVD to the decoding end.

[0072] The specific operation process of the decoding end in AMVP mode will not be further described.

[0073] An embodiment of the present application provides an image motion compensation method 100. Figure 1 is a schematic flowchart of the image motion compensation method 100 of one embodiment of the present application. As shown in Figure 1, the method 100 includes: S110, obtaining an initial motion vector of a current image block; S120, if the reference image directed by this initial motion vector is a specific reference image, perform motion compensation on this current image block according to this initial motion vector; S130, if the reference image to which the initial motion vector points is a non-specific reference image, modifying the initial motion vector to obtain a modified motion vector, and performing motion compensation on the current image block based on the modified motion vector.

[0074] In some embodiments, when the initial motion vector points to a specific reference picture, motion compensation is performed directly; when the initial motion vector points to a non-specific reference picture, the initial motion vector is modified and motion compensation is performed based on the modified motion vector, thereby avoiding meaningless searches due to obvious shaking of the image block edges of the specific reference picture and improving encoding / decoding efficiency.

[0075] In a possible embodiment, the image motion compensation method 100 of the embodiment of the present application can be applied to PMMVD technology.

[0076] The PMMVD technique is a special merge mode based on the Frame Rate Up Conversion (FRUC) technique, in which the motion information (e.g., MV and MVD) of the current image block is not coded in the stream but is generated directly at the decoding end.

[0077] Optionally, in this possible embodiment, S130, if the reference picture to which the initial motion vector points is a non-specific reference picture, modifying the initial motion vector, obtaining a modified motion vector, and performing motion compensation on the current image block based on the modified motion vector may include obtaining a motion vector candidate list for the current image block, if the reference picture to which any candidate motion vector in the motion vector candidate list points is a non-specific reference picture, determining the initial motion vector based on the motion vector candidate list, modifying the initial motion vector, obtaining a modified motion vector, and performing motion compensation on the current image block based on the modified motion vector.

[0078] In this embodiment, an initial motion vector is determined from a motion vector candidate list. If the reference image pointed to by a candidate vector in this motion vector candidate list is a non-specific reference image, the initial motion vector is modified to obtain a modified motion vector. Then, motion compensation is performed on the current image block based on the modified motion vector. This can avoid meaningless searches due to obvious shaking of the image block edge of the specific reference image, and improve the coding / decoding efficiency.

[0079] Optionally, in a possible embodiment, obtaining a motion vector candidate list for the current image block may include determining a candidate motion vector to add to the motion vector candidate list, and if the reference image to which the candidate motion vector points is a non-specific reference image, adding the candidate motion vector to the motion vector candidate list.

[0080] Specifically, in one embodiment, the method 100 may further include determining, based on a frame mark of the reference picture to which the candidate motion vector points, that the reference picture to which the candidate motion vector points is a non-specific reference picture. Specifically, based on the frame mark of the reference picture to which the candidate motion vector points, it may be determined whether the reference picture to which the candidate motion vector points is a specific reference picture. If the reference picture to which the candidate motion vector points is a specific reference picture, the candidate motion vector corresponding to the specific reference picture does not need to be added to the motion vector candidate list. Thus, when selecting an initial motion vector from the motion vector candidate list, the motion vector corresponding to the specific reference picture is not selected.

[0081] It is understood that the motion vector in each embodiment of the present application includes three parameters: a horizontal component v x , vertical component v y and a frame mark of the reference picture to which it points. For example, the frame mark may be a Picture Order Count (POC) or another type of mark. The encoding end and the decoding end can determine the attribute of the reference picture by the frame mark and determine whether the reference picture is a specific reference picture or a non-specific reference picture.

[0082] Optionally, in one embodiment, determining an initial motion vector based on the motion vector candidate list may include determining whether the selected initial motion vector is a motion vector pointing to a specific reference image, and if at least one of the selected initial motion vectors points to a specific reference image, reselecting an initial motion vector until the reference image to be pointed to selects an initial motion vector of a reference image other than the specific reference image.

[0083] Specifically, if the determined initial motion vector points to a specific reference image based on the motion vector candidate list, a candidate motion vector can be reselected from the motion vector candidate list as the initial motion vector based on a preset rule. For example, the preset rule may select the candidate motion vector with the next lowest matching cost, and if this candidate motion vector points to a specific reference image, this candidate motion vector can be used as the initial motion vector. This embodiment is for illustrative purposes only and does not limit the present application.

[0084] Taking the encoding end as an example, the motion vector derivation process in FRUC merge mode is divided into two steps. The first step is coding unit (CU) level motion search, and the second step is sub-CU level motion refinement. Similarly, the decoding end can also realize functions similar to those of the encoding end.

[0085] In the CU-level motion search, a CU-level motion vector candidate list is generated. Based on the bidirectional matching method, the CU-level motion vector candidate list is searched for the motion vector with the smallest matching cost, e.g., MV-A. Based on the template matching method, the CU-level motion vector candidate list is also searched for the motion vector with the smallest matching cost, e.g., MV-B. Then, based on the rate distortion cost (RD-Cost) strategy used in the merge mode strategy, it is determined whether the current CU should use the FRUC merge mode. That is, the RD-Cost strategy is used to check the results of the two matching methods (bidirectional matching and template matching). The result obtained by the matching method with the smallest matching cost is then compared with the results of the other CU modes. The matching method with the smallest matching cost among the two matching methods is the one with the smallest final matching cost. The FRUC flag of the current CU is set to TRUE, and the decoding end is instructed to use the corresponding matching method.

[0086] In this specific embodiment, the CU-level motion vector candidate list may correspond to the motion vector candidate list in the method 100 of the embodiment of the present application. When generating the motion vector candidate list, the motion vectors to be added to the list are scanned. If the motion vector points to a non-specific reference picture, the motion vector can be added to the motion vector candidate list. If the motion vector points to a specific reference picture, the motion vector is rejected and not added to the motion vector candidate list. In this embodiment, when the current image block is in an advanced motion vector prediction (AMVP) mode, the motion vector candidate list may include at least one candidate motion vector among the original AMVP candidate motion vector obtained for the non-specific reference picture, the merge candidate motion vector obtained for the non-specific reference picture, the motion vector obtained by interpolation from the non-specific reference picture, and the upper neighboring motion vector and the left neighboring motion vector of the current block for the non-specific reference picture. This motion vector candidate list may also be a CU-level motion vector candidate list. It should be understood that in this embodiment, the non-specific reference picture may specifically be a short-term reference picture or a short-term reference frame.

[0087] Determining the initial motion vector based on the motion vector candidate list may include determining distortion costs corresponding to candidate motion vectors in the motion vector candidate list based on at least one of a bidirectional matching method and a template matching method, and selecting the motion vector in the motion vector candidate list with the smallest distortion cost as the initial motion vector.

[0088] Specifically, taking the encoding end as an example, determining an initial motion vector based on the motion vector candidate list can correspond to CU-level motion search. In the CU-level motion search, a bidirectional matching method is used to search for a motion vector with the smallest matching cost from the CU-level motion vector candidate list, for example, MV-A. The template matching method also searches for a motion vector with the smallest matching cost from the CU-level motion vector candidate list, for example, MV-B. Then, based on the RD-Cost strategy used in the merge mode strategy, it is determined whether the current CU will use the FRUC merge mode. That is, the RD-Cost strategy is used to check the results of the two matching methods (bidirectional matching and template matching). The result obtained by the matching method with the smallest matching cost is then compared with the results of the other CU modes. If the matching method with the smallest matching cost among the two matching methods has the smallest final matching cost, the FRUC flag of the current CU is set to TRUE, and the decoding end is instructed to use the corresponding matching method. Similarly, the decoding end can also realize functions similar to those of the encoding end, but the decoding end is less complex than the encoding end. For example, the decoding end directly receives a flag to instruct the matching method of the encoding end, and does not need to implement the RD-Cost strategy.

[0089] Optionally, modifying the initial motion vector and obtaining a modified motion vector may include generating a sub-motion vector candidate list for the current image block, the sub-motion vector candidate list including the initial motion vector, and determining the motion vector with the smallest distortion cost from the sub-motion vector candidate list as the modified motion vector.

[0090] Specifically, modifying the initial motion vector and obtaining the modified motion vector can accommodate sub-CU level motion refinement.

[0091] In the CU-level motion search, the motion vector with the smallest matching cost is the initial motion vector, which is used as the starting point for CU-level motion refinement. A local search is performed around the starting point based on the matching method (bidirectional matching or template matching) determined by the CU level. Specifically, in the sub-CU-level motion refinement, a sub-CU-level motion vector candidate list can be generated. In the sub-CU-level motion vector candidate list, the motion vector with the smallest matching cost is searched for and used as the motion vector for the current CU.

[0092] The sub-CU level motion vector candidate list may include a motion vector determined by a CU-level motion search (an MV determined from a CU-level search), the top, left, top-left and top-right neighboring motion vectors of the current image block, motion vectors after collocation of corresponding positions of the current image block in reference pictures (scaled versions of collocated MVs from reference pictures), and candidate motion vectors derived from other temporal domains (up to 4 ATMVP candidates and up to 4 STMVP candidates).

[0093] In the above-mentioned specific embodiment, if the reference picture to which the candidate motion vector points is a specific reference picture, the candidate motion vector corresponding to the specific reference picture is not added to the CU level motion vector candidate list, so that the candidate motion vector corresponding to the specific reference picture has no chance of becoming the initial motion vector.

[0094] In another specific embodiment, if the initial motion vector determined based on the CU-level motion vector candidate list points to a specific reference image, a candidate motion vector can be reselected from the CU-level motion vector candidate list as the initial motion vector based on a preset rule, for example, the preset rule can be to select the candidate motion vector with the next smallest matching cost, and if the candidate motion vector does not point to a specific reference image, the candidate motion vector can be used as the initial motion vector.

[0095] FIG. 2 is a principle diagram of a bidirectional matching method according to an embodiment of the present application. As shown in FIG. 2, the bidirectional matching method searches for the closest match between two predicted image blocks in different reference images in the motion trajectory of the current CU to derive the motion vector of the current CU. The bidirectional matching method is based on the assumption that the motion trajectory of the current image block is continuous. The motion vectors MV0 and MV1 of the two predicted image blocks are proportional to the time domain distances (TD0 and TD1) between the current image and the two reference images (reference image 0 and reference image 1). The motion vector candidate list is scanned, and for example, for MV0, a ​​motion vector pair MV0 and MV1 can be generated. Here, MV1 can be generated based on MV0, TD0, and TD1. If the motion vector pair corresponding to MV0 points to the smallest distortion between the two predicted image blocks, this motion vector (i.e., MV0) is the motion vector of the current CU. If TD0=TD1, the bidirectional matching changes to bidirectional matching based on mirror images.

[0096] It should be understood that if one of the two reference images is a specific reference image, the temporal distance between the current image (the image currently being coded or the image currently being decoded) and the specific reference image is unclear, and therefore bidirectional matching cannot be performed. Moreover, the specific reference image is artificially constructed or derived from a specific reference image that is earlier in time order. Since there is not necessarily a spatial relationship between the image blocks in the specific reference image, and there is obvious shaking at the image block edges, it is not very meaningful for bidirectional matching to search for a motion vector based on such a specific reference image. In a possible embodiment of the present application, the candidate motion vector corresponding to the specific reference image is not added to the CU-level motion vector candidate list, thereby avoiding the above-mentioned problem.

[0097] Fig. 3 is a principle diagram of the template matching method of one embodiment of the present application. As shown in Fig. 3, the template matching method searches for the closest match between the template of the current image (at least one of the upper image block and the left image block of the current CU) and a block in the reference image (same size as the template) to derive the motion vector of the current CU. After obtaining the template, if the distortion between the template and the predicted image block to which the motion vector of the candidate points is the smallest, the motion vector of the candidate is the motion vector of the current CU.

[0098] When the reference image is a specific reference image, the specific reference image is artificially constructed or is derived from a specific reference image that is earlier in time order, and there is not necessarily a spatial relationship between the image blocks in the specific reference image, and there is obvious shaking at the image block edges, so there is little point in searching for a motion vector based on such a specific reference image, that is, the result of the template matching method is inaccurate and therefore meaningless. In a possible embodiment of the present application, the candidate motion vector corresponding to the specific reference image is not added to the CU-level motion vector candidate list, thereby avoiding the above-mentioned problem.

[0099] In a specific embodiment, the image motion compensation method 100 of the embodiment of the present application can be applied to DMVR technology.

[0100] The DMVR technique is a refinement technique used to make a more accurate prediction for a current image block when performing bidirectional prediction. The DMVR technique will be described in detail below, taking the decoding end as an example. The DMVR technique mainly includes two major steps: the first step is to construct a template based on a decoded image block corresponding to a plurality of initial motion vectors; and the second step is to modify the plurality of initial motion vectors based on the template.

[0101] Specifically, the decoding end can generate a motion vector candidate list. For example, this motion vector candidate list may be the motion vector candidate list of the AMVP mode described above or the motion vector candidate list of the Merge template. The decoding end can receive multiple indexes for indicating initial motion vectors sent from the encoding end. The decoding end obtains multiple initial motion vectors from the motion vector candidate list based on the indexes. The decoding end generates templates (e.g., by weighted summing of pixels) based on the decoded image blocks corresponding to the multiple initial motion vectors, and modifies the multiple initial motion vectors using the generated templates. Finally, the decoding end performs motion compensation on the current image block based on the modified motion vectors.

[0102] In this specific embodiment, the initial motion vector can include a first initial motion vector and a second initial motion vector. If a specific reference picture exists in the initial motion vector, the following processing can be adopted: S120: When the reference picture pointed to by the current initial motion vector is a specific reference picture, performing motion compensation on the current image block based on the initial motion vector can include: when the reference picture pointed to by at least one of the first and second initial motion vectors is a specific reference picture, performing motion compensation on the current image block based on the first and second initial motion vectors; and when the reference pictures pointed to by the first and second initial motion vectors are both non-specific reference pictures, modifying the first and second initial motion vectors to obtain modified first and second motion vectors, and performing motion compensation on the current image block based on the modified first and second motion vectors. If a specific reference picture does not exist in the initial motion vector, i.e., when both initial motion vectors are non-specific reference pictures, an existing DMVR processing method can be adopted.

[0103] 4 is a principle diagram of the DMVR technology of one embodiment of the present application. Specifically, assuming that the initial motion vectors include a first initial motion vector (e.g., MV0) and a second initial motion vector (e.g., MV1), and the decoded image block corresponding to the first initial motion vector belongs to the first decoded image block of the first frame, the first frame may be the first reference image, and the first decoded image block may be the first reference image block. Assuming that the decoded image block corresponding to the second motion vector belongs to the second decoded image block of the second frame, the second frame may be the second reference image, and the second decoded image block may be the second reference image block. A template can be obtained by weighted summing the first reference image block and the second reference image block. Here, this template can be referred to as a bidirectional template.

[0104] Optionally, in one embodiment, the initial motion vectors include a first initial motion vector and a second initial motion vector. Modifying the first initial motion vector and the second initial motion vector to obtain a modified first motion vector and a modified second motion vector can include generating a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial motion vector and belongs to the first reference image, and the second reference image block corresponds to the second initial motion vector and belongs to the second reference image, and modifying the first initial motion vector and the second initial motion vector based on the template to obtain a modified first motion vector and a modified second motion vector.

[0105] Specifically, modifying the first initial motion vector and the second initial motion vector based on the template and obtaining the modified first motion vector and the modified second motion vector may include: using N third reference image blocks to match with the template, respectively, where the N third reference image blocks correspond to N third initial motion vectors and belong to the first reference image; using M fourth reference image blocks to match with the template, respectively, where the M fourth reference image blocks correspond to M fourth initial motion vectors and belong to the second reference image; and selecting one third initial motion vector from the N third initial motion vectors and one fourth initial motion vector from the M fourth initial motion vectors based on the matching result, and using the one third initial motion vector and the one fourth initial motion vector as the motion vector of the current image block (i.e., the modified first motion vector and the modified second motion vector) or to determine the motion vector of the current image block.

[0106] Optionally, the selected third initial motion vector may be the motion vector corresponding to the smallest distortion cost, or the selected third initial motion vector may be the motion vector corresponding to a distortion cost less than a certain value.

[0107] Optionally, the selected fourth initial motion vector may be the motion vector corresponding to the smallest distortion cost, or the selected fourth initial motion vector may be the motion vector corresponding to a distortion cost less than a certain value.

[0108] Here, the one third initial motion vector and the one fourth initial motion vector are set as the motion vectors of the current image block, and in this case, a predicted image block can be obtained by weighted summing known image blocks (i.e., initial predicted image blocks) corresponding to the one third initial motion vector and the one fourth initial motion vector.

[0109] Alternatively, the one third initial motion vector and the one fourth initial motion vector can be used to determine the motion vector of the current image block, that is, the one third initial motion vector and the one fourth initial motion vector can be MVPs, respectively. In this case, the third MVP can be used as a starting point for search optimization to obtain an optimized motion vector, and the fourth MVP can be used as a starting point for search optimization to obtain another optimized motion vector. A predicted image block can be obtained by weighted summing known image blocks (i.e., initial predicted image blocks) corresponding to the two optimized motion vectors.

[0110] Alternatively, N and M may be equal.

[0111] Optionally, the third initial motion vector includes the first initial motion vector, and the fourth initial motion vector includes the second initial motion vector, i.e., the reference image block corresponding to the first initial motion vector for generating a template and the reference image block corresponding to the second motion vector also need to be matched with the template, respectively.

[0112] Optionally, in an embodiment of the present application, at least some of the N third initial motion vectors are obtained by shifting based on the first initial motion vector, and at least some of the M fourth initial motion vectors are obtained by shifting based on the second initial motion vector.

[0113] For example, the initial motion vectors other than the first initial motion vector among the N third initial motion vectors are obtained by shifting based on the first initial motion vector, for example, N may be equal to 9, and eight of the initial motion vectors are obtained by shifting based on the first initial motion vector, for example, by shifting in eight directions, or by shifting different pixels in the vertical or horizontal direction.

[0114] For example, the initial motion vectors other than the second initial motion vector among the N fourth initial motion vectors are obtained by shifting based on the second initial motion vector, for example, N may be equal to 9, and eight of the initial motion vectors are obtained by shifting based on the second initial motion vector, for example, by shifting in eight directions, or by shifting different pixels in the vertical or horizontal direction.

[0115] Optionally, in the embodiments of the present application, the first reference image may be a forward frame of the current image block and the second reference image may be a backward frame of the current image block, or the first reference image may be a forward frame of the current image block and the second reference image may be a forward frame of the current image block.

[0116] In a specific embodiment, the decoding end generates a motion vector candidate list and receives two indexes for indicating the initial motion vectors sent from the encoding end. The decoding end determines the DMVP condition, which requires that neither of the two initial motion vectors (for example, MV0 and MV1) points to a specific reference picture, and the prediction directions of the two initial motion vectors are opposite, i.e., one is forward and the other is backward. If the DMVP condition is met, the image block corresponding to MV0 and the image block corresponding to MV1 are weighted and summed to generate a bidirectional template.

[0117] In a possible embodiment of the present application, the candidate motion vector corresponding to the specific reference picture is not added to the motion vector candidate list, thereby preventing the reference picture pointed to by the initial motion vector indicated by the two indexes from being the specific reference picture. In another possible embodiment, if the reference picture pointed to by the initial motion vector indicated by the two indexes is the specific reference picture, the decoding end can request the encoding end to again indicate a motion vector that does not point to the specific reference picture as the initial motion vector, or the decoding end returns or marks the DMVR algorithm as invalid.

[0118] In the first reference image, for an image block corresponding to MV0, one luminance pixel is shifted in at least one of the horizontal and vertical directions, and the motion vectors of the retrieved eight adjacent pixel blocks can form a reference list list0 together with a total of nine motion vectors of MV0. In the second reference image, for an image block corresponding to MV1, one luminance pixel is shifted in at least one of the horizontal and vertical directions, and the motion vectors of the retrieved eight adjacent pixel blocks can form a reference list list1 together with a total of nine motion vectors of MV1.

[0119] The decoding end uses bidirectional template matching to perform distortion-based search between the bidirectional template and the reconstructed block in the reference image, ultimately obtaining a refined motion vector without additional motion information. Among the motion vectors in the two reference images (the motion vectors in List0 and the motion vectors in List1), the motion vector with the smallest matching cost replaces the original motion vector as the updated motion vector. Finally, two new motion vectors (MV0' and MV1' shown in FIG. 3) replace the original MV0 and MV1. The final bidirectional prediction result is generated based on the predicted image block corresponding to MV0' and the predicted image block corresponding to MV1'.

[0120] It should be understood that in the DMVR technology, when the reference image is a specific reference image, the specific reference image is artificially constructed or is derived from a specific reference image that is earlier in time order, and there is not necessarily a spatial relationship between the image blocks in the specific reference image, and there is obvious shaking at the image block edges, so there is little point in searching for a motion vector based on such a specific reference image, that is, the DMVR technology is inaccurate and therefore meaningless.In the embodiment of the present application, the motion vector corresponding to the specific reference image is not refined but is directly used for motion compensation, thereby avoiding the above-mentioned problems.

[0121] An embodiment of the present application provides an image processing method 200. Figure 5 is a schematic flowchart of the image processing method 200 of another embodiment of the present application. As shown in Figure 5, the method 200 includes: Obtaining a first initial motion vector and a second initial motion vector S210, where the first initial motion vector points to a first reference image and the second initial motion vector points to a second reference image; If at least one of the first reference image and the second reference image is a specific reference image, calculating a predicted image block of the current image block according to the first initial motion vector and the second initial motion vector (S220); If both the first reference image and the second reference image are non-specific reference images, the method includes: calculating and obtaining a motion vector of a current image block based on the gradient values ​​of pixel points pointed to by the first initial motion vector and the second initial motion vector; and calculating and obtaining a predicted image block of the current image block based on the motion vector of the current image block (S230).

[0122] In the image processing method of the embodiment of the present application, when the reference pictures pointed to by the two initial motion vectors are both non-specific reference pictures, the motion vector of the current image block is calculated by using the gradient value of the pixel point and the optimization principle, and the predicted image block of the current image block is calculated.When a specific reference picture exists in the reference pictures pointed to by the two initial motion vectors, the predicted image block of the current image block is directly calculated based on the two initial motion vectors, so as to avoid prediction failure due to the unclear definition of the time distance between the current image and the specific reference picture, and to improve the coding / decoding efficiency.

[0123] Optionally, S230 calculating and obtaining an MV of the current image block based on the gradient values ​​of pixel points to which the first initial MV and the second initial MV are directed may include calculating and obtaining an MV of the current image block based on the first initial MV, the gradient values ​​of pixel points to which the second initial MV are directed, and an optimization principle.

[0124] In a specific embodiment, the image processing method 200 of the embodiment of the present application can be applied to an improved bidirectional motion prediction mode. The bidirectional motion prediction mode refers to encoding a current image block by simultaneously using two initial predicted image blocks from two different reference images to predict the current image block, and combining the two initial predicted image blocks into one image block to form a predicted image block for the current image block. The BIO prediction technology is a technology used in the bidirectional motion prediction mode. In the BIO prediction technology, the motion vector has fewer coding marks than in the conventional bidirectional motion prediction mode, but the process of deriving the predicted image block is different. The BIO prediction technology is motion optimization based on block motion compensation, calculates motion vectors using optical flow mode, and is motion optimization at the sampling point level.

[0125] The BIO prediction technique of the embodiment of the present application can include two steps. The first step is to calculate and obtain an MV of a current image block based on two initial MVs (e.g., a first initial MV and a second initial MV). Specifically, the MV of the current image block is calculated based on the gradient values ​​of pixel points pointed to by the first initial MV and the second initial MV and an optimization principle. The second step is to calculate and obtain a predicted image block of the current image block based on the MV of the current image block, the first reference image block pointed to by the first initial MV, and the second reference image block pointed to by the second initial MV. Specifically, the BIO prediction technique interpolates two reference image blocks pointed to by two initial MVs to obtain two initial predicted image blocks of the same size as the current image block, and then weighted-sums the two initial predicted image blocks and combines them to form the predicted image block of the current image block.

[0126] FIG. 6 is a schematic diagram of the principle of the BIO technology according to one embodiment of the present application. The BIO technology assumes that the movement of an object is uniform in both the horizontal and vertical directions within a short time period, and conforms to the analytical conditions of the optical flow method. As shown in FIG. 6, the motion vector (v) of the reference point k (k=0, 1) to which the two initial MVs point is calculated. x , v y ), and the brightness value I (k) can be assumed to conform to the optical flow formula: ∂I (k) / ∂t+v x ∂I (k) / ∂x+v y ∂I (k) / ∂y=0 Formula 1 In the formula, ∂I (k) / ∂x and ∂I (k) / ∂y denotes the horizontal and vertical components of the gradient.

[0127] The optical flow formula and Hermite are interpolated to obtain the following polynomial, which is the BIO predicted value when t=0. pred BIO =1 / 2 (I(0) +I (1) +v x / 2·(τ1∂I (1) / ∂x-τ0∂I (0) / ∂x)+v y / 2·(τ1∂I (1) / ∂y-τ0∂I (0) / ∂y)) Formula 2 In the formula, τ0 and τ1 represent the distances from the current image to reference image 0 and reference image 1, respectively, and both can be calculated using the POCs of the current image and the two reference images. τ0=POC(current)-POC(Ref0) Formula 3 τ1=POC(Ref1)-POC(current) Formula 4 where POC(current) is the picture order count of the current picture, POC(Ref0) is the picture order count of reference picture 0, and POC(Ref1) is the picture order count of reference picture 1.

[0128] The reference images can be of different orientations, i.e., one from the past and one from the future. Two reference images can also be of the same orientation, i.e., both from the past or both from the future. If two reference images have the same orientation, τ0 and τ1 have opposite signs. In such a situation, the two reference images cannot be the same, i.e., τ0 ≠ τ1, and the reference region has a non-zero motion (MV x0 ,MV y0 ,MV x1 ,MV y1 ≠0), and the motion vector is proportional to the distance in the time domain (MV x0 / MV x1 =MV y0 / MV y1 =-τ0 / τ1).

[0129] Assuming that the motion within a small region is the same, the BIO motion vector can satisfy the following first-order Taylor expansion formula: Δ=(I (0) -I (1) +v x (τ1∂I(1) / ∂x+τ0∂I (0) / ∂x)+v y (τ1∂I (1) / ∂y+τ0∂I (0) / ∂y)) Formula 5 In the formula, Δ is the Taylor linear expansion of the pixel difference between two corresponding reference points (e.g., points A and B in FIG. 4) in two reference images. By analyzing the gradient value and the optimization method, the optimal motion vector of the current image block satisfies the square of Δ within the entire region and is the minimum, so that the optimal motion vector (v x , v y ) can be calculated, and considering the robustness of the algorithm, v x and v y is assumed to be within a certain threshold range.

[0130] Based on the above formula, the process of BIO prediction technology is as follows: For the current image block, the pixel values ​​of the corresponding positions of the two reference images have already been obtained. (0) and I (1) indicate the pixel values ​​in the two reference images, respectively, and the known values ​​in the above formula are I (0) , I (1) and τ0 and τ1, the horizontal and vertical components of the gradient are calculated in the reference image, and the unknowns are v x , v y and Δ. For all pixel points in a region, one Δ can be calculated, and then, using an optimization method, Δ is minimized to v x and v y , which is the optimal motion vector we need. x and v y When calculating v x and v yAn interval is given to the current image, and the threshold of this interval is determined according to the direction of the two reference images relative to the current image. After obtaining the optimal motion vector, the image block to which this optimal motion vector points plus the residual is not directly taken as the current image block, but each pixel of the current image block is calculated according to formula 2, which is also called BIO predicted value, and the predicted value of each pixel is combined together to form a predicted image block.

[0131] Optionally, in an embodiment of the present application, calculating and obtaining a predicted image block of the current image block based on the first initial motion vector and the second initial motion vector includes weighted summing a first reference image block pointed to by the first initial motion vector and a second reference image block pointed to by the second initial motion vector to obtain a predicted image block of the current image block.

[0132] Specifically, a judgment condition is added before starting BIO prediction, or a necessary condition is added to the BIO condition. If the first reference image pointed to by the first initial motion vector and the second reference image pointed to by the second initial motion vector are not specific reference images and meet the original BIO condition, BIO prediction can be performed. Otherwise, a predicted image block of the current image block is directly calculated based on the two initial motion vectors, or the BIO prediction algorithm is marked as invalid.

[0133] The BIO condition may further include that the prediction directions of the first initial motion vector and the second initial motion vector are different. Alternatively, the BIO condition may further include that the prediction directions of the first initial motion vector and the second initial motion vector are the same, neither of the first initial motion vector and the second initial motion vector is zero, and the first reference image and the second reference image are different. At the same time, the ratios of the directional components of the first initial motion vector and the second initial motion vector are the same, and are both equal to the ratio of the distance between the first reference image to which the motion vector is directed and the current image and the distance between the second reference image and the current image.

[0134] It is understood that the motion vector in each embodiment of the present application includes three parameters: a horizontal component v x , vertical component v y , and a frame mark of the reference image to which it points. For example, the frame mark may be a POC or other types of mark. The encoding end and the decoding end can determine the attribute of the reference image by the frame mark and determine whether the reference image is a specific reference image.

[0135] Optionally, in an embodiment of the present application, determining whether the reference picture to which the initial motion vector points is a specific reference picture may include determining whether the reference picture to which the initial motion vector points is a specific reference picture based on a frame mark of the reference picture to which the initial motion vector points.

[0136] Optionally, in an embodiment of the present application, calculating and obtaining a predicted image block of the current image block based on the motion vector of the current image block in S230 may include calculating and obtaining a predicted image block of the current image block based on the motion vector of the current image block, a first reference image block pointed to by the first initial motion vector, and a second reference image block pointed to by the second initial motion vector. The specific calculation formula has been described in detail above and will not be further described here.

[0137] A specific embodiment of the method 200 may include the following steps.

[0138] 1. Determine whether the current two initial motion vectors meet the BIO conditions. The conditions for determination are as follows:

[0139] a) The reference pictures to which the two initial motion vectors point are both non-specific reference pictures, i.e., the first reference picture to which the first initial motion vector points and the second reference picture to which the second initial motion vector points are both non-specific reference pictures.

[0140] b) One of the following two conditions is met: i) The two initial motion vectors are from different prediction directions (forward and backward, respectively). ii) The two initial motion vectors are from different reference pictures with the same prediction direction, and neither of the two initial motion vectors is zero, and the ratios of the directional components of the two initial motion vectors are the same, and both are equal to the ratio of the distance between the reference picture to which the motion vector points and the current picture.

[0141] An initial motion vector that simultaneously satisfies a) and b) satisfies the BIO condition and allows BIO prediction.

[0142] 2. Determine the calculation threshold value according to the prediction direction of the two initial motion vectors, i.e., v x and v y is the threshold for the interval.

[0143] 3. Calculate the gradient values ​​of the pixel points pointed by the two initial motion vectors.

[0144] 4. Based on the gradient value and the optimization principle, calculate the optimal motion vector, which is the motion vector of the current image block.

[0145] 5. Obtain a BIO prediction value according to the motion vector of the current image block and the reference image blocks pointed to by the two initial motion vectors.

[0146] If only b) i) is satisfied, a first reference image block pointed to by the first initial motion vector and a second reference image block pointed to by the second initial motion vector are weighted-summed to obtain a predicted image block of the current image block.

[0147] The principles, formulas, and steps referred to in the BIO predictions in the examples of the present application can be referred to in the above descriptions and will not be further described here.

[0148] It should be understood that in the BIO prediction technique, if the reference image is a specific reference image, the definition of the time distance between the current image (the image currently being coded or the image currently being decoded) and the specific reference image is unclear, so the BIO prediction technique cannot be implemented. In the embodiment of the present application, if the reference image pointed to by two initial motion vectors includes a specific reference image, the predicted image block of the current image block is directly calculated based on the two initial motion vectors, thereby avoiding the above-mentioned problems.

[0149] It should be understood that the methods in the embodiments of the present application can be applied to both the encoding end and the decoding end, and the current image block in the embodiments of the present application can be either the image block to be encoded or the image block to be decoded.

[0150] It should be understood that the embodiments of the present application merely exemplify image processing methods applied to PMMVD technology, DMVR technology, and BIO prediction technology, and that the methods of the embodiments of the present application can also be applied to other existing or future video encoding / decoding technologies, and that the embodiments of the present application are not limited thereto.

[0151] 7 is a schematic frame diagram of an image motion compensation device 70 according to one embodiment of the present application. As shown in FIG. 7, the image motion compensation device 70 includes: at least one memory 701 for storing computer executable commands; and at least one processor 702 adapted to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining an initial motion vector MV of a current image block; If the reference image indicated by the initial motion vector is a specific reference image, performing motion compensation on the current image block based on the initial motion vector; If the reference picture to which the initial MV points is a non-specific reference picture, modify the initial MV to obtain a modified MV, and perform motion compensation on the current image block based on the modified MV.

[0152] In some embodiments, the processor 702 specifically: Obtain a MV candidate list for the current image block, and a reference image to which any candidate MV in the MV candidate list points is a non-specific reference image; determining the initial MV based on the MV candidate list; The initial MV is modified to obtain a modified MV, and performing motion compensation on the current image block based on the modified MV.

[0153] In some embodiments, the processor 702 further comprises: A candidate MV is determined to be added to the MV candidate list, and if the reference image to which the candidate MV points is a non-specific reference image, the candidate MV is added to the MV candidate list.

[0154] In some embodiments, the processor 702 further comprises: The purpose is to determine that the reference picture to which the candidate MV points is a non-specific reference picture based on the frame mark of the reference picture to which the candidate MV points.

[0155] In some embodiments, the MV candidate list includes at least one of the following candidate MVs: If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for a non-specific reference image; The merge candidate MV obtained for the non-specific reference image, MVs obtained by interpolation from non-specific reference images, and The upper and left neighboring MVs for the non-specific reference image of the current block.

[0156] In some embodiments, the processor 702 specifically: determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method; This is to select the MV with the smallest distortion cost in the MV candidate list as the initial MV.

[0157] In some embodiments, the processor 702 specifically: Generate a sub-MV candidate list for the current image block, the sub-MV candidate list including the initial MV; The MV with the smallest distortion cost from the sub-MV candidate list is determined as the modified MV.

[0158] In some embodiments, the initial MV includes a first initial MV and a second initial MV; Specifically, the processor 702: performing motion compensation on the current image block based on the first initial MV and the second initial MV when a reference image indicated by at least one of the first initial MV and the second initial MV is a specific reference image; When the reference images to which the first initial MV and the second initial MV are directed are non-specific reference images, the first initial MV and the second initial MV are modified to obtain a modified first MV and a modified second MV, and motion compensation is performed on the current image block based on the modified first MV and the modified second MV.

[0159] In some embodiments, the processor 702 specifically: Generate a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial MV and belongs to a first reference image, and the second reference image block corresponds to the second initial MV and belongs to a second reference image; The first initial MV and the second initial MV are modified based on the template to obtain a modified first MV and a modified second MV.

[0160] In some embodiments, the processor 702 specifically: Using N third reference image blocks to match the template respectively, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; Using M fourth reference image blocks to match the template respectively, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; Based on the matching result, one third initial MV is selected from the N third initial MVs, and one fourth initial MV is selected from the M fourth initial MVs, and the one third initial MV and the one fourth initial MV are used as the MV of the current image block or to determine the MV of the current image block.

[0161] In some embodiments, the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV.

[0162] In some embodiments, at least some of the N third initial MVs are obtained by shifting based on the first initial MVs, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MVs.

[0163] In some embodiments, N is equal to M.

[0164] In some embodiments, the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block; or The first reference image is a forward frame of the current image block, and the second reference image is a forward frame of the current image block.

[0165] It is to be understood that the image motion compensation unit 70 can also be realized by corresponding software modules, which will not be further described here.

[0166] 8 is a schematic block diagram of an image processing device 80 according to another embodiment of the present application. As shown in FIG. 8, the image processing device 80 includes: at least one memory 801 for storing computer executable commands; and at least one processor 802 adapted to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining a first initial motion vector MV and a second initial motion vector MV, the first initial motion vector MV pointing to a first reference image, and the second initial motion vector MV pointing to a second reference image; When at least one of the first reference image and the second reference image is a specific reference image, calculating and obtaining a predicted image block of the current image block based on the first initial MV and the second initial MV; If the first reference image and the second reference image are both non-specific reference images, the MV of the current image block is calculated based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and the predicted image block of the current image block is calculated based on the MV of the current image block.

[0167] In some embodiments, the processor 802 specifically: The MV of the current image block is calculated based on the gradient values ​​of the pixel points to which the first initial MV and the second initial MV are directed and an optimization principle.

[0168] In some embodiments, the processor 802 specifically: A first reference image block pointed to by the first initial MV and a second reference image block pointed to by the second initial MV are weighted and summed to obtain a predicted image block of the current image block.

[0169] In some embodiments, the prediction directions of the first initial MV and the second initial MV are different.

[0170] In some embodiments, the prediction directions of the first initial MV and the second initial MV are the same, neither the first initial MV nor the second initial MV is 0, and the first reference image and the second reference image are different.

[0171] In some embodiments, the processor 802 specifically: It is for calculating and obtaining a predicted image block of the current image block based on the MV of the current image block, the first reference image block to which the first initial MV points, and the second reference image block to which the second initial MV points.

[0172] It is to be understood that the image processing device 80 may further be realized by corresponding software modules, which will not be further described here.

[0173] In some techniques using motion vector derivation, if a motion vector points to a specific reference picture, scaling of this motion vector is meaningless, which reduces search efficiency and encoding / decoding efficiency. This is because, when scaling a motion vector, the motion vector is scaled based on the distance between images, while a specific reference picture may be artificially constructed and may not have a distance from other reference pictures, so scaling a motion vector based on such a specific reference picture is not very meaningful.

[0174] The present application further provides an image motion compensation method. Figure 9 is a schematic flowchart of an image motion compensation method 900 according to another embodiment of the present application. As shown in Figure 9, the method 900 includes: S910 obtaining an initial motion vector MV of the current image block; S920: determining a scaling ratio of the initial MV, where if the initial MV is directed to a specific reference image, the scaling ratio of the initial MV is 1; S930 scales the initial MV based on a scaling ratio of the initial MV; and S940 performing motion compensation on the current image block based on the scaled MV.

[0175] Here, the method for obtaining the initial MV may be the same as the method for obtaining the initial MV described above, and will not be further described here.

[0176] In the image motion compensation method of the embodiment of the present application, when the initial MV points to a specific reference image, its scaling ratio is 1. When the initial MV points to a non-specific reference image, its scaling ratio is not limited, and the motion vector can be scaled based on the distance between the current image and the non-specific reference image and the scaling strategy, thereby performing motion compensation and improving the coding / decoding efficiency.

[0177] In some embodiments, motion compensation is performed directly on the current image block based on the scaled MV.

[0178] In some embodiments, for the scaled MV, the scaled MV can be further modified to obtain a modified MV, and then perform motion compensation on the current image block according to the modified MV, where the method for modifying the scaled MV may be the same as the method for modifying the initial MV described above, and will not be further described here.

[0179] For example, as shown in FIG. 5, the first initial MV points to a first reference image, and the second initial MV points to a second reference image. If both the first and second reference images point to a non-specific reference image, the first and second initial MVs are scaled based on the distance between the first and second reference images. Then, motion compensation is performed on the current image block based on the scaled first and second initial MVs. In one embodiment, a first reference image block is determined, and the first reference image block corresponds to the scaled first initial MV and belongs to the second reference image. A second reference image block is determined, and the second reference image block corresponds to the scaled second initial MV and belongs to the first reference image. A template is generated based on the first and second reference image blocks, and the first and second initial MVs are modified based on the template to obtain the modified first and second MVs. The motion vector of the current image block is calculated using the modified first and second MVs.

[0180] Here, the method for modifying the first initial MV and the second initial MV based on this template may be the same as the method for modifying the first initial MV and the second initial MV based on the template described above, and will not be described further here.

[0181] This application provides another image motion compensation method, in which an initial motion vector MV of a current image block is obtained, and when the initial MV points to a specific reference image and when the initial MV points to a non-specific reference image, different operations are adopted to perform motion compensation on the current image block based on the initial MV.

[0182] Here, the method for obtaining the initial MV may be the same as the method for obtaining the initial MV described above, and will not be further described here.

[0183] In one embodiment, when the initial MV points to a specific reference image, the initial MV is used to perform motion compensation on the current image block. When the initial MV points to a non-specific reference image, there are four embodiments: 1) an embodiment in which the initial MV is scaled and the current image block is motion compensated based on the scaled initial MV; 2) an embodiment in which the initial MV is modified and the current image block is motion compensated based on the modified initial MV; 3) an embodiment in which the initial MV is scaled and the scaled MV is modified to obtain the modified MV, and the current image block is motion compensated based on the modified MV; and 4) an embodiment in which the initial MV is modified and the modified MV is scaled and the scaled MV is scaled to obtain the scaled MV, and the current image block is motion compensated based on the scaled MV.

[0184] Here, the method for modifying the scaled MV or the method for modifying the initial MV may be the same as the method for modifying the initial MV described above, and will not be further described here.

[0185] In one embodiment, when the initial MV points to a specific reference picture, the initial MV can be modified and motion compensation can be performed based on the modified MV. When the initial MV points to a non-specific reference picture, the initial MV can be scaled, or the initial MV can be scaled and modified before motion compensation is performed on the current image block. That is, when the initial MV points to a specific reference picture, the scaling step can be skipped and the initial MV can be directly modified, or the initial MV can be directly scaled and modified after the scaling ratio is set to 1. When the initial MV points to a non-specific reference picture, the initial MV can be scaled, or the initial MV can be scaled and modified to improve encoding and decoding efficiency.

[0186] In other embodiments of the present application, when the initial MV points to a specific reference image, the initial MV is directly used for motion compensation, or the initial MV is modified and motion compensation is performed based on the modified MV; and when the initial MV points to a non-specific reference image, the initial MV is scaled and / or modified for motion compensation; various embodiments may be arbitrarily combined, and the embodiments of the present application are not limited thereto.

[0187] In some embodiments, obtaining an initial motion vector MV of the current image block includes obtaining an MV candidate list for the current image block, and determining the initial MV based on the MV candidate list. That is, after scaling the initial MV (including scaling with a scaling ratio of 1), an MV pointing to a non-specific reference picture may be selected and modified, and regardless of which type of reference picture the initial MV points to, any of them may be modified.

[0188] Optionally, the reference picture to which any candidate MV in the MV candidate list points is a non-specific reference picture.

[0189] Optionally, obtaining an MV candidate list for the current image block may include determining a candidate MV to add to the MV candidate list, and adding the candidate MV to the MV candidate list if the reference image to which the candidate MV points is a non-specific reference image.

[0190] Optionally, the method may further include determining, based on a frame mark of the reference image to which the candidate MV is directed, that the reference image to which the candidate MV is directed is a non-specific reference image.

[0191] Optionally, the MV candidate list may include at least one of the following candidate MVs: an original AMVP candidate MV obtained for a non-specific reference picture when the current image block is in advanced motion vector prediction AMVP mode; a merge candidate MV obtained for a non-specific reference picture; an MV obtained by interpolation from a non-specific reference picture; and an upper adjacent MV and a left adjacent MV for a non-specific reference picture of the current block.

[0192] Optionally, determining the initial MV based on the MV candidate list may include determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method, and selecting the MV with the smallest distortion cost in the MV candidate list as the initial MV.

[0193] Optionally, modifying the scaled MV and obtaining a modified MV may include generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the scaled MV, and determining the MV with the smallest distortion cost from the sub-MV candidate list as the modified MV.

[0194] Optionally, the initial MV includes a first initial MV and a second initial MV, and the method includes at least one of: when a reference image to which at least one of the first initial MV and the second initial MV points is a specific reference image, setting scaling ratios of the first initial MV and the second initial MV to 1, and performing motion compensation on the current image block based on the first initial MV and the second initial MV; when a reference image to which the first initial MV and the second initial MV point is a non-specific reference image, determining a scaling ratio of the first initial MV and a scaling ratio of the second initial MV, scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and performing motion compensation on the current image block based on the first initial MV after the scaling and the second initial MV after the scaling.

[0195] Optionally, performing motion compensation on the current image block based on the first initial MV after scaling and the second initial MV after scaling includes generating a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial MV after scaling and belongs to a second reference image, the second reference image block corresponds to the second initial MV after scaling and belongs to the first reference image, the first reference image is the reference image to which the first initial MV is directed, and the second reference image is the reference image to which the second initial MV is directed; and modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV.

[0196] Optionally, modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV may include: using N third reference image blocks to match with the template, respectively, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; and using M fourth reference image blocks to match with the template, respectively, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; and selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs based on the matching result, and using the one third initial MV and the one fourth initial MV as the MV of the current image block or to determine the MV of the current image block.

[0197] Optionally, the third initial MV may include the first initial MV, and the fourth initial MV may include the second initial MV.

[0198] Optionally, at least some of the N third initial MVs may be obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs may be obtained by shifting based on the second initial MV.

[0199] Optionally, N may be equal to M.

[0200] Optionally, the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block, or the first reference image is a forward frame of the current image block and the second reference image is a forward frame of the current image block.

[0201] It is understood that the above alternative embodiments can be implemented in similar manner to the details of method 100, and will not be further described here.

[0202] 10 is a schematic frame diagram of an image motion compensation device 1000 according to one embodiment of the present application. As shown in FIG. 10, the image motion compensation device 1000 includes: at least one memory 1001 for storing computer executable commands; and at least one processor 1002 adapted to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining an initial motion vector MV of a current image block; Determining a scaling ratio of the initial MV, where if the initial MV is directed to a specific reference image, the scaling ratio of the initial MV is 1; Scaling the initial MV based on a scaling ratio of the initial MV; performing motion compensation on the current image block based on the scaled MV.

[0203] In some embodiments, the processor 1002 performing motion compensation on the current image block based on the scaled motion vector comprises: modifying the scaled MV to obtain a modified MV; and performing motion compensation on the current image block based on the modified MV.

[0204] In some embodiments, the processor 1002 obtaining the initial motion vector MV of the current image block comprises: obtaining a list of MV candidates for the current image block; determining the initial MV based on the MV candidate list.

[0205] In some embodiments, the reference picture to which any candidate MV in the MV candidate list points is a non-specific reference picture.

[0206] In some embodiments, the processor 1002 obtaining the MV candidate list for the current image block comprises: The method includes determining a candidate MV to add to the MV candidate list, and adding the candidate MV to the MV candidate list if the reference image to which the candidate MV points is a non-specific reference image.

[0207] In some embodiments, the processor 1002 further comprises: The purpose is to determine that the reference picture to which the candidate MV points is a non-specific reference picture based on the frame mark of the reference picture to which the candidate MV points.

[0208] In some embodiments, the MV candidate list comprises: If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for a non-specific reference image; The merge candidate MV obtained for the non-specific reference image, MVs obtained by interpolation from non-specific reference images, and The candidate MV includes at least one of the upper neighboring MV and the left neighboring MV for the non-specific reference image of the current block.

[0209] In some embodiments, the processor 1002 determining the initial MV based on the MV candidate list includes: determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method; and selecting the MV with the smallest distortion cost in the MV candidate list as the initial MV.

[0210] In some embodiments, the processor 1002 modifying the scaled MV to obtain a modified MV comprises: generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the scaled MV; determining the MV with the smallest distortion cost from the sub-MV candidate list as the modified MV.

[0211] In some embodiments, the initial MV includes a first initial MV and a second initial MV, and the processor 1002 specifically: If the reference image pointed to by at least one of the first initial MV and the second initial MV is a specific reference image, setting the scaling ratios of the first initial MV and the second initial MV to 1, and performing motion compensation on the current image block based on the first initial MV and the second initial MV; When the reference images to which the first initial MV and the second initial MV are directed are both non-specific reference images, the method is for at least one of determining a scaling ratio of the first initial MV and a scaling ratio of the second initial MV, scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and performing motion compensation on the current image block based on the first initial MV after scaling and the second initial MV after scaling.

[0212] In some embodiments, the processor 1002 performing motion compensation on the current image block based on the scaled first initial MV and the scaled second initial MV includes: Generate a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to a first initial MV after scaling and belongs to a second reference image, the second reference image block corresponds to a second initial MV after scaling and belongs to the first reference image, the first reference image is a reference image directed by the first initial MV, and the second reference image is a reference image directed by the second initial MV; and modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV.

[0213] In some embodiments, the processor 1002 modifies the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV, Using N third reference image blocks to match with the template respectively, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; Using M fourth reference image blocks to match the template respectively, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; Based on the matching result, selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs, and using the one third initial MV and the one fourth initial MV as the MV of the current image block or to determine the MV of the current image block.

[0214] In some embodiments, the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV.

[0215] In some embodiments, at least some of the N third initial MVs are obtained by shifting based on the first initial MVs, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MVs.

[0216] In some embodiments, N is equal to M.

[0217] In some embodiments, the first reference image is a forward frame of the current image block and the second reference image is a backward frame of the current image block; or The first reference image is a forward frame of the current image block, and the second reference image is a forward frame of the current image block.

[0218] In some embodiments, the specific reference image comprises at least one of a long-term reference image, a synthesized frame, and a non-output frame.

[0219] In some embodiments, the processor 1002 further comprises: When determining that the reference image is a frame that will not be output, and further determining that the reference image is a composite frame, determining that the reference image is the specific reference image.

[0220] In some embodiments, the non-specific reference image comprises a short-term reference image.

[0221] It is to be understood that the image motion compensation device 1000 can also be realized by corresponding software modules, which will not be further described here.

[0222] It should be understood that the apparatus of each embodiment of the present application can be realized by a memory and a processor, each memory being for storing commands for executing the method of each embodiment of the present application, and the processor executing the commands, causing the apparatus to execute the method of each embodiment of the present application.

[0223] It should be understood that the processor referred to in the embodiments of this application may be a Central Processing Unit (CPU), or may be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), off-the-shelf Field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0224] It should be further understood that the memory referred to in the embodiments of the present application may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory. Here, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM) and is used as an external high-speed buffer memory. By way of example and not limitation, many forms of RAM may be used, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct Rambus RAM (DR RAM).

[0225] It should be noted that if the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) may be integrated into the processor.

[0226] Note that memory as described herein may include, without being limited to, these and any other suitable types of memory.

[0227] An embodiment of the present application further provides a computer-readable storage medium having stored thereon instructions that, when executed by a computer, cause the computer to perform the method of each of the method embodiments described above.

[0228] An embodiment of the present application further provides a computer device, which includes the computer-readable storage medium described above.

[0229] An embodiment of the present application further provides an encoding device, which includes at least one of an image motion compensation unit 70 , an image processing unit 80 , and an image motion compensation unit 1000 .

[0230] An embodiment of the present application further provides a decoding device, which includes at least one of an image motion compensation unit 70 , an image processing unit 80 , and an image motion compensation unit 1000 .

[0231] Embodiments of the present application have application in aviation, particularly in the field of unmanned aerial vehicles.

[0232] It should be understood that the division of the electrical circuits, sub-electrical circuits, and sub-units in each embodiment of the present application is only schematic, and those skilled in the art will recognize that each exemplary electrical circuit, sub-electrical circuit, and sub-unit illustrated by the embodiments disclosed herein can be separated or combined.

[0233] The above-described embodiments may be implemented, in whole or in part, by software, hardware, firmware, or any other combination. When implemented using software, they may be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer commands. When the computer commands are loaded or executed by a computer, they generate, in whole or in part, processes or functions according to the embodiments of the present application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer commands may be stored in a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer commands may be transmitted from a site, computer, server, or data center to another site, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, radio, microwave, etc.) methods. The computer-readable storage medium may be any available medium accessible by a computer, or may be a data storage device including a server, a data center, etc. in which one or more available media are integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)).

[0234] It should be understood that references throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic associated with an embodiment is included in at least one embodiment of the present application. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, these particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0235] In each embodiment of the present application, it should be understood that the magnitude of the numbers of the above-mentioned processes does not indicate the order of execution, and the execution order of each process is determined by its function and inherent logic, and does not impose any restrictions on the implementation process of the embodiments of the present application.

[0236] It should be understood that "B corresponding to A" in the examples of the present application indicates that B is related to A and B can be determined based on A. However, it should be understood that determining B based on A does not mean that B is determined only by A, and that B can be determined based on at least one of A and other information.

[0237] It should be understood that the term "at least one" in this document merely describes the relationship between related objects, and that three relationships can exist, for example, there are three situations for at least one of A and B: A exists alone, A and B exist simultaneously, or B exists alone. In addition, the symbol " / " in this document usually indicates that the related objects before and after it are in an "or" relationship.

[0238] Those skilled in the art can recognize that each exemplary unit and algorithm step described in the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are ultimately implemented in hardware or software is determined by the specific application of the technical solution and design constraints. Those skilled in the art can realize the described functions by using different methods for each specific application, but such realization should not be recognized as going beyond the scope of this application.

[0239] Those skilled in the art can clearly understand that, for ease of explanation and concise description, the specific operating processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be further described here.

[0240] It should be understood that the disclosed systems, devices, and methods in the various embodiments provided by this application can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a division of logical functions, and in actual implementation, other division methods may exist. For example, multiple units or components may be combined or integrated into other systems, or some features may be ignored or not implemented. Furthermore, the couplings, direct couplings, or communication connections between devices or units shown or discussed may be through some interfaces, and indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0241] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units, and some or all of the units can be selected according to actual needs to achieve the objectives of the solution of this embodiment.

[0242] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically alone, or two or more units may be integrated into one unit.

[0243] The above are only specific embodiments of the present application, and the scope of protection of the present application is not limited thereto. Anyone familiar with the art can easily think of modifications or replacements within the technical scope disclosed by the present application, and all of these shall be included in the scope of protection of the present application. Therefore, the scope of protection of the present application shall be in accordance with the scope of protection of the claims. (Item 1) 1. A method for image motion compensation, comprising: Obtaining an initial motion vector MV of a current image block; If the reference image indicated by the initial MV is a specific reference image, performing motion compensation on the current image block based on the initial MV; If the reference image to which the initial MV points is a non-specific reference image, modifying the initial MV to obtain a modified MV, and performing motion compensation on the current image block based on the modified MV. (Item 2) When the reference image indicated by the initial motion vector is a non-specific reference image, modifying the initial motion vector to obtain a modified motion vector, and performing motion compensation on the current image block according to the modified motion vector; Obtaining a list of candidate MVs for the current image block, and determining that the reference image pointed to by any candidate MV in the list of candidate MVs is a non-specific reference image; determining the initial MV based on the MV candidate list; Modifying the initial MV to obtain a modified MV; and performing motion compensation on the current image block based on the modified MV. (Item 3) To obtain the MV candidate list for the current image block, The method according to item 2, comprising determining a candidate MV to add to the MV candidate list, and adding the candidate MV to the MV candidate list if the reference image to which the candidate MV points is a non-specific reference image. (Item 4) The above method is 4. The method according to item 3, further comprising determining that the reference image to which the candidate MV is directed is a non-specific reference image based on a frame mark of the reference image to which the candidate MV is directed. (Item 5) The above MV candidate list is: If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for a non-specific reference image; The merge candidate MV obtained for the non-specific reference image, MVs obtained by interpolation from non-specific reference images, and The method according to any one of items 2 to 4, including at least one candidate MV among an upper neighboring MV and a left neighboring MV for the non-specific reference image of the current block. (Item 6) Determining the initial MV based on the MV candidate list is determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method; and selecting the MV with the smallest distortion cost in the MV candidate list as the initial MV. (Item 7) Modifying the initial MV and obtaining the modified MV is generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the initial MV; and determining the MV with the smallest distortion cost from the subMV candidate list as the modified MV. (Item 8) The initial MV includes a first initial MV and a second initial MV; When the reference image indicated by the initial MV is a specific reference image, performing motion compensation on the current image block based on the initial MV includes: When a reference image indicated by at least one of the first initial MV and the second initial MV is a specific reference image, performing motion compensation on the current image block based on the first initial MV and the second initial MV; The method described in item 1 includes, when the reference images to which the first initial MV and the second initial MV are directed are both non-specific reference images, modifying the first initial MV and the second initial MV to obtain a modified first MV and a modified second MV, and performing motion compensation on the current image block based on the modified first MV and the modified second MV. (Item 9) Modifying the first initial MV and the second initial MV to obtain a modified first MV and a modified second MV includes: Generate a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial MV and belongs to the first reference image, and the second reference image block corresponds to the second initial MV and belongs to the second reference image; Item 9. The method according to item 8, further comprising: modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV. (Item 10) modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV; Using N third reference image blocks to match with the template respectively, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; Using M fourth reference image blocks to match with the template respectively, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; The method described in item 9 includes selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs based on the matching result, and using the one third initial MV and the one fourth initial MV as the MV of the current image block or to determine the MV of the current image block. (Item 11) Item 11. The method according to item 10, wherein the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV. (Item 12) The method described in item 10 or 11, wherein at least some of the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MV. (Item 13) 13. The method according to any one of items 10 to 12, wherein N is equal to M. (Item 14) The first reference image is a forward frame of the current image block, and the second reference image is a backward frame of the current image block; or 14. The method according to any one of items 10 to 13, wherein the first reference image is a forward frame of the current image block, and the second reference image is a forward frame of the current image block. (Item 15) 15. The method according to any one of items 1 to 14, wherein the specific reference image includes at least one of a long-term reference image, a synthesized frame, and a frame that is not output. (Item 16) The above method is 16. The method according to any one of items 1 to 15, further comprising determining that the reference image is the specific reference image when the reference image is determined to be a frame that will not be output and the reference image is determined to be a composite frame. (Item 17) 17. The method according to any one of items 1 to 16, wherein the non-specific reference image comprises a short-term reference image. (Item 18) An image motion compensation device, comprising: at least one memory for storing computer executable commands; and at least one processor adapted, alone or jointly, to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining an initial motion vector MV of a current image block; If the reference image indicated by the initial MV is a specific reference image, performing motion compensation on the current image block based on the initial MV; If the reference picture pointed to by the initial MV is a non-specific reference picture, the initial MV is modified to obtain a modified MV, and motion compensation is performed on the current image block based on the modified MV. (Item 19) Specifically, the processor Obtain a list of candidate MVs for the current image block, and a reference image to which any candidate MV in the list of candidate MVs points is a non-specific reference image; Based on the MV candidate list, the initial MV is determined; Modify the initial MV above, and obtain the modified MV. Item 19. The image motion compensation device according to item 18, for performing motion compensation on the current image block based on the modified MV. (Item 20) The processor further comprises: An image motion compensation device as described in item 19, which determines a candidate MV to add to the MV candidate list, and adds the candidate MV to the MV candidate list if the reference image to which the candidate MV points is a non-specific reference image. (Item 21) The processor further comprises: An image motion compensation device according to any one of items 18 to 20, for determining that the reference image to which the candidate MV points is a non-specific reference image based on a frame mark of the reference image to which the candidate MV points. (Item 22) The above MV candidate list is: If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for a non-specific reference image; The merge candidate MV obtained for the non-specific reference image, MVs obtained by interpolation from non-specific reference images, and Item 22. The image motion compensation apparatus according to item 21, comprising at least one candidate MV among an upper neighboring MV and a left neighboring MV for the non-specific reference image of the current block. (Item 23) Specifically, the processor determining distortion costs corresponding to the candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method; Item 23. The image motion compensation device according to item 22, wherein the MV with the smallest distortion cost in the MV candidate list is set as the initial MV. (Item 24) Specifically, the processor Generate a sub-MV candidate list for the current image block, and the sub-MV candidate list includes the initial MV; 21. The image motion compensation device according to any one of items 18 to 20, wherein the image motion compensation device is configured to determine the image motion compensation device with the smallest distortion cost from the sub-image motion compensation candidate list as the corrected image motion compensation device. (Item 25) The initial MV includes a first initial MV and a second initial MV; Specifically, the processor If a reference image indicated by at least one of the first initial MV and the second initial MV is a specific reference image, performing motion compensation on the current image block based on the first initial MV and the second initial MV; An image motion compensation device as described in item 18, which, when the reference images pointed to by the first initial MV and the second initial MV are both non-specific reference images, modifies the first initial MV and the second initial MV to obtain a modified first MV and a modified second MV, and performs motion compensation on the current image block based on the modified first MV and the modified second MV. (Item 26) Specifically, the processor Generate a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial MV and belongs to a first reference image, and the second reference image block corresponds to the second initial MV and belongs to a second reference image; 26. An image motion compensation device according to item 25, for modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV. (Item 27) Specifically, the processor Using N third reference image blocks to match with the template respectively, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; Using M fourth reference image blocks to match with the template respectively, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; An image motion compensation device as described in item 26, which is for selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs based on the matching result, and using the one third initial MV and the one fourth initial MV as the MV of the current image block or for determining the MV of the current image block. (Item 28) 28. The image motion compensation device according to item 27, wherein the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV. (Item 29) An image motion compensation device described in item 27 or 28, wherein at least some of the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MV. (Item 30) 30. The image motion compensation device according to any one of items 27 to 29, wherein N is equal to M. (Item 31) The first reference image is a forward frame of the current image block, and the second reference image is a backward frame of the current image block; or 31. The image motion compensation device according to any one of items 27 to 30, wherein the first reference image is a forward frame of the current image block, and the second reference image is a forward frame of the current image block. (Item 32) 32. The image motion compensation device according to any one of items 18 to 31, wherein the specific reference image includes at least one of a long-term reference image, a synthesized frame, and a frame that is not output. (Item 33) The processor further comprises: 33. An image motion compensation device according to any one of items 18 to 32, for determining that the reference image is the specific reference image when the reference image is determined to be a frame that will not be output and the reference image is further determined to be a composite frame. (Item 34) 34. The image motion compensation device according to any one of items 18 to 33, wherein the non-specific reference image includes a short-term reference image. (Item 35) A computer-readable storage medium, A computer-readable storage medium having stored thereon commands that, when executed by a computer, cause the computer to execute the image motion compensation method according to any one of items 1 to 17. (Item 36) 35. An encoding device including the image motion compensation apparatus according to any one of items 18 to 34. (Item 37) A decoding device including the image motion compensation apparatus according to any one of items 18 to 34. (Item 38) 1. A method for image motion compensation, comprising: Obtaining an initial motion vector MV of a current image block; Determining a scaling ratio of the initial MV, where if the initial MV is directed to a specific reference image, the scaling ratio of the initial MV is 1; Scaling the initial MV based on a scaling ratio of the initial MV; and performing motion compensation on the current image block based on the scaled MV. (Item 39) performing motion compensation on the current image block based on the scaled MV, Correcting the scaled MV to obtain a corrected MV; Item 39. The method of item 38, comprising: performing motion compensation on the current image block based on the modified MV. (Item 40) Obtaining the initial motion vector MV of the current image block includes: Obtaining a list of MV candidates for the current image block; 40. The method according to item 38 or 39, comprising determining the initial MV based on the MV candidate list. (Item 41) Item 41. The method according to item 40, wherein the reference image to which any candidate MV in the MV candidate list points is a non-specific reference image. (Item 42) To obtain the MV candidate list for the current image block, Item 42. The method of item 41, comprising determining a candidate MV to add to the MV candidate list, and adding the candidate MV to the MV candidate list if the reference image to which the candidate MV points is a non-specific reference image. (Item 43) The above method is Item 43. The method according to item 42, further comprising determining that the reference image to which the candidate MV is directed is a non-specific reference image based on a frame mark of the reference image to which the candidate MV is directed. (Item 44) The above MV candidate list is: If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for a non-specific reference image; The merge candidate MV obtained for the non-specific reference image, MVs obtained by interpolation from non-specific reference images, and 44. The method according to any one of items 40 to 43, comprising at least one candidate MV among an upper neighboring MV and a left neighboring MV for the non-specific reference image of the current block. (Item 45) Determining the initial MV based on the MV candidate list is determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method; Item 45. The method according to item 44, comprising: selecting the MV in the MV candidate list with the smallest distortion cost as the initial MV. (Item 46) To correct the MV after scaling and obtain the corrected MV, generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the scaled MV; Item 46. The method according to item 45, further comprising determining the MV with the smallest distortion cost from the sub-MV candidate list as the modified MV. (Item 47) The initial MV includes a first initial MV and a second initial MV, and the method includes: If the reference image pointed to by at least one of the first initial MV and the second initial MV is a specific reference image, setting the scaling ratios of the first initial MV and the second initial MV to 1, and performing motion compensation on the current image block according to the first initial MV and the second initial MV; The method described in item 38 includes at least one of determining a scaling ratio of the first initial MV and a scaling ratio of the second initial MV when the reference images to which the first initial MV and the second initial MV are directed are both non-specific reference images, scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and performing motion compensation on the current image block based on the first initial MV after scaling and the second initial MV after scaling. (Item 48) performing motion compensation on the current image block based on the scaled first initial motion vector and the scaled second initial motion vector; Generate a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial MV after scaling and belongs to a second reference image, the second reference image block corresponds to the second initial MV after scaling and belongs to the first reference image, the first reference image is a reference image directed by the first initial MV, and the second reference image is a reference image directed by the second initial MV; Item 48. The method according to item 47, further comprising: modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV. (Item 49) modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV; Using N third reference image blocks to match with the template respectively, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; Using M fourth reference image blocks to match with the template respectively, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; Item 48. The method described in Item 47, comprising: based on the matching result, selecting one third initial MV from the N third initial MVs and selecting one fourth initial MV from the M fourth initial MVs, and using the one third initial MV and the one fourth initial MV as the MV of the current image block or to determine the MV of the current image block. (Item 50) Item 49. The method according to item 49, wherein the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV. (Item 51) A method as described in item 49 or 50, wherein at least some of the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MV. (Item 52) 52. The method according to any one of items 49 to 51, wherein N is equal to M. (Item 53) The first reference image is a forward frame of the current image block, and the second reference image is a backward frame of the current image block; or 53. The method according to any one of items 49 to 52, wherein the first reference image is a forward frame of the current image block, and the second reference image is a forward frame of the current image block. (Item 54) 54. The method according to any one of items 38 to 53, wherein the specific reference image includes at least one of a long-term reference image, a synthesized frame, and a frame that is not output. (Item 55) The above method is 55. The method according to any one of items 38 to 54, further comprising determining that the reference image is the specific reference image when the reference image is determined to be a frame that will not be output and the reference image is further determined to be a composite frame. (Item 56) 56. The method according to any one of items 38 to 55, wherein the non-specific reference image comprises a short-term reference image. (Item 57) An image motion compensation device, comprising: at least one memory for storing computer executable commands; and at least one processor adapted, alone or jointly, to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining an initial motion vector MV of a current image block; Determining a scaling ratio of the initial MV, where if the initial MV is directed to a specific reference image, the scaling ratio of the initial MV is 1; Scaling the initial MV based on a scaling ratio of the initial MV; and performing motion compensation on the current image block based on the scaled MV. (Item 58) The processor performing motion compensation on the current image block based on the scaled MV, Correcting the scaled MV to obtain a corrected MV; Item 58. The image motion compensation device according to item 57, further comprising: performing motion compensation on the current image block based on the modified MV. (Item 59) The processor obtaining the initial motion vector MV of the current image block includes: Obtaining a list of MV candidates for the current image block; 59. The image motion compensation device according to item 57 or 58, further comprising: determining the initial MV based on the MV candidate list. (Item 60) Item 59. The image motion compensation device according to Item 59, wherein the reference image to which any candidate MV in the MV candidate list points is a non-specific reference image. (Item 61) The processor obtaining the MV candidate list for the current image block includes: Item 61. An image motion compensation device as described in Item 60, comprising determining a candidate MV to add to the MV candidate list, and adding the candidate MV to the MV candidate list if the reference image to which the candidate MV points is a non-specific reference image. (Item 62) The processor further comprises: Item 62. An image motion compensation device according to item 61, for determining that the reference image to which the candidate MV points is a non-specific reference image based on a frame mark of the reference image to which the candidate MV points. (Item 63) The above MV candidate list is: If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for a non-specific reference image; The merge candidate MV obtained for the non-specific reference image, MVs obtained by interpolation from non-specific reference images, and 63. The image motion compensation device according to any one of items 59 to 62, comprising at least one candidate MV from among an upper adjacent MV and a left adjacent MV for the non-specific reference image of the current block. (Item 64) The processor determining the initial MV based on the MV candidate list, determining distortion costs corresponding to candidate MVs in the MV candidate list based on at least one of a bidirectional matching method and a template matching method; Item 64. The image motion compensation device according to Item 63, further comprising: selecting the MV with the smallest distortion cost in the MV candidate list as the initial MV. (Item 65) The processor modifies the scaled MV to obtain a modified MV, generating a sub-MV candidate list for the current image block, the sub-MV candidate list including the scaled MV; Item 65. The image motion compensation device according to item 64, further comprising determining the MV with the smallest distortion cost from the sub-MV candidate list as the modified MV. (Item 66) The initial MV includes a first initial MV and a second initial MV, and the processor specifically: If the reference image pointed to by at least one of the first initial MV and the second initial MV is a specific reference image, setting the scaling ratios of the first initial MV and the second initial MV to 1, and performing motion compensation on the current image block according to the first initial MV and the second initial MV; Item 57. An image motion compensation device for at least one of the following: when the reference images to which the first initial MV and the second initial MV are directed are both non-specific reference images, determining a scaling ratio for the first initial MV and a scaling ratio for the second initial MV, scaling the first initial MV based on the scaling ratio for the first initial MV, scaling the second initial MV based on the scaling ratio for the second initial MV, and performing motion compensation on the current image block based on the first initial MV after scaling and the second initial MV after scaling. (Item 67) The processor performing motion compensation on the current image block based on the scaled first initial MV and the scaled second initial MV, Generate a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial MV after scaling and belongs to a second reference image, the second reference image block corresponds to the second initial MV after scaling and belongs to the first reference image, the first reference image is a reference image directed by the first initial MV, and the second reference image is a reference image directed by the second initial MV; Item 67. An image motion compensation device according to item 66, further comprising: modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV. (Item 68) The processor modifies the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV, Using N third reference image blocks to match with the template respectively, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; Using M fourth reference image blocks to match with the template respectively, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; Item 66. An image motion compensation device comprising: based on the matching result, selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs, and using the one third initial MV and the one fourth initial MV as the MV of the current image block or to determine the MV of the current image block. (Item 69) Item 69. The image motion compensation device according to item 68, wherein the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV. (Item 70) An image motion compensation device described in item 68 or 69, wherein at least some of the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MV. (Item 71) 71. The image motion compensation device according to any one of items 68 to 70, wherein N is equal to M. (Item 72) The first reference image is a forward frame of the current image block, and the second reference image is a backward frame of the current image block; or 72. An apparatus for image motion compensation according to any one of items 68 to 71, wherein the first reference image is a forward frame of the current image block and the second reference image is a forward frame of the current image block. (Item 73) 73. The image motion compensation device according to any one of items 57 to 72, wherein the specific reference image includes at least one of a long-term reference image, a synthesized frame, and a frame that is not output. (Item 74) The processor further comprises: An image motion compensation device according to any one of items 57 to 73, which is for determining that the reference image is the specific reference image when the reference image is determined to be a frame that will not be output and the reference image is further determined to be a composite frame. (Item 75) 75. The image motion compensation device according to any one of items 57 to 74, wherein the non-specific reference image includes a short-term reference image. (Item 76) A computer-readable storage medium, A computer-readable storage medium having stored thereon commands that, when executed by a computer, cause the computer to perform the image motion compensation method according to any one of items 38 to 56. (Item 77) 76. An encoding device including the image motion compensation apparatus according to any one of items 57 to 75. (Item 78) A decoding device including the image motion compensation apparatus according to any one of items 57 to 75. (Item 79) 1. An image processing method, comprising: Obtaining a first initial motion vector MV and a second initial motion vector MV, where the first initial motion vector MV points to a first reference image and the second initial motion vector MV points to a second reference image; When at least one of the first reference image and the second reference image is a specific reference image, calculating and obtaining a predicted image block of the current image block based on the first initial MV and the second initial MV; When the first reference image and the second reference image are both non-specific reference images, a MV of a current image block is calculated based on the gradient values ​​of pixel points pointed to by the first initial MV and the second initial MV, and a predicted image block of the current image block is calculated based on the MV of the current image block. (Item 80) Calculating a motion vector (MV) of a current image block according to the gradient values ​​of the pixel points indicated by the first initial motion vector (MV) and the second initial motion vector (MV) is: 80. The method according to item 79, comprising calculating and obtaining a MV of the current image block based on the gradient values ​​of pixel points to which the first initial MV and the second initial MV point, and an optimization principle. (Item 81) Calculating a predicted image block of the current image block based on the first initial MV and the second initial MV includes: A method as described in 79 or 80, comprising weighting and summing a first reference image block to which the first initial MV points and a second reference image block to which the second initial MV points to obtain a predicted image block of the current image block. (Item 82) 82. The method according to any one of items 79 to 81, wherein the first initial MV and the second initial MV have different prediction directions. (Item 83) A method according to any one of items 79 to 82, wherein the prediction directions of the first initial MV and the second initial MV are the same, neither of the first initial MV and the second initial MV is 0, and the first reference image and the second reference image are different. (Item 84) Calculating a predicted image block of the current image block based on the MV of the current image block to obtain: A method according to any one of items 79 to 83, comprising calculating and obtaining a predicted image block of the current image block based on the MV of the current image block, a first reference image block to which the first initial MV points, and a second reference image block to which the second initial MV points. (Item 85) 85. The method according to any one of items 79 to 84, wherein the specific reference image includes at least one of a long-term reference image, a synthesized frame, and a frame that is not output. (Item 86) The above method is A method according to any one of items 79 to 85, further comprising determining that at least one of the first reference image and the second reference image is the specific reference image when at least one of the first reference image and the second reference image is determined to be a frame that will not be output and at least one of the first reference image and the second reference image is further determined to be a composite frame. (Item 87) 87. The method according to any one of items 79 to 86, wherein the non-specific reference image comprises a short-term reference image. (Item 88) An image processing device, at least one memory for storing computer executable commands; and at least one processor adapted, alone or jointly, to access said at least one memory and execute said computer-executable commands to perform the following operations: Obtaining a first initial motion vector MV and a second initial motion vector MV, where the first initial motion vector MV points to a first reference image and the second initial motion vector MV points to a second reference image; When at least one of the first reference image and the second reference image is a specific reference image, calculating and obtaining a predicted image block of the current image block based on the first initial MV and the second initial MV; When the first reference image and the second reference image are both non-specific reference images, the apparatus calculates and obtains an MV of a current image block based on the gradient values ​​of pixel points pointed to by the first initial MV and the second initial MV, and calculates and obtains a predicted image block of the current image block based on the MV of the current image block. (Item 89) Specifically, the processor Item 89. The device according to item 88, for calculating and obtaining a MV of the current image block based on the gradient values ​​of pixel points to which the first initial MV and the second initial MV point, and on an optimization principle. (Item 90) Specifically, the processor An apparatus as described in 88 or 89, which is for weightedly summing a first reference image block to which the first initial MV points and a second reference image block to which the second initial MV points to obtain a predicted image block of the current image block. (Item 91) The device according to any one of items 88 to 90, wherein the first initial MV and the second initial MV have different prediction directions. (Item 92) An apparatus described in any one of items 88 to 91, wherein the prediction direction of the first initial MV and the second initial MV is the same, neither of the first initial MV and the second initial MV is 0, and the first reference image and the second reference image are different. (Item 93) Specifically, the processor An apparatus according to any one of items 88 to 92, for calculating and obtaining a predicted image block of the current image block based on the MV of the current image block, a first reference image block to which the first initial MV points, and a second reference image block to which the second initial MV points. (Item 94) 94. The device according to any one of items 88 to 93, wherein the specific reference image includes at least one of a long-term reference image, a synthesized frame, and a frame that is not output. (Item 95) The processor further comprises: An apparatus described in any one of items 88 to 94, which is for determining that at least one of the first reference image and the second reference image is a frame that will not be output and further determining that at least one of the first reference image and the second reference image is a composite frame, and determining that at least one of the first reference image and the second reference image is the specific reference image. (Item 96) 96. The apparatus of any one of items 88 to 95, wherein the non-specific reference image comprises a short-term reference image. (Item 97) A computer-readable storage medium, A computer-readable storage medium that stores commands that, when executed by a computer, cause the computer to execute the image processing method according to any one of items 79 to 87. (Item 98) An encoding device including the image processing device according to any one of Items 88 to 96. (Item 99) A decoding device including the image processing device according to any one of Items 88 to 96.

Claims

1. obtaining a first initial motion vector and a second initial motion vector, the first initial motion vector pointing to a first reference image and the second initial motion vector pointing to a second reference image; determining whether the first reference image and the second reference image are specific reference images that are long-term reference images or non-specific reference images that are short-term reference images; if the first reference image and the second reference image are both the non-specific reference image, calculating a motion compensation of a current image block based on gradient values ​​of pixel points to which the first initial motion vector and the second initial motion vector point, and calculating a predicted image block of the current image block based on the motion compensation of the current image block; An image processing method comprising:

2. An image processing method as described in claim 1, comprising the steps of: when either the first reference image or the second reference image is the specific reference image, calculating motion compensation for a current image block using the first initial motion vector and the second initial motion vector; and calculating a predicted image block for the current image block based on the motion compensation of the current image block.

3. 3. The image processing method according to claim 1, wherein the first reference image is a forward frame of a current image block, the second reference image is a backward frame of the current image block, and the distance from the first reference image to the current image including the current image block is the same as the distance from the second reference image to the current image.

4. at least one memory for storing computer executable instructions; and at least one processor adapted, alone or jointly, to access said at least one memory and execute said computer-executable instructions to perform the following operations: obtaining a first initial motion vector and a second initial motion vector, wherein the first initial motion vector points to a first reference image and the second initial motion vector points to a second reference image; determining whether the first reference image and the second reference image are specific reference images that are long-term reference images or non-specific reference images that are short-term reference images; When the first reference image and the second reference image are both the non-specific reference image, an image processing device calculates motion compensation for a current image block based on gradient values ​​of pixel points pointed to by the first initial motion vector and the second initial motion vector, and calculates a predicted image block for the current image block based on the motion compensation for the current image block.

5. obtaining a first initial motion vector and a second initial motion vector, the first initial motion vector pointing to a first reference image and the second initial motion vector pointing to a second reference image; determining whether the first reference image and the second reference image are specific reference images that are long-term reference images or non-specific reference images that are short-term reference images; If the first reference image and the second reference image are both the non-specific reference image, calculating a motion compensation for a current image block based on gradient values ​​of pixel points to which the first initial motion vector and the second initial motion vector point, and calculating a predicted image block for the current image block based on the motion compensation for the current image block; encoding the current image block to generate a bitstream; A bitstream generation method comprising:

Images