Image processing method, image processing apparatus, and bitstream generation method

Abstract

The present invention provides an image motion compensation method and an image motion compensation device that improve encoding / decoding efficiency. [Solution] The video processing method comprises the step of acquiring a first initial motion vector (MV) and a second initial MV. The first initial MV points to a first reference image, and the second initial MV points to a second reference image. The method also comprises the step of calculating the motion offset of the current image block based on the gradient values ​​of the sampling points indicated by the first initial MV and the second initial MV, depending on whether both the first and second reference images are short-term reference images, and calculating a predicted image block for the current image block based on the motion offset of the current image block.

Description

[Technical Field]

[0001] Copyright Declaration The information disclosed in this patent document includes copyrighted material. Such copyright is owned by the copyright holder. The copyright holder does not object to anyone reproducing this patent document or the disclosure of this patent as it exists in the official records and public archives of the Patent and Trademark Office.

[0002] This application relates to the field of image processing, and more particularly to a method and apparatus for compensating for image motion. [Background technology]

[0003] In recent years, the popularity of portable, handheld, and wearable devices has led to a continuous increase in video file sizes. As video formats become more complex, storing and transmitting them also becomes more difficult. To reduce the bandwidth used for storing and transmitting video, video data is typically encoded and compressed at the encoding end and decoded at the decoding end.

[0004] The encoding and compression process includes operations such as prediction, transformation, quantification, and entropy coding. Here, prediction includes two types: intra-frame prediction and inter-frame prediction, which aims to remove redundant information from the current image block to be encoded using predicted block data. Intra-frame prediction obtains predicted block data using information from the current frame image. Inter-frame prediction obtains predicted block data using information from a reference image. This process involves dividing the current image to be encoded into multiple image blocks, dividing each image block into multiple sub-image blocks, and then, for each sub-image block, searching for the image block in the reference image that best matches the current sub-image block to obtain the predicted image block. The relative displacement between the predicted image block and the current sub-image block is the motion vector. Subsequently, the corresponding pixel values ​​of this sub-image block and the predicted image block are subtracted to obtain the residual. The residuals corresponding to each obtained sub-image block are combined to obtain the residual of the image block to be encoded. After processing such as transformation, quantification, and entropy coding, the residuals are converted to an entropy-coded bitstream, and the encoded mode information, such as the entropy-coded bitstream and information such as the in-frame prediction mode and motion vector (or motion vector residual), is stored or transmitted to the decoding end.

[0005] At the image decoding end, an entropy-encoded bitstream is obtained, followed by entropy decoding to obtain the corresponding residual. Based on information such as the motion vector and in-frame prediction obtained by decoding, a predicted image block corresponding to the image block to be decoded is searched, and the value of each pixel point in the image block to be decoded is obtained based on the predicted image block and the residual.

[0006] From the above explanation, it can be seen that when performing interframe prediction, the more similar the selected reference image and the current image to be encoded are, the smaller the residuals resulting from the interframe prediction become, thereby improving the encoding efficiency of the interframe prediction. Specifically, some existing technologies can construct high-quality specific reference images of background content, including scenes, using each image in a video. When performing interframe prediction, the background portion of the current image to be encoded or the current image to be decoded can be used to reduce the residual information of the interframe prediction by referring to the aforementioned high-quality specific reference image, thereby improving the encoding efficiency. In other words, this specific reference image is the reference image for interframe prediction. The long-term reference image is not a decoded image, but an artificially constructed image. The long-term reference image contains multiple image blocks, and each image block is extracted from a decoded image, and different image blocks in the long-term reference image may come from different decoded images.

[0007] To improve encoding efficiency and reduce the amount of information transmitted at the encoding end, some existing technologies allow for the direct derivation of motion vectors at the decoding end. This eliminates the need for the encoding end to transmit motion vector information or motion vector residual information, and the decoding end can obtain the true motion vector without decoding motion vector information or motion vector residual information. [Overview of the project] [Problems that the invention aims to solve]

[0008] In existing technologies that use the derivation of motion vectors and technologies that use bidirectional motion prediction, there are some that do not consider the particularity of long-term reference images. Among the technologies that use the derivation of motion vectors, there are some that do not consider whether the reference image pointed to by the motion vector is a long-term reference image. When correcting the motion vector, there is a possibility of performing a motion search in the long-term reference image, which may instead reduce the search efficiency and the encoding efficiency. The technology that uses bidirectional motion prediction operates the motion vector based on the temporal correlation of the images. When the reference image pointed to by the relevant motion vector is a long-term reference image, since the definition of the temporal distance between the current image to be encoded or the current image to be decoded and the long-term reference image is unclear, these operations may become ineffective.

Means for Solving the Problem

[0009] This application provides an image motion compensation method and an image motion compensation device, which can improve the encoding / decoding efficiency.

[0010] In the first aspect, an image motion compensation method is provided, and this method includes: obtaining an initial motion vector MV of a current image block; when the reference image pointed to by the initial MV is a specific reference image, performing motion compensation on the current image block based on the initial MV; when the reference image pointed to by the initial MV is a non-specific reference image, modifying the initial MV to obtain a modified MV, and performing motion compensation on the current image block based on the modified MV.

[0011] In the second aspect, an image motion compensation device is provided, and this device includes: at least one memory for storing computer-executable commands; at least one processor that is used alone or jointly to perform the following operations by accessing the at least one memory and executing the computer-executable commands, and the operations include: Obtain the initial motion vector MV of the current image block, and If the reference image pointed to by the initial MV is a specific reference image, motion compensation is performed on the current image block based on the initial MV. If the reference image pointed to by the initial MV is an unspecified reference image, the initial MV is modified to obtain a modified MV, and motion compensation is performed on the current image block based on the modified MV.

[0012] In a third embodiment, a computer-readable storage medium is provided, which stores commands, and when the commands are operated by the computer, the computer is instructed to execute the image motion compensation method according to the first embodiment.

[0013] In a fourth embodiment, an encoding device is provided that includes an image motion compensation device according to the second embodiment.

[0014] In a fifth embodiment, a decoding device is provided that includes an image motion compensation device according to the second embodiment.

[0015] In the sixth aspect, a method for compensating for image motion is provided, which is: Obtain the initial motion vector MV of the current image block, and The scaling ratio of the initial MV is determined, and here, if the initial MV points to a specific reference image, the scaling ratio of the initial MV is 1. Scaling the initial MV based on the scaling ratio of the initial MV, This includes performing motion compensation on the current image block based on the scaled MV.

[0016] In the seventh aspect, an image motion compensation device is provided, which device is At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable by the computer, the operations being: Obtain the initial motion vector MV of the current image block, and The scaling ratio of the initial MV is determined, and here, if the initial MV points to a specific reference image, the scaling ratio of the initial MV is 1. Scaling the initial MV based on the scaling ratio of the initial MV, This involves performing motion compensation on the current image block based on the scaled MV.

[0017] In the eighth aspect, a computer-readable storage medium is provided, which stores commands, and when the commands are operated by the computer, the computer is instructed to execute the image motion compensation method according to the sixth aspect.

[0018] In the ninth aspect, an encoding device is provided that includes an image motion compensation device according to the seventh aspect.

[0019] In the tenth aspect, a decoding device is provided that includes an image motion compensation device according to the seventh aspect.

[0020] In the eleventh aspect, an image processing method is provided, which is: A first initial motion vector MV and a second initial MV are obtained, wherein the first initial MV points to the first reference image, and the second initial MV points to the second reference image. If at least one of the first reference image and the second reference image is a specific reference image, then the predicted image block of the current image block is calculated and obtained based on the first initial MV and the second initial MV, If both the first reference image and the second reference image are unspecified reference images, the method includes calculating and obtaining the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and calculating and obtaining a predicted image block of the current image block based on the MV of the current image block.

[0021] In the twelfth aspect, an image processing apparatus is provided, which apparatus is At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable by the computer, the operations being: A first initial motion vector MV and a second initial MV are obtained, wherein the first initial MV points to the first reference image, and the second initial MV points to the second reference image. If at least one of the first reference image and the second reference image is a specific reference image, then the predicted image block of the current image block is calculated and obtained based on the first initial MV and the second initial MV, If both the first reference image and the second reference image are unspecified reference images, the MV of the current image block is calculated and obtained based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and the predicted image block of the current image block is calculated and obtained based on the MV of the current image block.

[0022] In the 13th aspect, a computer-readable storage medium is provided, which stores commands, and when the commands are operated by the computer, the computer is made to execute the image processing method according to the 11th aspect.

[0023] In the fourteenth aspect, an encoding device is provided that includes an image processing apparatus according to the twelfth aspect.

[0024] In the 15th aspect, a decoding device is provided that includes an image processing apparatus according to the 12th aspect. [Brief explanation of the drawing]

[0025] [Figure 1] This is a schematic flowchart of an image motion compensation method as one embodiment of this application. [Figure 2] This is a schematic diagram illustrating the principle of a bidirectional matching method according to one embodiment of this application. [Figure 3] This is a schematic diagram illustrating the principle of a template matching method according to one embodiment of this application. [Figure 4] This is a schematic diagram illustrating the principle of DMVR technology in one embodiment of this application. [Figure 5] This is a schematic flowchart of an image processing method in another embodiment of this application. [Figure 6] This is a schematic diagram illustrating the principle of a BIO technology in one embodiment of this application. [Figure 7] This is a schematic frame diagram of an image motion compensation device according to one embodiment of this application. [Figure 8] This is a schematic frame diagram of an image processing apparatus as one embodiment of this application. [Figure 9] This is a schematic flowchart of an image motion compensation method in another embodiment of this application. [Figure 10] This is a schematic frame diagram of an image motion compensation device according to another embodiment of this application. [Modes for carrying out the invention]

[0026] The technical solutions in the embodiments of this application will be described below with reference to the drawings.

[0027] Unless otherwise defined, all technical and academic terms used herein have the same meaning as those commonly understood by those skilled in the art of the present application. In this text, the technical terms used in the specification are solely for the purpose of describing specific embodiments and are not intended to limit this application.

[0028] First, we will describe the relevant technologies and concepts relating to the embodiments of this application.

[0029] A video consists of multiple images. When encoding / decoding a video, different prediction methods can be used for different images in the video. The prediction method used based on the image can divide the image into an in-frame prediction image and an inter-frame prediction image, where the inter-frame prediction image includes a forward prediction image and a bidirectional prediction image. Image I is an in-frame prediction image, also called a keyframe, and Image P is a forward prediction image, using a previously encoded / decoded Image P or Image I as a reference image. Image B is a bidirectional prediction image, using preceding and succeeding images as reference images. In one embodiment, the encoding / decoding end generates a Group of Pictures (GOP) in one-stage increments after encoding / decoding multiple images. This GOP is a Group of Pictures consisting of one Image I and at least one of multiple Images B (or bidirectional prediction images) and Images P (or forward prediction images). The decoding end reads and decodes each GOP in stages during broadcast, then reads the screen and renders it for display.

[0030] In modern video encoding / decoding standards, an image can be divided into multiple small blocks and encoded / decoded for images of different resolutions; in other words, an image can be divided into multiple image blocks. An image can be divided into any number of image blocks. For example, this image can be divided into an m × n image block matrix. Image blocks can have rectangular, square, circular, or any other shape. Image blocks can have any size, for example, p × q pixels. Each image block can have at least one of the same size and shape. Substitutable, two or more image blocks can have at least one of different sizes and shapes. Image blocks may or may not have any overlapping portions. In some embodiments, this image block is called a macroblock or Largest Coding Unit (LCU) in some encoding / decoding standards. In the H.264 standard, this image block is called a macroblock and may be 16 × 16 pixels in size. In the High Efficiency Video Coding (HEVC) standard, an image block is called a maximum coding unit, and its size may be 64x64 pixels.

[0031] In some other embodiments, an image block does not have to be a macroblock or a maximum encoding unit, but may include a part of a macroblock or a maximum encoding unit, or include at least two complete macroblocks (or maximum encoding units), or include at least one complete macroblock (or maximum encoding unit) and a part of one macroblock (or maximum encoding unit), or include at least two complete macroblocks (or maximum encoding units) and several parts of macroblocks (or maximum encoding units). In this way, after the image is divided into multiple image blocks, each of these image blocks in the image data can be encoded / decoded.

[0032] The encoding process includes operations such as prediction, transformation, quantification, and entropy coding. Here, prediction includes two types: intra-frame prediction and inter-frame prediction, which aims to remove redundant information from the current image block to be encoded using predicted block data. Intra-frame prediction obtains predicted block data using information from the current frame image. Inter-frame prediction obtains predicted block data using information from a reference image. This process involves dividing the current image to be encoded into multiple image blocks, dividing each image block into multiple sub-image blocks, and then, for each sub-image block, searching for the image block in the reference image that best matches the current sub-image block to obtain the predicted image block. The relative displacement between the predicted image block and the current sub-image block is the motion vector. Subsequently, the corresponding pixel values ​​of this sub-image block and the predicted image block are subtracted to obtain the residual. The residuals corresponding to each obtained sub-image block are combined to obtain the residual of the image block to be encoded.

[0033] In each embodiment of this application, the relationship between residuals of image blocks can be removed by the transformation matrix, that is, the coding efficiency is improved by removing redundant information in the image blocks. The transformation of data blocks in image blocks usually employs a two-dimensional transformation, where the coding end multiplies the residual information of each data block by a single N×M transformation matrix and its transpose, and the result is a transformation coefficient. After quantifying the transformation coefficient, the quantified coefficient is obtained, and finally the quantified coefficient is entropy encoded to obtain an entropy encoded bitstream. The entropy encoded bitstream and encoded mode information, such as the in-frame prediction mode and motion vector (or motion vector residual), are stored or transmitted to the decoding end.

[0034] At the image decoding end, an entropy-encoded bitstream is obtained, then entropy-encoded to obtain the corresponding residual. Based on information such as the motion vector and in-frame prediction obtained through decoding, a predicted image block corresponding to the image block is searched for. Based on the predicted image block and residual, the value of each pixel point in the current sub-image block is obtained.

[0035] In the preamble, it was mentioned that an already encoded / decoded image is used as the reference image to be encoded / decoded. In some embodiments, the similarity between the reference image and the current image to be encoded / decoded can be increased by further constructing the reference image.

[0036] For example, if a video content contains a specific encoding / decoding scene where the background remains essentially unchanged, then only the foreground in the video will change or move. Video monitoring, for instance, belongs to this type of scene. In a video monitoring scene, the monitoring camera is typically fixed and motionless, or only moves slowly, and the background can be considered essentially unchanged. In contrast, objects such as people or vehicles captured by the video monitoring camera are constantly moving or changing, and the foreground is constantly changing. In this type of scene, a specific reference image can be constructed, and this specific reference image contains only high-quality background information. This specific reference image can contain multiple image blocks, each of which is extracted from a decoded image, and different image blocks in this specific reference image may come from different decoded images. When performing interframe prediction, the background portion of the current image to be encoded / decoded can refer to this specific reference image, which can be used to reduce residual information in interframe prediction, thereby improving encoding / decoding efficiency.

[0037] The above is a specific example of a particular reference image. In some embodiments, the particular reference image has at least one of the following properties: a composite reference, a long-term reference image, or an image that is not output. For example, this particular reference image may be a composite long-term reference image, a composite frame that is not output, or a long-term reference image that is not output, etc. In some embodiments, a composite frame is called a composite reference frame.

[0038] In some embodiments, the non-specific reference image may be a reference image that does not have at least one of the following properties: a composite frame, a long-term reference image, or a reference image that is not output. For example, this specific reference image may include reference images other than composite frames, or reference images other than long-term reference images, or reference images other than reference images that are not output, or reference images other than composite frames that are not output, or reference images other than long-term reference images that are not output.

[0039] In some embodiments, when images in a video can be used as reference images, they can be divided into long-term reference images and short-term reference images. Here, the short-term reference image is a concept corresponding to the long-term reference image. The short-term reference image exists in the buffer area of ​​the reference image for a certain period of time, and after the decoded reference image has passed through the buffer area of ​​the reference image, the short-term reference image is moved out of the buffer area of ​​the reference image. The buffer area of ​​the reference image can be called a reference image list buffer memory, a reference image list, a reference frame list buffer memory, or a reference frame list, but in this document, it will be referred to as the buffer area of ​​the reference image.

[0040] A long-term reference image (or a portion of the data in a long-term reference image) can remain in the buffer area of ​​the reference image. This long-term reference image (or a portion of the data in a long-term reference image) is not affected by ingress / ingress operations in the buffer area of ​​the decoded reference image. Only when the decoding end sends an update command operation does this long-term reference image (or a portion of the data in a long-term reference image) move out of the buffer area of ​​the reference image.

[0041] The terminology for short-term and long-term reference images may differ depending on the standard. For example, in standards such as H.264 / Advanced Video Coding (AVC) or H.265 / HEVC, short-term reference images are called short-term reference frames, while long-term reference images are called long-term reference frames. In standards such as AVS (Audio Video Coding Standard), 1-P2, AVS2-P2, and IEEE (Institute of Electrical and Electronics Engineers) 1857.9-P4, long-term reference images are called background pictures. In standards such as VP8 and VP9, ​​long-term reference images are called golden frames.

[0042] It should be understood that the adoption of specific technical terms in the embodiments of this application does not indicate that they must be applied to a specific scenario. For example, referring to a long-range reference image as a long-range reference frame does not indicate that it must be used in technologies corresponding to standards such as H.264 / AVC or H.265 / HEVC.

[0043] The long-term reference image described above may be constructed from image blocks extracted from multiple decoded images, or it may be obtained by updating an existing reference frame (for example, a pre-memorized reference frame) using multiple decoded images. Naturally, this constructed specific reference image may also be a short-term reference image. Alternatively, the long-term reference image may not be a constructed reference image.

[0044] In the embodiments described above, the specific reference image may include a long-term reference image, and the non-specific reference image may include a short-term reference image.

[0045] The type of reference frame can be selected and identified from a special field in the stream structure.

[0046] If the reference image is determined to be a long-term reference image, then this reference image is determined to be a specific reference image. Alternatively, if the reference image is determined to be a frame in which no output is generated, then this reference image is determined to be a specific reference image. Alternatively, if the reference image is determined to be a composite frame, then this reference image is determined to be a specific reference image. Alternatively, if the reference image is determined to be a frame in which no output is generated, AND this reference image is determined to be a composite frame, then this reference image is determined to be a specific reference image.

[0047] Selectable reference images of various types can each have a corresponding mark, and in this case, the decoding end can determine whether or not a reference image is a specific reference image based on the mark it has.

[0048] In some embodiments, if a reference image is determined to have a long-term reference image mark, this reference image is determined to be a specific reference image.

[0049] In some embodiments, if it is determined that a reference image has a mark that does not result in output, this reference image is determined to be a specific reference image.

[0050] In some embodiments, if it is determined that a reference image has a composite frame mark, this reference image is determined to be a specific reference image.

[0051] In some embodiments, a reference image is determined to be a specific reference image if it has at least two of the following three marks: a long-term reference image mark, a non-output mark, a composite frame mark, or a composite reference frame mark. For example, if a reference image is determined to have a non-output mark and also to have a composite frame mark, the reference image is determined to be a specific reference image.

[0052] Specifically, an image may have a mark indicating whether or not it is a frame to be output. If an image is indicated not to be output, it indicates that this frame is a reference image. Furthermore, it is determined whether or not this frame has a mark indicating it is a composite frame. If it does, the reference image is confirmed to be a specific reference image. If an image is indicated to be output, it is not necessary to determine whether or not it is a composite frame; this frame is directly confirmed not to be a specific reference image. Alternatively, if an image is indicated not to be output but has a mark indicating it is not a composite frame, this frame can be confirmed not to be a specific reference image.

[0053] To be selectable, the parameters are analyzed from the picture header, picture parameter set (PPS), and slice header, and if it is determined that the reference image satisfies one of the following conditions, then the reference image is determined to be a specific reference image.

[0054] The aforementioned reference image is a long-term reference image.

[0055] The aforementioned reference image is a composite reference image.

[0056] The aforementioned reference image is an image that is not output.

[0057] If the aforementioned reference image is an image that is not output, it is further determined whether the aforementioned reference image is a composite reference image.

[0058] In the preamble, we mentioned a technique that uses the derivation of motion vectors. However, when modifying motion vectors, performing motion search on a specific reference image can actually decrease search efficiency and encoding / decoding efficiency. A specific reference image is artificially constructed or derived from an earlier, earlier reference image in chronological order. There is not necessarily a spatial relationship between image blocks in the specific reference image, and there is clear shaking at the image block edges. Therefore, there is little value in searching for motion vectors based on such a specific reference image.

[0059] Pattern Matching Motion Vector Derivation (PMMVD) and Decode Motion Vector Refinement (DMVR) are both techniques that utilize the derivation of motion vectors.

[0060] The preamble further mentions techniques that use bidirectional motion prediction, manipulating motion vectors based on the temporal correlation of images. These operations may fail if the reference image pointed to by the associated motion vector is a specific reference image, because the definition of the temporal distance between the current image to be encoded or decoded and the specific reference image is unclear. Bidirectional optical flow (BIO) prediction techniques are techniques that use bidirectional motion prediction.

[0061] Next, the image motion compensation method of the present invention will be explained with examples, taking into account PMMVD, DMVR, and BIO. It is necessary to explain that the image motion compensation method of the present invention is not limited to these three technologies.

[0062] Before describing the image motion compensation method of the embodiment of this application, we will first briefly explain the video encoding and decoding process in the HEVC standard.

[0063] The HEVC standard defines three modes of inter-frame prediction: Inter mode, Merge mode, and Skip mode. The purpose of inter-frame prediction is to obtain a motion vector (MV) and then determine the position of the predicted image block in the reference image based on this motion vector. Adjacent image blocks have similar motion schemes; for example, if an image block (e.g., at least one of the image block to be encoded and the image block to be decoded) and an adjacent image block belong to the same object, and the camera moves, their movement distance and direction will naturally be similar or the same, so it is often unnecessary to calculate the motion vector, and the motion vector of the adjacent image block is directly used as the motion vector of the current image block. Here, in Merge mode and Skip mode, the motion vector difference (MVD) is 0, meaning that the motion vector is obtained directly based on the adjacent encoded or decoded image block.

[0064] When at least one of the image blocks to be encoded or decoded is in Merge mode, the operating principle is as follows: A list of Motion Vector Prediction (MVP) candidates is constructed from adjacent image blocks, the optimal MVP is selected from the MVP candidate list as the motion vector for the current image block, the position of the predicted image block is determined based on this motion vector, and the residual is calculated after determining the predicted image block. In Merge mode, since the motion vector is selected from the MVP candidate list, there is no MVD. The encoding end only needs to encode the residual and the index of the selected motion vector in the MVP candidate list; it is not necessary to encode the MVD. The decoding end can construct an MVP candidate list in a similar manner and then obtain the motion vector based on the index transmitted by the encoding end. Based on the motion vector, the decoding end determines the predicted image block, incorporates the residual, and decodes to obtain the current image block.

[0065] The specific operation process of the encoded end in Merge mode is as follows: 1. Obtain the MVP candidate list. 2. Select the most suitable MVP from the MVP candidate list and obtain the index of this MVP within the MVP candidate list. 3. The selected MVP is used as the motion vector for the current image block. 4. Based on the motion vector, determine the predicted image block from the reference image. 5. Subtract the predicted image block from the current image block to obtain the residual. 6. Since the motion vector is selected from the MVP candidate list, there is no MVD, and only the residual and the index of the selected MVP in the MVP candidate list need to be sent to the decoding end.

[0066] The specific operation process of the decoding end in Merge mode is as follows: 1. Receive the residuals and the index of the motion vector in the MVP candidate list. 2. Obtain the MVP candidate list. 3. Based on the index, search for the motion vector in the MVP candidate list and use it as the motion vector for the current image block. 4. Based on the motion vector, the predicted image block is determined, and the residual is incorporated and decoded to obtain the current image block.

[0067] The above describes the processing steps in normal Merge mode.

[0068] Skip mode is a special case of Merge mode. After obtaining motion vectors using Merge mode, if the encoder determines, based on a certain method, that the current image block and the predicted image block are essentially the same, then it is not necessary to transmit residual data, only the index in the MVP candidate list of motion vectors needs to be transmitted, and the current image block can obtain a mark directly from the predicted image block.

[0069] In Inter mode, the MVP is determined first, then the MVP is modified to obtain the MVD. The coding end needs to transmit not only the index and residual to the decoding end, but also the MVD to the decoding end. Advanced Motion Vector Prediction (AMVP) is a tool that achieves motion vector prediction through a competitive mechanism.

[0070] In AMVP mode, there is also an MVP candidate list, and the motion vectors in this MVP candidate list are obtained from adjacent blocks in the spatial or temporal domain of the current image block. The MVP candidate list in AMVP mode may differ from the MVP candidate list in Merge mode. The encoding end or decoding end selects the optimal MVP from the MVP candidate list. Using this MVP as a search starting point, the vicinity is searched to obtain the optimal motion vector, which is the motion vector of the current image block. Based on this motion vector, the position of the predicted image block is determined, and after determining the predicted image block, the residual is calculated. Simultaneously, the MV and MVP are subtracted from each other to obtain the MVD. The encoding end encodes the residual, the index of the MVP in the MVP candidate list, and the MVD and transmits them to the decoding end. The decoding end can construct an MVP candidate list in a similar manner and then obtain the MVP based on the index transmitted by the encoding end. The decoding end determines the MV based on the MVP and MVD, determines the predicted image block based on the MV, incorporates the residuals, and decodes to obtain the current image block.

[0071] The specific operation process of the coding end in AMVP mode is as follows: 1. Obtain the MVP candidate list. 2. Select the most suitable MVP from the MVP candidate list and obtain the index of this MVP within the MVP candidate list. 3. Determine the starting point of the search based on the MVP. 4. Search near the starting point to obtain the optimal motion vector. 5. Based on the motion vector, determine the predicted image block from the reference image. 6. Subtract the predicted image block from the current image block to obtain the residual. 7. Subtract the MVP from the motion vector to obtain the MVD. 8. Send the residual, the index of the selected MVP in the MVP candidate list, and the MVD to the decoding end.

[0072] The specific operational process of the decoding end in AMVP mode will not be explained further.

[0073] The embodiments of this application provide an image motion compensation method 100. Figure 1 is a schematic flowchart of the image motion compensation method 100 according to one embodiment of this application. As shown in Figure 1, this method 100 is S110, obtain the initial motion vector of the current image block, S120, if the reference image pointed to by this initial motion vector is a specific reference image, motion compensation is performed on this current image block based on this initial motion vector. S130 includes, if the reference image pointed to by this initial motion vector is an unspecified reference image, modifying this initial motion vector to obtain a modified motion vector, and performing motion compensation on the current image block based on the modified motion vector.

[0074] In some embodiments, if the initial motion vector points to a specific reference image, motion compensation is performed directly. If the initial motion vector points to a non-specific reference image, the initial motion vector is modified, and motion compensation is performed based on the modified motion vector. This avoids performing meaningless searches due to obvious shaking of the image block edges of the specific reference image, thereby improving encoding and decoding efficiency.

[0075] In possible embodiments, the image motion compensation method 100 of the embodiment of this application can be applied to PMMVD technology.

[0076] PMMVD technology is a special merge mode based on Frame Rate Up Conversion (FRUC) technology. In this special merge mode, motion information for the current image block (e.g., MV and MVD) is not encoded in the stream but is generated directly at the decoding end.

[0077] Selectively, in this possible embodiment, if the reference image pointed to by the initial motion vector is an unspecified reference image, modifying the initial motion vector to obtain a modified motion vector and performing motion compensation on the current image block based on the modified motion vector may include obtaining a list of motion vector candidates for the current image block, determining that any of the candidate motion vectors in the motion vector candidate list is pointed to an unspecified reference image, determining the initial motion vector based on the motion vector candidate list, modifying the initial motion vector to obtain a modified motion vector, and performing motion compensation on the current image block based on the modified motion vector.

[0078] In this embodiment, determining an initial motion vector from a motion vector candidate list, and if the reference image pointed to by the candidate vector in this motion vector candidate list is a non-specific reference image, modifying the initial motion vector to obtain a modified motion vector, and performing motion compensation on the current image block based on the modified motion vector, can avoid performing meaningless searches due to obvious shaking of the image block edges of the specific reference image and improve encoding and decoding efficiency.

[0079] Selectively, in a possible embodiment, obtaining a list of motion vector candidates for the current image block may include determining a candidate motion vector to add to the motion vector candidate list, and adding the candidate motion vector to the motion vector candidate list if the reference image pointed to by the candidate motion vector is an unspecified reference image.

[0080] Specifically, in one embodiment, the method 100 may further include determining that the reference image pointed to by the candidate motion vector is a non-specific reference image based on the frame marks of the reference image pointed to by the candidate motion vector. Specifically, it can be determined whether or not the reference image pointed to by the candidate motion vector is a specific reference image based on the frame marks of the reference image pointed to by the candidate motion vector. If the reference image pointed to by the candidate motion vector is a specific reference image, it is not necessary to add the candidate motion vector corresponding to the specific reference image to the motion vector candidate list. In this way, when selecting an initial motion vector from the motion vector candidate list, the motion vector corresponding to the specific reference image is not selected.

[0081] It should be understood that the motion vectors of each embodiment of this application include three parameters, the horizontal component v x , vertical component v y and the frame mark of the reference image being pointed to. For example, this frame mark may be a Picture Order Count (POC), or it may be any other form of mark. The encoding end and the decoding end can determine the attributes of the reference image from this frame mark and decide whether this reference image is a specific reference image or a non-specific reference image.

[0082] In one embodiment, the selection of an initial motion vector based on the motion vector candidate list may include determining whether the selected initial motion vector points to a specific reference image, and if at least one of the selected initial motion vectors points to a specific reference image, selecting the initial motion vector again until an initial motion vector for a reference image other than the specific reference image is selected.

[0083] Specifically, if the determined initial motion vector points to a specific reference image based on the motion vector candidate list, a candidate motion vector can be selected again from the motion vector candidate list and used as the initial motion vector, based on a pre-set rule. This pre-set rule may, for example, select the candidate motion vector with the next smallest matching cost, and if this candidate motion vector does not point to a specific reference image, use this candidate motion vector as the initial motion vector. This embodiment is illustrative and does not limit the present application.

[0084] Taking the coded end as an example, the motion vector derivation process in FRUC merge mode is divided into two steps. The first step is coding unit (CU) level motion lookup, and the second step is sub-CU level motion refinement. Similarly, the decoding end can achieve similar functionality to the coded end.

[0085] In CU-level motion lookup, a list of CU-level motion vector candidates is generated. Based on the bidirectional matching method, the motion vector with the smallest matching cost is searched from the list of CU-level motion vector candidates, e.g., MV-A. Similarly, the motion vector with the smallest matching cost is searched from the list of CU-level motion vector candidates using the template matching method, e.g., MV-B. Then, based on the Rate Distortion Cost (RD-Cost) strategy used in the Merge mode strategy, it is determined whether the current CU uses the FRUC merge mode. That is, the results of the two matching methods (bidirectional matching method and template matching method) are checked using the RD-Cost strategy. The results obtained from the matching method with the smallest matching cost are further compared with the results of the other CU modes. The matching method with the smallest matching cost among the two matching methods is the one with the smallest final matching cost, and the FRUC flag of the current CU is set to TRUE, instructing the decoding end to use the corresponding matching method.

[0086] In this specific embodiment, the CU-level motion vector candidate list can correspond to the motion vector candidate list in Method 100 of the embodiment of this application. When generating the motion vector candidate list, motion vectors to be added to the list are scanned, and if a motion vector points to a non-specific reference image, this motion vector can be added to the motion vector candidate list; if it points to a specific reference image, this motion vector is rejected and not added to the motion vector candidate list. The motion vector candidate list in this embodiment may include, when the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate motion vector obtained for the non-specific reference image, the merge candidate motion vector obtained for the non-specific reference image, the motion vector obtained by interpolation from the non-specific reference image, and at least one candidate motion vector from the upper adjacent motion vector and the left adjacent motion vector for the non-specific reference image of the current block. This motion vector candidate list may also be a CU-level motion vector candidate list. In this embodiment, it should be understood that the non-specific reference image may specifically be a short-term reference image or a short-term reference frame.

[0087] Determining the initial motion vector based on the motion vector candidate list may include determining the strain cost corresponding to the candidate motion vector in the motion vector candidate list based on at least one of the bidirectional matching method and the template matching method, and selecting the motion vector with the smallest strain cost in the motion vector candidate list as the initial motion vector.

[0088] Specifically, taking the encoded end as an example, determining the initial motion vector based on the motion vector candidate list corresponds to CU-level motion search. In CU-level motion search, the motion vector with the smallest matching cost is searched from the CU-level motion vector candidate list based on the bidirectional matching method, for example, MV-A. Similarly, the motion vector with the smallest matching cost is searched from the CU-level motion vector candidate list using the template matching method, for example, MV-B. Subsequently, it is determined whether the current CU uses FRUC merge mode based on the RD-Cost strategy used in the merge mode strategy. That is, the results of the two matching methods (bidirectional matching method and template matching method) are checked using the RD-Cost strategy. The result obtained by the matching method with the smallest matching cost is further compared with the results of the other CU modes. The matching method with the smallest matching cost among the two matching methods is the one with the smallest final matching cost, the FRUC flag of the current CU is set to TRUE, and the decoding end is instructed to use the corresponding matching method. Similarly, the decoding end can perform functions similar to the encoding end, but it is less complex. For example, the decoding end directly receives flags to instruct the encoding end on how to match and does not need to execute an RD-Cost strategy.

[0089] Selectively modifying the initial motion vector and obtaining the modified motion vector may include generating a list of sub-motion vector candidates in the current image block, the sub-motion vector candidate list including the initial motion vector, and determining the motion vector with the smallest distortion cost from the sub-motion vector candidate list as the modified motion vector.

[0090] Specifically, modifying the initial motion vector and obtaining the modified motion vector can accommodate sub-CU level motion refinement.

[0091] In CU-level motion search, the motion vector with the lowest matching cost is the initial motion vector, which is used as the starting point for CU-level motion refinement. Around the starting point, a local search is performed based on the matching method determined by the CU level (bidirectional matching or template matching). Specifically, in sub-CU-level motion refinement, a list of sub-CU-level motion vector candidates can be generated. From the sub-CU-level motion vector candidate list, the motion vector with the lowest matching cost is searched for and set as the motion vector for the current CU.

[0092] The subCU-level motion vector candidate list may include motion vectors determined by a CU-level motion search (an MV determined from a CU-level search), top, left, top-left, and top-right neighboring motion vectors for the current image block, scaled versions of collocated motion vectors for the corresponding positions of the current image block in reference images, and candidate motion vectors derived from other time domains (up to 4 ATMVP candidates and up to 4 STMVP candidates).

[0093] In the specific embodiment described above, if the reference image pointed to by the candidate motion vector is a specific reference image, the candidate motion vector corresponding to the specific reference image is not added to the CU-level motion vector candidate list, so that the candidate motion vector corresponding to the specific reference image does not have the opportunity to become the initial motion vector.

[0094] In another specific embodiment, if the initial motion vector determined based on the CU-level motion vector candidate list points to a specific reference image, a candidate motion vector can be selected again from the CU-level motion vector candidate list and used as the initial motion vector, based on a pre-set rule. This pre-set rule may be, for example, selecting the candidate motion vector with the next smallest matching cost, and if this candidate motion vector does not point to a specific reference image, then using this candidate motion vector as the initial motion vector.

[0095] Figure 2 is a schematic diagram of the principle of a bidirectional matching method according to one embodiment of this application. As shown in Figure 2, the bidirectional matching method searches for the closest matching between two predicted image blocks in different reference images in the motion trajectory of the current CU and derives the motion vector of the current CU. The bidirectional matching method is based on the assumption that the motion trajectory of the current image block is continuous. The motion vectors MV0 and MV1 of the two predicted image blocks are assumed to be proportional to the time-domain distance (TD0 and TD1) between the current image and two reference images (reference image 0 and reference image 1). By scanning the motion vector candidate list, for example, for MV0, the motion vector pair MV0 and MV1 can be generated. Here, MV1 can be generated based on MV0, TD0, and TD1. When the distortion between the two predicted image blocks pointed to by the motion vector pair corresponding to MV0 is smallest, this motion vector (i.e., MV0) is the motion vector of the current CU. If TD0 = TD1, the bidirectional matching changes to a mirror-image-based bidirectional matching.

[0096] It should be understood that if one of the two reference images is a specific reference image, bidirectional matching cannot be performed because the definition of the time distance between the current image (the current image to be encoded, or the current image to be decoded) and the specific reference image is unclear. Moreover, the specific reference image is artificially constructed or originates from an earlier specific reference image in the time order, and there is not necessarily a spatial relationship between the image blocks in the specific reference image, and there is clear shaking at the image block edges, so there is little significance in bidirectional matching to search for motion vectors based on such a specific reference image. In possible embodiments of this application, the above-mentioned problems are avoided by not adding candidate motion vectors corresponding to the specific reference image to the CU-level motion vector candidate list.

[0097] Figure 3 is a schematic diagram of the principle of a template matching method according to one embodiment of this application. As shown in Figure 3, the template matching method searches for the closest match between the template of the current image (at least one of the upper image block and the left image block of the current CU) and a block in the reference image (the same size as the template) to derive the motion vector of the current CU. After obtaining the template, if the distortion between the template and the predicted image block pointed to by the candidate motion vector is smallest, this candidate motion vector is the motion vector of the current CU.

[0098] When a reference image is a specific reference image, it is artificially constructed or derived from an earlier specific reference image in chronological order, and there is not necessarily a spatial relationship between image blocks in the specific reference image, and there is clear shaking at the image block edges. Therefore, there is little value in searching for motion vectors based on such a specific reference image, and the results of the template matching method are inaccurate and, consequently, meaningless. In possible embodiments of this application, the above-mentioned problems are avoided by not adding candidate motion vectors corresponding to the specific reference image to the CU-level motion vector candidate list.

[0099] In a specific embodiment, the image motion compensation method 100 of the embodiment of this application can be applied to DMVR technology.

[0100] DMVR technology is a densification technique used to make more accurate predictions on the current image block when performing bidirectional prediction. Next, we will explain DMVR technology in detail using the decoding end as an example. DMVR technology mainly involves two major steps: the first step is to construct a template based on the decoded image block corresponding to multiple initial motion vectors; the second step is to modify these multiple initial motion vectors based on the template.

[0101] Specifically, the decoding end can generate a motion vector candidate list. For example, this motion vector candidate list may be the AMVP mode motion vector candidate list described in the preceding paragraph, or the Merge template motion vector candidate list. The decoding end can receive multiple indices to indicate initial motion vectors transmitted from the encoding end. Based on the indices, the decoding end obtains multiple initial motion vectors from the motion vector candidate list. Based on the decoded image blocks corresponding to these multiple initial motion vectors, the decoding end generates a template (for example, a method of weighting pixels) and uses the generated template to modify each of these multiple initial motion vectors. Finally, motion compensation is performed on the current image block based on these modified motion vectors.

[0102] In this specific embodiment, the initial motion vectors may include a first initial motion vector and a second initial motion vector. If a specific reference image exists in the initial motion vectors, the following processing can be employed. S120, if the reference image pointed to by the current initial motion vector is a specific reference image, performing motion compensation on the current image block based on the initial motion vectors may include: if the reference image pointed to by at least one of the first and second initial motion vectors is a specific reference image, performing motion compensation on the current image block based on the first and second initial motion vectors; and if the reference images pointed to by the first and second initial motion vectors are both non-specific reference images, modifying the first and second initial motion vectors to obtain a modified first motion vector and a modified second motion vector, and performing motion compensation on the current image block based on the modified first and modified second motion vectors. If no specific reference image exists in the initial motion vectors, i.e., if all initial motion vectors are non-specific reference images, existing DMVR processing methods can be employed.

[0103] Figure 4 is a schematic diagram of the principle of DMVR technology according to one embodiment of this application. Specifically, assuming that the initial motion vector includes a first initial motion vector (which may be MV0) and a second initial motion vector (which may be MV1), and that the decoded image block corresponding to the first initial motion vector belongs to the first decoded image block of the first frame, then this first frame may be the first reference image, and this first decoded image block may be the first reference image block. Assuming that the decoded image block corresponding to the second motion vector belongs to the second decoded image block of the second frame, then this second frame may be the second reference image, and this second decoded image block may be the second reference image block. A template can be obtained by weighting and summing this first reference image block and this second reference image block. Here, this template can be called a bidirectional template.

[0104] Optionally, in one embodiment, the initial motion vector includes a first initial motion vector and a second initial motion vector. Modifying the first initial motion vector and the second initial motion vector to obtain a modified first motion vector and a modified second motion vector may include generating a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial motion vector and belongs to the first reference image, and the second reference image block corresponds to the second initial motion vector and belongs to the second reference image, and modifying the first initial motion vector and the second initial motion vector based on the template to obtain a modified first motion vector and a modified second motion vector.

[0105] Specifically, modifying the first initial motion vector and the second initial motion vector based on the template to obtain the modified first motion vector and the modified second motion vector may include: matching N third reference image blocks with the template, where the N third reference image blocks correspond to N third initial motion vectors and belong to the first reference image; matching M fourth reference image blocks with the template, where the M fourth reference image blocks correspond to M fourth initial motion vectors and belong to the second reference image; selecting one third initial motion vector from the N third initial motion vectors and one fourth initial motion vector from the M fourth initial motion vectors based on the matching results; and using the one third initial motion vector and the one fourth initial motion vector as the motion vectors of the current image block (i.e., the modified first motion vector and the modified second motion vector), or determining the motion vectors of the current image block.

[0106] Selectively, this selected third initial motion vector may be the motion vector corresponding to the smallest strain cost. Alternatively, this selected third initial motion vector may be the motion vector corresponding to a strain cost smaller than a certain value.

[0107] Selectively, this selected fourth initial motion vector may be the motion vector corresponding to the smallest strain cost. Alternatively, this selected fourth initial motion vector may be the motion vector corresponding to a strain cost smaller than a certain value.

[0108] Here, the one third initial motion vector and the one fourth initial motion vector are defined as the motion vectors of the current image block. In this case, the known image blocks corresponding to the one third initial motion vector and the one fourth initial motion vector (i.e., the initial predicted image blocks) can be weighted and summed to obtain the predicted image block.

[0109] Alternatively, the one third initial motion vector and the one fourth initial motion vector can be used to determine the motion vector of the current image block; that is, the one third initial motion vector and the one fourth initial motion vector can each be an MVP. In this case, the third MVP can be used as a starting point for search optimization to obtain an optimized motion vector, and the fourth MVP can be used as a starting point for search optimization to obtain another optimized motion vector. The known image blocks (i.e., initial predicted image blocks) corresponding to these two optimized motion vectors can be weighted and summed to obtain a predicted image block.

[0110] For selection to be possible, N and M may be equal.

[0111] For selection to be possible, this third initial motion vector includes this first initial motion vector, and this fourth initial motion vector includes this second initial motion vector; that is, the reference image block corresponding to the first initial motion vector and the reference image block corresponding to the second motion vector for generating the template must also be matched with the template.

[0112] Selectively, in the embodiments of this application, at least some of the N third initial motion vectors are obtained by shifting based on the first initial motion vector, and at least some of the M fourth initial motion vectors are obtained by shifting based on the second initial motion vector.

[0113] For example, in these N third initial motion vectors, the initial motion vectors other than the first initial motion vector are obtained by shifting them based on the first initial motion vector. For example, N may be equal to 9, and eight of these initial motion vectors are obtained by shifting them based on the first initial motion vector, for example, by shifting them in eight directions, or by shifting different pixels in the vertical or horizontal direction.

[0114] For example, the initial motion vectors other than the second initial motion vector in these N fourth initial motion vectors are obtained by shifting them based on the second initial motion vector. For example, N may be equal to 9, and eight of these initial motion vectors are obtained by shifting them based on the second initial motion vector. For example, they may be obtained by shifting them in eight directions, or by shifting different pixels in the vertical or horizontal direction.

[0115] Selectively, in the embodiments of this application, the first reference image may be a forward-facing frame of the current image block, and the second reference image may be a backward-facing frame of the current image block. Alternatively, the first reference image may be a forward-facing frame of the current image block, and the second reference image may be a forward-facing frame of the current image block.

[0116] In a specific embodiment, the decoding end generates a motion vector candidate list and receives two indices to indicate the initial motion vectors transmitted from the encoding end. The decoding end determines the DMVP condition, which requires that neither of the two initial motion vectors (which may be MV0 and MV1) points to a specific reference image, and that the predicted directions of the two initial motion vectors are opposite, i.e., one is forward and the other is backward. If the DMVR condition is satisfied, the image blocks corresponding to MV0 and the image blocks corresponding to MV1 are weighted together to generate a bidirectional template.

[0117] In a possible embodiment of this application, the reference image pointed to by the initial motion vector indicated by the two indices is avoided by not adding a candidate motion vector corresponding to a specific reference image to the motion vector candidate list. In another possible embodiment, if the reference image pointed to by the initial motion vector indicated by the two indices is the specific reference image, the decoding end can ask the encoding end to again point to a motion vector that does not point to the specific reference image as the initial motion vector, or the decoding end can return or mark the DMVR algorithm as expired.

[0118] In the first reference image, for the image block corresponding to MV0, one luminance pixel is shifted in at least one of the horizontal or vertical directions, and the motion vectors of the eight adjacent pixel blocks found, along with the total of nine motion vectors of MV0, can form a reference list list0. In the second reference image, for the image block corresponding to MV1, one luminance pixel is shifted in at least one of the horizontal or vertical directions, and the motion vectors of the eight adjacent pixel blocks found, along with the total of nine motion vectors of MV1, can form a reference list list1.

[0119] The bidirectional template matching used by the decoding end performs a distortion-based search between the bidirectional template and the reconstructed block in the reference image, ultimately obtaining a refined motion vector without additional motion information. For the motion vectors in the two reference images (the motion vector in List0 and the motion vector in List1), the motion vector with the smallest matching cost replaces the original motion vector as the updated motion vector. Finally, the original MV0 and MV1 are replaced with two new motion vectors (MV0' and MV1' shown in Figure 3). Based on the predicted image block corresponding to MV0' and the predicted image block corresponding to MV1', the final bidirectional prediction result is generated.

[0120] In DMVR technology, when the reference image is a specific reference image, the specific reference image is artificially constructed or originates from an earlier specific reference image in chronological order. There is not necessarily a spatial relationship between the image blocks in the specific reference image, and there is clear shaking at the image block edges. Therefore, there is little value in searching for motion vectors based on such a specific reference image, meaning that DMVR technology is inaccurate and, consequently, meaningless. In the embodiment of this application, the above-mentioned problems are avoided by not refining the motion vectors corresponding to the specific reference image and using them directly for motion compensation.

[0121] The embodiments of this application provide an image processing method 200. Figure 5 is a schematic flowchart of another embodiment of the image processing method 200 of this application. As shown in Figure 5, this method 200 is A first initial motion vector and a second initial motion vector are obtained, and the first initial motion vector points to the first reference image, and the second initial M motion vector points to the second reference image, S210, If at least one of the first reference image and the second reference image is a specific reference image, then S220 is obtained by calculating the predicted image block of the current image block based on the first initial motion vector and the second initial motion vector, If both the first reference image and the second reference image are unspecified reference images, the method includes S230, which calculates and obtains the motion vector of the current image block based on the first initial motion vector and the gradient value of the pixel point pointed to by the second initial motion vector, and calculates and obtains a predicted image block of the current image block based on the motion vector of the current image block.

[0122] In the image processing method of the embodiment of this application, when the reference images pointed to by two initial motion vectors are both unspecified reference images, the motion vector of the current image block is calculated by employing the gradient value of the pixel points and an optimization principle, and the predicted image block of the current image block is calculated. When a specific reference image exists among the reference images pointed to by the two initial motion vectors, the predicted image block of the current image block is calculated directly based on the two initial motion vectors, thereby avoiding the inability to perform prediction due to the unclear definition of the time distance between the current image and the specific reference image, and improving the encoding and decoding efficiency.

[0123] Selectively, obtaining the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV may include obtaining the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and an optimization principle.

[0124] In a specific embodiment, the image processing method 200 of the embodiment of this application can be applied to improve the bidirectional motion prediction mode. The bidirectional motion prediction mode refers to a method in which, when encoding a current image block, two initial prediction image blocks derived from two different reference images are used simultaneously to predict the current image block, and the two initial prediction image blocks are combined into a single image block to form the predicted image block of the current image block. BIO prediction technology is a technology used in the bidirectional motion prediction mode. In BIO prediction technology, the motion vector has even fewer encoding marks compared to the normal bidirectional motion prediction mode, but the derivation process of the predicted image block is different. BIO prediction technology is motion optimization based on block motion compensation, calculates the motion vector using optical flow mode, and is motion optimization at the sampling point level.

[0125] The BIO prediction technology of the embodiments of this application can include two steps. The first step is to calculate and obtain the motion vector (MV) of the current image block based on two initial MVs (for example, the first initial MV and the second initial MV). Specifically, it is to calculate and obtain the MV of the current image block based on the gradient values of the pixel points pointed to by the first initial MV and the second initial MV, and the optimization principle. The second step is to calculate and obtain the predicted image block of the current image block based on the MV of the current image block, the first reference image block pointed to by the first initial MV, and the second reference image block pointed to by the second initial MV. Specifically, the BIO prediction technology interpolates the two reference image blocks pointed to by the two initial MVs to obtain two initial predicted image blocks of the same size as the current image block, and then weighted sum these two initial predicted image blocks and combine them as the predicted image block of the current image block.

[0126] FIG. 6 is a schematic diagram of the principle of the BIO technology of one embodiment of this application. The BIO technology assumes that the movement of an object is a uniform motion in both the horizontal and vertical directions within a short time and conforms to the analysis conditions of the optical flow method. As shown in FIG. 6, the motion vectors (v x 、v y ) of the reference points k (k = 0, 1) pointed to by the two initial MVs, and the luminance value I (k) can be assumed to conform to the following optical flow formula. ∂I (k) / ∂t + v x ∂I (k) / ∂x + v y ∂I (k) / ∂y = 0 Formula 1 In the formula, ∂I (k) / ∂x and ∂I (k) / ∂y represent the horizontal and vertical components of the gradient.

[0127] By联立 the optical flow formula and Hermite interpolation, a polynomial as follows is obtained, and when t = 0, it is the BIO prediction value. pred BIO = 1 / 2·(I(0) +I (1) +v x / 2·(τ1∂I (1) / ∂x-τ0∂I (0) / ∂x)+v y / 2·(τ1∂I (1) / ∂y-τ0∂I (0) / ∂y)) Official 2 In the formula, τ0 and τ1 represent the distances from the current image to reference image 0 and reference image 1, respectively, and both can be obtained by calculating the point of view (POC) of the current image and the two reference images. τ0=POC(current)-POC(Ref0) Formula 3 τ1=POC(Ref1)-POC(current) Formula 4 In the formula, POC(current) is the picture order count of the current image, POC(Ref0) is the picture order count of reference image 0, and POC(Ref1) is the picture order count of reference image 1.

[0128] The reference images can be in different directions, i.e., one from the past and one from the future. The two reference images can also be in the same direction, i.e., both from the past or both from the future. If the two reference images are in the same direction, τ0 and τ1 have opposite signs. In such a situation, the two reference images cannot be the same, i.e., τ0 ≠ τ1, and the reference region has non-zero motion (MV). x0 MV y0 MV x1 MV y1 There exists a value (≠0), and the motion vector is proportional to the distance in the time domain (MV). x0 / MV x1 =MV y0 / MV y1 = -τ0 / τ1).

[0129] Assuming that the motion within a small region is the same, the motion vector of BIO can satisfy the following formula for a first-order Taylor expansion. Δ=(I (0) -I (1) +v x (τ1∂I(1) / ∂x+τ0∂I (0) / ∂x)+v y (τ1∂I (1) / ∂y+τ0∂I (0) / ∂y)) Official 5 In the formula, Δ is the Taylor linear expansion of the pixel difference between two corresponding reference points (for example, points A and B in Figure 4) in two reference images. By analyzing the gradient value and optimization method, the optimal motion vector of the current image block satisfies the square of Δ and its minimum value throughout the entire region, thereby determining the optimal motion vector (v x , v y ) can be calculated, and considering the robustness of the algorithm, v x and v y It is assumed that it is within a certain threshold range.

[0130] Based on the formula above, the process of the BIO prediction technique is as follows: For the current image block, the pixel values ​​at the positions corresponding to the two reference images have already been obtained. (0) and I (1) These represent the pixel values ​​in the two reference images, and the known ones in the above formula are: (0) , I (1) and τ0 and τ1, the horizontal and vertical components of the gradient are obtained by calculation in the reference image, and the unknown is v x , v y And Δ. For all pixel points in a single region, one Δ can be calculated for each, and using an optimization method, the v with the minimum value of Δ is obtained. x and v y The goal is to obtain the optimal motion vector, which is the one needed. Here, v x and v y When calculating v x and v yA single interval is given, and the threshold of this interval is determined by the orientation of the two reference images relative to the current image. After obtaining the optimal motion vector, the image block + residual pointed to by this optimal motion vector is not directly used as the current image block. Instead, each pixel of the current image block is calculated using formula 2, which is also called the BIO predicted value, and the predicted values ​​of each pixel are combined to form the predicted image block.

[0131] Selectively, in an embodiment of the present application, obtaining a predicted image block of the current image block based on the first initial motion vector and the second initial motion vector includes weighting and summing the first reference image block pointed to by the first initial motion vector and the second reference image block pointed to by the second initial motion vector to obtain a predicted image block of the current image block.

[0132] Specifically, a decision condition is added before the start of the BIO prediction, or a necessary condition is added to the BIO condition. If neither the first reference image pointed to by the first initial motion vector nor the second reference image pointed to by the second initial motion vector is a specific reference image, and the initial BIO condition is met, then the BIO prediction can be performed. Otherwise, the predicted image block for the current image block is directly calculated and obtained based on the two initial motion vectors, or returned, or the BIO prediction algorithm is marked as expired.

[0133] The BIO condition may further include that the predicted directions of the first initial motion vector and the second initial motion vector are different. Alternatively, the BIO condition may further include that the predicted directions of the first initial motion vector and the second initial motion vector are the same, neither the first initial motion vector nor the second initial motion vector is zero, and the first reference image and the second reference image are different. At the same time, the ratio of the directional components of the first initial motion vector and the second initial motion vector is the same, and both are equal to the ratio of the distance between the first reference image and the current image pointed to by the motion vector, and the distance between the second reference image and the current image.

[0134] It should be understood that the motion vectors of each embodiment of this application include three parameters, the horizontal component v x , vertical component v y , and the frame mark of the reference image being pointed to. For example, this frame mark may be a POC, or it may be any other form of mark. The encoding end and the decoding end can determine the attributes of the reference image from this frame mark and determine whether or not this reference image is a specific reference image.

[0135] Selectively, in the embodiments of this application, determining whether the reference image pointed to by the initial motion vector is a specific reference image may include determining whether the reference image pointed to by the initial motion vector is a specific reference image based on the frame marks of the reference image pointed to by the initial motion vector.

[0136] Optionally, in the embodiments of this application, calculating a predicted image block of the current image block based on the motion vector of the current image block in S230 may include calculating a predicted image block of the current image block based on the motion vector of the current image block, a first reference image block pointed to by the first initial motion vector, and a second reference image block pointed to by the second initial motion vector. The specific calculation formula is described in detail in the preamble and will not be described further here.

[0137] A specific embodiment of Method 200 may include the following steps:

[0138] 1. Determine whether the two current initial motion vectors satisfy the BIO conditions, and the criteria for this determination are as follows:

[0139] a) Both reference images pointed to by the two initial motion vectors are unspecified reference images, that is, the first reference image pointed to by the first initial motion vector and the second reference image pointed to by the second initial motion vector are both unspecified reference images.

[0140] b) Satisfying one of the following two conditions, i) The two initial motion vectors are from different predicted directions (forward and backward, respectively). ii) The two initial motion vectors are from different reference images with the same predicted direction, neither of the two initial motion vectors is zero, and the ratio of the directional components of the two initial motion vectors is the same, and both are equal to the ratio of the distance between the reference image pointed to by the motion vector and the current image.

[0141] An initial motion vector that satisfies both a) and b) simultaneously meets the BIO conditions and allows for BIO prediction.

[0142] 2. Based on the predicted directions of the two initial motion vectors, the calculation threshold is determined, i.e., v x and v y This is the threshold value for the interval.

[0143] 3. Calculate the gradient value of the pixel point pointed to by the two initial motion vectors.

[0144] 4. Based on the gradient value and the optimization principle, the optimal motion vector is calculated and set as the motion vector for the current image block.

[0145] 5. Obtain the BIO prediction value using the motion vector of the current image block and the reference image block pointed to by the two initial motion vectors.

[0146] If only b)i) is satisfied, the first reference image block pointed to by the first initial motion vector and the second reference image block pointed to by the second initial motion vector are weighted and summed to obtain the predicted image block of the current image block.

[0147] The principles, formulas, and steps referenced by the BIO predictions in the embodiments of this application can be found in the description above and will not be explained further here.

[0148] In BIO prediction technology, if the reference image is a specific reference image, the definition of the time distance between the current image (the current image to be encoded, or the current image to be decoded) and the specific reference image is unclear, so it should be understood that BIO prediction technology cannot be performed. In the embodiment of this application, if a specific reference image exists in the reference image pointed to by the two initial motion vectors, the predicted image block of the current image block is directly calculated and obtained based on the two initial motion vectors, thereby avoiding the above-mentioned problem.

[0149] It should be understood that the methods of each embodiment of this application can be applied to both the encoding end and the decoding end. The current image block in each embodiment of this application may be an image block to be encoded or an image block to be decoded.

[0150] The embodiments of this application merely exemplify image processing methods applicable to PMMVD technology, DMVR technology, and BIO prediction technology. The methods of the embodiments of this application can be further applied to other existing or future video encoding / decoding technologies, and the embodiments of this application are not limited thereto.

[0151] Figure 7 is a schematic frame diagram of an image motion compensation device 70 according to one embodiment of this application. As shown in Figure 7, the image motion compensation device 70 is At least one memory 701 for storing executable commands for the computer, The system includes at least one processor 702 used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable by the computer, the operations being: Obtain the initial motion vector MV of the current image block, and If the reference image pointed to by the initial MV is a specific reference image, motion compensation is performed on the current image block based on the initial MV. If the reference image pointed to by the initial MV is an unspecified reference image, the initial MV is modified to obtain a modified MV, and motion compensation is performed on the current image block based on the modified MV.

[0152] In some embodiments, the processor 702 specifically, Obtain the MV candidate list for the current image block, and if any of the candidate MVs in the MV candidate list point to a reference image that is not a specific reference image, Based on the aforementioned list of MV candidates, the initial MV is determined, The initial MV was modified, and the modified MV was obtained. This is for performing motion compensation on the current image block based on the aforementioned modified MV.

[0153] In some embodiments, the processor 702 further, This process determines a candidate MV to be added to the aforementioned MV candidate list, and if the reference image pointed to by the candidate MV is an unspecified reference image, it adds the candidate MV to the aforementioned MV candidate list.

[0154] In some embodiments, the processor 702 further, This is to determine that the reference image pointed to by the candidate MV is an unspecified reference image, based on the frame marks of the reference image pointed to by the candidate MV.

[0155] In some embodiments, the MV candidate list includes at least one candidate MV from the following: If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for the unspecified reference image, Merge candidate MV obtained for a non-specific reference image, MV obtained by interpolation from a non-specific reference image, and These are the upper adjacent MV and left adjacent MV for the unspecified reference image of the current block.

[0156] In some embodiments, the processor 702 specifically, Based on at least one of the bidirectional matching method and the template matching method, the distortion cost corresponding to the candidate MV in the MV candidate list is determined. The purpose is to select the MV with the smallest distortion cost in the aforementioned MV candidate list as the initial MV.

[0157] In some embodiments, the processor 702 specifically, A sub-MV candidate list is generated for the current image block, and the sub-MV candidate list includes the initial MV. This is to determine the MV with the smallest distortion cost from the aforementioned sub-MV candidate list as the corrected MV.

[0158] In some embodiments, the initial MV includes a first initial MV and a second initial MV, The processor 702, specifically, If the reference image pointed to by at least one of the first initial MV and the second initial MV is a specific reference image, motion compensation is performed on the current image block based on the first initial MV and the second initial MV. If the reference image pointed to by the first initial MV and the second initial MV is an unspecified reference image, the first initial MV and the second initial MV are modified to obtain a modified first MV and a modified second MV, and motion compensation is performed on the current image block based on the modified first MV and the modified second MV.

[0159] In some embodiments, the processor 702 specifically, A template is generated based on the first reference image block and the second reference image block, where the first reference image block corresponds to the first initial MV and belongs to the first reference image, and the second reference image block corresponds to the second initial MV and belongs to the second reference image. This is for modifying the first initial MV and the second initial MV based on the aforementioned template, and obtaining the modified first MV and the modified second MV.

[0160] In some embodiments, the processor 702 specifically, N third reference image blocks are used to match each of the templates, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image. M fourth reference image blocks are used to match each of the templates, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image. Based on the matching results, one third initial MV is selected from the N third initial MVs, and one fourth initial MV is selected from the M fourth initial MVs. These three third initial MVs and the fourth initial MV are used as the MV of the current image block, or to determine the MV of the current image block.

[0161] In some embodiments, the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV.

[0162] In some embodiments, at least some of the initial MVs in the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the initial MVs in the M fourth initial MVs are obtained by shifting based on the second initial MV.

[0163] In some embodiments, N is equal to M.

[0164] In some embodiments, the first reference image is the forward frame of the current image block, the second reference image is the backward frame of the current image block, or The first reference image is the forward-facing frame of the current image block, and the second reference image is the forward-facing frame of the current image block.

[0165] It is understood that the image motion compensation device 70 can also be implemented using a corresponding software module, and this will not be explained further here.

[0166] Figure 8 is a schematic frame diagram of an image processing apparatus 80 of another embodiment of this application. As shown in Figure 8, the image processing apparatus 80 is At least one memory 801 for storing executable commands for the computer, The system includes at least one processor 802 used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable by the computer, the operations being: A first initial motion vector MV and a second initial MV are obtained, wherein the first initial MV points to the first reference image, and the second initial MV points to the second reference image. If at least one of the first reference image and the second reference image is a specific reference image, then the predicted image block of the current image block is calculated and obtained based on the first initial MV and the second initial MV, If both the first reference image and the second reference image are unspecified reference images, the MV of the current image block is calculated and obtained based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and the predicted image block of the current image block is calculated and obtained based on the MV of the current image block.

[0167] In some embodiments, the processor 802 specifically, This is for calculating and obtaining the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and the optimization principle.

[0168] In some embodiments, the processor 802 specifically, This method involves weighting and summing the first reference image block pointed to by the first initial MV and the second reference image block pointed to by the second initial MV to obtain a predicted image block for the current image block.

[0169] In some embodiments, the prediction directions of the first initial MV and the second initial MV are different.

[0170] In some embodiments, the prediction directions of the first initial MV and the second initial MV are the same, neither the first initial MV nor the second initial MV is 0, and the first reference image and the second reference image are different.

[0171] In some embodiments, the processor 802 specifically, This is for calculating and obtaining a predicted image block for the current image block based on the MV of the current image block, the first reference image block pointed to by the first initial MV, and the second reference image block pointed to by the second initial MV.

[0172] It is understood that the image processing device 80 can be further realized by corresponding software modules, and this will not be explained further here.

[0173] In some techniques that use motion vector derivation, scaling a motion vector when it points to a specific reference image is meaningless and reduces search efficiency and encoding / decoding efficiency. This is because scaling a motion vector is based on the distance between images, while a specific reference image may be artificially constructed and not at a distance from other reference images. Therefore, scaling a motion vector based on such a specific reference image is not very meaningful.

[0174] This application further provides an image motion compensation method. Figure 9 is a schematic flowchart of an image motion compensation method 900, which is another embodiment of this application. As shown in Figure 9, this method 900 is S910 obtains the initial motion vector MV of the current image block, The scaling ratio of this initial MV is determined, and here, if this initial MV points to a specific reference image, the scaling ratio of this initial MV is 1 for S920, and Based on the scaling ratio of this initial MV, the S930 scales this initial MV, This includes S940, which performs motion compensation on the current image block based on the scaled MV.

[0175] The method for obtaining the initial MV here can be the same as the method for obtaining the initial MV described above, and will not be explained further here.

[0176] In the image motion compensation method of the embodiment of this application, the scaling ratio is 1 when the initial MV points to a specific reference image. When the initial MV points to a non-specific reference image, the scaling ratio is not limited, and the motion vector is scaled based on the distance between the current image and the non-specific reference image, and the scaling strategy, thereby performing motion compensation and improving the encoding and decoding efficiency.

[0177] In some embodiments, motion compensation is performed directly on the current image block based on the scaled MV.

[0178] In some embodiments, the scaled MV can be further modified to obtain a modified MV, and motion compensation is performed on the current image block based on the modified MV. Here, the method for modifying the scaled MV may be the same as the method for modifying the initial MV described above, and will not be described further here.

[0179] For example, as shown in Figure 5, the first initial MV points to the first reference image, and the second initial MV points to the second reference image. If both the first and second reference images point to an unspecified reference image, the first and second initial MVs are scaled based on the distance between the first and second reference images. Then, motion compensation is performed on the current image block based on the scaled first and second initial MVs. In one embodiment, a first reference image block is determined, which corresponds to the scaled first initial MV and belongs to the second reference image. A second reference image block is determined, which corresponds to the scaled second initial MV and belongs to the first reference image. A template is generated based on the first and second reference image blocks, and the first and second initial MVs are modified based on this template to obtain the modified first and second MVs. The motion vector of the current image block is calculated using the modified first and second MVs.

[0180] Here, the method for modifying the first initial MV and the second initial MV based on this template may be the same as the method for modifying the first initial MV and the second initial MV based on the template described above, and will not be explained further here.

[0181] This application provides yet another image motion compensation method. In this image motion compensation method, an initial motion vector MV of the current image block is obtained, and motion compensation is performed on the current image block based on the initial MV by employing different operations depending on whether the initial MV points to a specific reference image or to a non-specific reference image.

[0182] The method for obtaining the initial MV here can be the same as the method for obtaining the initial MV described above, and will not be explained further here.

[0183] In one embodiment, if the initial MV points to a specific reference image, motion compensation is performed on the current image block using this initial MV. If the initial MV points to a non-specific reference image, there are four embodiments: 1) an embodiment in which the initial MV is scaled and motion compensation is performed on the current image block based on the scaled initial MV; 2) an embodiment in which the initial MV is modified and motion compensation is performed on the current image block based on the modified initial MV; 3) an embodiment in which the initial MV is scaled, the scaled MV is modified to obtain the modified MV, and motion compensation is performed on the current image block based on the modified MV; and 4) an embodiment in which the initial MV is modified, the modified MV is scaled to obtain the scaled MV, and motion compensation is performed on the current image block based on the scaled MV.

[0184] Here, the method for correcting the MV after scaling, or the method for correcting the initial MV, may be the same as the method for correcting the initial MV described above, and will not be explained further here.

[0185] In one embodiment, if the initial MV points to a specific reference image, the initial MV can be modified, and motion compensation can be performed based on the modified MV. If the initial MV points to a non-specific reference image, motion compensation can be performed on the current image block after scaling the initial MV, or after scaling and modifying the initial MV. That is, in situations where the initial MV points to a specific reference image, the scaling step can be skipped and the initial MV can be modified directly, or the scaling ratio of the initial MV can be set directly to 1 and then scaled and modified. In situations where the initial MV points to a non-specific reference image, the initial MV can be scaled, or the initial MV can be scaled and modified to improve encoding and decoding efficiency.

[0186] In other embodiments of this application, when the initial MV points to a specific reference image, motion compensation is performed using the initial MV directly, or the initial MV is modified and motion compensation is performed based on the modified MV. When the initial MV points to a non-specific reference image, at least one of scaling and modification is performed on the initial MV and motion compensation is performed. Various embodiments may be combined in any way, and the embodiments of this application are not limited thereto.

[0187] In some embodiments, obtaining the initial motion vector MV of the current image block includes obtaining a list of candidate MVs for the current image block and determining the initial MV based on the list of candidate MVs. That is, after scaling the initial MV (including scaling with a scaling ratio of 1), an MV may be selected and modified so that the initial MV points to an unspecified reference image, and any of the initial MVs may be modified regardless of the type of reference image they point to.

[0188] For selection to be possible, the reference image pointed to by any of the candidate MVs in the MV candidate list is an unspecified reference image.

[0189] Selectively, obtaining the MV candidate list for the current image block may include determining a candidate MV to add to the MV candidate list, and if the reference image pointed to by the candidate MV is an unspecified reference image, adding the candidate MV to the MV candidate list.

[0190] Optionally, the method may further include determining that the reference image pointed to by the candidate MV is an unspecified reference image based on the frame marks of the reference image pointed to by the candidate MV.

[0191] Selectable, the MV candidate list may include at least one of the following candidate MVs: if the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for the unspecified reference image, the merge candidate MV obtained for the unspecified reference image, the MV obtained by interpolation from the unspecified reference image, and the upper adjacent MV and left adjacent MV of the current block with respect to the unspecified reference image.

[0192] Selectively determining the initial MV based on the MV candidate list may include determining the strain cost corresponding to the candidate MV in the MV candidate list based on at least one of the bidirectional matching method and the template matching method, and selecting the MV with the smallest strain cost in the MV candidate list as the initial MV.

[0193] Selectively, modifying the scaled MV and obtaining the modified MV may include generating a sub-MV candidate list in the current image block, the sub-MV candidate list containing the scaled MV, and determining the MV with the smallest distortion cost from the sub-MV candidate list as the modified MV.

[0194] Selectively, the initial MV includes a first initial MV and a second initial MV, and the method includes at least one of the following: if the reference image pointed to by at least one of the first and second initial MVs is a specific reference image, setting the scaling ratios of both the first and second initial MVs to 1 and performing motion compensation on the current image block based on the first and second initial MVs; or if the reference image pointed to by both the first and second initial MVs is a non-specific reference image, determining the scaling ratio of the first initial MV and the scaling ratio of the second initial MV, scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and performing motion compensation on the current image block based on the scaled first initial MV and the scaled second initial MV.

[0195] Selectively, motion compensation for the current image block based on the scaled first initial MV and the scaled second initial MV includes generating a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the scaled first initial MV and belongs to the second reference image, the second reference image block corresponds to the scaled second initial MV and belongs to the first reference image, the first reference image is a reference image pointed to by the first initial MV, and the second reference image is a reference image pointed to by the second initial MV, and modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV.

[0196] Selectively modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV includes using N third reference image blocks to match each with the template, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; using M fourth reference image blocks to match each with the template, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; and based on the matching result, selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs, and using the one third initial MV and the one fourth initial MV as the MV of the current image block, or determining the MV of the current image block.

[0197] Selectively, the third initial MV may include the first initial MV, and the fourth initial MV may include the second initial MV.

[0198] Selectively, at least some of the initial MVs in the N third initial MVs may be obtained by shifting based on the first initial MV, and at least some of the initial MVs in the M fourth initial MVs may be obtained by shifting based on the second initial MV.

[0199] For selection to be possible, N may be equal to M.

[0200] Selectively, the first reference image is the forward frame of the current image block and the second reference image is the backward frame of the current image block, or the first reference image is the forward frame of the current image block and the second reference image is the forward frame of the current image block.

[0201] It is understood that the selectable embodiments described above can be implemented in a manner similar to the details of Method 100, and will not be described further here.

[0202] Figure 10 is a schematic frame diagram of an image motion compensation device 1000 according to one embodiment of this application. As shown in Figure 10, the image motion compensation device 1000 is A memory 1001 for storing executable commands for the computer, The system includes at least one processor 1002 used alone or jointly to perform the following operations by accessing the at least one memory and executing a command executable by the computer, the operations being: Obtain the initial motion vector MV of the current image block, and The scaling ratio of the initial MV is determined, and here, if the initial MV points to a specific reference image, the scaling ratio of the initial MV is 1. Scaling the initial MV based on the scaling ratio of the initial MV, This involves performing motion compensation on the current image block based on the scaled MV.

[0203] In some embodiments, the processor 1002 performs motion compensation on the current image block based on the scaled MV, The scaling of the MV is corrected, and the corrected MV is obtained. This includes performing motion compensation on the current image block based on the modified MV.

[0204] In some embodiments, the processor 1002 obtains the initial motion vector MV of the current image block, Obtaining the MV candidate list for the current image block, This includes determining the initial MV based on the aforementioned MV candidate list.

[0205] In some embodiments, the reference image pointed to by any of the candidate MVs in the MV candidate list is an unspecified reference image.

[0206] In some embodiments, the processor 1002 obtains the MV candidate list for the current image block, This includes determining a candidate MV to be added to the aforementioned MV candidate list, and if the reference image pointed to by the candidate MV is an unspecified reference image, adding the candidate MV to the aforementioned MV candidate list.

[0207] In some embodiments, the processor 1002 further, This is to determine that the reference image pointed to by the candidate MV is an unspecified reference image, based on the frame marks of the reference image pointed to by the candidate MV.

[0208] In some embodiments, the MV candidate list is, If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for the unspecified reference image, Merge candidate MV obtained for a non-specific reference image, MV obtained by interpolation from a non-specific reference image, and The current block includes at least one candidate MV from among the upper adjacent MV and the left adjacent MV with respect to the unspecified reference image.

[0209] In some embodiments, the processor 1002 determines the initial MV based on the MV candidate list, Based on at least one of the bidirectional matching method and the template matching method, the distortion cost corresponding to the candidate MV in the MV candidate list is determined, This includes selecting the MV with the smallest distortion cost in the MV candidate list as the initial MV.

[0210] In some embodiments, the processor 1002 modifies the scaled MV and obtains the modified MV, A sub-MV candidate list is generated for the current image block, and the sub-MV candidate list includes the scaled MV. This includes determining the MV with the smallest distortion cost from the sub-MV candidate list as the corrected MV.

[0211] In some embodiments, the initial MV includes a first initial MV and a second initial MV, and the processor 1002 specifically, If the reference image pointed to by at least one of the first initial MV and the second initial MV is a specific reference image, the scaling ratios of both the first initial MV and the second initial MV are set to 1, and motion compensation is performed on the current image block based on the first initial MV and the second initial MV. If the reference image pointed to by the first initial MV and the second initial MV is an unspecified reference image, this is for at least one of the following: determining the scaling ratio of the first initial MV and the scaling ratio of the second initial MV, scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and performing motion compensation on the current image block based on the scaled first initial MV and the scaled second initial MV.

[0212] In some embodiments, the processor 1002 performs motion compensation on the current image block based on the first initial MV after scaling and the second initial MV after scaling. A template is generated based on a first reference image block and a second reference image block, wherein the first reference image block corresponds to the scaled first initial MV and belongs to the second reference image, the second reference image block corresponds to the scaled second initial MV and belongs to the first reference image, the first reference image is the reference image pointed to by the first initial MV, and the second reference image is the reference image pointed to by the second initial MV. This includes modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV.

[0213] In some embodiments, the processor 1002 modifies the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV. N third reference image blocks are used to match each of the templates, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image. Using M fourth reference image blocks, each is matched with the template, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image. The method includes selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs based on the matching results, and using the one third initial MV and the one fourth initial MV as the MV of the current image block, or determining the MV of the current image block.

[0214] In some embodiments, the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV.

[0215] In some embodiments, at least some of the initial MVs in the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the initial MVs in the M fourth initial MVs are obtained by shifting based on the second initial MV.

[0216] In some embodiments, N is equal to M.

[0217] In some embodiments, the first reference image is the forward frame of the current image block, the second reference image is the backward frame of the current image block, or The first reference image is the forward-facing frame of the current image block, and the second reference image is the forward-facing frame of the current image block.

[0218] In some embodiments, the specific reference image includes at least one of a long-term reference image, a composite frame, and a non-output frame.

[0219] In some embodiments, the processor 1002 further, This is for determining that the aforementioned reference image is the specific reference image when it is determined that the aforementioned reference image is a frame in which no output is generated, and further determined that the aforementioned reference image is a composite frame.

[0220] In some embodiments, the non-specific reference image includes a short-term reference image.

[0221] It is understood that the image motion compensation device 1000 can also be implemented using a corresponding software module, and this will not be explained further here.

[0222] Each embodiment of this application can be implemented by a memory and a processor, each memory being for storing commands for performing the methods of the embodiments of this application, and the processor is for executing the aforementioned commands, causing the device to perform the methods of each embodiment of this application.

[0223] The processor referred to in the embodiments of this application may be a CPU (Central Processing Unit), but may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates, or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or this processor may be any ordinary processor, etc.

[0224] It should be further understood that the memory referred to in the embodiments of the present application may be a volatile memory, a non-volatile memory, or may include both volatile and non-volatile memories. Here, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM) and is used as an external high-speed buffer memory. By way of illustrative but not limiting description, many forms of RAM can be used, for example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0225] In the case where the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate, or transistor logic device, discrete hardware component, it is necessary to explain that the memory (memory module) is integrated into the processor.

[0226] It should be noted that the memory described in this text includes these and any other suitable types of memory, but is not limited thereto.

[0227] Embodiments of the present application further provide a computer-readable storage medium, in which commands are stored, and when the commands are run on a computer, the computer is caused to execute the methods of the embodiments of the above-described respective methods.

[0228] Embodiments of the present application further provide a computer device, which includes the above-described computer-readable storage medium.

[0229] Embodiments of the present application further provide an encoding device, which includes at least one of an image motion compensation device 70, an image processing device 80, and an image motion compensation device 1000.

[0230] Embodiments of the present application further provide a decoding device, which includes at least one of an image motion compensation device 70, an image processing device 80, and an image motion compensation device 1000.

[0231] Embodiments of the present application can be applied to aircraft, particularly in the field of unmanned aircraft.

[0232] It should be understood that the classification of the electric circuits, sub-electric circuits, and sub-units in each embodiment of the present application is only schematic. Those skilled in the art can recognize that each exemplary electric circuit, sub-electric circuit, and sub-unit described by the embodiments disclosed in the present text can be disassembled or combined.

[0233] In the embodiments described above, implementation can be carried out in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, it can be carried out in whole or in part by a computer program product. The computer program product includes one or more computer commands. When a computer loads or executes computer commands, it generates, in whole or in part, a process or function based on the embodiments of this application. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable device. Computer commands can be stored in a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, computer commands can be transmitted from a site, computer, server, or data center to another site, computer, server, or data center by wired means (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless means (e.g., infrared, radio, microwave, etc.). A computer-readable storage medium may be any available medium accessible by a computer, or it may be a data storage device including a server, data center, etc., which integrates one or more available media. Available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs (Digital Video Discs)), or semiconductor media (e.g., SSDs (Solid State Disks)).

[0234] Throughout this specification, the terms "one embodiment" or "embodiment" refer to a particular feature, structure, or characteristic associated with that embodiment, and should be understood to mean that such feature, structure, or characteristic is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" in any other part of this specification do not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics can be combined in any suitable manner into one or more embodiments.

[0235] In each embodiment of this application, the numbering of each process described above does not indicate the order of execution. The execution order of each process is determined by its function and inherent logic, and should be understood as not imposing any restrictions on the implementation of the embodiments of this application.

[0236] In the embodiments of this application, "B corresponding to A" should be understood as indicating that B is related to A and that B can be determined based on A. However, it should be understood that determining B based on A does not mean determining B solely by A, but rather that B can be determined based on at least one of A and other information.

[0237] In this text, the term "at least one" merely describes the relationship between related objects; three relationships are possible. For example, "at least one of A and B" can be in three situations: A exists alone, A and B exist simultaneously, or B exists alone. In addition, the symbol " / " in this text usually indicates that the related objects before and after it are in an "or" relationship.

[0238] Those skilled in the art will recognize, based on the exemplary units and algorithmic steps described in the embodiments disclosed herein, that these can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are ultimately performed in hardware or software will depend on the specific application of the technical solution and the design constraints. Those skilled in the art will recognize that the functions described can be implemented by using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

[0239] Those skilled in the art will clearly understand, for the sake of clarity and conciseness, that the specific operating processes of the systems, apparatus, and units described above can be referenced to the corresponding processes in the embodiments of the methods described above, and will not be described further here.

[0240] In the various embodiments provided in this application, it should be understood that the systems, apparatus, and methods disclosed can be implemented in other ways. For example, the embodiments of the apparatus described above are illustrative only. For example, the division of the units is merely a division of logic functions, and in actual implementation, there may be other methods of division, for example, multiple units or components may be combined or integrated into other systems, or some features may be ignored or not performed. Also, the shown or considered connections, direct connections, or communication connections between them may be by some interface, and the indirect connections or communication connections between apparatus or units may be electrical, mechanical, or in other forms.

[0241] The unit described as the separating member may or may not be physically separated, and the member shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed among multiple network units. Some or all of these units can be selected as needed to achieve the objective of the solution of this embodiment.

[0242] In addition, each functional unit in each embodiment of this application can be integrated into a single processing unit, each unit can exist physically independently, or two or more units can be integrated into a single unit.

[0243] The foregoing describes only specific embodiments of this application; however, the scope of protection of this application is not limited thereto. Any person skilled in the art familiar with the art can easily conceive of any modification or substitution within the scope of the technology disclosed herein, and such modifications or substitutions will be included within the scope of protection of this application. Accordingly, the scope of protection of this application shall be the same as the scope of protection of the claims described herein. (Item 1) Image motion compensation method, Obtain the initial motion vector MV of the current image block, and If the reference image pointed to by the initial MV is a specific reference image, motion compensation is performed on the current image block based on the initial MV. An image motion compensation method that includes, if the reference image pointed to by the initial MV is an unspecified reference image, modifying the initial MV to obtain a modified MV, and performing motion compensation on the current image block based on the modified MV. (Item 2) If the reference image pointed to by the initial MV is an unspecified reference image, then modifying the initial MV, obtaining the modified MV, and performing motion compensation on the current image block based on the modified MV is possible. Obtain the MV candidate list for the current image block above, and confirm that the reference image pointed to by any of the candidate MVs in the above MV candidate list is an unspecified reference image. Based on the above list of MV candidates, the initial MV will be finalized as described above. The initial MV mentioned above will be revised, and the revised MV will be obtained. The method described in item 1, which includes performing motion compensation on the current image block based on the above-mentioned modified MV. (Item 3) Obtaining the MV candidate list of the current image block is The method according to item 2, including determining a candidate MV for adding to the MV candidate list, and adding the candidate MV to the MV candidate list when the reference image pointed to by the candidate MV is an unspecified reference image. (Item 4) The method is The method according to item 3, further including determining that the reference image pointed to by the candidate MV is an unspecified reference image based on the frame mark of the reference image pointed to by the candidate MV. (Item 5) The MV candidate list is When the current image block is in the advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for the unspecified reference image, The merge candidate MV obtained for the unspecified reference image, The MV obtained by interpolation from the unspecified reference image, and The method according to any one of items 2 to 4, including at least one candidate MV of the upper adjacent MV and the left adjacent MV of the current block for the unspecified reference image. (Item 6) Determining the initial MV based on the MV candidate list is Determining the distortion cost corresponding to the candidate MV in the MV candidate list based on at least one of the bidirectional matching method and the template matching method, The method according to item 5, including setting the MV with the smallest distortion cost in the MV candidate list as the initial MV. (Item 7) Modifying the initial MV to obtain the modified MV is Generating a sub-MV candidate list for the current image block, and including the initial MV in the sub-MV candidate list, The method according to item 6, including determining the MV with the smallest distortion cost from the sub-MV candidate list as the modified MV. (Item 8) The initial MV includes a first initial MV and a second initial MV. If the reference image pointed to by the initial MV above is a specific reference image, then performing motion compensation on the current image block based on the initial MV above is: If the reference image pointed to by at least one of the first initial MV and the second initial MV is a specific reference image, motion compensation is performed on the current image block based on the first initial MV and the second initial MV. The method according to item 1, which includes, if the reference image pointed to by the first initial MV and the second initial MV is an unspecified reference image, modifying the first initial MV and the second initial MV to obtain a modified first MV and a modified second MV, and performing motion compensation on the current image block based on the modified first MV and the modified second MV. (Item 9) Modifying the above-mentioned first initial MV and second initial MV to obtain the modified first MV and modified second MV is, A template is generated based on the first reference image block and the second reference image block, wherein the first reference image block corresponds to the first initial MV and belongs to the first reference image, and the second reference image block corresponds to the second initial MV and belongs to the second reference image. The method according to item 8, which includes modifying the above-mentioned first initial MV and the above-mentioned second initial MV based on the above-mentioned template to obtain the above-mentioned modified first MV and the above-mentioned modified second MV. (Item 10) Based on the above template, modifying the above first initial MV and the above second initial MV to obtain the above modified first MV and the above modified second MV is: Using N third reference image blocks, each is matched with the above template, where the above N third reference image blocks correspond to N third initial MVs and belong to the above first reference image. Using M fourth reference image blocks, each is matched with the above template, where the above M fourth reference image blocks correspond to M fourth initial MVs and belong to the above second reference image. The method according to item 9, which includes selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs based on the matching results, and using the one third initial MV and the one fourth initial MV as the MV of the current image block, or to determine the MV of the current image block. (Item 11) The method described in item 10, wherein the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV. (Item 12) The method according to item 10 or 11, wherein at least some of the initial MVs in the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the initial MVs in the M fourth initial MVs are obtained by shifting based on the second initial MV. (Item 13) The above N is the method described in any one of items 10 to 12, which is equal to the above M. (Item 14) The first reference image above is the forward frame of the current image block, the second reference image above is the backward frame of the current image block, or, The method described in any one of items 10 to 13, wherein the first reference image above is the forward frame of the current image block, and the second reference image above is the forward frame of the current image block. (Item 15) The specified reference image described above is the method described in any one of items 1 to 14, which includes at least one of a long-term reference image, a composite frame, or a frame that is not output. (Item 16) The above method, The method according to any one of items 1 to 15, further comprising determining that the above reference image is the specific reference image if it is determined that the above reference image is not output in the frame and that the above reference image is a composite frame. (Item 17) The above non-specific reference image is the method described in any one of items 1 to 16, including the short-term reference image. (Item 18) Image motion compensation device, At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable on the computer, wherein the operations are: Obtain the initial motion vector MV of the current image block, and If the reference image pointed to by the initial MV is a specific reference image, motion compensation is performed on the current image block based on the initial MV. An image motion compensation device that, when the reference image pointed to by the initial MV is an unspecified reference image, modifies the initial MV to obtain a modified MV, and performs motion compensation on the current image block based on the modified MV. (Item 19) The above processor, specifically, Obtain the MV candidate list for the current image block above, and if any of the candidate MVs in the MV candidate list point to a reference image that is not a specific reference image, Based on the above list of MV candidates, the initial MV will be finalized as follows: After correcting the initial MV mentioned above and obtaining the corrected MV, Image motion compensation device as described in item 18, which is for performing motion compensation on the current image block based on the above-mentioned modified MV. (Item 20) The above processor further, The image motion compensation device described in item 19, which determines a candidate MV to be added to the above MV candidate list, and adds the candidate MV to the above MV candidate list if the reference image pointed to by the above candidate MV is an unspecified reference image. (Item 21) The above processor further, An image motion compensation device according to any one of items 18 to 20, which determines that the reference image pointed to by the candidate MV is an unspecified reference image based on the frame marks of the reference image pointed to by the candidate MV. (Item 22) The above list of MV candidates is: If the above current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for the unspecified reference image, Merge candidate MV obtained for a non-specific reference image, MV obtained by interpolation from a non-specific reference image, and Image motion compensation device according to item 21, which includes at least one candidate MV from the upper adjacent MV and the left adjacent MV for the unspecified reference image of the current block. (Item 23) The above processor, specifically, Based on at least one of the bidirectional matching method and the template matching method, the distortion cost corresponding to the candidate MV in the above MV candidate list is determined. The image motion compensation device described in item 22 is intended to select the MV with the smallest distortion cost in the above MV candidate list as the initial MV. (Item 24) The above processor, specifically, A sub-MV candidate list is generated in the current image block above, and the sub-MV candidate list includes the initial MV above. An image motion compensation device described in any one of items 18-20, which is used to determine the MV with the lowest distortion cost from the above sub-MV candidate list as the corrected MV. (Item 25) The above initial MVs include the first initial MV and the second initial MV. The above processor, specifically, If the reference image pointed to by at least one of the first initial MV and the second initial MV is a specific reference image, motion compensation is performed on the current image block based on the first initial MV and the second initial MV. The image motion compensation device described in item 18, which, when the reference image pointed to by the first initial MV and the second initial MV is an unspecified reference image, modifies the first initial MV and the second initial MV to obtain a modified first MV and a modified second MV, and performs motion compensation on the current image block based on the modified first MV and the modified second MV. (Item 26) The above processor, specifically, A template is generated based on the first reference image block and the second reference image block, where the first reference image block corresponds to the first initial MV and belongs to the first reference image, and the second reference image block corresponds to the second initial MV and belongs to the second reference image. The image motion compensation device described in item 25 is for modifying the above-mentioned first initial MV and the above-mentioned second initial MV based on the above-mentioned template, in order to obtain the above-mentioned modified first MV and the above-mentioned modified second MV. (Item 27) The above processor, specifically, Using N third reference image blocks, each is matched with the above template, where the above N third reference image blocks correspond to N third initial MVs and belong to the above first reference image. Using M fourth reference image blocks, each is matched with the above template, where the above M fourth reference image blocks correspond to M fourth initial MVs and belong to the above second reference image. An image motion compensation device according to item 26, which, based on the matching results described above, selects one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs, and uses the one third initial MV and the one fourth initial MV as the MV of the current image block, or to determine the MV of the current image block. (Item 28) The image motion compensation device described in item 27, wherein the above-mentioned third initial MV includes the above-mentioned first initial MV, and the above-mentioned fourth initial MV includes the above-mentioned second initial MV. (Item 29) The image motion compensation device according to item 27 or 28, wherein at least some of the initial MVs in the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the initial MVs in the M fourth initial MVs are obtained by shifting based on the second initial MV. (Item 30) The above N is an image motion compensation device as described in any one of items 27 to 29, which is equivalent to the above M. (Item 31) The first reference image above is the forward frame of the current image block, the second reference image above is the backward frame of the current image block, or, An image motion compensation device according to any one of items 27 to 30, wherein the first reference image above is a forward-facing frame of the current image block, and the second reference image above is a forward-facing frame of the current image block. (Item 32) The above-mentioned specific reference image is an image motion compensation device according to any one of items 18 to 31, which includes at least one of a long-term reference image, a composite frame, or a frame that is not output. (Item 33) The above processor further, An image motion compensation device described in any one of items 18 to 32, which is used to determine that the above-mentioned reference image is the above-mentioned specific reference image when it is determined that the above-mentioned reference image is a frame in which no output is given, and further determined that the above-mentioned reference image is a composite frame. (Item 34) The above non-specific reference image is an image motion compensation device as described in any one of items 18 to 33, including a short-term reference image. (Item 35) A computer-readable storage medium, This includes a computer-readable storage medium that stores commands and, when the commands are executed by a computer, causes the computer to execute one of the image motion compensation methods described in item 1 to 17. (Item 36) An encoding device including an image motion compensation device as described in any one of items 18 to 34. (Item 37) A decoding device including an image motion compensation device as described in any one of items 18 to 34. (Item 38) Image motion compensation method, Obtain the initial motion vector MV of the current image block, and The scaling ratio of the initial MV is determined, and here, if the initial MV points to a specific reference image, the scaling ratio of the initial MV is 1. Based on the scaling ratio of the initial MV described above, the initial MV is scaled, An image motion compensation method, which includes performing motion compensation on the current image block based on the above-mentioned scaled MV. (Item 39) Performing motion compensation on the current image block based on the above scaled MV is, The above scaling process is modified to obtain the modified MV, The method described in item 38, which includes performing motion compensation on the current image block based on the above-mentioned modified MV. (Item 40) To obtain the initial motion vector MV of the current image block, To obtain the list of MV candidates for the current image block mentioned above, The method described in item 38 or 39, which includes determining the initial MV based on the above MV candidate list. (Item 41) The reference image pointed to by any of the candidate MVs in the above MV candidate list is a non-specific reference image, as described in item 40. (Item 42) To obtain the list of MV candidates for the current image block mentioned above, The method described in item 41, which includes determining a candidate MV to be added to the above MV candidate list, and adding the candidate MV to the above MV candidate list if the reference image pointed to by the candidate MV is an unspecified reference image. (Item 43) The above method, The method of item 42, further comprising determining that the reference image pointed to by the candidate MV is an unspecified reference image based on the frame marks of the reference image pointed to by the candidate MV. (Item 44) The above list of MV candidates is: If the above current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for the unspecified reference image, Merge candidate MV obtained for a non-specific reference image, MV obtained by interpolation from a non-specific reference image, and The method described in any one of items 40 to 43, which includes at least one candidate MV from the upper adjacent MV and the left adjacent MV for the unspecified reference image of the current block. (Item 45) Based on the above list of MV candidates, determining the initial MV is as follows: Based on at least one of the bidirectional matching method and the template matching method, the distortion cost corresponding to the candidate MV in the above MV candidate list is determined, The method described in item 44, which includes setting the MV with the smallest distortion cost in the above MV candidate list as the initial MV. (Item 46) To modify the MV after scaling as described above and obtain the modified MV, A sub-MV candidate list is generated in the current image block mentioned above, and this sub-MV candidate list includes the scaled MV mentioned above. The method described in item 45, which includes determining the MV with the smallest distortion cost from the above sub-MV candidate list as the corrected MV. (Item 47) The above initial MV includes the first initial MV and the second initial MV, and the above method is If the reference image pointed to by at least one of the first initial MV and the second initial MV is a specific reference image, the scaling ratios of both the first and second initial MVs are set to 1, and motion compensation is performed on the current image block based on the first and second initial MVs. The method according to item 38, which includes at least one of the following: if the reference image pointed to by the first initial MV and the second initial MV is an unspecified reference image, determining the scaling ratio of the first initial MV and the scaling ratio of the second initial MV, scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and performing motion compensation on the current image block based on the first initial MV after scaling and the second initial MV after scaling. (Item 48) Performing motion compensation on the current image block based on the first initial MV after scaling and the second initial MV after scaling is: A template is generated based on the first reference image block and the second reference image block, wherein the first reference image block corresponds to the first initial MV after scaling and belongs to the second reference image, the second reference image block corresponds to the second initial MV after scaling and belongs to the first reference image, the first reference image is the reference image pointed to by the first initial MV, and the second reference image is the reference image pointed to by the second initial MV. The method described in item 47, which includes modifying the above-mentioned first initial MV and the above-mentioned second initial MV based on the above-mentioned template to obtain the above-mentioned modified first MV and the above-mentioned modified second MV. (Item 49) Based on the above template, modifying the above first initial MV and the above second initial MV to obtain the above modified first MV and the above modified second MV is: Using N third reference image blocks, each is matched with the above template, where the above N third reference image blocks correspond to N third initial MVs and belong to the above first reference image. Using M fourth reference image blocks, each is matched with the above template, where the above M fourth reference image blocks correspond to M fourth initial MVs and belong to the above second reference image. The method according to item 47, which includes selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs based on the matching results, and using the one third initial MV and the one fourth initial MV as the MV of the current image block, or to determine the MV of the current image block. (Item 50) The method described in item 49, wherein the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV. (Item 51) The method according to item 49 or 50, wherein at least some of the initial MVs in the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the initial MVs in the M fourth initial MVs are obtained by shifting based on the second initial MV. (Item 52) The above N is the method described in any one of items 49 to 51, which is equal to the above M. (Item 53) The first reference image above is the forward frame of the current image block, the second reference image above is the backward frame of the current image block, or, The method described in any one of items 49 to 52, wherein the first reference image above is the forward frame of the current image block, and the second reference image above is the forward frame of the current image block. (Item 54) The specified reference image described above is the method described in any one of items 38 to 53, which includes at least one of a long-term reference image, a composite frame, or a frame that is not output. (Item 55) The above method, The method according to any one of items 38 to 54, further comprising determining that the above-mentioned reference image is the specific reference image if it is determined that the above-mentioned reference image is a frame in which no output is generated, and if it is further determined that the above-mentioned reference image is a composite frame. (Item 56) The above non-specific reference image is the method described in any one of items 38 to 55, including the short-term reference image. (Item 57) Image motion compensation device, At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable on the computer, wherein the operations are: Obtain the initial motion vector MV of the current image block, and The scaling ratio of the initial MV is determined, and here, if the initial MV points to a specific reference image, the scaling ratio of the initial MV is 1. Based on the scaling ratio of the initial MV described above, the initial MV is scaled, The apparatus performs motion compensation on the current image block based on the above-mentioned scaled MV. (Item 58) The above processor performs motion compensation on the current image block based on the above scaled MV, The above scaling process is modified to obtain the modified MV, Image motion compensation device as described in item 57, which includes performing motion compensation on the current image block based on the above-mentioned modified MV. (Item 59) The above processor obtains the initial motion vector MV of the current image block, To obtain the list of MV candidates for the current image block mentioned above, Image motion compensation device according to item 57 or 58, which includes determining the initial MV based on the above MV candidate list. (Item 60) The reference image pointed to by any of the candidate MVs in the above MV candidate list is an unspecified reference image, as described in item 59, which is the image motion compensation device. (Item 61) The above processor obtains the MV candidate list for the current image block, The image motion compensation device according to item 60, which includes determining a candidate MV to be added to the above MV candidate list, and adding the candidate MV to the above MV candidate list if the reference image pointed to by the above candidate MV is an unspecified reference image. (Item 62) The above processor further, The image motion compensation device described in item 61, which determines that the reference image pointed to by the candidate MV is an unspecified reference image based on the frame marks of the reference image pointed to by the candidate MV. (Item 63) The above list of MV candidates is: If the above current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for the unspecified reference image, Merge candidate MV obtained for a non-specific reference image, MV obtained by interpolation from a non-specific reference image, and An image motion compensation device according to any one of items 59 to 62, which includes at least one candidate MV from the upper adjacent MV and the left adjacent MV for the unspecified reference image of the current block. (Item 64) The above processor determines the above initial MV based on the above MV candidate list. Based on at least one of the bidirectional matching method and the template matching method, the distortion cost corresponding to the candidate MV in the above MV candidate list is determined, The image motion compensation device described in item 63, which includes setting the MV with the smallest distortion cost in the above MV candidate list as the initial MV. (Item 65) The above processor corrects the scaled MV and obtains the corrected MV. A sub-MV candidate list is generated in the current image block mentioned above, and this sub-MV candidate list includes the scaled MV mentioned above. The image motion compensation device described in item 64, which includes determining the MV with the smallest distortion cost from the above sub-MV candidate list as the corrected MV. (Item 66) The above initial MV includes the first initial MV and the second initial MV, and the above processor specifically, If the reference image pointed to by at least one of the first initial MV and the second initial MV is a specific reference image, the scaling ratios of both the first and second initial MVs are set to 1, and motion compensation is performed on the current image block based on the first and second initial MVs. Image motion compensation device according to item 57, for at least one of the following: determining the scaling ratio of the first initial MV and the scaling ratio of the second initial MV when both the first initial MV and the second initial MV are directed to unspecified reference images; scaling the first initial MV based on the scaling ratio of the first initial MV; scaling the second initial MV based on the scaling ratio of the second initial MV; and performing motion compensation on the current image block based on the scaled first initial MV and the scaled second initial MV. (Item 67) The above processor performs motion compensation on the current image block based on the first initial MV after scaling and the second initial MV after scaling. A template is generated based on the first reference image block and the second reference image block, wherein the first reference image block corresponds to the first initial MV after scaling and belongs to the second reference image, the second reference image block corresponds to the second initial MV after scaling and belongs to the first reference image, the first reference image is the reference image pointed to by the first initial MV, and the second reference image is the reference image pointed to by the second initial MV. The image motion compensation device according to item 66, which includes modifying the above-mentioned first initial MV and the above-mentioned second initial MV based on the above-mentioned template to obtain the above-mentioned modified first MV and the above-mentioned modified second MV. (Item 68) The above processor modifies the above first initial MV and the above second initial MV based on the above template to obtain the above modified first MV and the above modified second MV, Using N third reference image blocks, each is matched with the above template, where the above N third reference image blocks correspond to N third initial MVs and belong to the above first reference image. Using M fourth reference image blocks, each is matched with the above template, where the above M fourth reference image blocks correspond to M fourth initial MVs and belong to the above second reference image. An image motion compensation device according to item 66, which includes selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs based on the matching results above, and using the one third initial MV and the one fourth initial MV as the MV of the current image block, or to determine the MV of the current image block. (Item 69) The image motion compensation device described in item 68, wherein the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV. (Item 70) The image motion compensation device according to item 68 or 69, wherein at least some of the initial MVs in the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the initial MVs in the M fourth initial MVs are obtained by shifting based on the second initial MV. (Item 71) The above N is an image motion compensation device as described in any one of items 68 to 70, which is equivalent to the above M. (Item 72) The first reference image above is the forward frame of the current image block, the second reference image above is the backward frame of the current image block, or, The first reference image above is a forward-facing frame of the current image block, and the second reference image above is a forward-facing frame of the current image block, as described in any one of items 68 to 71, for the image motion compensation apparatus. (Item 73) The above-mentioned specific reference image is an image motion compensation device as described in any one of items 57 to 72, which includes at least one of a long-term reference image, a composite frame, or a frame that is not output. (Item 74) The above processor further, An image motion compensation device described in any one of items 57 to 73, which is used to determine that the above-mentioned reference image is the above-mentioned specific reference image when it is determined that the above-mentioned reference image is a frame in which no output is given, and further determined that the above-mentioned reference image is a composite frame. (Item 75) The above non-specific reference image is an image motion compensation device as described in any one of items 57 to 74, including a short-term reference image. (Item 76) A computer-readable storage medium, This includes a computer-readable storage medium on which commands are stored, and which, when the commands are executed by a computer, causes the computer to execute one of the image motion compensation methods described in any one of items 38 to 56. (Item 77) An encoding device including an image motion compensation device as described in any one of items 57 to 75. (Item 78) A decoding device including an image motion compensation device as described in any one of items 57 to 75. (Item 79) An image processing method, A first initial motion vector MV and a second initial MV are obtained, wherein the first initial MV points to the first reference image, and the second initial MV points to the second reference image. If at least one of the above first reference image and the above second reference image is a specific reference image, then the predicted image block of the current image block is calculated and obtained based on the above first initial MV and the above second initial MV, A method that, when both the first reference image and the second reference image are unspecified reference images, calculates and obtains the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and calculates and obtains a predicted image block of the current image block based on the MV of the current image block. (Item 80) Based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV mentioned above, the MV of the current image block can be calculated and obtained as follows: The method according to item 79, which includes calculating and obtaining the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and an optimization principle. (Item 81) Based on the above first initial MV and the above second initial MV, the predicted image block of the current image block is calculated and obtained as follows: The method according to 79 or 80, which includes weighting and summing the first reference image block pointed to by the first initial MV and the second reference image block pointed to by the second initial MV to obtain a predicted image block for the current image block. (Item 82) The prediction direction of the first initial MV and the second initial MV described above is determined by the method described in any one of the different items 79 to 81. (Item 83) The prediction direction of the first initial MV and the second initial MV is the same, neither the first initial MV nor the second initial MV is 0, and the first reference image and the second reference image are different, as described in any one of items 79 to 82. (Item 84) Based on the MV of the current image block above, the predicted image block obtained by calculating the current image block is: The method according to any one of items 79 to 83, which includes calculating and obtaining a predicted image block of the current image block based on the MV of the current image block, the first reference image block pointed to by the first initial MV, and the second reference image block pointed to by the second initial MV. (Item 85) The above-mentioned specific reference image is the method described in any one of items 79 to 84, which includes at least one of a long-term reference image, a composite frame, or a frame that is not output. (Item 86) The above method, The method according to any one of items 79 to 85, further comprising determining that at least one of the above-mentioned first reference image and second reference image is a frame that is not output, and further determining that at least one of the above-mentioned first reference image and second reference image is a composite frame, and then determining that at least one of the above-mentioned first reference image and second reference image is the above-mentioned specific reference image. (Item 87) The above non-specific reference image is the method described in any one of items 79 to 86, which includes a short-term reference image. (Item 88) An image processing device, At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable on the computer, wherein the operations are: A first initial motion vector MV and a second initial MV are obtained, wherein the first initial MV points to the first reference image, and the second initial MV points to the second reference image. If at least one of the above first reference image and the above second reference image is a specific reference image, then the predicted image block of the current image block is calculated and obtained based on the above first initial MV and the above second initial MV, The apparatus is characterized in that, when both the first reference image and the second reference image are unspecified reference images, it calculates and obtains the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and then calculates and obtains a predicted image block of the current image block based on the MV of the current image block. (Item 89) The above processor, specifically, The apparatus described in item 88 is for calculating and obtaining the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and the optimization principle. (Item 90) The above processor, specifically, The apparatus according to 88 or 89, which weights and sums the first reference image block pointed to by the first initial MV and the second reference image block pointed to by the second initial MV to obtain a predicted image block for the current image block. (Item 91) The prediction direction of the first initial MV and the second initial MV described above is determined by the apparatus described in any one of the items 88 to 90. (Item 92) The prediction direction of the first initial MV and the second initial MV is the same, neither the first initial MV nor the second initial MV is 0, and the first reference image and the second reference image are different devices as described in any one of items 88 to 91. (Item 93) The above processor, specifically, An apparatus according to any one of items 88 to 92, for calculating and obtaining a predicted image block of the current image block based on the MV of the current image block, the first reference image block pointed to by the first initial MV, and the second reference image block pointed to by the second initial MV. (Item 94) The above-mentioned specific reference image is an apparatus according to any one of items 88 to 93, which includes at least one of a long-term reference image, a composite frame, or a frame that is not output. (Item 95) The above processor further, The apparatus according to any one of items 88 to 94, which determines that at least one of the above-mentioned first reference image and the above-mentioned second reference image is a frame that is not output, and further determines that at least one of the above-mentioned first reference image and the above-mentioned second reference image is a composite frame, and for determining that at least one of the above-mentioned first reference image and the above-mentioned second reference image is the above-mentioned specific reference image. (Item 96) The above non-specific reference image is a device described in any one of items 88 to 95, including a short-term reference image. (Item 97) A computer-readable storage medium, This is a computer-readable storage medium that stores commands and, when the commands are executed by a computer, causes the computer to perform any of the image processing methods described in items 79 to 87. (Item 98) An encoding device including an image processing device as described in any one of items 88 to 96. (Item 99) A decoding device including an image processing device as described in any one of items 88 to 96.

Claims

1. A step of acquiring a first initial motion vector and a second initial motion vector, wherein the first initial motion vector points to a first reference image, and the second initial motion vector points to a second reference image. If both the first and second reference images are non-specific reference images that are short-term reference images, the process involves calculating motion compensation for the current image block based on the gradient values ​​of the pixel points pointed to by the first and second initial motion vectors, and calculating a predicted image block for the current image block based on the motion compensation for the current image block. An image processing method comprising:

2. The image processing method according to claim 1, wherein the first reference image is a forward frame of the current image block, the second reference image is a backward frame of the current image block, and the distance from the first reference image to the current image including the current image block is the same as the distance from the second reference image to the current image.

3. A computer has at least one memory for storing executable instructions, The system includes at least one processor used alone or jointly to perform the following operations by accessing at least one memory and executing instructions executable by the computer, the operations being: The step of acquiring a first initial motion vector and a second initial motion vector, wherein the first initial motion vector points to a first reference image, and the second initial motion vector points to a second reference image. An image processing apparatus that, when both the first reference image and the second reference image are non-specific reference images that are short-term reference images, calculates motion compensation for the current image block based on the gradient values ​​of the pixel points pointed to by the first initial motion vector and the second initial motion vector, and calculates a predicted image block for the current image block based on the motion compensation for the current image block.

4. A step of acquiring a first initial motion vector and a second initial motion vector, wherein the first initial motion vector points to a first reference image, and the second initial motion vector points to a second reference image. If both the first and second reference images are unspecified reference images that are short-term reference images, the process involves calculating motion compensation for the current image block based on the gradient values ​​of the pixel points pointed to by the first and second initial motion vectors, and calculating a predicted image block for the current image block based on the motion compensation for the current image block. The steps include encoding the current image block to generate a bitstream and A bitstream generation method comprising the following features.

Images