Video processing method, bitstream generation method, and video processing apparatus

Abstract

To provide a method and a device for image motion compensation that can increase coding efficiency.SOLUTION: An image processing method includes: acquiring an initial motion vector MV of the current image block including a first initial MV directing a first reference image and a second initial MV directing a second reference image; correcting the first initial MV and the second initial MV to obtain a first MV after correction and a second MV after correction; and executing motion prediction for the current image block on the basis of the first MV after correction and the second MV after correction. The first reference image and the second reference image are a short-term reference image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] Copyright Declaration The information disclosed in this patent document includes copyrighted material. Such copyright is owned by the copyright holder. The copyright holder does not object to anyone reproducing this patent document or the disclosure of this patent as it exists in the official records and public archives of the Patent and Trademark Office.

[0002] This application relates to the field of image processing, and more particularly to a method and apparatus for image motion compensation. [Background technology]

[0003] In recent years, the popularity of portable, handheld, and wearable devices has led to a continuous increase in video file sizes. As video formats become more complex, storing and transmitting them also becomes more difficult. To reduce the bandwidth used for storing and transmitting video, video data is typically encoded and compressed at the encoding end and decoded at the decoding end.

[0004] The symbolic compression process includes processes such as prediction, transformation, quantization, and entropy coding. Here, the prediction includes two types: intra-frame prediction and inter-frame prediction, which aim to remove redundant information of the current image block to be coded by using prediction block data. Intra-frame prediction obtains prediction block data by using the information of the current frame image. Inter-frame prediction obtains prediction block data by using the information of the reference image. The process is to divide the current image to be coded into a plurality of image blocks to be coded, divide the image blocks to be coded into a plurality of sub-image blocks, and then, for each sub-image block, search for the image block in the reference image that most matches the current sub-image block as the predicted image block. The relative displacement between the predicted image block and the current sub-image block is the motion vector. Then, the corresponding pixel values of this sub-image block and the predicted image block are subtracted to obtain the residual. The residuals corresponding to each obtained sub-image block are combined together to obtain the residual of the image block to be coded. After the residuals have undergone processes such as transformation, quantization, and entropy coding, an entropy-coded bitstream is obtained, and the entropy-coded bitstream and coded mode information such as information like the intra-frame prediction mode, motion vector (or motion vector residual), etc. are stored or transmitted to the decoding end.

[0005] At the decoding end of the image, after obtaining the entropy-coded bitstream, entropy decoding is performed to obtain the corresponding residual. Based on information such as the motion vector and intra-frame prediction obtained by decoding, the predicted image block corresponding to the image block to be decoded is searched, and the value of each pixel point in the image block to be decoded is obtained based on the predicted image block and the residual.

[0006] From the above description, when performing inter-frame prediction, it can be seen that the more similar the selected reference image is to the current image to be encoded, the smaller the residual generated by the inter-frame prediction, thereby improving the encoding efficiency of the inter-frame prediction. Specifically, in existing technologies, there are some that can construct a high-quality specific reference image of background content including a scene by using each image in a video. When performing inter-frame prediction, the background part of the current image to be encoded or the current image to be decoded can be used to reduce the residual information of the inter-frame prediction by referring to the high-quality specific reference image, thereby improving the encoding efficiency. That is, this specific reference image is a reference image for inter-frame prediction. The long-term reference image is an artificially constructed image rather than a decoded image. The long-term reference image includes a plurality of image blocks, and any of the image blocks is extracted from a certain decoded image, and different image blocks in the long-term reference image may be from different decoded images.

[0007] In order to improve the encoding efficiency and reduce the amount of information transmitted by the encoding end, in existing technologies, there are some that can directly derive the motion vector at the decoding end. The encoding end does not need to transmit the motion vector information or the motion vector residual information, and the decoding end can also obtain the true motion vector without decoding the motion vector information or the motion vector residual information. Summary of the Invention Problems to be Solved by the Invention

[0008] Existing techniques that use motion vector derivation and techniques that use bidirectional motion prediction sometimes fail to consider the specific characteristics of long-range reference images. Techniques that use motion vector derivation sometimes fail to consider whether the reference image pointed to by the motion vector is a long-range reference image, and when modifying the motion vector, motion search may be performed on the long-range reference image, which can actually reduce search efficiency and encoding efficiency. Techniques that use bidirectional motion prediction manipulate motion vectors based on the temporal correlation of images, and if the reference image pointed to by the associated motion vector is a long-range reference image, these operations may become invalid because the definition of the temporal distance between the current image to be encoded or decoded and the long-range reference image is unclear. [Means for solving the problem]

[0009] This application provides a method and apparatus for image motion compensation, which can improve encoding / decoding efficiency.

[0010] In the first aspect, a method for compensating for image motion is provided, and this method is, Obtain the initial motion vector MV of the current image block, and If the reference image pointed to by the initial MV is a specific reference image, motion compensation is performed on the current image block based on the initial MV. If the reference image pointed to by the initial MV is an unspecified reference image, the process includes modifying the initial MV to obtain a modified MV, and performing motion compensation on the current image block based on the modified MV.

[0011] In a second embodiment, an apparatus for image motion compensation is provided, which apparatus At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable by the computer, the operations being: Obtain the initial motion vector MV of the current image block, and If the reference image pointed to by the initial MV is a specific reference image, motion compensation is performed on the current image block based on the initial MV. If the reference image pointed to by the initial MV is an unspecified reference image, the initial MV is modified to obtain a modified MV, and motion compensation is performed on the current image block based on the modified MV.

[0012] In a third aspect, a computer-readable storage medium is provided, which stores commands, and when the commands are operated by the computer, the computer is instructed to execute the method for image motion compensation according to the first aspect.

[0013] In a fourth aspect, an encoding device is provided that includes an apparatus for image motion compensation according to the second aspect.

[0014] A fifth embodiment provides a decoding device that includes an apparatus for image motion compensation according to the second embodiment.

[0015] In a sixth aspect, a method for image motion compensation is provided, which is Obtain the initial motion vector MV of the current image block, and The scaling ratio of the initial MV is determined, and here, if the initial MV points to a specific reference image, the scaling ratio of the initial MV is 1. Scaling the initial MV based on the scaling ratio of the initial MV, This includes performing motion compensation on the current image block based on the MV after the initial scaling.

[0016] In the seventh aspect, an apparatus for image motion compensation is provided, which apparatus At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable by the computer, the operations being: Obtain the initial motion vector MV of the current image block, and The scaling ratio of the initial MV is determined, and here, if the initial MV points to a specific reference image, the scaling ratio of the initial MV is 1. Scaling the initial MV based on the scaling ratio of the initial MV, This involves performing motion compensation on the current image block based on the scaled MV.

[0017] In the eighth aspect, a computer-readable storage medium is provided, which stores commands, and when the commands are operated by the computer, the computer is instructed to execute the method for image motion compensation according to the sixth aspect.

[0018] In the ninth aspect, an encoding device is provided that includes an apparatus for image motion compensation according to the seventh aspect.

[0019] In a tenth aspect, a decoding device is provided that includes an apparatus for image motion compensation according to a seventh aspect.

[0020] In the eleventh aspect, a method for image processing is provided, which is A first initial motion vector MV and a second initial MV are obtained, wherein the first initial MV points to the first reference image, and the second initial MV points to the second reference image. If at least one of the first reference image and the second reference image is a specific reference image, then the predicted image block of the current image block is calculated and obtained based on the first initial MV and the second initial MV, If both the first reference image and the second reference image are unspecified reference images, the method includes calculating and obtaining the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and calculating and obtaining a predicted image block of the current image block based on the MV of the current image block.

[0021] In the twelfth aspect, an apparatus for image processing is provided, which apparatus is At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable by the computer, the operations being: A first initial motion vector MV and a second initial MV are obtained, wherein the first initial MV points to the first reference image, and the second initial MV points to the second reference image. If at least one of the first reference image and the second reference image is a specific reference image, then the predicted image block of the current image block is calculated and obtained based on the first initial MV and the second initial MV, If both the first reference image and the second reference image are unspecified reference images, the MV of the current image block is calculated and obtained based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and the predicted image block of the current image block is calculated and obtained based on the MV of the current image block.

[0022] In the 13th aspect, a computer-readable storage medium is provided, which stores commands, and when the commands are operated by the computer, the computer is instructed to execute the image processing method according to the 11th aspect.

[0023] In the fourteenth aspect, an encoding device is provided that includes an apparatus for image processing according to the twelfth aspect.

[0024] In the 15th aspect, a decoding device is provided that includes an apparatus for image processing according to the 12th aspect. [Brief explanation of the drawing]

[0025] [Figure 1] This is a schematic flowchart of a method for image motion compensation in one embodiment of this application. [Figure 2] This is a schematic diagram illustrating the principle of a bidirectional matching method according to one embodiment of this application. [Figure 3] This is a schematic diagram illustrating the principle of a template matching method according to one embodiment of this application. [Figure 4] This is a schematic diagram illustrating the principle of DMVR technology in one embodiment of this application. [Figure 5] This is a schematic flowchart of a method for image processing in another embodiment of the present application. [Figure 6] This is a schematic diagram illustrating the principle of a BIO technology in one embodiment of this application. [Figure 7] This is a schematic frame diagram of an apparatus for image motion compensation in one embodiment of the present application. [Figure 8] This is a schematic frame diagram of an image processing apparatus according to one embodiment of this application. [Figure 9] This is a schematic flowchart of a method for image motion compensation in another embodiment of the present application. [Figure 10] This is a schematic frame diagram of an apparatus for image motion compensation in another embodiment of the present application. [Figure 11] This is a schematic flowchart of a video processing method 1100, which is another embodiment of this application. [Figure 12] This is a schematic flowchart of a video processing method 1200, which is another embodiment of this application. [Figure 13] This is a schematic flowchart of a video processing method 1200, which is another embodiment of this application. [Figure 14] This diagram shows the current image block A divided into four sub-image blocks of the same size. [Modes for carrying out the invention]

[0026] The technical solution in the embodiment of this application will be described below with reference to the drawings.

[0027] Unless otherwise defined, all technical and academic terms used herein have the same meaning as those commonly understood by those skilled in the art of the present application. In this text, the technical terms used in the specification are solely for the purpose of describing specific embodiments and are not intended to limit this application.

[0028] First, we will describe the relevant technologies and concepts relating to the embodiments of this application.

[0029] A video consists of multiple images. When encoding / decoding a video, different prediction methods can be used for different images in the video. The prediction method used based on the image can divide the image into an in-frame prediction image and an inter-frame prediction image, where the inter-frame prediction image includes a forward prediction image and a bidirectional prediction image. Image I is an in-frame prediction image, also called a keyframe, and Image P is a forward prediction image, using a previously encoded / decoded Image P or Image I as a reference image. Image B is a bidirectional prediction image, using the preceding and succeeding images as reference images. In one embodiment, the encoding / decoding end generates a Group of Pictures (GOP) in stages after encoding / decoding multiple images, and this GOP is a Group of Pictures consisting of one Image I and at least one of multiple Images B (or bidirectional prediction images) and Images P (or forward prediction images). The decoding end reads and decodes each GOP in stages during playback, then reads the screen and renders it.

[0030] In modern video encoding / decoding standards, an image can be divided into multiple small blocks and encoded / decoded for images of different resolutions; in other words, an image can be divided into multiple image blocks. An image can be divided into any number of image blocks. For example, this image can be divided into an m × n image block matrix. Image blocks can have rectangular, square, circular, or any other shape. Image blocks can have any size, for example, p × q pixels. Each image block can have at least one of the same size and shape. Substitutable, two or more image blocks can have at least one of different sizes and shapes. Image blocks may or may not have any overlapping portions. In some embodiments, this image block is called a macroblock or Largest Coding Unit (LCU) in some encoding / decoding standards. In the H.264 standard, this image block is called a macroblock and may be 16 × 16 pixels in size. In the High Efficiency Video Coding (HEVC) standard, an image block is called a Coding Tree Unit (CTU), and its size may range from 8x8 pixels to 64x64 pixels.

[0031] In some other embodiments, an image block does not have to be a macroblock or coding tree unit, but may include a part of a macroblock or coding tree unit, or include at least two complete macroblocks (or coding tree units), or include at least one complete macroblock (or coding tree unit) and a part of one macroblock (or coding tree unit), or include at least two complete macroblocks (or coding tree units) and parts of several macroblocks (or coding tree units). In this way, after the image is divided into multiple image blocks, each of these image blocks in the image data can be encoded / decoded.

[0032] In some other embodiments, an image block may be part of a macroblock or coding tree unit. For example, an image block may be a coding unit (CU) in some video encoding / decoding standards.

[0033] The encoding process includes operations such as prediction, transformation, quantification, and entropy coding. Here, prediction includes two types: intra-frame prediction and inter-frame prediction, which aims to remove redundant information from the current image block to be encoded using prediction block data. Intra-frame prediction obtains prediction block data using information from the current frame image. Inter-frame prediction obtains prediction block data using information from a reference image, and this process involves dividing the current image to be encoded into multiple image blocks, and dividing each image block at least once (for example, using the quadtree partitioning method in the HEVC standard, or using the quadtree-binary tree partitioning method) to divide it into multiple image blocks. Then, for each image block, the image block that best matches the current image block in the reference image is searched for and designated as the prediction image block, and the relative displacement between the prediction image block and the current image block is the motion vector. Then, the corresponding pixel values ​​of this image block and the prediction image block are subtracted to obtain the residual. The residuals corresponding to each obtained image block are combined together to obtain the residual of the image block to be encoded.

[0034] In each embodiment of this application, the relationship between residuals of image blocks can be removed by the transformation matrix, that is, the coding efficiency is improved by removing redundant information in the image blocks. The transformation of data blocks in image blocks usually employs a two-dimensional transformation, where the coding end multiplies the residual information of each data block by a single N×M transformation matrix and its transpose, and the result is a transformation coefficient. After quantifying the transformation coefficient, the quantified coefficient is obtained, and finally the quantified coefficient is entropy encoded to obtain an entropy encoded bitstream. The entropy encoded bitstream and encoded mode information, such as the in-frame prediction mode and motion vector (or motion vector residual), are stored or transmitted to the decoding end.

[0035] At the image decoding end, an entropy-encoded bitstream is obtained, then entropy-encoded to obtain the corresponding residual. Based on information such as the motion vector and in-frame prediction obtained through decoding, a predicted image block corresponding to the image block is searched for. Based on the predicted image block and residual, the value of each pixel point in the current sub-image block is obtained.

[0036] In the preamble, it was mentioned that an already encoded / decoded image is used as the reference image to be encoded / decoded. In some embodiments, the similarity between the reference image and the current image to be encoded / decoded can be increased by further constructing the reference image.

[0037] For example, if a video content contains a specific encoding / decoding scene where the background remains essentially unchanged, then only the foreground in the video will change or move. Video monitoring, for instance, belongs to this type of scene. In a video monitoring scene, the monitoring camera is typically fixed and motionless, or only moves slowly, and the background can be considered essentially unchanged. In contrast, objects such as people or vehicles captured by the video monitoring camera are constantly moving or changing, and the foreground is constantly changing. In this type of scene, a specific reference image can be constructed, and this specific reference image contains only high-quality background information. This specific reference image can contain multiple image blocks, each of which is extracted from a decoded image, and different image blocks in this specific reference image may come from different decoded images. When performing interframe prediction, the background portion of the current image to be encoded / decoded can refer to this specific reference image, which can be used to reduce residual information in interframe prediction, thereby improving encoding / decoding efficiency.

[0038] The above is a specific example of a particular reference image. In some embodiments, the particular reference image has at least one of the properties of a composite frame, a long-term reference image, or a non-output image. Here, a non-output image is an image that is not output and displayed. Typically, this non-output image exists as a reference image for other images. For example, this particular reference image may be a composite long-term reference image, or a non-output composite frame, or a non-output long-term reference image, etc. In some embodiments, a composite frame is called a composite reference frame.

[0039] In some embodiments, the non-specific reference image may be a reference image that does not have at least one of the following properties: a composite frame, a long-term reference image, or an image that is not output. For example, this specific reference image may include reference images other than composite frames, or reference images other than long-term reference images, or reference images other than images that are not output, or reference images other than composite frames that are not output, or reference images other than long-term reference images that are not output.

[0040] In some embodiments, when images in a video can be used as reference images, they can be divided into long-term reference images and short-term reference images. Here, the short-term reference image is a concept corresponding to the long-term reference image. The short-term reference image exists in the buffer area of ​​the reference image for a certain period of time, and after the decoded reference image has passed through the buffer area of ​​the reference image, the short-term reference image is moved out of the buffer area of ​​the reference image. The buffer area of ​​the reference image can be called a reference image list buffer memory, a reference image list, a reference frame list buffer memory, or a reference frame list, but in this document, it will be referred to as the buffer area of ​​the reference image.

[0041] A long-term reference image (or a portion of the data in a long-term reference image) can remain in the buffer area of ​​the reference image, and this long-term reference image (or a portion of the data in a long-term reference image) is not affected by ingress / ingress operations in the buffer area of ​​the decoded reference image. This long-term reference image (or a portion of the data in a long-term reference image) is only moved out of the buffer area of ​​the reference image when the decoding end sends an update command operation.

[0042] The terminology for short-term and long-term reference images may differ depending on the standard. For example, in standards such as H.264 / Advanced Video Coding (AVC) or H.265 / HEVC, short-term reference images are called short-term reference frames, while long-term reference images are called long-term reference frames. In standards such as AVS (Audio Video Coding Standard), 1-P2, AVS2-P2, and IEEE (Institute of Electrical and Electronics Engineers) 1857.9-P4, long-term reference images are called background pictures. In standards such as VP8 and VP9, ​​long-term reference images are called golden frames.

[0043] It should be understood that the adoption of specific technical terms in the embodiments of this application does not indicate that they must be applied to a specific scenario. For example, referring to a long-range reference image as a long-range reference frame does not indicate that it must be used in technologies corresponding to standards such as H.264 / AVC or H.265 / HEVC.

[0044] The long-term reference image described above may be constructed from image blocks extracted from multiple decoded images, or it may be obtained by updating an existing reference frame (for example, a pre-memorized reference frame) using multiple decoded images. Naturally, this constructed specific reference image may also be a short-term reference image. Alternatively, the long-term reference image may not be a constructed reference image.

[0045] In the embodiments described above, the specific reference image may include a long-term reference image, and the non-specific reference image may include a short-term reference image.

[0046] The type of reference frame can be selected and identified from a special field in the stream structure.

[0047] If the reference image is determined to be a long-term reference image, then this reference image is determined to be a specific reference image. Alternatively, if the reference image is determined to be a frame in which no output is generated, then this reference image is determined to be a specific reference image. Alternatively, if the reference image is determined to be a composite frame, then this reference image is determined to be a specific reference image. Alternatively, if the reference image is determined to be a frame in which no output is generated, AND this reference image is determined to be a composite frame, then this reference image is determined to be a specific reference image.

[0048] Selectable reference images can all have corresponding marks, and in this case, the decoding end can determine whether or not a reference image is a specific reference image based on the marks it has.

[0049] In some embodiments, if a reference image is determined to have a long-term reference image mark, this reference image is determined to be a specific reference image.

[0050] In some embodiments, if it is determined that a reference image has a mark that does not result in output, this reference image is determined to be a specific reference image.

[0051] In some embodiments, if it is determined that a reference image has a composite frame mark, this reference image is determined to be a specific reference image.

[0052] In some embodiments, a reference image is determined to be a specific reference image if it has at least two of the following three marks: a long-term reference image mark, a non-output mark, a composite frame mark, or a composite reference frame mark. For example, if a reference image is determined to have a non-output mark and also to have a composite frame mark, the reference image is determined to be a specific reference image.

[0053] Specifically, an image may have a mark indicating whether or not it is a frame to be output. If an image is indicated not to be output, it indicates that this frame is a reference image. Furthermore, it is determined whether or not this frame has a mark indicating it is a composite frame. If it does, the reference image is confirmed to be a specific reference image. If an image is indicated to be output, it is not necessary to determine whether or not it is a composite frame; this frame is directly confirmed not to be a specific reference image. Alternatively, if an image is indicated not to be output but has a mark indicating it is not a composite frame, this frame can be confirmed not to be a specific reference image.

[0054] Selectable means that the reference image is determined to be a specific reference image if the parameters are analyzed from the image header, image parameter set (PPS), and slice header and it is determined that the reference image satisfies one of the following conditions.

[0055] The aforementioned reference image is a long-term reference image.

[0056] The aforementioned reference image is a composite reference image.

[0057] The aforementioned reference image is an image that is not output.

[0058] If the aforementioned reference image is an image that is not output, it is further determined whether the aforementioned reference image is a composite reference image.

[0059] In the preamble, we mentioned a technique that uses the derivation of motion vectors. However, when modifying motion vectors, performing motion search on a specific reference image can actually decrease search efficiency and encoding / decoding efficiency. A specific reference image is artificially constructed or derived from an earlier, earlier reference image in chronological order. There is not necessarily a spatial relationship between image blocks in the specific reference image, and there is clear shaking at the image block edges. Therefore, there is little value in searching for motion vectors based on such a specific reference image.

[0060] Pattern Matching Motion Vector Derivation (PMMVD) and Decode Motion Vector Refinement (DMVR) are both techniques that utilize the derivation of motion vectors.

[0061] The preamble further mentions techniques that use bidirectional motion prediction, manipulating motion vectors based on the temporal correlation of images. These operations may fail if the reference image pointed to by the associated motion vector is a specific reference image, because the definition of the temporal distance between the current image to be encoded or decoded and the specific reference image is unclear. Bidirectional optical flow (BIO) prediction techniques are techniques that use bidirectional motion prediction.

[0062] Next, based on PMMVD, DMVR, and BIO, the method for image motion compensation of the present invention will be explained with examples. It is important to note that the method for image motion compensation in the present invention is not limited to these three technologies.

[0063] Before describing the method for image motion compensation in the embodiments of this application, we will first briefly explain the video encoding and decoding process in the HEVC standard.

[0064] The HEVC standard defines three modes of inter-frame prediction: Inter mode, Merge mode, and Skip mode. The purpose of inter-frame prediction is to obtain a motion vector (MV) and then determine the position of the predicted image block in the reference image based on this motion vector. Adjacent image blocks have similar motion schemes; for example, if an image block (e.g., at least one of the image block to be encoded and the image block to be decoded) and an adjacent image block belong to the same object, and the camera moves, their movement distance and direction will naturally be similar or the same, so it is often unnecessary to calculate the motion vector, and the motion vector of the adjacent image block is directly used as the motion vector of the current image block. Here, in Merge mode and Skip mode, the motion vector difference (MVD) is 0, meaning that the motion vector is obtained directly based on the adjacent encoded or decoded image block.

[0065] When at least one of the image blocks to be encoded or decoded is in Merge mode, the operating principle is as follows: A list of Motion Vector Prediction (MVP) candidates is constructed from adjacent image blocks, the optimal MVP is selected from the MVP candidate list as the motion vector for the current image block, the position of the predicted image block is determined based on this motion vector, and the residual is calculated after determining the predicted image block. In Merge mode, since the motion vector is selected from the MVP candidate list, there is no MVD. The encoding end only needs to encode the residual and the index of the selected motion vector in the MVP candidate list; it is not necessary to encode the MVD. The decoding end can construct an MVP candidate list in a similar manner and then obtain the motion vector based on the index transmitted by the encoding end. Based on the motion vector, the decoding end determines the predicted image block, incorporates the residual, and decodes to obtain the current image block.

[0066] The specific operation process of the encoded end in Merge mode is as follows: 1. Obtain the MVP candidate list. 2. Select the most suitable MVP from the MVP candidate list and obtain the index of this MVP within the MVP candidate list. 3. The selected MVP is used as the motion vector for the current image block. 4. Based on the motion vector, determine the predicted image block from the reference image. 5. Subtract the predicted image block from the current image block to obtain the residual. 6. Since the motion vector is selected from the MVP candidate list, there is no MVD, and only the residual and the index of the selected MVP in the MVP candidate list should be sent to the decoding end.

[0067] The specific operation process of the decoding end in Merge mode is as follows: 1. Receive the residuals and the index of the motion vector in the MVP candidate list. 2. Obtain the MVP candidate list. 3. Based on the index, search for the motion vector in the MVP candidate list and use it as the motion vector for the current image block. 4. Based on the motion vector, the predicted image block is determined, and the residual is incorporated and decoded to obtain the current image block.

[0068] The above describes the processing steps in normal Merge mode.

[0069] Skip mode is a special case of Merge mode. After obtaining motion vectors using Merge mode, if the encoder determines, based on a certain method, that the current image block and the predicted image block are essentially the same, it is not necessary to transmit residual data; only the index in the MVP candidate list of motion vectors needs to be transmitted, indicating that the current image block can obtain its mark directly from the predicted image block.

[0070] In Inter mode, the MVP is determined first, then the MVP is modified to obtain the MVD. The coding end needs to transmit not only the index and residual to the decoding end, but also the MVD to the decoding end. Advanced Motion Vector Prediction (AMVP) is a tool that achieves motion vector prediction through a competitive mechanism.

[0071] In AMVP mode, there is also an MVP candidate list, and the motion vectors in this MVP candidate list are obtained from adjacent blocks in the spatial or temporal domain of the current image block. The MVP candidate list in AMVP mode may differ from the MVP candidate list in Merge mode. The encoding end or decoding end selects the optimal MVP from the MVP candidate list. Using this MVP as a search starting point, the vicinity is searched to obtain the optimal motion vector, which is the motion vector of the current image block. Based on this motion vector, the position of the predicted image block is determined, and after determining the predicted image block, the residual is calculated. Simultaneously, the MV and MVP are subtracted from each other to obtain the MVD. The encoding end encodes the residual, the index of the MVP in the MVP candidate list, and the MVD and transmits them to the decoding end. The decoding end can construct an MVP candidate list in a similar manner and then obtain the MVP based on the index transmitted by the encoding end. The decoding end determines the MV based on the MVP and MVD, determines the predicted image block based on the MV, incorporates the residuals, and decodes to obtain the current image block.

[0072] The specific operation process of the coding end in AMVP mode is as follows: 1. Obtain the MVP candidate list. 2. Select the most suitable MVP from the MVP candidate list and obtain the index of this MVP within the MVP candidate list. 3. Determine the starting point of the search based on the MVP. 4. Search near the starting point to obtain the optimal motion vector. 5. Based on the motion vector, determine the predicted image block from the reference image. 6. Subtract the predicted image block from the current image block to obtain the residual. 7. Subtract the MVP from the motion vector to obtain the MVD. 8. Send the residual, the index of the selected MVP in the MVP candidate list, and the MVD to the decoding end.

[0073] The specific operational process of the decoding end in AMVP mode will not be explained further.

[0074] Embodiments of this application provide a method 100 for image motion compensation. Figure 1 is a schematic flowchart of method 100 for image motion compensation according to one embodiment of this application. As shown in Figure 1, method 100 is S110 obtains the initial motion vector of the current image block, If the reference image pointed to by this initial motion vector is a specific reference image, then S120 performs motion compensation on this current image block based on this initial motion vector. S130 includes, if the reference image pointed to by this initial motion vector is an unspecified reference image, modifying this initial motion vector to obtain a modified motion vector, and performing motion compensation on the current image block based on the modified motion vector.

[0075] In some embodiments, if the initial motion vector points to a specific reference image, motion compensation is performed directly. If the initial motion vector points to a non-specific reference image, the initial motion vector is modified, and motion compensation is performed based on the modified motion vector. This avoids performing meaningless searches due to obvious shaking of the image block edges of the specific reference image, thereby improving encoding and decoding efficiency.

[0076] In possible embodiments, the method 100 for image motion compensation of the embodiments of this application can be applied to PMMVD technology.

[0077] PMMVD technology is a special merge mode based on Frame Rate Up Conversion (FRUC) technology. In this special merge mode, motion information for the current image block (e.g., MV and MVD) is not encoded in the stream but is generated directly at the decoding end.

[0078] Selectively, in this possible embodiment, if the reference image pointed to by the initial motion vector is an unspecified reference image, modifying the initial motion vector to obtain a modified motion vector and performing motion compensation on the current image block based on the modified motion vector may include obtaining a list of motion vector candidates for the current image block, determining that any of the motion vector candidates in the motion vector candidate list are pointed to an unspecified reference image, determining the initial motion vector based on the motion vector candidate list, modifying the initial motion vector to obtain a modified motion vector, and performing motion compensation on the current image block based on the modified motion vector.

[0079] In this embodiment, determining an initial motion vector from a motion vector candidate list, and if the reference image pointed to by the candidate vector in this motion vector candidate list is a non-specific reference image, modifying the initial motion vector to obtain a modified motion vector, and performing motion compensation on the current image block based on the modified motion vector, can avoid performing meaningless searches due to obvious shaking of the image block edges of a specific reference image, and can improve encoding and decoding efficiency.

[0080] Selectively, in a possible embodiment, obtaining a list of motion vector candidates for the current image block may include determining a motion vector candidate to add to the motion vector candidate list, and adding the motion vector candidate to the motion vector candidate list if the reference image pointed to by the motion vector candidate is an unspecified reference image.

[0081] Specifically, in one embodiment, the method 100 may further include determining that the reference image pointed to by the motion vector candidate is a non-specific reference image based on the frame marks of the reference image pointed to by the motion vector candidate. Specifically, it can be determined whether or not the reference image pointed to by the motion vector candidate is a specific reference image based on the frame marks of the reference image pointed to by the motion vector candidate. If the reference image pointed to by the motion vector candidate is a specific reference image, it is not necessary to add the motion vector candidate corresponding to the specific reference image to the motion vector candidate list. In this way, when selecting an initial motion vector from the motion vector candidate list, the motion vector corresponding to the specific reference image is not selected.

[0082] It should be understood that the motion vector of each embodiment of this application includes three parameters, namely the horizontal component v x , vertical component v y and the frame mark of the reference image being pointed to. For example, this frame mark may be a Picture Order Count (POC), or it may be any other form of mark. The encoding end and the decoding end can determine the attributes of the reference image from this frame mark and decide whether this reference image is a specific reference image or a non-specific reference image.

[0083] In one embodiment, the selection of an initial motion vector based on the motion vector candidate list may include determining whether the selected initial motion vector points to a specific reference image, and if at least one of the selected initial motion vectors points to a specific reference image, selecting the initial motion vector again until an initial motion vector for a reference image other than the specific reference image is selected.

[0084] Specifically, if the determined initial motion vector points to a specific reference image based on the motion vector candidate list, a motion vector candidate can be selected again from the motion vector candidate list and used as the initial motion vector, based on a pre-set rule. This pre-set rule may, for example, select the motion vector candidate with the next smallest matching cost, and if this motion vector candidate does not point to a specific reference image, use this motion vector candidate as the initial motion vector. This embodiment is illustrative and does not limit the present application.

[0085] Taking the coded end as an example, the motion vector derivation process in FRUC merge mode is divided into two steps. The first step is coding unit (CU) level motion lookup, and the second step is sub-CU level motion refinement. Similarly, the decoding end can achieve similar functionality to the coded end.

[0086] In CU-level motion lookup, a list of CU-level motion vector candidates is generated. Based on the bidirectional matching method, the motion vector with the smallest matching cost is searched from the list of CU-level motion vector candidates, e.g., MV-A. Similarly, the motion vector with the smallest matching cost is searched from the list of CU-level motion vector candidates using the template matching method, e.g., MV-B. Then, based on the Rate Distortion Cost (RD-Cost) policy used in the Merge mode policy, it is determined whether the current CU uses the FRUC merge mode. That is, the results of the two matching methods (bidirectional matching method and template matching method) are checked using the RD-Cost policy. The results obtained from the matching method with the smallest matching cost are further compared with the results of the other CU modes. The matching method with the smallest matching cost among the two matching methods is the one with the smallest final matching cost, and the FRUC flag of the current CU is set to TRUE, instructing the decoding end to use the corresponding matching method.

[0087] In this specific embodiment, the CU-level motion vector candidate list can correspond to the motion vector candidate list in Method 100 of the embodiment of this application. When generating the motion vector candidate list, motion vectors to be added to the list are scanned, and if a motion vector points to a non-specific reference image, this motion vector can be added to the motion vector candidate list; if it points to a specific reference image, this motion vector is rejected and not added to the motion vector candidate list. The motion vector candidate list in this embodiment may include, when the current image block is in advanced motion vector prediction AMVP mode, original AMVP motion vector candidates obtained for a non-specific reference image, merged motion vector candidates obtained for a non-specific reference image, motion vectors obtained by interpolation from a non-specific reference image, and at least one motion vector candidate from the upper adjacent motion vector and left adjacent motion vector for the non-specific reference image of the current block. This motion vector candidate list may be a CU-level motion vector candidate list. In this embodiment, it is understood that the non-specific reference image may specifically be a short-term reference image or a short-term reference frame.

[0088] Determining the initial motion vector based on the motion vector candidate list may include determining the strain cost corresponding to the motion vector candidate in the motion vector candidate list based on at least one of the bidirectional matching method and the template matching method, and selecting the motion vector with the smallest strain cost in the motion vector candidate list as the initial motion vector.

[0089] Specifically, taking the encoded end as an example, determining the initial motion vector based on the motion vector candidate list corresponds to CU-level motion search. In CU-level motion search, the motion vector with the smallest matching cost is searched from the CU-level motion vector candidate list based on the bidirectional matching method, for example, MV-A. Similarly, the motion vector with the smallest matching cost is searched from the CU-level motion vector candidate list using the template matching method, for example, MV-B. Subsequently, based on the RD-Cost policy used in the Merge mode policy, it is determined whether the current CU uses FRUC merge mode. That is, the results of two matching methods (bidirectional matching method and template matching method) are checked using the RD-Cost policy. The result obtained by the matching method with the smallest matching cost is further compared with the results of the other CU modes. If the matching method with the smallest matching cost among the two matching methods has the smallest final matching cost, the FRUC flag of the current CU is set to TRUE, and the decoding end is instructed to use the corresponding matching method. Similarly, the decoding end can perform functions similar to the encoding end, but it is less complex. For example, the decoding end directly receives flags to instruct the encoding end on how to match and does not need to execute an RD-Cost policy.

[0090] Selectively modifying the initial motion vector and obtaining the modified motion vector may include generating a list of sub-motion vector candidates in the current image block, the sub-motion vector candidate list including the initial motion vector, and determining the motion vector with the smallest distortion cost from the sub-motion vector candidate list as the modified motion vector.

[0091] Specifically, modifying the initial motion vector and obtaining the modified motion vector can accommodate sub-CU level motion refinement.

[0092] In CU-level motion search, the motion vector with the lowest matching cost is the initial motion vector, which is used as the starting point for CU-level motion refinement. Around the starting point, a local search is performed based on the matching method determined by the CU level (bidirectional matching or template matching). Specifically, in sub-CU-level motion refinement, a list of sub-CU-level motion vector candidates can be generated. From the sub-CU-level motion vector candidate list, the motion vector with the lowest matching cost is searched for and set as the motion vector for the current CU.

[0093] The subCU-level motion vector candidate list may include motion vectors determined by a CU-level motion search (an MV determined from a CU-level search), top, left, top-left, and top-right neighboring motion vectors for the current image block, scaled versions of collocated MVs from reference pictures, and other motion vector candidates derived from other time domains (up to 4 ATMVP candidates and up to 4 STMVP candidates).

[0094] In the specific embodiment described above, if the reference image pointed to by the motion vector candidate is a specific reference image, the motion vector candidate corresponding to the specific reference image is not added to the CU-level motion vector candidate list, so that the motion vector candidate corresponding to the specific reference image does not have the opportunity to become the initial motion vector.

[0095] In another specific embodiment, if the initial motion vector determined based on the CU-level motion vector candidate list points to a specific reference image, a motion vector candidate can be selected again from the CU-level motion vector candidate list and used as the initial motion vector, based on a pre-set rule. This pre-set rule may be, for example, selecting the motion vector candidate with the next smallest matching cost, and if this motion vector candidate does not point to a specific reference image, then using this motion vector candidate as the initial motion vector.

[0096] Figure 2 is a schematic diagram of the principle of a bidirectional matching method according to one embodiment of this application. As shown in Figure 2, the bidirectional matching method searches for the closest matching between two predicted image blocks in different reference images in the motion trajectory of the current CU and derives the motion vector of the current CU. The bidirectional matching method is based on the assumption that the motion trajectory of the current image block is continuous. The motion vectors MV0 and MV1 of the two predicted image blocks are assumed to be proportional to the time-domain distance (TD0 and TD1) between the current image and two reference images (reference image 0 and reference image 1). By scanning the motion vector candidate list, for example, for MV0, the motion vector pair MV0 and MV1 can be generated. Here, MV1 can be generated based on MV0, TD0, and TD1. When the distortion between the two predicted image blocks pointed to by the motion vector pair corresponding to MV0 is smallest, this motion vector (i.e., MV0) is the motion vector of the current CU. If TD0 = TD1, the bidirectional matching changes to a mirror-image-based bidirectional matching.

[0097] It should be understood that if one of the two reference images is a specific reference image, bidirectional matching cannot be performed because the definition of the time distance between the current image (the current image to be encoded, or the current image to be decoded) and the specific reference image is unclear. Moreover, the specific reference image is artificially constructed or originates from an earlier specific reference image in the time order, and there is not necessarily a spatial relationship between the image blocks in the specific reference image, and there is clear shaking at the image block edges, so there is little significance in bidirectional matching to search for motion vectors based on such a specific reference image. In possible embodiments of this application, the above-mentioned problems are avoided by not adding motion vector candidates corresponding to the specific reference image to the CU-level motion vector candidate list.

[0098] Figure 3 is a schematic diagram of the principle of a template matching method according to one embodiment of this application. As shown in Figure 3, the template matching method searches for the closest match between the template of the current image (at least one of the upper image block and the left image block of the current CU) and a block in the reference image (the same size as the template) to derive the motion vector of the current CU. After obtaining the template, if the distortion between the template and the predicted image block pointed to by the candidate motion vector is smallest, this candidate motion vector is the motion vector of the current CU.

[0099] When a reference image is a specific reference image, it is artificially constructed or derived from an earlier specific reference image in chronological order, and there is not necessarily a spatial relationship between image blocks in the specific reference image, and there is clear shaking at the image block edges. Therefore, there is little value in searching for motion vectors based on such a specific reference image, and the results of the template matching method are inaccurate and, consequently, meaningless. In possible embodiments of this application, the above-mentioned problems are avoided by not adding motion vector candidates corresponding to the specific reference image to the CU-level motion vector candidate list.

[0100] In a specific embodiment, the method 100 for image motion compensation according to the embodiment of this application can be applied to DMVR technology.

[0101] DMVR technology is a densification technique used to make more accurate predictions on the current image block when performing bidirectional prediction. Next, we will explain DMVR technology in detail using the decoding end as an example. DMVR technology mainly involves two major steps: the first step is to construct a template based on the decoded image block corresponding to multiple initial motion vectors; the second step is to modify these multiple initial motion vectors based on the template.

[0102] Specifically, the decoding end can generate a motion vector candidate list. For example, this motion vector candidate list may be the AMVP mode motion vector candidate list described in the preceding paragraph, or the Merge template motion vector candidate list. The decoding end can receive multiple indices to indicate initial motion vectors transmitted from the encoding end. Based on the indices, the decoding end obtains multiple initial motion vectors from the motion vector candidate list. Based on the decoded image blocks corresponding to these multiple initial motion vectors, the decoding end generates a template (for example, a method of weighting pixels) and uses the generated template to modify each of these multiple initial motion vectors. Finally, motion compensation is performed on the current image block based on these modified motion vectors.

[0103] In this specific embodiment, the initial motion vectors may include a first initial motion vector and a second initial motion vector. If a specific reference image exists in the initial motion vectors, the following processing can be employed. S120, if the reference image pointed to by the current initial motion vector is a specific reference image, performing motion compensation on the current image block based on the initial motion vectors may include: if the reference image pointed to by at least one of the first and second initial motion vectors is a specific reference image, performing motion compensation on the current image block based on the first and second initial motion vectors; and if the reference images pointed to by the first and second initial motion vectors are both non-specific reference images, modifying the first and second initial motion vectors to obtain a modified first motion vector and a modified second motion vector, and performing motion compensation on the current image block based on the modified first and modified second motion vectors. If no specific reference image exists in the initial motion vectors, i.e., if all initial motion vectors are non-specific reference images, existing DMVR processing methods can be employed.

[0104] Figure 4 is a schematic diagram of the principle of DMVR technology according to one embodiment of this application. Specifically, assuming that the initial motion vector includes a first initial motion vector (which may be MV0) and a second initial motion vector (which may be MV1), and that the decoded image block corresponding to the first initial motion vector belongs to the first decoded image block of the first frame, then this first frame may be the first reference image, and this first decoded image block may be the first reference image block. Assuming that the decoded image block corresponding to the second motion vector belongs to the second decoded image block of the second frame, then this second frame may be the second reference image, and this second decoded image block may be the second reference image block. A template can be obtained by weighting and summing this first reference image block and this second reference image block. Here, this template can be called a bidirectional template.

[0105] Optionally, in one embodiment, the initial motion vector includes a first initial motion vector and a second initial motion vector. Modifying the first initial motion vector and the second initial motion vector to obtain a modified first motion vector and a modified second motion vector may include generating a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the first initial motion vector and belongs to the first reference image, and the second reference image block corresponds to the second initial motion vector and belongs to the second reference image, and modifying the first initial motion vector and the second initial motion vector based on the template to obtain a modified first motion vector and a modified second motion vector.

[0106] Specifically, modifying the first initial motion vector and the second initial motion vector based on the template to obtain the modified first motion vector and the modified second motion vector may include: matching N third reference image blocks with the template, where the N third reference image blocks correspond to N third initial motion vectors and belong to the first reference image; matching M fourth reference image blocks with the template, where the M fourth reference image blocks correspond to M fourth initial motion vectors and belong to the second reference image; selecting one third initial motion vector from the N third initial motion vectors and one fourth initial motion vector from the M fourth initial motion vectors based on the matching results; and using the one third initial motion vector and the one fourth initial motion vector as the motion vectors of the current image block (i.e., the modified first motion vector and the modified second motion vector), or determining the motion vectors of the current image block.

[0107] Selectively, this selected third initial motion vector may be the motion vector corresponding to the smallest strain cost. Alternatively, this selected third initial motion vector may be the motion vector corresponding to a strain cost smaller than a certain value.

[0108] Selectively, this selected fourth initial motion vector may be the motion vector corresponding to the smallest strain cost. Alternatively, this selected fourth initial motion vector may be the motion vector corresponding to a strain cost smaller than a certain value.

[0109] Here, the one third initial motion vector and the one fourth initial motion vector are defined as the motion vectors of the current image block. In this case, the known image blocks corresponding to the one third initial motion vector and the one fourth initial motion vector (i.e., the initial predicted image blocks) can be weighted and summed to obtain the predicted image block.

[0110] Alternatively, the one third initial motion vector and the one fourth initial motion vector can be used to determine the motion vector of the current image block; that is, the one third initial motion vector and the one fourth initial motion vector can each be an MVP. In this case, the search can be optimized starting from this third MVP to obtain an optimized motion vector, and the search can be optimized starting from this fourth MVP to obtain another optimized motion vector. The known image blocks (i.e., initial predicted image blocks) corresponding to these two optimized motion vectors can be weighted and summed to obtain a predicted image block.

[0111] For selection to be possible, N and M may be equal.

[0112] For selection to be possible, this third initial motion vector includes this first initial motion vector, and this fourth initial motion vector includes this second initial motion vector; that is, the reference image block corresponding to the first initial motion vector and the reference image block corresponding to the second motion vector for generating the template must also be matched with the template.

[0113] Selectively, in the embodiments of this application, at least some of the N third initial motion vectors are obtained by shifting based on the first initial motion vector, and at least some of the M fourth initial motion vectors are obtained by shifting based on the second initial motion vector.

[0114] For example, in these N third initial motion vectors, the initial motion vectors other than the first initial motion vector are obtained by shifting them based on the first initial motion vector. For example, N may be equal to 9, and eight of these initial motion vectors are obtained by shifting them based on the first initial motion vector, for example, by shifting them in eight directions, or by shifting different pixels in the vertical or horizontal direction.

[0115] For example, the initial motion vectors other than the second initial motion vector in these N fourth initial motion vectors are obtained by shifting them based on the second initial motion vector. For example, N may be equal to 9, and eight of these initial motion vectors are obtained by shifting them based on the second initial motion vector. For example, they may be obtained by shifting them in eight directions, or by shifting different pixels in the vertical or horizontal direction.

[0116] Selectively, in the embodiments of this application, the first reference image may be a forward-facing frame of the current image block, and the second reference image may be a backward-facing frame of the current image block. Alternatively, the first reference image may be a forward-facing frame of the current image block, and the second reference image may be a forward-facing frame of the current image block.

[0117] In a specific embodiment, the decoding end generates a motion vector candidate list and receives two indices to indicate the initial motion vectors transmitted from the encoding end. The decoding end determines the DMVP condition, which requires that neither of the two initial motion vectors (which may be MV0 and MV1) points to a specific reference image, and that the predicted directions of the two initial motion vectors are opposite, i.e., one is forward and the other is backward. If the DMVR condition is satisfied, the image blocks corresponding to MV0 and the image blocks corresponding to MV1 are weighted together to generate a bidirectional template.

[0118] In a possible embodiment of this application, the reference image pointed to by the initial motion vector indicated by the two indices is avoided by not adding motion vector candidates corresponding to a specific reference image to the motion vector candidate list. In another possible embodiment, if the reference image pointed to by the initial motion vector indicated by the two indices is the specific reference image, the decoding end can ask the encoding end to again point to a motion vector that does not point to the specific reference image as the initial motion vector, or the decoding end can return or mark the DMVR algorithm as expired.

[0119] In the first reference image, for the image block corresponding to MV0, one luminance pixel is shifted in at least one of the horizontal or vertical directions, and the motion vectors of the eight adjacent pixel blocks found, along with the total of nine motion vectors of MV0, can form a reference list list0. In the second reference image, for the image block corresponding to MV1, one luminance pixel is shifted in at least one of the horizontal or vertical directions, and the motion vectors of the eight adjacent pixel blocks found, along with the total of nine motion vectors of MV1, can form a reference list list1.

[0120] The bidirectional template matching used by the decoding end performs a distortion-based search between the bidirectional template and the reconstructed block in the reference image, ultimately obtaining a refined motion vector without additional motion information. For the motion vectors in the two reference images (the motion vector in List0 and the motion vector in List1), the motion vector with the smallest matching cost replaces the original motion vector as the updated motion vector. Finally, the original MV0 and MV1 are replaced with two new motion vectors (MV0' and MV1' shown in Figure 3). Based on the predicted image block corresponding to MV0' and the predicted image block corresponding to MV1', the final bidirectional prediction result is generated.

[0121] In DMVR technology, when the reference image is a specific reference image, the specific reference image is artificially constructed or originates from an earlier specific reference image in chronological order, and there is not necessarily a spatial relationship between the image blocks in the specific reference image, and there is clear shaking at the image block edges. Therefore, it should be understood that there is little value in searching for motion vectors based on such a specific reference image, and that is, DMVR technology is inaccurate and, consequently, meaningless. In the embodiment of this application, the above-mentioned problems are avoided by not refining the motion vectors corresponding to the specific reference image and using them directly for motion compensation.

[0122] Embodiments of this application provide a method 200 for image processing. Figure 5 is a schematic flowchart of another embodiment of the method 200 for image processing of this application. As shown in Figure 5, this method 200 is A first initial motion vector and a second initial motion vector are obtained, and the first initial motion vector points to the first reference image, and the second initial M motion vector points to the second reference image, S210, If at least one of the first reference image and the second reference image is a specific reference image, then S220 is obtained by calculating the predicted image block of the current image block based on the first initial motion vector and the second initial motion vector, If both the first reference image and the second reference image are unspecified reference images, the method includes S230, which calculates and obtains the motion vector of the current image block based on the first initial motion vector and the gradient value of the pixel point pointed to by the second initial motion vector, and calculates and obtains a predicted image block of the current image block based on the motion vector of the current image block.

[0123] In the image processing method of the embodiment of this application, when the reference images pointed to by two initial motion vectors are both unspecified reference images, the motion vector of the current image block is calculated by employing the gradient value of the pixel points and an optimization principle, and a predicted image block for the current image block is calculated. When a specific reference image exists among the reference images pointed to by the two initial motion vectors, the predicted image block for the current image block is directly calculated based on the two initial motion vectors, thereby avoiding the inability to perform prediction due to the unclear definition of the time distance between the current image and the specific reference image, and improving the encoding and decoding efficiency.

[0124] Selectively, in S230, calculating the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV may include calculating the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and an optimization principle.

[0125] In a specific embodiment, the image processing method 200 of the embodiment of this application can be applied to an improvement of the bidirectional motion prediction mode. The bidirectional motion prediction mode refers to a method in which, when encoding a current image block, two initial prediction image blocks derived from two different reference images are used simultaneously to predict the current image block, and the two initial prediction image blocks are combined into a single image block to form the predicted image block of the current image block. The BIO prediction technique is a technique in the bidirectional motion prediction mode. In the BIO prediction technique, the motion vector has even fewer encoding marks compared to the normal bidirectional motion prediction mode, but the derivation process of the predicted image block is different. The BIO prediction technique is a motion optimization based on block motion compensation, calculates the motion vector using optical flow mode, and is a motion optimization at the sampling point level.

[0126] The BIO prediction technology of the embodiment of this application can include two steps. The first step is to calculate and obtain the motion vector (MV) of the current image block based on two initial MVs (for example, the first initial MV and the second initial MV). Specifically, it is to calculate and obtain the MV of the current image block based on the gradient values of the pixel points pointed to by the first initial MV and the second initial MV, and the optimization principle. The second step is to calculate and obtain the predicted image block of the current image block based on the MV of the current image block, the first reference image block pointed to by the first initial MV, and the second reference image block pointed to by the second initial MV. Specifically, the BIO prediction technology interpolates two reference image blocks pointed to by two initial MVs to obtain two initial predicted image blocks of the same size as the current image block, and then weighted-sums these two initial predicted image blocks and combines them as the predicted image block of the current image block.

[0127] FIG. 6 is a schematic diagram of the principle of the BIO technology of one embodiment of this application. The BIO technology assumes that the movement of an object is a uniform motion in both the horizontal and vertical directions within a short period of time and conforms to the analysis conditions of the optical flow method. As shown in FIG. 6, the motion vectors (v x 、v y ) of the reference points k (k = 0, 1) pointed to by two initial MVs, and the luminance value I (k) can be assumed to conform to the following optical flow formula. ∂I (k) / ∂t + v x ∂I (k) / ∂x + v y ∂I (k) / ∂y = 0 Formula 1 In the formula, ∂I (k) / ∂x and ∂I (k) / ∂y represent the horizontal component and the vertical component of the gradient.

[0128] By联立 the optical flow formula and Hermite interpolation, a polynomial as follows is obtained, and when t = 0, it is the BIO prediction value. pred BIO = 1 / 2·(I(0) +I (1) +v x / 2·(τ1∂I (1) / ∂x-τ0∂I (0) / ∂x)+v y / 2·(τ1∂I (1) / ∂y-τ0∂I (0) / ∂y)) Official 2 In the formula, τ0 and τ1 represent the distances from the current image to reference image 0 and reference image 1, respectively, and both can be obtained by calculating the point of view (POC) of the current image and the two reference images. τ0=POC(current)-POC(Ref0) Formula 3 τ1=POC(Ref1)-POC(current) Formula 4 In the formula, POC(current) is the picture order count of the current image, POC(Ref0) is the picture order count of reference image 0, and POC(Ref1) is the picture order count of reference image 1.

[0129] The reference images can be in different directions, i.e., one from the past and one from the future. The two reference images can also be in the same direction, i.e., both from the past or both from the future. If the two reference images are in the same direction, τ0 and τ1 have opposite signs. In such a situation, the two reference images cannot be the same, i.e., τ0 ≠ τ1, and the reference region has non-zero motion (MV). x0 MV y0 MV x1 MV y1 There exists a value (≠0), and the motion vector is proportional to the distance in the time domain (MV). x0 / MV x1 =MV y0 / MV y1 = -τ0 / τ1).

[0130] Assuming that the motion within a small region is the same, the motion vector of BIO can satisfy the following formula for a first-order Taylor expansion. Δ=(I (0) -I (1) +v x (τ1∂I(1) / ∂x+τ0∂I (0) / ∂x)+v y (τ1∂I (1) / ∂y+τ0∂I (0) / ∂y)) Official 5 In the formula, Δ is the Taylor linear expansion of the pixel difference between two corresponding reference points (for example, points A and B in Figure 4) in two reference images. By analyzing the gradient value and optimization method, the optimal motion vector of the current image block satisfies the square of Δ and its minimum value throughout the entire region, thereby determining the optimal motion vector (v x , v y ) can be calculated, and considering the robustness of the algorithm, v x and v y It is assumed that it is within a certain threshold range.

[0131] Based on the formula above, the process of the BIO prediction technique is as follows: For the current image block, the pixel values ​​at the positions corresponding to the two reference images have already been obtained. (0) and I (1) These represent the pixel values ​​in the two reference images, and the known ones in the above formula are: (0) , I (1) and τ0 and τ1, the horizontal and vertical components of the gradient are obtained by calculation in the reference image, and the unknown is v x , v y And Δ. For all pixel points in a single region, one Δ can be calculated for each, and using an optimization method, the v with the minimum value of Δ is obtained. x and v y The goal is to obtain the optimal motion vector, which is the one needed. Here, v x and v y When calculating v x and v yA single interval is given, and the threshold of this interval is determined by the orientation of the two reference images relative to the current image. After obtaining the optimal motion vector, the image block + residual pointed to by this optimal motion vector is not directly used as the current image block. Instead, each pixel of the current image block is calculated using formula 2, which is also called the BIO predicted value, and the predicted values ​​of each pixel are combined to form the predicted image block.

[0132] Selectively, in an embodiment of the present application, obtaining a predicted image block of the current image block based on the first initial motion vector and the second initial motion vector includes weighting and summing the first reference image block pointed to by the first initial motion vector and the second reference image block pointed to by the second initial motion vector to obtain a predicted image block of the current image block.

[0133] Specifically, a decision condition is added before the start of the BIO prediction, or a necessary condition is added to the BIO condition. If neither the first reference image pointed to by the first initial motion vector nor the second reference image pointed to by the second initial motion vector is a specific reference image, and the initial BIO condition is met, then the BIO prediction can be performed. Otherwise, the predicted image block for the current image block is directly calculated and obtained based on the two initial motion vectors, or returned, or the BIO prediction algorithm is marked as expired.

[0134] The BIO condition may further include: the predicted directions of the first initial motion vector and the second initial motion vector are different. Alternatively, the BIO condition may further include: the predicted directions of the first initial motion vector and the second initial motion vector are the same, neither the first initial motion vector nor the second initial motion vector is zero, and the first reference image and the second reference image are different. At the same time, the ratios of the directional components of the first initial motion vector and the second initial motion vector are the same, and both are equal to the ratio of the distance between the first reference image and the current image pointed to by the motion vector, and the distance between the second reference image and the current image.

[0135] It should be understood that the motion vectors of each embodiment of this application include three parameters, the horizontal component v x , vertical component v y , and the frame mark of the reference image being pointed to. For example, this frame mark may be a POC, or it may be any other form of mark. The encoding end and the decoding end can determine the attributes of the reference image from this frame mark and determine whether or not this reference image is a specific reference image.

[0136] Selectively, in the embodiments of this application, determining whether the reference image pointed to by the initial motion vector is a specific reference image may include determining whether the reference image pointed to by the initial motion vector is a specific reference image based on the frame marks of the reference image pointed to by the initial motion vector.

[0137] Optionally, in the embodiments of this application, calculating a predicted image block of the current image block based on the motion vector of the current image block in S230 may include calculating a predicted image block of the current image block based on the motion vector of the current image block, a first reference image block pointed to by the first initial motion vector, and a second reference image block pointed to by the second initial motion vector. The specific calculation formula is described in detail in the preamble and will not be described further here.

[0138] A specific embodiment of Method 200 may include the following steps:

[0139] 1. Determine whether the two current initial motion vectors satisfy the BIO conditions, and the criteria for this determination are as follows:

[0140] a) The reference images pointed to by the two initial motion vectors are both unspecified reference images, that is, the first reference image pointed to by the first initial motion vector and the second reference image pointed to by the second initial motion vector are both unspecified reference images.

[0141] b) Satisfying one of the following two conditions, i) The two initial motion vectors are from different predicted directions (forward and backward, respectively). ii) The two initial motion vectors are from different reference images with the same predicted direction, neither of the two initial motion vectors is zero, and the ratio of the directional components of the two initial motion vectors is the same, and both are equal to the ratio of the distance between the reference image pointed to by the motion vector and the current image.

[0142] An initial motion vector that satisfies both a) and b) simultaneously meets the BIO conditions and allows for BIO prediction.

[0143] 2. Based on the predicted directions of the two initial motion vectors, the calculation threshold is determined, i.e., v x and v y This is the threshold value for the interval.

[0144] 3. Calculate the gradient value of the pixel point pointed to by the two initial motion vectors.

[0145] 4. Based on the gradient value and the optimization principle, the optimal motion vector is calculated and set as the motion vector for the current image block.

[0146] 5. Obtain the BIO prediction value using the motion vector of the current image block and the reference image block pointed to by the two initial motion vectors.

[0147] If only b)i) is satisfied, the first reference image block pointed to by the first initial motion vector and the second reference image block pointed to by the second initial motion vector are weighted and summed to obtain the predicted image block of the current image block.

[0148] The principles, formulas, and steps referenced by the BIO predictions in the embodiments of this application can be found in the description above and will not be explained further here.

[0149] In BIO prediction technology, if the reference image is a specific reference image, the definition of the time distance between the current image (the current image to be encoded, or the current image to be decoded) and the specific reference image is unclear, and therefore, BIO prediction technology cannot be performed. In the embodiment of this application, if a specific reference image exists in the reference image pointed to by the two initial motion vectors, the predicted image block of the current image block is calculated and obtained directly based on the two initial motion vectors, thereby avoiding the above-mentioned problem.

[0150] It should be understood that the methods of each embodiment of this application can be applied to both the encoding end and the decoding end. The current image block in each embodiment of this application may be an image block to be encoded or an image block to be decoded.

[0151] The embodiments of this application merely exemplify methods for image processing that can be applied to PMMVD technology, DMVR technology, and BIO prediction technology. The methods of the embodiments of this application may be further applied to other existing or future video encoding / decoding technologies, and the embodiments of this application are not limited thereto.

[0152] In the embodiments of this application, in some embodiments, it is necessary to determine the motion vector of an image block by utilizing the motion vector of another image block. For ease of explanation, this image block will be referred to as the first image block, and the other image block that must be utilized will be referred to as the time-domain reference block of the first image block. It can be understood that the first image block and its time-domain reference block are located in different images. Therefore, in the process of determining the motion vector of the first image block by utilizing the motion vector of the time-domain reference block, the motion vector of the time-domain reference block must be scaled.

[0153] Typically, the scaling ratio of the motion vector of the time-domain reference block is determined based on the time distance between the reference image pointed to by the motion vector of the time-domain reference block and the image in which the time-domain reference block is located, and the time distance between the reference image of the first image block and the image in which the first image block is located. Specifically, the scaling ratio of the motion vector of the time-domain reference block is determined based on the time distance between the reference image pointed to by the motion vector of the time-domain reference block and the image in which this time-domain reference block is located, and the time distance between the reference image of the first image block and the image in which the first image block is located.

[0154] In this example, let MV2 be the motion vector of the time-domain reference block, and let x be the reference frame index value of the reference image pointed to by this motion vector MV2. Here, this reference frame index value x is the difference between the sequence number of the reference image pointed to by MV2 (e.g., POC) and the sequence number of the image in which this time-domain reference block resides. Let y be the reference frame index value of the reference image of the first image block. Here, this reference frame index value y is the difference between the sequence number of the reference image of the first image block and the sequence number of the image in which this first image block resides. Then, the scaling ratio of the motion vector MV2 is y / x. Optionally, the product of the motion vector MV2 and y / x may be used as the motion vector of the first image block.

[0155] However, if the motion vector MV2 of the time-domain reference block points to a specific reference image, or if the reference image of the first image block is a specific reference image, the definition of the time distance between the specific reference image and the image in which the first image block is located becomes unclear, and scaling the motion vector MV2 of the time-domain reference block becomes meaningless.

[0156] The embodiments of this application provide a video processing method 1100. Figure 11 is a schematic flowchart of another embodiment of the video processing method 1100 of this application. As shown in Figure 11, this method 1100 is The motion vector of the time-domain reference block of the first image block is determined, where the time-domain reference block and the first image block are located in different images, S1101 S1102 includes the case where the motion vector of the time-domain reference block points to a specific reference image, or where the reference image of the first image block is a specific reference image, and the motion vector of the first image block is determined based on the motion vector of the time-domain reference block after processing, and the motion vector of the time-domain reference block after processing and the motion vector of the time-domain reference block before processing are the same.

[0157] Figure 12 is a schematic flowchart of a video processing method 1200 of another embodiment of this application. As shown in Figure 12, this method 1200 is The motion vector of the time-domain reference block of the first image block is determined, where the time-domain reference block and the first image block are located in different images, S1201 The method includes S1202, which, if the motion vector of the time-domain reference block points to a specific reference image, or if the reference image of the first image block is a specific reference image, obtains the motion vector of the first image block by abandoning the reference of the motion vector of the time-domain reference block.

[0158] Figure 13 is a schematic flowchart of a video processing method 1200 of another embodiment of this application. As shown in Figure 13, this method 1300 is The motion vector of the time-domain reference block of the first image block is determined, where the time-domain reference block and the first image block are located in different images, S1301, S1302 includes setting the motion vector of the time-domain reference block to the motion vector of the first image block if the motion vector of the time-domain reference block points to a specific reference image, or if the reference image of the first image block is a specific reference image.

[0159] In the example, if the motion vector MV2 of the time-domain reference block points to a specific reference image, or if the reference image of the first image block is a specific reference image, the motion vector of the first image block is determined based on the processed motion vector of the time-domain reference block. The processed motion vector of the time-domain reference block described here can be said to be different from the operation on this motion vector under normal circumstances. Here, the operation on the motion vector under normal circumstances refers to the operation on the motion vector MV2 of the time-domain reference block in at least one of the cases where it does not point to a specific reference image, or where the reference image of the first image block is not a specific reference image.

[0160] In the example, if it is determined that the motion vector MV2 of the time-domain reference block points to a specific reference image, or if it is determined that the reference image of the first image block is a specific reference image, the scaling ratio of the motion vector MV2 of the time-domain reference block defaults to 1. The processed motion vector of the time-domain reference block described here may be the motion vector MV2 after being scaled by this default scaling ratio of 1.

[0161] In the example, if it is determined that the motion vector MV2 of the time-domain reference block points to a specific reference image, or if it is determined that the reference image of the first image block is a specific reference image, the scaling step for this motion vector MV2 is skipped. The motion vector of the time-domain reference block after processing described here may be the motion vector MV2 after this scaling step has been skipped.

[0162] In the example, if it is determined that the motion vector MV2 of the time-domain reference block points to a specific reference image, or if it is determined that the reference image of the first image block is a specific reference image, the motion vector of the first image block is obtained by abandoning the reference to the motion vector of the time-domain reference block.

[0163] In the example, the first image block may be a coding unit (CU), or a sub-image block within a coding unit, for example, a sub-CU, and is not limited to that here.

[0164] Here, the method for determining the time-domain reference block of the first image block can be found by referring to the method for determining the associated reference block of the current image block, which will be described below. This will not be explained here.

[0165] Next, based on specific scenarios, we will explain the embodiments of this application with concrete examples.

[0166] In some embodiments where AMVP mode, Merge mode, etc. derive motion vectors, when constructing a candidate list of motion information for the current image block (e.g., an MVP candidate list, a reference frame index value candidate list, or other motion information candidate list), the candidate list is obtained not only based on at least one of the motion vectors of adjacent image blocks in the spatial domain and the reference frame index value of the current image block, but also, for example, based on at least one of the motion vectors of adjacent image blocks in an image in which the current image block is located and the reference frame index value of the current image block, and further based on the motion vectors of adjacent image blocks in the temporal domain (corresponding to the temporal domain reference block described above). For example, the temporal domain reference block of the current image block may be an image block related to the position of the current image block in another image.

[0167] In some embodiments, "an image block related to the position of the current image block in the other image" refers to a co-located image block (hereinafter referred to as a related reference block) whose position is related to the current image block. A reference image containing this related reference block is referred to as a co-located reference picture (hereinafter referred to as a related reference picture).

[0168] The associated reference block of the current image block may be the same image block as the current image block in the associated reference image, or it may be an image block that has other associations with the current image block in the associated reference image, but is not limited to that.

[0169] There are several ways to identify the associated reference blocks of the current image block, some of which are explained below with examples.

[0170] In this example, we first determine the reference image of the current image block. For ease of explanation, we will refer to this reference image as the first reference image. This first reference image may be the image in the reference image set of the current image block that is closest to the image containing the current image block (i.e., the current image). Here, "closest distance" as described here may mean the closest distance in encoding order, or the closest distance in display order. Here, the reference image set of the current image block may be the reference image set of the slice containing the current image block, or it may be the reference image set of the frame containing the current image block.

[0171] Selectively, the associated reference block of the current image block may be the same image block as the current image block in the first reference image, or it may be the image block pointed to by the motion vector of a specific adjacent image block of the current image block. Or it may be the image block corresponding to the numerically cutoff position of the position pointed to by the motion vector of a specific adjacent image block of the current image block. Or it may be the image block pointed to by the motion vector of the same image block as the current image block in the first reference image. Or it may be a specific adjacent image block of the same image block as the current image block in the first reference image. Or it may be the image block pointed to by the motion vector of a specific adjacent image block of the same image block as the current image block in the first reference image.

[0172] In the example, this first reference image may be a reference frame of a specific adjacent image block in the current image of the current image block. For example, this first reference image may be the image pointed to by the motion vector of a specific adjacent image block in the current image of the current image block. Optionally, the associated reference block of the current image block may be the image block pointed to by the motion vector of this specific adjacent image block. Or it may be the same image block as the current image block in the first reference image. Or it may be a specific adjacent image block of the same image block as the current image block in the first reference image. Or it may be the image block pointed to by the motion vector of a specific adjacent image block of the same image block as the current image block in the first reference image.

[0173] For example, this first reference image may be the image closest to the current image in the reference image set of a specific adjacent image block in the current image of the current image block. Selectively, the associated reference block of the current image block may be the same image block as the current image block or the specific adjacent image block in this first reference image. Or it may be the image block pointed to by the motion vector of the image block adjacent to the current image block. Or it may be the image block corresponding to the numerically cutoff position of the position pointed to by the motion vector of the image block adjacent to the current image block. Or it may be the image block pointed to by the motion vector of the same image block as the current image block or the specific adjacent image block in this first reference image. Or it may be the specific adjacent image block of the image block at the same position as the current image block in the first reference image. Or it may be the image block pointed to by the motion vector of the specific adjacent image block of the image block at the same position as the current image block in the first reference image.

[0174] In the example, the first reference image may be a reference image defaulted by the encoding / decoding end, for example, a reference image defaulted by the rules described in the above example. Alternatively, the encoding end may add index information for the first reference image of the current image block in the stream, and the decoding end may determine the associated reference image of the current image block using this index information.

[0175] Here, the defaulted reference image may be determined by the encoding / decoding system, and may also be defined in at least one of the following: sequence header, picture header, video parameter set, sequence parameter set (SPS), picture parameter set (PPS), or slice header.

[0176] Here, in each of the above examples, the specific adjacent image block in the current image of the current image block may have a size that is the same as or different from the size of the current image block. There are multiple ways to determine the specific adjacent image block. In the examples, the method of determining this specific adjacent image block may be to obtain a first adjacent image block in the current image by a specific scan order. In the examples, the method of determining this specific adjacent image block may be to obtain a first adjacent image block in the current image that has the same reference frame index value as the current image block by a specific scan order. In the examples, the method of determining this specific adjacent image block may be to obtain a first image block containing motion information by a specific scan order. In the examples, the method of determining this specific adjacent image block may be to obtain a first adjacent image block that is the same as the defaulted reference image of the current image by a specific scan order.

[0177] Here, the specific scanning order may be a scanning order that starts from the current image block and follows the direction "bottom to top, left to right", or a scanning order that starts from the current image block and follows the direction "right to left, top to bottom", or any other scanning order. In the example, the image blocks obtained by scanning with this specific scanning order may include consecutive image blocks or discrete image blocks. This specific scanning order is determined by the encoding / decoding end.

[0178] When obtaining a candidate list based on the motion vectors of adjacent image blocks in the time domain, for example, when obtaining a candidate list of motion information for the current image block based on the motion vector of the associated reference block of the current image block, it is necessary to scale the motion vector of the associated reference block. For example, the motion vector of this associated reference block needs to be scaled based on the distance between the image pointed to by this motion vector of the associated reference block and the image in which this associated reference block is located, and the distance between the reference frame of the current image block and the current image. Then, the scaled motion vector of the associated reference block is added to the candidate list.

[0179] If the motion vector of this associated reference block points to a specific reference image, there is no point in scaling the motion vector as described above, because the definition of the time distance between the current image and the specific reference image is unclear.

[0180] In one embodiment, if the motion vector of the acquired associated reference block points to a specific reference image, the motion vector of the associated reference block is not scaled and is directly added to the candidate list, or the scaling ratio of the motion vector of the associated reference block is defaulted to 1, and then the scaled motion vector based on the default scaling ratio is added to the candidate list, or the motion information of the associated reference block is directly used as the motion information of the current image block.

[0181] In one embodiment, if the reference frame of the current image block is a specific reference image, the motion vector of this associated reference block is not scaled, and is directly added to the candidate list, or the scaling ratio of the motion vector of this associated reference block is defaulted to 1, and then the scaled motion vector based on the default scaling ratio is added to the candidate list, or the motion information of this associated reference image block is directly used as the motion information of the current image block.

[0182] In one embodiment, if the motion vector of this associated reference block obtained points to a specific reference image, the motion vector of this associated reference block is not added to the candidate list.

[0183] In one embodiment, if the reference frame of the current image block is a specific reference image, the motion vector of this associated reference block is not added to the candidate list.

[0184] To improve the accuracy of the motion vector of the current image block, the current image block is optionally further divided into multiple sub-image blocks. In some embodiments, the current CU (coding unit) is said to be divided into multiple sub-CUs. For each sub-image block in the current image block, the image block to which the position of this sub-image block belongs is obtained. Then, the motion vector of this sub-image block is obtained based on the motion vector of the reference block to which this position belongs.

[0185] In other words, the current image block may be a specific example of the first image block described above, or a sub-image block of the current image block may be a specific example of the first image block described above.

[0186] In one embodiment, when the current image block is divided into multiple sub-image blocks, a motion information candidate list can be optionally constructed for each sub-image block (e.g., an MVP candidate list, a reference frame index value candidate list, or another motion information candidate list). Specifically, when constructing a motion information candidate list for each sub-image block, the candidate list can be obtained based on at least one of the motion vectors of adjacent image blocks in the spatial domain of the current image block and the reference frame index value of the current sub-image block. For example, the motion information candidate list for the current sub-image block is constructed by obtaining a candidate list based on at least one of the motion vectors of adjacent image blocks in an image in which the current image block exists and the reference frame index value of the current sub-image block. Here, the reference frame index value of each sub-image block is the same as the reference frame index value of the current image block. Alternatively, the reference frame index value of each sub-image block is determined based on the reference frame index value of the current image block.

[0187] The motion vector of the current image block is determined based on a list of candidate motion information for each sub-image block within this current image block.

[0188] Taking the current image block as an example, if we divide it into four sub-image blocks, each sub-image block in the current image block has one motion information candidate list, and the current image block has a total of four motion information candidate lists. The motion vector of this current image block is obtained based on these four motion information candidate lists.

[0189] In one embodiment, "obtaining a candidate list based on the motion vectors of adjacent image blocks in the time domain" as described above includes obtaining a candidate list based on the motion vectors of adjacent image blocks in the time domain of at least some of the sub-image blocks in the current image block. In the example, a candidate list can be obtained based on the motion vectors of adjacent image blocks in the time domain of each sub-image block in the current image block, for example, by adding the motion vectors of adjacent image blocks in the time domain of each sub-image block in the current image block to the candidate list. In the example, a candidate list can be obtained based on the motion vectors of adjacent image blocks in the time domain of some of the sub-image blocks in the current image block, for example, by adding the motion vectors of adjacent image blocks in the time domain of some of the sub-image blocks in the current image block to the candidate list.

[0190] Here, there are several ways to determine the adjacent image blocks in the time domain of at least some of the sub-image blocks within the current image block. For example, the above describes how to determine the associated reference blocks of the current image block.

[0191] In the example, the associated reference block of the current image block is divided into multiple sub-image blocks in the same way that the current image block is divided into multiple sub-image blocks, so that each sub-image block in the current image block corresponds to each sub-image block in this associated reference block. The image blocks adjacent to each sub-image block in the time domain in the current image block may each be the corresponding sub-image block in the associated reference block of the current image block.

[0192] In the example, similar to the method for determining all "associated reference images and associated reference blocks of the current image block" described above, all current image blocks in each method are replaced with the current sub-image block, and by adopting this method, the associated reference images and associated reference blocks of the current sub-image block can be determined.

[0193] After determining the associated reference blocks of at least some sub-image blocks in the current image block, if a candidate list is obtained based on the motion vectors of these associated reference blocks, the process involves scaling the motion vectors of these associated reference blocks. For example, the motion vectors of these sub-image blocks are scaled based on the distance between the image pointed to by the motion vector of the associated reference block and the image in which the associated reference block resides, and the distance between the reference frame of the current image block and the current image. The scaled motion vectors are then added to the candidate list.

[0194] In one embodiment, if the motion vector of the associated reference block of this acquired sub-image block points to a specific reference image, this motion vector is not scaled, and is directly added to the candidate list, or the scaling ratio of this motion vector is defaulted to 1, and then the scaled motion vector based on the default scaling ratio is added to the candidate list, or the motion vector of the associated reference block of this sub-image block is directly used as the motion vector of this sub-image block.

[0195] In one embodiment, if the reference frame of the current image block is a specific reference image, the motion vector of this associated reference block is not scaled, and is directly added to the candidate list, or the scaling ratio of the motion vector of this associated reference block is defaulted to 1, and then the scaled motion vector based on the default scaling ratio is added to the candidate list, or the motion vector of the associated reference block of this sub-image block is directly used as the motion vector of this sub-image block.

[0196] In one embodiment, if the motion information of the associated reference block of this acquired sub-image block points to a specific reference image, this motion vector is not added to the candidate list.

[0197] In one embodiment, if the reference frame of the current image block is a specific reference image, the motion vector of this associated reference block is not added to the candidate list.

[0198] Next, I will explain by giving examples based on specific cases.

[0199] As shown in Figure 14, the current image block A is divided into four sub-image blocks of the same size. The associated reference block Col A (co-located A) is obtained in the associated reference image of the current image block A. The associated reference image and the said associated reference image are obtained by the method described above. Specifically, in the example, this associated reference block Col A is divided into four sub-image blocks of the same size as the four sub-image blocks of the current image block A. The four sub-image blocks in associated reference block Col A and the four sub-image blocks in the current image block A correspond to each other. The motion vector of each sub-image block in associated reference block Col A is used as a reference to obtain the motion vector of the corresponding sub-image block in the current image block A.

[0200] The method for calculating the motion vector of the sub-image blocks of the current image block, after obtaining the motion vector of the sub-image block whose position is related to the current image block from the related reference block, is as follows:

[0201] The motion vector of the related sub-image block is obtained by the position in the related reference image block. ColSubblock , and the reference frame index value of this sub-image block Ref ColSubblock This is confirmed. This reference frame index value Ref ColSubblock The position in this associated reference block indicates the distance between the reference frame of the associated sub-image block and the image in which this associated reference block is located.

[0202] The current image block's reference frame index value is RefCurrSubblock This is confirmed. This reference frame index value Ref CurrSubblock This indicates the distance between the reference frame of the current reference block and the image in which this current reference block is located.

[0203] Based on the motion mode of the object, Ref ColSubblock and Ref CurrSubblock If the values ​​are not equal, MV ColSubblock It is necessary to scale the motion vector to fit the assumption of the object's motion mode. Scale the motion vector MV of the sub-image block of the current image block. CurrSubblock The method for obtaining it is as follows: MV CurrSubblock =( Ref CurrSubblock / Ref ColSubblock ) × MV ColSubblock

[0204] Specifically, considering that floating-point numbers and decimals are not typically used in video encoding and decoding systems, the above motion vector scaling formula merely reflects the basic principle of motion vector scaling; in practical implementation, it is necessary to convert to integers for calculations.

[0205] Reference frame index value Ref ColSubblock When pointing to a specific reference image, the motion vector MV ColSubblock Without performing a scaling operation, if the reference frame index of the current image block points to a specific reference image, the motion vector MV ColSubblock Do not perform scaling operations.

[0206] In the example, the video processing device is At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable by the computer, the operations being: The motion vector of the time-domain reference block of the first image block is determined, where the time-domain reference block and the first image block are located in different images. If the motion vector of the time-domain reference block points to a specific reference image, or if the reference image of the first image block is a specific reference image, the motion vector of the first image block is determined based on the motion vector of the time-domain reference block after processing, and the motion vector of the time-domain reference block after processing and the motion vector of the time-domain reference block before processing are the same, or If the motion vector of the time-domain reference block points to a specific reference image, or if the reference image of the first image block is a specific reference image, the motion vector of the first image block is obtained by abandoning the reference of the motion vector of the time-domain reference block, or If the motion vector of the time-domain reference block points to a specific reference image, or if the reference image of the first image block is a specific reference image, the motion vector of the time-domain reference block is set to be the motion vector of the first image block.

[0207] Selectable, the motion vector of the time-domain reference block after the processing is, Based on a scaling ratio where the value is 1, the motion vector of the time-domain reference block is scaled to obtain the motion vector, or Includes the motion vector of the time-domain reference block with the scaling step skipped.

[0208] In the selectable state, the processor further, When the motion vector of the time-domain reference block points to an unspecified reference image, and the reference image of the first image block is an unspecified reference image, the motion vector of the first image block is determined based on the time distance between the reference image pointed to by the motion vector of the time-domain reference block and the image in which the time-domain reference block is located, and the time distance between the reference image of the first image block and the image in which the first image block is located.

[0209] To be selectable, the processor is The scaling ratio of the motion vector of the time-domain reference block is determined based on the time distance between the reference image pointed to by the motion vector of the time-domain reference block and the image in which the time-domain reference block is located, and the time distance between the reference image of the first image block and the image in which the first image block is located. It must be used to determine the motion vector of the first image block based on the motion vector of the time-domain reference block after scaling by the aforementioned scaling ratio.

[0210] In the selectable state, the processor further, A list of motion information candidates for the first image block is constructed, and the list of motion information candidates includes at least one of the motion vector of the spatial region reference block of the first image block and the reference frame index value of the first image block, wherein the spatial region reference block and the first image block are located in the same image. This is for determining the motion vector of the first image block based on the list of candidate motion information for the first image block.

[0211] Selectively, determining the motion vector of the first image block based on the motion vector of the time-domain reference block after processing is: This includes adding the motion vector of the time-domain reference block after the processing to the motion vector candidate list, Alternatively, obtaining the motion vector of the first image block by abandoning the reference of the motion vector of the time-domain reference block is: This includes abandoning the addition of the motion vector of the time-domain reference block to the motion vector candidate list.

[0212] Selectively, the first image block is an encoding unit CU.

[0213] Selectable, the first image block is part of the second image block, and the processor further, A list of motion information candidates for the first image block is constructed, and the list of motion information candidates includes at least one of the motion vector of the spatial region reference block of the second image block and the reference frame index value of the first image block, wherein the spatial region reference block and the second image block are located in the same image. This is for determining the motion vector of the second image block based on the list of motion information candidates for the first image block.

[0214] Selectable means that the reference frame index value of the first image block and the reference frame index value of the second image block are the same, or The reference frame index value of the first image block is determined based on the reference frame index value of the second image block.

[0215] Selectively, determining the motion vector of the first image block based on the motion vector of the time-domain reference block after processing is: This includes adding the motion vector of the time-domain reference block after the processing to the motion information candidate list of the second image block, Alternatively, obtaining the motion vector of the first image block by abandoning the reference of the motion vector of the time-domain reference block is: This includes abandoning the addition of the motion vector of the time-domain reference block to the motion information candidate list of the second image block.

[0216] Selectively, the second image block is an encoding unit CU, the first image block is a sub-image block in the encoding unit, and the sub-image block is obtained by partitioning from the encoding unit.

[0217] Selectively, determining the motion vector of the time-domain reference block of the first image block is possible. In the reference image set of the first image block, determine the image that is closest in distance to the image containing the first image block, This includes determining the time-domain reference block of the first image block and the motion vector of the time-domain reference block based on the determined image.

[0218] Selectable means that the time-domain reference block is the same image block as the first image block in the determined image, or The time-domain reference block is an image block whose motion vector is pointed to by the determined image, or, The time-domain reference block is an image block that corresponds to a numerically cutoff position of the position pointed to by the motion vector of an image block adjacent to the first image block in the determined image, or, The time-domain reference block is an image block in the determined image that is pointed to by the motion vector of the same image block as the position of the first image block, or The time-domain reference block is a specific adjacent image block of the same position as the first image block in the determined image, or The time-domain reference block is an image block in the determined image that is pointed to by the motion vector of a specific adjacent image block of the same position as the first image block.

[0219] Selectively, determining the motion vector of the time-domain reference block of the first image block is possible. In an image containing the first image block, the specific adjacent image block to the first image block is determined, This includes determining the time-domain reference block of the first image block and the motion vector of the time-domain reference block based on the reference image of the specified adjacent image block.

[0220] Selectable means that the time-domain reference block is the same image block as the first image block in the reference image of the specific adjacent image block, or The aforementioned time-domain reference block is, in the reference image of the specified adjacent image block, the same image block as the position of the specified adjacent image block, or The time-domain reference block is the image block pointed to by the motion vector of the specific adjacent image block.

[0221] To make it selectable, it is possible to determine a specific adjacent image block of the first image block. In an image containing the aforementioned first image block, to acquire a first adjacent image block by a specific scanning order, or, In an image containing the first image block, to obtain a first adjacent image block having the same reference frame index value as the first image block by a specific scanning order, or, To obtain a first image block containing motion information by a specific scanning sequence, or, This includes obtaining the same first adjacent image block as the default reference image of the first image by a specific scanning sequence.

[0222] Selectable, the specific scan sequence is a scan sequence that starts with the first block as the search starting point and follows one of the following directions: From bottom to top, from left to right, from right to left, and from top to bottom.

[0223] Selectively, the specified reference image includes at least one of the following: a long-term reference image, a composite frame, or a frame that is not output.

[0224] In the selectable state, the processor further, This is for determining that the aforementioned reference image is the specific reference image when it is determined that the frame is not output and when it is determined that the aforementioned reference image is a composite frame.

[0225] Optionally, the non-specific reference image includes a short-term reference image.

[0226] FIG. 7 is a schematic frame diagram of an apparatus 70 for image motion compensation according to an embodiment of the present application. As shown in FIG. 7, the apparatus 70 for image motion compensation includes at least one memory 701 for storing computer-executable commands, at least one processor 702 that is used alone or jointly to perform the following operations by accessing the at least one memory and executing the computer-executable commands. The operations include obtaining an initial motion vector MV of a current image block, when the reference image pointed to by the initial MV is a specific reference image, performing motion compensation on the current image block based on the initial MV, when the reference image pointed to by the initial MV is a non-specific reference image, modifying the initial MV to obtain a modified MV, and performing motion compensation on the current image block based on the modified MV.

[0227] In some embodiments, the processor 702 specifically obtains an MV candidate list of the current image block, and when any candidate MV in the MV candidate list points to a non-specific reference image, determines the initial MV based on the MV candidate list, modifies the initial MV to obtain a modified MV, and is for performing motion compensation on the current image block based on the modified MV.

[0228] In some embodiments, the processor 702 further determines a candidate MV for adding to the MV candidate list, and when the reference image pointed to by the candidate MV is a non-specific reference image, is for adding the candidate MV to the MV candidate list.

[0229] In some embodiments, the processor 702 further, This is to determine that the reference image pointed to by the candidate MV is an unspecified reference image, based on the frame marks of the reference image pointed to by the candidate MV.

[0230] In some embodiments, the MV candidate list includes at least one candidate MV from the following: If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for the unspecified reference image, Merge candidate MV obtained for a non-specific reference image, MV obtained by interpolation from a non-specific reference image, and These are the upper adjacent MV and left adjacent MV for the unspecified reference image of the current block.

[0231] In some embodiments, the processor 702 specifically, Based on at least one of the bidirectional matching method and the template matching method, the distortion cost corresponding to the candidate MV in the MV candidate list is determined. The purpose is to select the MV with the smallest distortion cost in the aforementioned MV candidate list as the initial MV.

[0232] In some embodiments, the processor 702 specifically, A sub-MV candidate list is generated for the current image block, and the sub-MV candidate list includes the initial MV. This is to determine the MV with the smallest distortion cost from the aforementioned sub-MV candidate list as the corrected MV.

[0233] In some embodiments, the initial MV includes a first initial MV and a second initial MV, The processor 702, specifically, If the reference image pointed to by at least one of the first initial MV and the second initial MV is a specific reference image, motion compensation is performed on the current image block based on the first initial MV and the second initial MV. If the reference image pointed to by the first initial MV and the second initial MV is an unspecified reference image, the first initial MV and the second initial MV are modified to obtain a modified first MV and a modified second MV, and motion compensation is performed on the current image block based on the modified first MV and the modified second MV.

[0234] In some embodiments, the processor 702 specifically, A template is generated based on the first reference image block and the second reference image block, where the first reference image block corresponds to the first initial MV and belongs to the first reference image, and the second reference image block corresponds to the second initial MV and belongs to the second reference image. This is for modifying the first initial MV and the second initial MV based on the aforementioned template, and obtaining the modified first MV and the modified second MV.

[0235] In some embodiments, the processor 702 specifically, N third reference image blocks are used to match each of the templates, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image. M fourth reference image blocks are used to match each of the templates, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image. Based on the matching results, one third initial MV is selected from the N third initial MVs, and one fourth initial MV is selected from the M fourth initial MVs. These three third initial MVs and the fourth initial MV are used as the MV of the current image block, or to determine the MV of the current image block.

[0236] In some embodiments, the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV.

[0237] In some embodiments, at least some of the initial MVs among the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the initial MVs among the M fourth initial MVs are obtained by shifting based on the second initial MV.

[0238] In some embodiments, N is equal to M.

[0239] In some embodiments, the first reference image is the forward frame of the current image block, the second reference image is the backward frame of the current image block, or the first reference image is the forward frame of the current image block, and the second reference image is the forward frame of the current image block.

[0240] It should be understood that the apparatus 70 for image motion compensation can also be realized by corresponding software modules, which will not be described further here.

[0241] FIG. 8 is a schematic frame diagram of an apparatus 80 for image processing according to another embodiment of the present application. As shown in FIG. 8, the apparatus 80 for image processing includes at least one memory 801 for storing computer-executable commands, and at least one processor 802 that accesses the at least one memory and executes the computer-executable commands to be used alone or jointly to perform the following operations, and the operations include obtaining a first initial motion vector MV and a second initial MV, where the first initial MV points to a first reference image and the second initial MV points to a second reference image, and If at least one of the first reference image and the second reference image is a specific reference image, then the predicted image block of the current image block is calculated and obtained based on the first initial MV and the second initial MV, If both the first reference image and the second reference image are unspecified reference images, the MV of the current image block is calculated and obtained based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and the predicted image block of the current image block is calculated and obtained based on the MV of the current image block.

[0242] In some embodiments, the processor 802 specifically, This is for calculating and obtaining the MV of the current image block based on the gradient values ​​of the pixel points pointed to by the first initial MV and the second initial MV, and the optimization principle.

[0243] In some embodiments, the processor 802 specifically, This method involves weighting and summing the first reference image block pointed to by the first initial MV and the second reference image block pointed to by the second initial MV to obtain a predicted image block for the current image block.

[0244] In some embodiments, the prediction directions of the first initial MV and the second initial MV are different.

[0245] In some embodiments, the prediction directions of the first initial MV and the second initial MV are the same, neither the first initial MV nor the second initial MV is 0, and the first reference image and the second reference image are different.

[0246] In some embodiments, the processor 802 specifically, This is for calculating and obtaining a predicted image block for the current image block based on the MV of the current image block, the first reference image block pointed to by the first initial MV, and the second reference image block pointed to by the second initial MV.

[0247] It is understood that the image processing device 80 can also be implemented using a corresponding software module, and this will not be explained further here.

[0248] In some techniques that use motion vector derivation, scaling a motion vector when it points to a specific reference image is meaningless and reduces search efficiency and encoding / decoding efficiency. This is because scaling a motion vector is based on the distance between images, while a specific reference image may be artificially constructed and not at a distance from other reference images. Therefore, scaling a motion vector based on such a specific reference image is not very meaningful.

[0249] This application further provides a method for image motion compensation. Figure 9 is a schematic flowchart of method 900 for image motion compensation in another embodiment of this application. As shown in Figure 9, method 900 is S910 obtains the initial motion vector MV of the current image block, S920 determines the scaling ratio of this initial MV, and here, if this initial MV points to a specific reference image, the scaling ratio of this initial MV is 1. S930, based on the scaling ratio of this initial MV, scales this initial MV, S940 includes performing motion compensation on the current image block based on the scaled MV.

[0250] The method for obtaining the initial MV here can be the same as the method for obtaining the initial MV described above, and will not be explained further here.

[0251] In the embodiment of this application, the method for compensating for image motion has a scaling ratio of 1 when the initial MV points to a specific reference image. When the initial MV points to a non-specific reference image, the scaling ratio is not limited, and the motion vector is scaled based on the distance between the current image and the non-specific reference image, and the scaling policy, thereby performing motion compensation and improving the encoding and decoding efficiency.

[0252] In some embodiments, motion compensation is performed directly on the current image block based on the scaled MV.

[0253] In some embodiments, the scaled MV can be further modified to obtain a modified MV, and motion compensation is performed on the current image block based on the modified MV. Here, the method for modifying the scaled MV may be the same as the method for modifying the initial MV described above, and will not be described further here.

[0254] For example, as shown in Figure 5, the first initial MV points to the first reference image, and the second initial MV points to the second reference image. If both the first and second reference images point to an unspecified reference image, the first and second initial MVs are scaled based on the distance between the first and second reference images. Then, motion compensation is performed on the current image block based on the scaled first and second initial MVs. In one embodiment, a first reference image block is determined, which corresponds to the scaled first initial MV and belongs to the second reference image. A second reference image block is determined, which corresponds to the scaled second initial MV and belongs to the first reference image. A template is generated based on the first and second reference image blocks, and the first and second initial MVs are modified based on this template to obtain the modified first and second MVs. The motion vector of the current image block is calculated using the modified first and second MVs.

[0255] Here, the method for modifying the first initial MV and the second initial MV based on this template may be the same as the method for modifying the first initial MV and the second initial MV based on the template described above, and will not be explained further here.

[0256] This application provides yet another method for image motion compensation. In this method for image motion compensation, an initial motion vector MV of the current image block is obtained, and motion compensation is performed on the current image block based on the initial MV by employing different operations depending on whether the initial MV points to a specific reference image or to a non-specific reference image.

[0257] The method for obtaining the initial MV here can be the same as the method for obtaining the initial MV described above, and will not be explained further here.

[0258] In one embodiment, if the initial MV points to a specific reference image, motion compensation is performed on the current image block using this initial MV. If the initial MV points to a non-specific reference image, there are four embodiments: 1. Scaling the initial MV and performing motion compensation on the current image block based on the scaled initial MV; 2. Modifying the initial MV and performing motion compensation on the current image block based on the modified initial MV; 3. Scaling the initial MV, modifying the scaled MV to obtain the modified MV, and performing motion compensation on the current image block based on the modified MV; 4. Modifying the initial MV, scaling the modified MV to obtain the scaled MV, and performing motion compensation on the current image block based on the scaled MV.

[0259] Here, the method for correcting the MV after scaling, or the method for correcting the initial MV, may be the same as the method for correcting the initial MV described above, and will not be explained further here.

[0260] In one embodiment, if the initial MV points to a specific reference image, the initial MV can be modified, and motion compensation can be performed based on the modified MV. If the initial MV points to a non-specific reference image, motion compensation can be performed on the current image block after scaling the initial MV, or after scaling and modifying the initial MV. That is, in situations where the initial MV points to a specific reference image, the scaling step can be skipped and the initial MV can be modified directly, or the scaling ratio of the initial MV can be set directly to 1 and then scaled and modified. In situations where the initial MV points to a non-specific reference image, the initial MV can be scaled, or the initial MV can be scaled and modified to improve encoding and decoding efficiency.

[0261] In other embodiments of this application, when the initial MV points to a specific reference image, motion compensation is performed using the initial MV directly, or the initial MV is modified and motion compensation is performed based on the modified MV. When the initial MV points to a non-specific reference image, at least one of scaling and modification is performed on the initial MV and motion compensation is performed. Various embodiments may be combined in any way, and the embodiments of this application are not limited thereto.

[0262] In some embodiments, obtaining the initial motion vector MV of the current image block includes obtaining a list of candidate MVs for the current image block and determining the initial MV based on the list of candidate MVs. That is, after scaling the initial MV (including scaling with a scaling ratio of 1), an MV may be selected and modified so that the initial MV points to an unspecified reference image, and any of the initial MVs may be modified regardless of the type of reference image they point to.

[0263] For selection to be possible, the reference image pointed to by any of the candidate MVs in the MV candidate list is an unspecified reference image.

[0264] Selectively, obtaining the MV candidate list for the current image block may include determining a candidate MV to add to the MV candidate list, and if the reference image pointed to by the candidate MV is an unspecified reference image, adding the candidate MV to the MV candidate list.

[0265] Optionally, the method may further include determining that the reference image pointed to by the candidate MV is an unspecified reference image based on the frame marks of the reference image pointed to by the candidate MV.

[0266] Selectable, the MV candidate list may include at least one of the following candidate MVs: if the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for the unspecified reference image, the merge candidate MV obtained for the unspecified reference image, the MV obtained by interpolation from the unspecified reference image, and the upper adjacent MV and left adjacent MV of the current block with respect to the unspecified reference image.

[0267] Selectively determining the initial MV based on the MV candidate list may include determining the strain cost corresponding to the candidate MV in the MV candidate list based on at least one of the bidirectional matching method and the template matching method, and selecting the MV with the smallest strain cost in the MV candidate list as the initial MV.

[0268] Selectively, modifying the scaled MV and obtaining the modified MV may include generating a sub-MV candidate list in the current image block, the sub-MV candidate list containing the scaled MV, and determining the MV with the smallest distortion cost from the sub-MV candidate list as the modified MV.

[0269] Selectively, the initial MV includes a first initial MV and a second initial MV, and the method includes at least one of the following: if the reference image pointed to by at least one of the first and second initial MVs is a specific reference image, setting the scaling ratios of both the first and second initial MVs to 1 and performing motion compensation on the current image block based on the first and second initial MVs; or if the reference image pointed to by both the first and second initial MVs is a non-specific reference image, determining the scaling ratio of the first initial MV and the scaling ratio of the second initial MV, scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and performing motion compensation on the current image block based on the scaled first initial MV and the scaled second initial MV.

[0270] Selectively, motion compensation for the current image block based on the scaled first initial MV and the scaled second initial MV includes generating a template based on a first reference image block and a second reference image block, where the first reference image block corresponds to the scaled first initial MV and belongs to the second reference image, the second reference image block corresponds to the scaled second initial MV and belongs to the first reference image, the first reference image is a reference image pointed to by the first initial MV, and the second reference image is a reference image pointed to by the second initial MV, and modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV.

[0271] Selectively modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV includes using N third reference image blocks to match each with the template, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image; using M fourth reference image blocks to match each with the template, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image; and based on the matching result, selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs, and using the one third initial MV and the one fourth initial MV as the MV of the current image block, or determining the MV of the current image block.

[0272] Selectively, the third initial MV may include the first initial MV, and the fourth initial MV may include the second initial MV.

[0273] Selectively, at least some of the initial MVs in the N third initial MVs may be obtained by shifting based on the first initial MV, and at least some of the initial MVs in the M fourth initial MVs may be obtained by shifting based on the second initial MV.

[0274] For selection to be possible, N may be equal to M.

[0275] Selectively, the first reference image is the forward frame of the current image block and the second reference image is the backward frame of the current image block, or the first reference image is the forward frame of the current image block and the second reference image is the forward frame of the current image block.

[0276] It is understood that the selectable embodiments described above can be implemented in a manner similar to the details of Method 100, and will not be described further here.

[0277] Figure 10 is a schematic frame diagram of an image motion compensation apparatus 1000 of one embodiment of the present application. As shown in Figure 10, the image motion compensation apparatus 1000 is A memory 1001 for storing executable commands for the computer, The system includes at least one processor 1002 used alone or jointly to perform the following operations by accessing the at least one memory and executing a command executable by the computer, the operations being: Obtain the initial motion vector MV of the current image block, and The scaling ratio of the initial MV is determined, and here, if the initial MV points to a specific reference image, the scaling ratio of the initial MV is 1. Scaling the initial MV based on the scaling ratio of the initial MV, This involves performing motion compensation on the current image block based on the scaled MV.

[0278] In some embodiments, the processor 1002 performs motion compensation on the current image block based on the scaled MV, The scaling of the MV is corrected, and the corrected MV is obtained. This includes performing motion compensation on the current image block based on the modified MV.

[0279] In some embodiments, the processor 1002 obtains the initial motion vector MV of the current image block, Obtaining the MV candidate list for the current image block, This includes determining the initial MV based on the aforementioned MV candidate list.

[0280] In some embodiments, the reference image pointed to by any of the candidate MVs in the MV candidate list is an unspecified reference image.

[0281] In some embodiments, the processor 1002 obtains the MV candidate list for the current image block, This includes determining a candidate MV to be added to the aforementioned MV candidate list, and if the reference image pointed to by the candidate MV is an unspecified reference image, adding the candidate MV to the aforementioned MV candidate list.

[0282] In some embodiments, the processor 1002 further, This is to determine that the reference image pointed to by the candidate MV is an unspecified reference image, based on the frame marks of the reference image pointed to by the candidate MV.

[0283] In some embodiments, the MV candidate list includes at least one candidate MV from the following: If the current image block is in advanced motion vector prediction AMVP mode, the original AMVP candidate MV obtained for the unspecified reference image, Merge candidate MV obtained for a non-specific reference image, MV obtained by interpolation from a non-specific reference image, and These are the upper adjacent MV and left adjacent MV for the unspecified reference image of the current block.

[0284] In some embodiments, the processor 1002 determines the initial MV based on the MV candidate list, Based on at least one of the bidirectional matching method and the template matching method, the distortion cost corresponding to the candidate MV in the MV candidate list is determined, This includes selecting the MV with the smallest distortion cost in the MV candidate list as the initial MV.

[0285] In some embodiments, the processor 1002 modifies the scaled MV and obtains the modified MV, A sub-MV candidate list is generated for the current image block, and the sub-MV candidate list includes the scaled MV. This includes determining the MV with the smallest distortion cost from the sub-MV candidate list as the corrected MV.

[0286] In some embodiments, the initial MV includes a first initial MV and a second initial MV, and the processor 1002 specifically, If the reference image pointed to by at least one of the first initial MV and the second initial MV is a specific reference image, the scaling ratios of both the first initial MV and the second initial MV are set to 1, and motion compensation is performed on the current image block based on the first initial MV and the second initial MV. If the reference image pointed to by the first initial MV and the second initial MV is an unspecified reference image, this is for at least one of the following: determining the scaling ratio of the first initial MV and the scaling ratio of the second initial MV, scaling the first initial MV based on the scaling ratio of the first initial MV, scaling the second initial MV based on the scaling ratio of the second initial MV, and performing motion compensation on the current image block based on the scaled first initial MV and the scaled second initial MV.

[0287] In some embodiments, the processor 1002 performs motion compensation on the current image block based on the first initial MV after scaling and the second initial MV after scaling. A template is generated based on a first reference image block and a second reference image block, wherein the first reference image block corresponds to the scaled first initial MV and belongs to the second reference image, the second reference image block corresponds to the scaled second initial MV and belongs to the first reference image, the first reference image is the reference image pointed to by the first initial MV, and the second reference image is the reference image pointed to by the second initial MV. This includes modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV.

[0288] In some embodiments, the processor 1002 modifies the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV, N third reference image blocks are used to match each of the templates, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image. Using M fourth reference image blocks, each is matched with the template, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image. The method includes selecting one third initial MV from the N third initial MVs and one fourth initial MV from the M fourth initial MVs based on the matching results, and using the one third initial MV and the one fourth initial MV as the MV of the current image block, or determining the MV of the current image block.

[0289] In some embodiments, the third initial MV includes the first initial MV, and the fourth initial MV includes the second initial MV.

[0290] In some embodiments, at least some of the initial MVs in the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the initial MVs in the M fourth initial MVs are obtained by shifting based on the second initial MV.

[0291] In some embodiments, N is equal to M.

[0292] In some embodiments, the first reference image is the forward frame of the current image block, the second reference image is the backward frame of the current image block, or The first reference image is a forward-facing frame of the current image block, and the second reference image is a forward-facing frame of the current image block.

[0293] In some embodiments, the specific reference image includes at least one of a long-term reference image, a composite frame, and a non-output frame.

[0294] In some embodiments, the processor 1002 further, This is for determining that the aforementioned reference image is the specific reference image when it is determined that the aforementioned reference image is a frame in which no output is generated, and further determined that the aforementioned reference image is a composite frame.

[0295] In some embodiments, the non-specific reference image includes a short-term reference image.

[0296] It is understood that the device 1000 for image motion compensation can also be implemented by a corresponding software module, and this will not be explained further here.

[0297] Each embodiment of the present application may be implemented by a memory and a processor, each memory being for storing commands for performing the methods of the embodiments of the present application, and the processor being understood to execute the aforementioned commands so that the device can perform the methods of each embodiment of the present application.

[0298] The processor referred to in the embodiments of this application may be a CPU (Central Processing Unit), but may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates, or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or this processor may be any ordinary processor, etc.

[0299] It should be further understood that the memories referred to in the embodiments of this application may be volatile memories, non-volatile memories, or may include both volatile and non-volatile memories. Here, non-volatile memories may be read-only memories (ROM), programmable read-only memories (PROM), erasable programmable read-only memories (Erasable PROM, EPROM), electrically erasable programmable read-only memories (Electrically EPROM, EEPROM), or flash memories. Volatile memories may be random-access memories (RAM) used as external high-speed buffer memories. By illustrative but non-limiting explanation, many forms of RAM can be used, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct Rambus RAM (DR RAM).

[0300] If the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate, or transistor logic device, or discrete hardware component, it needs to be explained that the memory (storage module) is integrated into the processor.

[0301] It should be noted that the memory described herein includes, but is not limited to, these and any other appropriate types of memory.

[0302] Embodiments of this application further provide a computer-readable storage medium on which commands are stored, and which causes the computer to execute the methods of each embodiment of the above-described method when the commands are operated on the computer.

[0303] Embodiments of this application further provide a computer device comprising a storage medium readable by the computer described above.

[0304] Embodiments of this application further provide an encoding device which includes at least one of an apparatus 70 for image motion compensation, an apparatus 80 for image processing, and an apparatus 1000 for image motion compensation.

[0305] Embodiments of this application further provide a decoding device which includes at least one of an apparatus 70 for image motion compensation, an apparatus 80 for image processing, and an apparatus 1000 for image motion compensation.

[0306] The embodiments described in this application can be applied to aircraft, particularly in the field of unmanned aerial vehicles.

[0307] It should be understood that the classifications of electrical circuits, sub-electrical circuits, and sub-units in each embodiment of this application are for illustrative purposes only. Those skilled in the art will recognize that each exemplary electrical circuit, sub-electrical circuit, and sub-unit described in the embodiments disclosed herein can be disassembled or combined.

[0308] In the embodiments described above, implementation can be carried out in whole or in part by software, hardware, firmware, or any other combination. When implemented using software, it can be carried out in whole or in part by a computer program product. The computer program product includes one or more computer commands. When a computer loads or executes computer commands, it generates, in whole or in part, a process or function based on the embodiments of this application. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable device. Computer commands can be stored in a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, computer commands can be transmitted from a site, computer, server, or data center to another site, computer, server, or data center by wired means (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless means (e.g., infrared, radio, microwave, etc.). A computer-readable storage medium may be any available medium accessible by a computer, or it may be a data storage device including a server, data center, etc., which integrates one or more available media. Available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs (Digital Video Discs)), or semiconductor media (e.g., SSDs (Solid State Disks)).

[0309] Throughout this specification, the terms "one embodiment" or "embodiment" refer to a particular feature, structure, or characteristic associated with that embodiment, and it should be understood that such feature is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" in any other part of this specification does not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics can be combined in any suitable manner into one or more embodiments.

[0310] In each embodiment of this application, the numbering of each process described above does not indicate the order of execution, and the execution order of each process is determined by its function and inherent logic, and does not impose any restrictions on the implementation of the embodiments of this application.

[0311] In the embodiments of this application, "B corresponding to A" is understood to mean that B is related to A and that B can be determined based on A. However, it is understood that determining B based on A does not mean determining B solely by A, but rather that B can be determined based on at least one of A and other information.

[0312] In this text, the term "at least one" merely describes the relationship between related objects, and it should be understood that three relationships are possible. For example, at least one of A and B can be in one of three situations: A exists alone, A and B exist simultaneously, or B exists alone. In addition, the symbol " / " in this text usually indicates that the related objects before and after it are in an "or" relationship.

[0313] Those skilled in the art will recognize, based on the exemplary units and algorithmic steps described in the embodiments disclosed herein, that these can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are ultimately performed in hardware or software will depend on the specific application of the technical solution and the design constraints. Those skilled in the art will recognize that the functions described can be implemented by using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

[0314] Those skilled in the art will clearly understand, for the sake of clarity and conciseness, that the specific operating processes of the systems, apparatus, and units described above can be referenced to the corresponding processes in the embodiments of the methods described above, and will not be described further here.

[0315] In the various embodiments provided in this application, it should be understood that the systems, apparatus, and methods disclosed can be implemented in other ways. For example, the embodiments of the apparatus described above are illustrative only. For example, the division of the units is merely a division of logic functions, and in actual implementation, there may be other methods of division, for example, multiple units or components may be combined or integrated into other systems, or some features may be ignored or not performed. Also, the connections between each other, direct connections or communication connections shown or considered may be by some interfaces, and the indirect connections or communication connections between apparatus or units may be electrical, mechanical or in other forms.

[0316] The unit described as the separating member may or may not be physically separated, and the member shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed among multiple network units. Some or all of these units can be selected as needed to achieve the objective of the solution of this embodiment.

[0317] In addition, each functional unit in each embodiment of this application can be integrated into a single processing unit, each unit can exist physically independently, or two or more units can be integrated into a single unit.

[0318] The above describes only specific embodiments of this application; however, the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of modifications or substitutions within the scope of the art disclosed herein, and such modifications or substitutions will be included within the scope of protection of this application. Accordingly, the scope of protection of this application is based on the scope of protection of the claims described herein. [Item 1] A video processing method, wherein the above method is The motion vector of the time-domain reference block of the first image block is determined, where the time-domain reference block and the first image block are located in different images. If the motion vector of the time-domain reference block points to a specific reference image, or if the reference image of the first image block is a specific reference image, the motion vector of the first image block is determined based on the motion vector of the time-domain reference block after processing, and the motion vector of the time-domain reference block after processing and the motion vector of the time-domain reference block before processing are the same, or If the motion vector of the time-domain reference block points to a specific reference image, or if the reference image of the first image block is a specific reference image, the motion vector of the first image block can be obtained by abandoning the reference to the motion vector of the time-domain reference block, or A video processing method that includes setting the motion vector of the time-domain reference block to the motion vector of the first image block when the motion vector of the time-domain reference block points to a specific reference image, or when the reference image of the first image block is a specific reference image. [Item 2] The motion vector of the time-domain reference block after the above processing is: Based on a scaling ratio where the value is 1, the motion vector of the time-domain reference block is scaled to obtain the motion vector, or The video processing method described in item 1, which includes the motion vector of the time-domain reference block described above, with the scaling step skipped. [Item 3] The above method, The video processing method according to item 1 or 2, further comprising determining the motion vector of the first image block based on the time distance between the reference image pointed to by the motion vector of the time-domain reference block and the image containing the time-domain reference block, and the time distance between the reference image of the first image block and the image containing the first image block, when the motion vector of the time-domain reference block points to an unspecified reference image and the reference image of the first image block is an unspecified reference image. [Item 4] The above method, Based on the time distance between the reference image pointed to by the motion vector of the time-domain reference block and the image in which the time-domain reference block is located, and the time distance between the reference image of the first image block and the image in which the first image block is located, the scaling ratio of the motion vector of the time-domain reference block is determined. The video processing method according to item 3, which includes determining the motion vector of the first image block based on the motion vector of the time-domain reference block after scaling with the above scaling ratio. [Item 5] The above method, A list of motion information candidates for the first image block is constructed, and this list of motion information candidates includes at least one of the motion vector of the spatial region reference block of the first image block and the reference frame index value of the first image block, wherein the spatial region reference block and the first image block are located in the same image. A video processing method according to any one of items 1 to 4, further comprising determining the motion vector of the first image block based on the list of motion information candidates for the first image block. [Item 6] Determining the motion vector of the first image block based on the motion vector of the time-domain reference block after the above processing is: This includes adding the motion vector of the time-domain reference block after the above processing to the motion vector candidate list, Alternatively, obtaining the motion vector of the first image block by abandoning the reference of the motion vector of the time-domain reference block is possible. The video processing method described in item 5, which includes abandoning the addition of the motion vector of the time-domain reference block to the motion vector candidate list. [Item 7] The above first image block is a video processing method according to item 5 or 6, wherein the encoding unit CU is used. [Item 8] The above first image block is part of the second image block, and the above method is A list of motion information candidates for the first image block is constructed, and this list of motion information candidates includes at least one of the motion vector of the spatial region reference block of the second image block and the reference frame index value of the first image block, where the spatial region reference block and the second image block are located in the same image. A video processing method according to any one of items 1 to 3, further comprising determining the motion vector of the second image block based on the list of motion information candidates for the first image block. [Item 9] The reference frame index value of the first image block and the reference frame index value of the second image block are the same, or The video processing method described in item 8, wherein the reference frame index value of the first image block is determined based on the reference frame index value of the second image block. [Item 10] Determining the motion vector of the first image block based on the motion vector of the time-domain reference block after the above processing is: This includes adding the motion vector of the time-domain reference block after the above processing to the motion information candidate list of the second image block, Alternatively, obtaining the motion vector of the first image block by abandoning the reference of the motion vector of the time-domain reference block is possible. A video processing method according to item 8 or 9, which includes abandoning the addition of the motion vector of the time-domain reference block to the motion information candidate list of the second image block. [Item 11] The above-mentioned second image block is an encoding unit CU, the above-mentioned first image block is a sub-image block in the above-mentioned encoding unit, and the above-mentioned sub-image block is obtained by separating it from the above-mentioned encoding unit, the video processing method according to item 1, 8, or 9. [Item 12] Determining the motion vector of the time-domain reference block in the first image block above is: In the reference image set of the first image block mentioned above, determine the image that is closest in distance to the image containing the first image block, A video processing method according to any one of items 1 to 11, comprising determining the time-domain reference block of the first image block and the motion vector of the time-domain reference block based on the determined image. [Item 13] The above time-domain reference block is the same image block as the first image block in the above-determined image, or The above time-domain reference block is an image block whose motion vector is pointed to by the above determined image, or, The above time-domain reference block is an image block that corresponds to a numerically cutoff position of the position pointed to by the motion vector of an image block adjacent to the above first image block in the above determined image, or, The above time-domain reference block is an image block pointed to by the motion vector of the same image block as the position of the above first image block in the above determined image, or, The above time-domain reference block is a specific adjacent image block of the same position as the first image block in the above-determined image, or The video processing method according to item 12, wherein the time-domain reference block is an image block pointed to by the motion vector of a specific adjacent image block of the same position as the first image block in the determined image. [Item 14] Determining the motion vector of the time-domain reference block in the first image block above is: In an image containing the above-mentioned first image block, the specific adjacent image block to the above-mentioned first image block is determined, A video processing method according to any one of items 1 to 11, which includes determining the time-domain reference block of the first image block and the motion vector of the time-domain reference block based on the reference image of the specified adjacent image block. [Item 15] The above time-domain reference block is the same image block as the first image block in the reference image of the above specific adjacent image block, or The above time-domain reference block is, in the reference image of the above specific adjacent image block, the same image block as the position of the above specific adjacent image block, or The above time-domain reference block is the image block pointed to by the motion vector of the above specific adjacent image block, as described in item 14 of the video processing method. [Item 16] Determining a specific adjacent image block of the above first image block is: In an image containing the above-mentioned first image block, the first adjacent image block is acquired by a specific scanning order, or, In an image containing the above-mentioned first image block, to obtain a first adjacent image block having the same reference frame index value as the above-mentioned first image block by a specific scanning order, or, To obtain a first image block containing motion information by a specific scanning sequence, or, The video processing method according to item 14, which includes obtaining the same first adjacent image block as the default reference image of the first image by a specific scanning order. [Item 17] The above specific scanning sequence starts with the above first block as the search starting point. The video processing method described in item 16, wherein the scanning order is along one of the following directions: bottom to top, left to right, right to left, or top to bottom. [Item 18] The specified reference image described above is the method described in any one of items 1 to 14, which includes at least one of a long-term reference image, a composite frame, or a frame that is not output. [Item 19] The above method, The method according to any one of items 1 to 15, further comprising determining that the above reference image is the specific reference image if it is determined that the above reference image is not output in the frame and that the above reference image is a composite frame. [Item 20] The above non-specific reference image is the method described in any one of items 1 to 16, including the short-term reference image. [Item 21] A video processing device, At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing at least one memory and executing a command executable on the computer, wherein the operations are: The motion vector of the time-domain reference block of the first image block is determined, where the time-domain reference block and the first image block are located in different images. If the motion vector of the time-domain reference block points to a specific reference image, or if the reference image of the first image block is a specific reference image, the motion vector of the first image block is determined based on the motion vector of the time-domain reference block after processing, and the motion vector of the time-domain reference block after processing and the motion vector of the time-domain reference block before processing are the same, or If the motion vector of the time-domain reference block points to a specific reference image, or if the reference image of the first image block is a specific reference image, the motion vector of the first image block can be obtained by abandoning the reference to the motion vector of the time-domain reference block, or An apparatus wherein, when the motion vector of the time-domain reference block points to a specific reference image, or when the reference image of the first image block is a specific reference image, the motion vector of the time-domain reference block is set to be the motion vector of the first image block. [Item 22] The motion vector of the time-domain reference block after the above processing is: Based on a scaling ratio where the value is 1, the motion vector of the time-domain reference block is scaled to obtain the motion vector, or A video processing apparatus according to item 21, which includes the motion vector of the time-domain reference block described above, with the scaling step skipped. [Item 23] The above processor further, The motion processing apparatus according to item 21 or 22, wherein, when the motion vector of the time-domain reference block points to an unspecified reference image, and the reference image of the first image block is an unspecified reference image, the motion vector of the first image block is determined based on the time distance between the reference image pointed to by the motion vector of the time-domain reference block and the image in which the time-domain reference block is located, and the time distance between the reference image of the first image block and the image in which the first image block is located. [Item 24] The above processor is Based on the time distance between the reference image pointed to by the motion vector of the time-domain reference block and the image in which the time-domain reference block is located, and the time distance between the reference image of the first image block and the image in which the first image block is located, the scaling ratio of the motion vector of the time-domain reference block is determined. A video processing apparatus according to item 23, used to determine the motion vector of the first image block based on the motion vector of the time-domain reference block after scaling with the above scaling ratio. [Item 25] The above processor further, A list of motion information candidates for the first image block is constructed, and the list of motion information candidates includes at least one of the motion vector of the spatial region reference block of the first image block and the reference frame index value of the first image block, wherein the spatial region reference block and the first image block are located in the same image. A video processing device according to any one of items 21 to 24, which determines the motion vector of the first image block based on the list of motion information candidates for the first image block. [Item 26] Determining the motion vector of the first image block based on the motion vector of the time-domain reference block after the above processing is: This includes adding the motion vector of the time-domain reference block after the above processing to the motion vector candidate list, Alternatively, obtaining the motion vector of the first image block by abandoning the reference of the motion vector of the time-domain reference block is possible. The motion processing apparatus according to item 25, which includes abandoning the addition of the motion vector of the time-domain reference block to the motion vector candidate list. [Item 27] The above-mentioned first image block is a video processing device according to item 25 or 26, which is an encoding unit CU. [Item 28] The above first image block is part of the second image block, and the above processor further, A list of motion information candidates for the first image block is constructed, and the list of motion information candidates includes at least one of the motion vector of the spatial region reference block of the second image block and the reference frame index value of the first image block, where the spatial region reference block and the second image block are located in the same image. A video processing apparatus according to any one of items 21 to 23, which determines the motion vector of the second image block based on the list of motion information candidates for the first image block. [Item 29] The reference frame index value of the first image block and the reference frame index value of the second image block are the same, or The reference frame index value of the first image block is determined based on the reference frame index value of the second image block, as described in item 28 of the video processing apparatus. [Item 30] Determining the motion vector of the first image block based on the motion vector of the time-domain reference block after the above processing is: This includes adding the motion vector of the time-domain reference block after the above processing to the motion information candidate list of the second image block, Alternatively, obtaining the motion vector of the first image block by abandoning the reference of the motion vector of the time-domain reference block is possible. The motion processing apparatus according to item 28 or 29, which includes abandoning the addition of the motion vector of the time-domain reference block to the motion information candidate list of the second image block. [Item 31] The above-mentioned second image block is an encoding unit CU, the above-mentioned first image block is a sub-image block in the encoding unit, and the above-mentioned sub-image block is obtained by separating it from the encoding unit, as described in item 21, 28, or 29. [Item 32] Determining the motion vector of the time-domain reference block in the first image block above is: In the reference image set of the first image block mentioned above, determine the image that is closest in distance to the image containing the first image block, A video processing apparatus according to any one of items 21 to 31, comprising determining the time-domain reference block of the first image block and the motion vector of the time-domain reference block based on the determined image. [Item 33] The above time-domain reference block is the same image block as the first image block in the above-determined image, or The above time-domain reference block is an image block whose motion vector is pointed to by the above determined image, or, The above time-domain reference block is an image block that corresponds to a numerically cutoff position of the position pointed to by the motion vector of an image block adjacent to the above first image block in the above determined image, or, The above time-domain reference block is an image block pointed to by the motion vector of the same image block as the position of the above first image block in the above determined image, or, The above time-domain reference block is a specific adjacent image block of the same position as the first image block in the above-determined image, or The video processing apparatus according to item 32, wherein the time-domain reference block is an image block pointed to by the motion vector of a specific adjacent image block of the same position as the first image block in the determined image. [Item 34] Determining the motion vector of the time-domain reference block in the first image block above is: In an image containing the above-mentioned first image block, the specific adjacent image block to the above-mentioned first image block is determined, A video processing apparatus according to any one of items 21 to 31, comprising determining the time-domain reference block of the first image block and the motion vector of the time-domain reference block based on the reference image of the specified adjacent image block. [Item 35] The above time-domain reference block is the same image block as the first image block in the reference image of the above specific adjacent image block, or The above time-domain reference block is, in the reference image of the above specific adjacent image block, the same image block as the position of the above specific adjacent image block, or The above time-domain reference block is the image block pointed to by the motion vector of the above specific adjacent image block, as described in item 34 of the video processing apparatus. [Item 36] Determining a specific adjacent image block of the above first image block is: In an image containing the above-mentioned first image block, to obtain a first adjacent image block by a specific scanning sequence, or, In an image containing the above-mentioned first image block, to obtain a first adjacent image block having the same reference frame index value as the above-mentioned first image block by a specific scanning order, or, To obtain a first image block containing motion information by a specific scanning sequence, or, The video processing apparatus according to item 34, comprising obtaining the same first adjacent image block as the default reference image of the first image by a specific scanning order. [Item 37] The video processing device described in item 36, wherein the specified scanning sequence is determined by starting the search from the first block and scanning in one of the following directions: from bottom to top, from left to right, from right to left, or from top to bottom. [Item 38] The above-mentioned specific reference image is an apparatus according to any one of items 21 to 34, which includes at least one of a long-term reference image, a composite frame, or a frame that is not output. [Item 39] The above processor further, A device described in any one of items 21 to 35 for determining that the above reference image is the specified reference image, when it is determined that the above reference image is a frame in which no output is generated, and when it is determined that the above reference image is a composite frame. [Item 40] The above non-specific reference image is a device described in any one of items 21 to 36, including a short-term reference image. [Item 41] A computer-readable storage medium, This is a computer-readable storage medium that stores commands and, when the commands are executed by a computer, causes the computer to perform one of the image processing methods described in item 1 to 20. [Item 42] An encoding device including a video processing device as described in any one of items 21 to 39. [Item 43] A decoding device including a video processing device as described in any one of items 21 to 39.

Claims

1. A video processing method, To obtain the initial motion vector MV of the current image block, which includes a first initial MV pointing to a first reference image and a second initial MV pointing to a second reference image, wherein the first and second reference images are reference images of the current image block in different directions. Under the condition that at least one of the first reference image and the second reference image is a long-term reference image, the motion prediction of the current image block is performed directly based on the first initial MV and the second initial MV without modifying the first initial MV and the second initial MV. A method that includes [a certain feature].

2. The method according to claim 1, wherein the first reference image is a forward-facing frame of the current image block, and the second reference image is a backward-facing frame of the current image block.

3. A video processing method, To obtain the initial motion vector MV of the current image block, which includes a first initial MV pointing to a first reference image and a second initial MV pointing to a second reference image, wherein the first and second reference images are reference images of the current image block in different directions. Under the condition that both the first and second reference images are short-term reference images, the first initial MV and the second initial MV are modified to obtain the modified first MV and the modified second MV, and motion prediction of the current image block is performed based on the modified first MV and the modified second MV, and Under the condition that at least one of the first reference image and the second reference image is a long-term reference image, the motion prediction of the current image block is performed directly based on the first initial MV and the second initial MV without modifying the first initial MV and the second initial MV. Equipped with, Modifying the first initial MV and the second initial MV to obtain the modified first MV and the modified second MV is, A template is generated based on a first reference image block corresponding to the first initial MV and belonging to the first reference image, and a second reference image block corresponding to the second initial MV and belonging to the second reference image. Based on the template, the first initial MV and the second initial MV are modified to obtain the modified first MV and the modified second MV. Methods that include...

4. Modifying the first initial MV and the second initial MV based on the template to obtain the modified first MV and the modified second MV is: N third reference image blocks are used to match each of the templates, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image. M fourth reference image blocks are used to match each of the templates, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image. Based on the matching results, one third initial MV is selected from the N third initial MVs, and one fourth initial MV is selected from the M fourth initial MVs. The method according to claim 3, comprising determining the one third initial MV and the one fourth initial MV as the MV of the current image block.

5. The method according to claim 4, wherein at least some of the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MV.

6. A video processing device, At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing the at least one memory and executing a command executable by the computer, the operations being: Obtaining the initial motion vector MV of the current image block, which includes a first initial MV pointing to a first reference image and a second initial MV pointing to a second reference image, wherein the first reference image and the second reference image are reference images of the current image block in different directions, and Under the condition that at least one of the first reference image and the second reference image is a long-term reference image, the motion prediction of the current image block is performed directly based on the first initial MV and the second initial MV without modifying the first initial MV and the second initial MV. A device including a device.

7. The apparatus according to claim 6, wherein the first reference image is a forward-facing frame of the current image block, and the second reference image is a backward-facing frame of the current image block.

8. A video processing device, At least one memory for storing executable commands on the computer, The system includes at least one processor used alone or jointly to perform the following operations by accessing the at least one memory and executing a command executable by the computer, the operations being: To obtain the initial motion vector MV of the current image block, which includes a first initial MV pointing to a first reference image and a second initial MV pointing to a second reference image, wherein the first and second reference images are reference images of the current image block in different directions, Under the condition that both the first and second reference images are short-term reference images, the first initial MV and the second initial MV are modified to obtain the modified first MV and the modified second MV, and motion prediction of the current image block is performed based on the modified first MV and the modified second MV, and Under the condition that at least one of the first reference image and the second reference image is a long-term reference image, the motion prediction of the current image block is performed directly based on the first initial MV and the second initial MV without modifying the first initial MV and the second initial MV. Includes, The at least one processor is A template is generated based on a first reference image block corresponding to the first initial MV and belonging to the first reference image, and a second reference image block corresponding to the second initial MV and belonging to the second reference image. Based on the template, the first initial MV and the second initial MV are modified to obtain the modified first MV and the modified second MV. The apparatus described, used for the purpose of [the purpose of the invention].

9. The at least one processor is N third reference image blocks are used to match each of the templates, where the N third reference image blocks correspond to N third initial MVs and belong to the first reference image. M fourth reference image blocks are used to match each of the templates, where the M fourth reference image blocks correspond to M fourth initial MVs and belong to the second reference image. Based on the matching results, one third initial MV is selected from the N third initial MVs, and one fourth initial MV is selected from the M fourth initial MVs. The apparatus according to claim 8, comprising determining the one third initial MV and the one fourth initial MV as the MV of the current image block.

10. The apparatus according to claim 9, wherein at least some of the N third initial MVs are obtained by shifting based on the first initial MV, and at least some of the M fourth initial MVs are obtained by shifting based on the second initial MV.

11. A method for generating a bitstream, To obtain the initial motion vector MV of the current image block, which includes a first initial MV pointing to a first reference image and a second initial MV pointing to a second reference image, wherein the first and second reference images are reference images of the current image block in different directions, Under the condition that at least one of the first reference image and the second reference image is a long-term reference image, the motion prediction of the current image block is performed directly based on the first initial MV and the second initial MV without modifying the first initial MV and the second initial MV, and Encoding the current image block based on the motion prediction to generate a bitstream containing motion information for the current image block, A method that includes [a certain feature].

Images