Local motion extension in video coding

By extending the motion model of adjacent blocks to the current block to generate a local warping model, the problem of insufficient motion model in inter prediction in the prior art is solved, and the encoding efficiency and prediction accuracy of video streams are improved.

CN119096548APending Publication Date: 2024-12-06GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280095849.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-05-11
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art is difficult to fully capture relatively large internal motions in inter-frame prediction, resulting in insufficient motion models, which in turn affects the encoding efficiency of video streams.

Method used

By extending the motion model of adjacent blocks to the current block to generate a local warping model, inter prediction is used to ensure motion continuity and reduce computational intensity.

Benefits of technology

Improves the accuracy and encoding efficiency of inter-frame prediction, reduces computational costs, and reduces the number of bits signaled.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096548A_ABST
    Figure CN119096548A_ABST
Patent Text Reader

Abstract

A warping model of a block to be coded is determined by using local motion extension. Motion of a neighboring block is locally extended into the block to determine motion for predicting the block. A motion model for inter prediction of a neighboring block sharing at least a portion of a boundary with a current block to be predicted is determined. The motion model is a first translational motion vector or a first warpage model. A second translational motion vector for inter prediction of the current block is determined. A second warpage model for inter prediction of the current block is determined using the motion model and the second translational motion vector. An inter prediction process is performed on the current block using the second warping model. For example, an inter prediction process may include determining a prediction block for encoding or decoding a current block.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] A digital video stream can represent a video using a sequence of frames or still images. Digital video can be used for a variety of applications, including, for example, video conferencing, high-definition video entertainment, video advertising, or sharing of user-generated videos. A digital video stream can contain a large amount of data and consume a considerable amount of computing or communication resources of a computing device for processing, transmitting, or storing video data. Various methods have been proposed to reduce the amount of data in a video stream, including compression and other encoding techniques.

[0002] A compression technique that uses a reference frame to generate a prediction block that corresponds to the current block to be encoded. The difference between the prediction block and the current block can be encoded instead of the current block value itself to reduce the amount of encoded data. Summary of the invention

[0003] The present disclosure relates generally to encoding and decoding video data using local motion extension, and more particularly to generating a warping model for inter prediction of a current block using local motion extension from blocks neighboring the current block.

[0004] One aspect of the present disclosure is a method comprising: determining a motion model for inter-frame prediction of a neighboring block that shares at least a portion of a boundary with a current block to be predicted; determining a translation motion vector for inter-frame prediction of the current block; generating a warping model for inter-frame prediction of the current block using the motion model and the translation motion vector; and performing an inter-frame prediction process on the current block using the warping model.

[0005] In some implementations, the motion model is one of a first translational motion vector or a first warping model. In some implementations, the warping model used for inter prediction of the current block is a second warping model.

[0006] In some implementations, generating the second warp model includes maintaining motion continuity of pixels across the boundary.

[0007] In some implementations, motion continuity exists when the per-pixel motion vectors of a neighboring block are continuous across the boundary between the neighboring block and the current block.

[0008] In some implementations, generating the second warping model includes establishing a common coordinate system for the motion model and the second translation motion vector, and solving for parameters of the second warping model using the common coordinate system.

[0009] In some implementations, generating the second warping model includes solving for parameters of the second warping model using the motion model and the second translational motion vector.

[0010] In some implementations, performing an inter-frame prediction process on the current block includes: determining a prediction block for the current block using a second warping model and a reference frame; generating a residual including differences between pixels of the current block and pixels of the prediction block; and encoding the residual into an encoded bitstream.

[0011] In some implementations, performing an inter-frame prediction process on the current block includes determining a prediction block for the current block using a second warping model and a reference frame; decoding a residual from an encoded bitstream; and reconstructing the current block by adding pixels of the residual to pixels of the prediction block.

[0012] In some implementations, the second warping model is a six-parameter equation that represents at least one of rotation, scaling, shearing, or stretching.

[0013] In some implementations, the boundary is a vertical line between the adjacent block and the current block, the vertical line having a horizontal coordinate x and multiple values ​​for the vertical coordinate y, and generating the second warping model includes: generating a first equation that makes a first coefficient of the vertical coordinate y of the motion model equal to a first coefficient of the vertical coordinate y of the second warping model; generating a second equation that makes a second coefficient of the vertical coordinate y of the motion model equal to a second coefficient of the vertical coordinate y of the second warping model; generating a third equation that makes the first coefficient and the first translation parameter of the horizontal coordinate x of the motion model equal to the first coefficient and the first translation parameter of the horizontal coordinate x of the second warping model; generating a fourth equation that makes the second coefficient and the second translation parameter of the horizontal coordinate x of the motion model equal to the second coefficient and the second translation parameter of the horizontal coordinate x of the second warping model. shift parameters; using a defined pixel position in the current block to generate a fifth equation, the fifth equation making the predicted horizontal pixel coordinates obtained using the second translation motion vector equal to the predicted horizontal pixel coordinates obtained using the second warping model; using a defined pixel position in the current block to generate a sixth equation, the sixth equation making the predicted vertical pixel coordinates obtained using the second translation motion vector equal to the predicted vertical pixel coordinates obtained using the second warping model; and solving the first equation, the second equation, the third equation, the fourth equation, the fifth equation and the sixth equation to obtain a first coefficient of the vertical coordinate y of the second warping model, a second coefficient of the vertical coordinate y of the second warping model, a first coefficient of the horizontal coordinate x of the second warping model, a first translation parameter of the second warping model, a second coefficient of the horizontal coordinate x of the second warping model and a second translation parameter of the second warping model.

[0014] In some implementations, the motion model is a first warping model, and generating the second warping model includes modifying the first warping model using the second translational motion vector.

[0015] In some implementations, the motion model is a first translational motion vector, and generating the second warping model includes determining a six-parameter warping model using the first translational motion vector and modifying the six-parameter warping model using the second translational motion vector.

[0016] In some implementations, the prediction mode of the current block is a composite prediction mode. The method includes determining a third translation motion vector for inter-frame prediction of the current block, and generating a third warp model for inter-frame prediction of the current block using the motion model and the third translation motion vector. Performing an inter-frame prediction process on the current block includes performing an inter-frame prediction process using the second warp model and the third warp model.

[0017] In some implementations, the prediction mode of the current block is a composite prediction mode having a first portion of the current block predicted using intra prediction and a second portion of the current block predicted using a second translation motion vector. Performing an inter prediction process on the current block includes performing an inter prediction process on only the second portion of the current block using the second warping model.

[0018] One aspect of the present disclosure is a device that performs any of the above methods, either alone or in combination. For example, the device may include a processor configured to perform any of the above methods.

[0019] These and other aspects of the present disclosure are disclosed in the following detailed description of the embodiments, the appended claims and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The description herein refers to the drawings described below, wherein like reference numerals refer to like parts throughout the several views unless otherwise specified.

[0021] Figure 1 is a schematic diagram of a video encoding and decoding system.

[0022] Figure 2 is a block diagram of an example of a computing device that may implement a transmitting station or a receiving station.

[0023] Figure 3 is an illustration of an example of a video stream to be encoded and then decoded.

[0024] Figure 4 is a block diagram of an encoder according to an implementation of the present disclosure.

[0025] Figure 5 is a block diagram of a decoder according to an implementation of the present disclosure.

[0026] Figure 6 is a flow chart of a process for inter-predicting a current block using a warping model generated using local motion expansion.

[0027] Fig. 7A and Figure 7B It is used to explain Figure 6 Diagram of the process. DETAILED DESCRIPTION

[0028] Video streams can be compressed by a variety of techniques to reduce the bandwidth required to send or store the video stream. The video stream can be encoded into a bitstream, which involves compression, and then the bitstream is sent to a decoder, which can decode or decompress the video stream to prepare it for viewing or further processing. Compression of video streams typically exploits the spatial and temporal correlations of video signals through spatial and / or motion compensated prediction. Motion compensated prediction may also be referred to as inter-frame prediction. Inter-frame prediction uses one or more motion vectors to generate a block (also called a prediction block) that is similar to the current block to be encoded using previously encoded and decoded pixels. By encoding the motion vector and the difference between the two blocks (i.e., the residual), a decoder receiving the encoded signal can reconstruct the current block by generating a prediction block and adding the pixels of the prediction block to the decoded residual block.

[0029] Each motion vector used to generate a prediction block in the inter-frame prediction process refers to a frame other than the current frame, i.e., a reference frame. The reference frame can be located before or after the current frame in the sequence of the video stream, and can be a frame reconstructed before being used as a reference frame. As used herein, a forward reference frame is a frame used for forward prediction relative to a sequence, and a backward reference frame is a frame used for backward prediction relative to a sequence. One or more forward reference frames and / or backward reference frames can be used to encode or decode a block. When used to encode or decode a block in the current frame, the effectiveness of the reference frame can be measured based on the resulting signal-to-noise ratio or other rate-distortion metrics.

[0030] The motion model indicates the motion of an object relative to a reference frame. In the previous examples, the motion model represented by the motion vectors indicated the motion of the object relative to the reference frame in two dimensions - traditionally referred to as X-axis and Y-axis motion. An accurate motion model for a block improves codec efficiency, for example, because it minimizes the residuals of the block that needs to be encoded and decoded (e.g., encoded and then decoded). However, many objects captured by a video sequence do not exhibit two-dimensional translational motion, so that translating pixel values ​​from the previous frame (i.e., the reference frame) does not accurately capture the motion for inter-frame prediction of the block. Instead of the motion model of the block being represented by a motion vector (also referred to herein as a translational motion vector), the motion model can be represented by a warping model.

[0031] The motion model in the form of a warp model may apply linear transformations (e.g., warping) such as rotation, shearing, and / or scaling (symmetrically or asymmetrically) to pixels of a reference frame in addition to translational motion for inter-frame prediction. As discussed in more detail below, the warp model may be determined in different ways. For example, the warp model may be determined as a global model that is signaled at the frame level. The warp model may also be a local model determined by fitting a model of motion to blocks near the current block.

[0032] Although the prior art can determine motion models that are effective for inter-frame prediction for many blocks, some blocks may have motion that is difficult to model. For example, the prior art may not adequately capture the internal motion of relatively large blocks. Video sequences encoded at relatively low bit rates may also result in motion models used for inter-frame prediction that do not adequately capture the motion of individual blocks.

[0033] The techniques described herein determine a local warp model by extending the motion model of neighboring blocks to the current block for inter prediction. The details of generating such a local warp model and using the local warp model to perform an inter prediction process on the current block are first described with reference to a system in which the teachings of this article can be implemented.

[0034] Figure 1 1 is a schematic diagram of a video encoding and decoding system 100. The transmitting station 102 may be, for example, Figure 2 The computer with internal hardware configuration is described. However, other suitable implementations of the sending station 102 are possible. For example, the processing of the sending station 102 can be distributed among multiple devices.

[0035] The network 104 can connect the sending station 102 and the receiving station 106 for encoding and decoding the video stream. Specifically, the video stream can be encoded in the sending station 102, and the encoded video stream can be decoded in the receiving station 106. The network 104 can be, for example, the Internet. The network 104 can also be a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a cellular phone network, or any other means for transmitting the video stream from the sending station 102 to the receiving station 106 (in this example).

[0036] In one example, receiving station 106 may be a Figure 2 A computer with an internal hardware configuration is described. However, other suitable implementations of receiving station 106 are possible. For example, the processing of receiving station 106 may be distributed among multiple devices.

[0037] Other implementations of the video encoding and decoding system 100 are possible. For example, an implementation may omit the network 104. In another implementation, the video stream may be encoded and then stored for transmission at a later time to a receiving station 106 or any other device having a non-transitory storage medium or memory. In one implementation, the receiving station 106 receives (e.g., via the network 104, a computer bus, and / or a communication pathway) the encoded video stream and stores the video stream for later decoding. In an example implementation, a real-time transport protocol (RTP) is used to send the encoded video over the network 104. In another implementation, a transport protocol other than RTP may be used, such as a video streaming protocol based on the hypertext transfer protocol (HTTP).

[0038] When used in a video conferencing system, for example, the transmitting station 102 and / or the receiving station 106 may include the capability to both encode and decode video streams as described below. For example, the receiving station 106 may be a video conference participant that receives an encoded video bitstream from a video conference server (e.g., the transmitting station 102) for decoding and viewing, and further encodes its own video bitstream and sends it to the video conference server for decoding and viewing by other participants.

[0039] Figure 2 is a block diagram of an example of a computing device 200 that can implement a sending station or a receiving station. For example, the computing device 200 can implement Figure 1 The computing device 200 may be in the form of a computing system including a plurality of computing devices, or in the form of a computing device (e.g., a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, etc.).

[0040] The CPU 202 in the computing device 200 may be a central processing unit. Alternatively, the CPU 202 may be any other type of device or devices now existing or later developed that are capable of manipulating or processing information. Although the disclosed implementations may be practiced with one processor (e.g., CPU 202) as shown, more than one processor may be used to achieve advantages in speed and efficiency.

[0041] In an implementation, the memory 204 in the computing device 200 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device or non-transitory storage medium may be used as the memory 204. The memory 204 may include code and data 206 accessed by the CPU 202 using the bus 212. The memory 204 may also include an operating system 208 and an application 210, which includes at least one program that permits the CPU 202 to perform the methods described herein. For example, the application 210 may include applications 1 to N, which also include a video codec application that performs the methods described herein. The computing device 200 may also include an auxiliary storage 214, which may be, for example, a memory card used with a mobile computing device. Since a video communication session may contain a considerable amount of information, they may be stored in whole or in part in the auxiliary storage 214 and loaded into the memory 204 as needed for processing.

[0042] The computing device 200 may also include one or more output devices, such as a display 218. In one example, the display 218 may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 218 may be coupled to the CPU 202 via the bus 212. In addition to or in lieu of the display 218, other output devices may be provided that permit a user to program the computing device 200 or otherwise use the computing device. When the output device is or includes a display, the display may be implemented in various ways, including by a liquid crystal display (LCD), a cathode ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0043] The computing device 200 may also include or be in communication with an image sensing device 220, such as a camera or any other image sensing device 220 now existing or later developed that can sense an image, such as an image of a user operating the computing device 200. The image sensing device 220 may be positioned so that it is pointed toward the user operating the computing device 200. In an example, the position and optical axis of the image sensing device 220 may be configured so that the field of view includes an area directly adjacent to the display 218 and from which the display 218 is visible.

[0044] The computing device 200 may also include or communicate with a sound sensing device 222, such as a microphone or any other sound sensing device now existing or later developed that can sense sounds near the computing device 200. The sound sensing device 222 may be positioned so that it is directed toward a user operating the computing device 200, and may be configured to receive sounds, such as voice or other utterances, uttered by the user while the user is operating the computing device 200.

[0045] although Figure 2 The CPU 202 and memory 204 of the computing device 200 are depicted as being integrated into a single unit, but other configurations may be utilized. The operation of the CPU 202 may be distributed across multiple machines (where each machine may have one or more of the processors), which may be directly coupled or coupled across a local area network or other network. The memory 204 may be distributed across multiple machines, such as network-based memory or memory in multiple machines that perform the operations of the computing device 200. Although depicted here as a single bus, the bus 212 of the computing device 200 may be composed of multiple buses. In addition, the auxiliary storage 214 may be directly coupled to other components of the computing device 200 or may be accessed via a network, and may include an integrated unit (such as a memory card) or multiple units (such as multiple memory cards). Therefore, the computing device 200 may be implemented in a wide variety of configurations.

[0046] Figure 3 300 is an illustration of an example of a video stream 300 to be encoded and subsequently decoded. The video stream 300 includes a video sequence 302. At the next level, the video sequence 302 includes a number of adjacent frames 304. Although three frames are depicted as adjacent frames 304, the video sequence 302 may include any number of adjacent frames 304. The adjacent frames 304 may then be further subdivided into individual frames, such as frame 306. At the next level, the frame 306 may be divided into a series of planes or fragments 308. For example, the fragment 308 may be a subgroup of a frame that permits parallel processing. The fragment 308 may also be a subgroup of a frame that can separate video data into individual colors. For example, a frame 306 of color video data may include a luminance plane and two chrominance planes. The fragment 308 may be sampled at different resolutions.

[0047] Regardless of whether the frame 306 is divided into segments 308, the frame 306 can be further subdivided into blocks 310, which can contain data corresponding to, for example, 16×16 pixels in the frame 306. The blocks 310 can also be arranged to include data from one or more segments 308 of pixel data. The blocks 310 can also be any other suitable size, such as 4×4 pixels, 8×8 pixels, 16×8 pixels, 8×16 pixels, 16×16 pixels, or larger. Unless otherwise specified, the terms block and macroblock are used interchangeably herein.

[0048] Figure 4 4 is a block diagram of an encoder 400 according to an implementation of the present disclosure. As described above, the encoder 400 may be implemented in the sending station 102, such as by providing a computer software program stored in a memory (e.g., the memory 204). The computer software program may include machine instructions that, when executed by a processor such as the CPU 202, cause the sending station 102 to generate a signal. Figure 4 The video data is encoded in the manner described. The encoder 400 may also be implemented as dedicated hardware included in, for example, the transmission station 102. The encoder 400 may be a hardware encoder.

[0049] The encoder 400 has the following stages for performing various functions in a forward path (shown by solid connecting lines) to produce an encoded or compressed bitstream 420 using the video stream 300 as input: intra / inter prediction 402, a transform stage 404, a quantization stage 406, and an entropy coding stage 408. The encoder 400 may also include a reconstruction path (shown by a dashed connecting line) to reconstruct frames for encoding future blocks. Figure 4 In FIG. 4 , the encoder 400 has the following stages for performing various functions in the reconstruction path: a dequantization stage 410, an inverse transform stage 412, a reconstruction stage 414, and a loop filter stage 416. Other structural variations of the encoder 400 may be used to encode the video stream 300.

[0050] When the video stream 300 is presented for encoding, the corresponding frame 304, such as frame 306, can be processed in units of blocks. At the intra / inter prediction stage 402, the corresponding block can be encoded using intra-frame prediction (also known as intra-prediction) or inter-frame prediction (also known as inter-prediction). In any case, a prediction block can be formed. In the case of intra-frame prediction, the prediction block can be formed by samples in the current frame that have been previously encoded and reconstructed. In the case of inter-frame prediction, the prediction block can be formed by samples in one or more previously constructed reference frames.

[0051] Next, refer to Figure 4 , the prediction block can be subtracted from the current block at the intra / inter prediction stage 402 to produce a residual block (also referred to as a residual). The transform stage 404 uses a block-based transform to transform the residual into, for example, a transform coefficient in the frequency domain. The quantization stage 406 uses a quantizer value or quantization level to convert the transform coefficient into a discrete quantum value, which is referred to as a quantized transform coefficient. For example, the transform coefficient can be divided by the quantizer value and truncated. The quantized transform coefficient is then entropy encoded by the entropy encoding stage 408. The entropy encoded coefficient is then output to a compressed bit stream 420 along with other information for decoding the block, which may include, for example, the prediction type used, the transform type, the motion vector, and the quantizer value. Various techniques, such as variable length coding (VLC) or arithmetic coding, may be used to format the compressed bit stream 420. The compressed bit stream 420 may also be referred to as an encoded video stream or an encoded video bit stream, and the terms will be used interchangeably herein.

[0052] Figure 4 400 and decoder 500 (described below) use the same reference frame to decode the compressed bitstream 420. The reconstruction path performs functions similar to those that occur during the decoding process discussed in more detail below, including dequantizing the quantized transform coefficients at the dequantization stage 410 and inverse transforming the dequantized transform coefficients at the inverse transform stage 412 to produce a derivative residual block (also referred to as a derivative residual). At the reconstruction stage 414, the prediction block predicted at the intra / inter prediction stage 402 can be added to the derivative residual to create a reconstructed block. A loop filtering stage 416 can be applied to the reconstructed block to reduce distortion, such as blocking artifacts.

[0053] Other variations of the encoder 400 may be used to encode the compressed bitstream 420. For example, for certain blocks or frames, a non-transform based encoder may directly quantize the residual signal without the transform stage 404. In another implementation, the encoder may have the quantization stage 406 and the dequantization stage 410 combined in a common stage.

[0054] Figure 5 2 is a block diagram of a decoder 500 according to an implementation of the present disclosure. The decoder 500 may be implemented in the receiving station 106, for example, by providing a computer software program stored in the memory 204. The computer software program may include machine instructions that, when executed by a processor such as the CPU 202, cause the receiving station 106 to decode the video data in the manner described herein. The decoder 500 can also be implemented in hardware included in, for example, the sending station 102 or the receiving station 106. The decoder 500 may be a hardware decoder.

[0055] Similar to the reconstruction path of the encoder 400 discussed above, in one example, the decoder 500 includes the following stages for performing various functions to produce an output video stream 516 from the compressed bitstream 420: an entropy decoding stage 502, a dequantization stage 504, an inverse transform stage 506, an intra / inter prediction stage 508, a reconstruction stage 510, a loop filtering stage 512, and a post-filtering stage 514. Other structural variations of the decoder 500 may be used to decode the compressed bitstream 420.

[0056] When the compressed bitstream 420 is presented for decoding, the data elements within the compressed bitstream 420 may be decoded by the entropy decoding stage 502 to produce a set of quantized transform coefficients. The dequantization stage 504 dequantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by a quantizer value), and the inverse transform stage 506 inversely transforms the dequantized transform coefficients to produce a derivative residual, which may be the same as the derivative residual created by the inverse transform stage 412 in the encoder 400. Using the header information decoded from the compressed bitstream 420, the decoder 500 may use the intra / inter prediction stage 508 to create a prediction block that is the same as the prediction block created in the encoder 400 (e.g., at the intra / inter prediction stage 402). At the reconstruction stage 510, the prediction block may be added to the derivative residual to create a reconstructed block. A loop filtering stage 512 may be applied to the reconstructed block to reduce blocking artifacts.

[0057] Other filtering can be applied to the reconstructed blocks. In this example, the post-filtering stage 514 can be a deblocking filter applied to the reconstructed blocks to reduce blocking distortion, and the result is output as an output video stream 516. The output video stream 516 can also be referred to as a decoded video stream, and the two terms will be used interchangeably herein. Other variations of the decoder 500 can be used to decode the compressed bit stream 420. For example, the decoder 500 can generate an output video stream 516 without the post-filtering stage 514.

[0058] As briefly described above, a local warp model is described herein that can be used to perform an inter prediction process on a current block. The local warp model is generated by extending the motion model used for inter prediction of neighboring blocks. By extending the motion model, relatively smooth motion (e.g., continuity of motion) between neighboring blocks is produced. In addition, the techniques herein are significantly less computationally intensive than known techniques for generating a warp model. Finally, the signaling cost of the new warp model is relatively low (i.e., only a few bits may be required to indicate the use of local motion extension).

[0059] Figure 6 is a flow chart of a method for process 600 of inter-predicting a current block using a warping model generated using local motion expansion. Fig. 7A and Figure 7B It is used to explain Figure 6 600. Process 600 may be implemented, for example, as a software program executable by a computing device, such as a transmitting station 102 or a receiving station 106. For example, the software program may include machine-readable instructions that may be stored in a memory, such as memory 204, or an auxiliary storage 214, and when executed by a processor, such as CPU 202, may cause a computing device to perform process 600. Process 600 may be implemented in whole or in part in the intra / inter prediction stage 402 of the encoder 400 and / or the intra / inter prediction stage 508 of the decoder 500. Process 600 may be implemented using dedicated hardware or firmware. Multiple processors, memories, or both may be used. Process 600 is first described with reference to the operation of a decoder, such as decoder 500. Thereafter, process 600 is described with reference to the operation of the encoder.

[0060] The process 600 at the decoder assumes that local motion extension is a codec tool available for the current block and is used by the current block. In fact, the decoder can determine whether local motion extension is an available codec tool frame by frame by reading a flag in the header of the frame. Alternatively, the decoder can determine whether local motion extension is an available codec tool frame by frame by reasoning based on the context. For example, local motion extension will not be used with frames that are not encoded using inter-frame prediction (e.g., key frames). In some implementations, whether the tool is available is not determined at the frame level. Whether or not it is determined at the frame level, it can be determined separately to use local motion extension to encode and decode the current block. That is, it can be determined block by block by reading a flag encoded in the header of the block whether local motion extension is used to encode the block (and whether it should be decoded). Alternatively, it can be inferred from the surrounding context whether the block uses local motion extension. For example, if a warping model is used to encode one or more adjacent blocks, the current block can use local motion extension regardless of whether it is generated using the prior art or the local motion extension described herein.

[0061] At operation 602, the process 600 determines a motion model (also referred to as a motion mode) of a block adjacent to a current block to be predicted for decoding. The block is hereinafter referred to as a neighboring block, and is a block that shares at least part of a border with the current block. A neighboring block is a previously encoded neighboring block that has been reconstructed. When encoding and decoding in a raster scan order, the neighboring block may be a block above or to the left of the current block. In other implementations or scan orders, the neighboring block may be located below or to the right of the current block.

[0062] When there is only one neighboring block, the decoder may determine the motion model of the one neighboring block to be the motion model at operation 602. In some implementations where more than one neighboring block is available, it may be explicitly signaled in the bitstream which block is the neighboring block for determining the motion model. Alternatively, the neighboring block may be determined based on the context. For example, the neighboring block may be the block that shares the largest boundary with the current block, may be the first block (or other defined block) adjacent to the current block after decoding, may be a block that shares a common reference frame with the motion vector of the current block (described in more detail below), etc., or some combination thereof.

[0063] The motion model of the neighboring block determined at 602 is a motion model used for decoding the neighboring block, and is obtained from header information of the neighboring block, for example, during decoding of the current block.

[0064] Fig. 7A and Figure 7BA portion of a frame 700 including a current block 702 and a neighboring block 704 is shown. In this example, the current block 702 and the neighboring block 704 have the same size (e.g., the same pixel size), but in other implementations described below, the neighboring block may be smaller or larger than the current block. The motion model of the neighboring block may be a translational motion vector or a warping model, also referred to as a first translational motion vector or a first warping model to distinguish it from other translational motion vectors or warping models discussed herein. That is, for clarity, the terms first, second, third, etc. are used herein to distinguish one element from another, and the presence of such terms does not indicate an order or sequence of elements, such as the order in which they are encoded, decoded, determined, identified, used, etc.

[0065] like Fig. 7A As shown by the arrows in , the motion model of the neighboring block 704 is a warping model. The warping model may be referred to as a six-parameter warping model and may be represented by the following six-parameter equation.

[0066]

[0067] parameter , , and Represents various transformations such as rotation, scaling, shearing, stretching, reflection, etc. They may also be referred to as coefficients in this article. and represents the vertical and horizontal translation offset. According to equation (1), the pixel position in the current block is The pixel value of the prediction block at position is taken from the pixel position in the reference frame When the motion model is a translation motion vector , the following equation applies.

[0068]

[0069]

[0070] Process 600 determines a motion vector for a current block at 604. At 604, the motion vector is a translational motion vector for inter-frame prediction of the current block. The motion vector may be a motion vector determined and encoded into an encoded bitstream that is subsequently decoded by a decoder. In some implementations, the motion vector is a motion vector reused from another block. In some implementations, the motion vector is differentially encoded similarly to the block. That is, a reference motion vector is determined, and the difference between the determined motion vector and the reference motion vector is encoded for later use by the decoder to reconstruct the determined motion vector. The decoder may determine the motion vector based on header information of the block included in the encoded bitstream. The motion vector for inter-frame prediction of the current block may be referred to as a second translational motion vector. Figure 7B , the motion vector of the current block 702 is a translation motion vector 706 .

[0071] At 606, a warp model is generated for the current block. The warp model may be a second warp model for inter prediction of the current block generated using the motion model for inter prediction of the neighboring block and the (e.g., second translation) motion vector for inter prediction of the current block. The warp model of the current block may be constructed to extend the motion of the neighboring block into the current block. To make the transition smooth, it is desirable to apply continuity constraints so that the per-pixel motion vectors (such as Figure 7B The per-pixel motion vectors 708 in ( ) are continuous across a boundary (edge, boundary, line, etc.) between two blocks. More specifically, if the motion models of two adjacent blocks do not line up completely at the boundary between them (e.g., the motion models are simple translations of different amounts), then the pixels on either side of the boundary can be predicted from very different places and can have very different values. In some cases, such as when the boundary matches an edge in an image, this is desirable, and otherwise the boundary pixels may need to be adjusted (e.g., using a deblocking filter). Using the continuity constraint, a motion model can be selected that causes the newly predicted block to blend smoothly into the predictions of the adjacent blocks, thereby reducing or eliminating such adjustments.

[0072] Because in this example the neighboring block 704 is to the left of the current block 702, the two warping models should make the motion vectors of the vertical lines of pixels just to the left of the boundary between blocks the same. Therefore, each point along the edge satisfies the following equation.

[0073]

[0074] In equation (3), the parameter , , , , and represents the first warping model (i.e., the warping model of the adjacent block), and the parameter and Denotes the second warping model (i.e., the warping model for the current block). If a continuity constraint is applied along a column of pixels with horizontal coordinate x (e.g., the column immediately to the left of the current block), then equation (3) is true for this single value of x but for many values ​​of y (one for each pixel along the edge). This means that the coefficients of y on both sides of the equation are the same, as shown in the equation below.

[0075]

[0076]

[0077] At the same time, the coefficients of x satisfy the following equations. These equations make the relationship between the x coefficients of the motion model and the translation parameters equal to the relationship between the x coefficients of the second warping model and the translation parameters.

[0078]

[0079]

[0080] Equations (4), (5), (6) and (7) are four equations with six unknowns (due to and is known, and is therefore equivalent to two equations with four unknowns), and may be referred to as the first equation, the second equation, the third equation, and the fourth equation, respectively. To solve for the parameters of the second warping model, the translation motion vector determined at 604 may be used The motion vectors are specified at additional points in the current block, resulting in six equations in six variables. More specifically, when translational motion vectors are the only motion model for a block, the translational motion vectors apply to all pixels in the block, but their origin (mathematically) can be specified as a single point within the block, referred to herein as In this example, the pixel position is the center of the block. In the coordinate system of the current block, the upper left pixel is at (0,0) and the lower right pixel is at , where w represents the width of the block in pixels and h represents the height of the block in pixels, with the center of the block located at The block can have a size of 4x4 pixels, 8x8 pixels, 8x16 pixels, etc. A common coordinate system needs to be established for the motion models and translational motion vectors of adjacent blocks. In one example, the common coordinate system can be defined and where x is the pixel coordinate of the vertical line for which continuity is to be maintained (e.g., relative to the left edge of the frame), and y is the vertical coordinate of the corresponding pixel along that line. In the example described above where the continuity constraint is applied along the column immediately to the left of the current block, the position of the column is , so that the horizontal distance from this line to the center of the block is exactly .

[0081] Two additional equations (e.g., the fifth and sixth equations) using the translational motion vector of the current block are included below as equations (8) and (9). As can be seen below, these equations are obtained by taking, for example, the pixel position obtained from the motion vector according to equations (2a) and (2b) The predicted pixel position of is equal to the predicted pixel position obtained by the second warping model according to equation (3).

[0082]

[0083]

[0084] According to equation (4), Substituting into equation (8) The following equation is obtained.

[0085]

[0086] Solve equation (10), where The following equation is obtained.

[0087]

[0088] Rearranging equation (11) yields the following equation.

[0089]

[0090] Substituting equation (6) into the equation below yields the following equation.

[0091]

[0092] For the only unknown variable Solving equation (13), we obtain the following equation.

[0093]

[0094] As mentioned above, and The value of and The value of depends on the coordinate system used. For relatively simple implementations, a common coordinate system can be used, where and The coordinate values ​​are given relative to the left edge of the frame, and and The coordinate values ​​of are given relative to the top edge of the frame (e.g., starting at 0). At the center of the block on a common coordinate system, Half the width of the current block , which is generally a power of 2. This simplifies computations, especially division. That is, the use of this convention allows division to be performed using a binary right shift operation, which is much simpler than division on a digital computer.

[0095] A similar calculation can be used to determine from equation (9) That is, according to equation (5), substitute D into equation (9) The following equation is obtained.

[0096]

[0097] Solve equation (15), where The following equation is obtained.

[0098]

[0099] Rearranging equation (16) yields the following equation.

[0100]

[0101] Substituting equation (7) into the equation below yields the following equation.

[0102]

[0103] For the only unknown variable, that is , solve equation (18), that is, get the following equation.

[0104]

[0105] Afterwards, equations (6) and (7) can be rearranged to solve and , as shown below.

[0106]

[0107]

[0108] The above is one technique for solving the warping model parameters for the current block, which generates the second warping model. Other cases are possible, and the order of calculations varies. Again, for ease of understanding, the solution is explained using the above equations. The calculations may be different from the one described, as long as the solution for the parameters of the second warping model is obtained from the motion models of the neighboring blocks and the translational motion vector of the current block.

[0109] For example, generating a warp model for the current block at 606 may include using a (e.g., first) translational motion vector and a second translational motion vector of a neighboring block to generate the warp model. That is, another implementation of generating a warp model at 606 may include a case where the motion model of a neighboring block (such as neighboring block 704) is a translational motion vector. To generate a warp model for the current block (e.g., current block 702), the translational motion vector of the neighboring block may be considered as a special case of a warp model where the non-translational portion of the model is the identity matrix. That is, the solution for the parameters of the second warp model may assume that the non-translational parameters of the warp model are , and the translation parameters of the adjacent blocks correspond to the translation motion vectors, i.e. and , where the motion vector of the adjacent block is .

[0110] After generating the warp model for the current block at 606, the (second) warp model may be used to perform an inter prediction process on the current block at 608. For example, the second warp model may be used to generate a prediction block for the current block at a prediction stage of a decoder, such as the intra / inter prediction stage 508 of the decoder 500. Thereafter, the prediction block is added to the residual of the current block to reconstruct the current block, such as at a reconstruction stage 510 of the decoder 500. In some examples, the residual is decoded from the encoded bitstream by entropy decoding, dequantization, and inverse transformation as described with respect to the decoder 500.

[0111] The above process 600 may be repeated for any block encoded using local motion extension. At the decoder, the reconstructed block, the reconstructed frame, or both may be filtered using in-loop filters, post-loop filters, or both.

[0112] As briefly mentioned above, the method or process 600 for inter-predicting a current block using a warp model generated using local motion extension can be performed at an encoder or a decoder. The implementation of the process 600 at the decoder is described above. Described next is the implementation of the process 600 for encoding the current block at the encoder.

[0113] At 602, a motion model of a neighboring block is determined. The neighboring block shares at least a portion of a border with the current block. The motion model is a (e.g., first) translation motion vector or a (e.g., first) warping model for inter-frame prediction of the neighboring block. The neighboring block is a previously encoded neighboring block that has been reconstructed. When encoding and decoding in raster scan order, the neighboring block may be a block above or to the left of the current block. In other implementations or scan orders, the neighboring block may be below or to the right of the current block. This implementation first discusses the case where only one neighboring block has a motion model (i.e., the other neighboring blocks are intra-frame predicted). The implementation described below addresses the case where more than one neighboring block is available.

[0114] The process 600 determines a motion vector for the current block at 604. At 604, the motion vector is a translational motion vector for inter prediction of the current block. The motion vector may be a motion vector determined by conventional techniques (e.g., in the intra / inter prediction stage 402 of the encoder 400), such as performing a motion search within one or more reference frames to find the best matching reference block among the reference blocks found within the corresponding reference frames. This may be implemented in a rate-distortion loop at the encoder. If the best prediction mode for the current block is instead an intra prediction mode, the process 600 may end for the current block. In some implementations, the search for a prediction mode for the current block may determine the prediction mode to be a composite prediction mode, e.g., a mode that uses at least two prediction modes to generate a prediction block.

[0115] At 606, a warping model is generated for the current block. The warping model may be a second warping model for inter prediction of the current block generated using the motion model for inter prediction of the neighboring block and the (e.g., second translation) motion vector for inter prediction of the current block. According to the above example, the warping model for the current block is generated using the warping model for the neighboring block and the translation motion vector (e.g., second translation) of the current block. Fig. 7A The warping model of the adjacent block 704 shown by the arrows and Figure 7B The warping model of the current block is generated by using the motion vector 706 of the current block 702 in the image. Another example described above is an implementation in which the motion model of the neighboring block (such as the neighboring block 704) is a translation motion vector.

[0116] After generating the warp model for the current block at 606, the encoder can use the warp model to perform an inter prediction process on the current block at 608. For example, the warp model can be used to generate a prediction block for the current block at a prediction stage of the encoder (such as the intra / inter prediction stage 402 of the encoder 400). Thereafter, the prediction block is subtracted from the current block to generate a residual for encoding into an encoded bitstream, such as by transform, quantization, and entropy coding and decoding. The warp model and the resulting prediction block can also be used in a rate-distortion loop to determine whether local motion extension as described herein provides a better predictor than the inter prediction mode.

[0117] Furthermore, and although the above description indicates that if the best prediction mode is an intra prediction mode at 604, then process 600 may end, this is not required. The best inter prediction mode for the current block determined at 604 (e.g., in a rate-distortion loop) may be used with the motion model determined at 602 to generate a warp model for the current block at 606. A better prediction mode of the intra prediction mode and the local motion extension mode (e.g., a prediction mode of the prediction block that results in lower bit cost, higher quality, or both for encoding the current block through the intra prediction mode or the warp model determined at 608) may then be selected as the prediction mode for the current block.

[0118] However, if the encoder determines that the warp model is to be used as a predictor for the current block after process 600 ends, the decoder must also know to generate the warp model. In some implementations, the encoder can indicate whether position motion extension is an available codec tool (also referred to herein as a prediction mode) on a frame-level basis. For example, the encoder can send a flag in the header of a frame where one or more blocks are encoded using a warp model generated using local motion extension. Alternatively, the encoder can omit the frame-level signal. Regardless of whether the encoder sends a frame-level signal or a flag, a per-block signal can be sent to indicate which blocks are encoded using a warp model generated using local motion extension. The signal can be a flag encoded into the header of a block (such as the current block).

[0119] In some implementations, it may not be possible to signal on a per-block basis. For example, an encoder may follow a set of rules that a decoder conforming to the bitstream may also use to infer whether a current block uses local motion extension based on the surrounding context. An example of a surrounding context may be the number of neighboring blocks that are encoded and decoded using a warp model. If a defined number of neighboring blocks use a warp model, the current block may also use a warp model. Another example of a surrounding context may be the number of neighboring blocks that are encoded and decoded using translation motion vectors compared to an intra-prediction mode. If a defined number of neighboring blocks use different translation motion vectors, the current block may use a warp model. Another example of a surrounding context may be the number of neighboring blocks that use the same reference frame for motion prediction as the current block. If a defined number of neighboring blocks use the same reference frame, the current block may use a warp model. Some combination of two or more of these contexts, or other contexts, may be used.

[0120] Any known technique can be used to encode the motion vector of the current block into the encoded bit stream for use by the decoder. For example, the encoder can signal that the motion vector is a motion vector reused from another block. In some implementations, the encoder can differentially encode the motion vector by determining a reference motion vector and encoding the difference between the determined motion vector and the reference motion vector. The encoder can encode a signal (flag, bit, etc.) about how to encode the motion vector, what motion vector to use, what reference motion vector to use, or some combination thereof into the header. In an example implementation, a dynamic reference list can be maintained that prioritizes adjacent (or neighboring) blocks based on defined criteria, and the encoder can signal which block the reference motion vector can be obtained from. In this latter implementation, the decoder uses the same criteria to generate a similar dynamic reference list and decodes the signal to determine the reference motion vector.

[0121] If there is only one neighboring block, or only one neighboring block is encoded using inter-frame prediction (e.g., using motion vectors or a warp model), the encoder does not need to signal which neighboring block is used to generate the warp model for the current block. If there are multiple neighboring blocks, identifying which neighboring block is selected can be explicitly signaled within the bitstream. In one implementation, each neighboring block that can be used for local motion expansion can be numbered, and the encoder can signal an index to specify which neighboring block to use. In another implementation, the dynamic reference list described above and used to identify reference motion vectors for neighboring blocks, current blocks, or both can be used to identify neighboring blocks for local motion expansion. Even if there is more than one neighboring block, fast signaling may not be required. As described above, the decoder can determine the neighboring blocks based on the context so that the encoder does not need to send an identifier.

[0122] The above description applies to extending the motion of only one neighboring block into the current block, but the teachings herein are not limited thereto. More complex motion can be captured by generating corresponding prediction blocks using two or more warping models developed from corresponding neighboring blocks and the motion vector of the current block, and then combining the prediction blocks into a single prediction block. The combination can be performed using an average, a weighted average (e.g., based on the pixel position relative to an edge), etc. This variation generates each warping model using the same motion vector of the current block.

[0123] The encoder may determine that the (e.g., initial) prediction mode of the current block is a composite prediction mode. In an implementation of the composite prediction mode (i.e., bidirectional prediction mode), two motion vectors may be determined for the current block. The motion models of the neighboring blocks and each motion vector of the current block may be used to determine a corresponding warping model. The prediction blocks generated from each warping model may be combined using an average, a weighted average, etc. to obtain a single prediction block for the current block.

[0124] In another implementation of the composite prediction mode, the encoder may determine that different parts of the current block should be predicted in different ways. For example, one part of the current block may be associated with a translational motion vector, while another part is associated with an intra-frame prediction mode. The generation and use of the warp model may be limited to the pixels of the portion of the current block associated with the translational motion vector. In another example, one part of the current block may be associated with a translational motion vector, while another part of the current block is associated with a different translational motion vector. In this example, each of the translational motion vectors may be used to generate a corresponding warp model, and the prediction blocks may be combined as described above. Alternatively, the motion model of the neighboring blocks adjacent to the pixels of each part of the current frame and the associated translational motion vector may be used to generate a warp model for the portion. The neighboring blocks may be a single neighboring block or different neighboring blocks. The prediction block may be limited to the size of the corresponding portion and connected along the same boundaries as the portions to form a single prediction block of the current block.

[0125] In the examples herein, a warping model is generated, derived, or otherwise created where a neighboring block is located to the left of a current block to be encoded or decoded. The continuity condition is maintained so that a vertical line of pixels just to the left of the boundary between blocks produces the same motion vector. This condition changes if the neighboring block is located to the right of the current block, above the current block, or below the current block. For example, if the neighboring block is above the current block, the equations can be similarly derived taking into account the case where points (pixels) along an edge or boundary between blocks have different x values ​​but all have the same y value.

[0126] In the examples herein, the current block and the neighboring block have the same size. Implementations taught herein can also generate a warp model for a current block of a different block size. If the neighboring block has a larger dimension (e.g., length or height in pixels) along the border than the current block, the generation of the warp model described above at 606 is unchanged. If the neighboring block has a smaller dimension (e.g., length or height in pixels) along the border than the current block, the generation of the warp model described above at 606 is limited to those pixels located on a shared (common, coincident, etc.) border.

[0127] The warping model of the neighboring block may be used to predict the warping model of the current block. One way to achieve this is to use the warping model of the neighboring block to predict the motion vector at a particular point (or more than one point) of the current block. The motion vector of the current block may be signaled as a corresponding offset from the predicted motion vector. If the motion vector of the current block matches the predicted motion vector of the warping model of the neighboring block, the warping model of the current block determined by the techniques described herein will remain unchanged from the warping model of the neighboring block. In this case, it may be desirable to directly signal that the warping model of the neighboring block should be reused without adjustment, rather than performing the full process 600, as this will provide the same result for the current block.

[0128] The techniques herein describe the use of local motion expansion to generate local warp models. The local warp models can be used for inter-frame prediction of blocks. These techniques represent an improvement over the use of least squares to fit models to motion vectors of nearby blocks. First, these techniques use the continuity of motion between adjacent blocks. This reduces the sharp edges between blocks, which can reduce the need for deblocking filtering and / or reduce the number of bits required to signal the deblocking filtering. In addition, these techniques are significantly less computationally intensive than full least squares solutions. Only a relatively small number of basic arithmetic operations may be required. These techniques do not simply copy the warp models between blocks. Instead, these techniques allow the warp models of adjacent blocks to be adjusted in a relatively low-cost signaling manner. There is greater flexibility in modeling complex motion results.

[0129] For ease of explanation, the process herein is depicted and described as a series of steps or operations. However, steps or operations according to the present disclosure may occur in various orders and / or concurrently. In addition, other steps or operations not presented and described herein may be used. In addition, all steps or operations of the description may not be required to implement the method according to the disclosed subject matter.

[0130] The encoding and decoding aspects described above illustrate some examples of encoding and decoding techniques. However, it should be understood that when those terms are used in the claims, encoding and decoding may mean compressing data, decompressing data, transforming data, or any other processing or change of data.

[0131] The word "example" is used herein to mean serving as an example, instance or illustration. Any aspect or design described herein as an "example" is not necessarily to be interpreted as being preferred or advantageous over other aspects or designs. Rather, the use of the word "example" is intended to present concepts in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clearly visible from the context, "X includes A or B" is intended to mean any one of the natural inclusive arrangements. That is, if X includes A; X includes B; or X includes both A and B, "X includes A or B" is satisfied under any of the above examples. In addition, the articles "one" and "a kind of" as used in this application and the appended claims should generally be interpreted as meaning "one or more", unless otherwise specified or clearly visible from the context to refer to the singular form. In addition, the use of the term "implementation" or "an implementation" throughout the text is not intended to mean the same embodiment or implementation, unless so described.

[0132] The implementation of the sending station 102 and / or the receiving station 106 (and the algorithms, methods, instructions, etc. stored thereon and / or executed by it (including by the encoder 400 and the decoder 500)) can be implemented in hardware, software, or any combination thereof. The hardware may include, for example, a computer, an intellectual property (IP) core, an application specific integrated circuit (ASIC), a programmable logic array, an optical processor, a programmable logic controller, a microcode, a microcontroller, a server, a microprocessor, a digital signal processor, or any other suitable circuit. In the claims, the term "processor" should be understood to cover any of the aforementioned hardware individually or in combination. The terms "signal" and "data" are used interchangeably. In addition, the parts of the sending station 102 and the receiving station 106 do not necessarily have to be implemented in the same manner.

[0133] Furthermore, in one aspect, for example, the sending station 102 or the receiving station 106 can be implemented using a general purpose computer or general purpose processor with a computer program that, when executed, performs any of the corresponding methods, algorithms and / or instructions described herein. Additionally or alternatively, for example, a special purpose computer / processor can be utilized that includes other hardware for performing any of the methods, algorithms or instructions described herein.

[0134] The sending station 102 and the receiving station 106 can be implemented, for example, on a computer in a video conference system. Alternatively, the sending station 102 can be implemented on a server, and the receiving station 106 can be implemented on a device (such as a handheld communication device) separated from the server. In this example, the sending station 102 can use an encoder 400 to encode the content into an encoded video signal and send the encoded video signal to the communication device. Then, the communication device can use a decoder 500 to decode the encoded video signal. Alternatively, the communication device can decode the content (for example, the content not sent by the sending station 102) stored locally on the communication device. Other suitable transmission and reception implementations are available. For example, the receiving station 106 can be a generally fixed personal computer rather than a portable communication device, and / or the device including the encoder 400 can also include a decoder 500.

[0135] In addition, all or part of the implementation of the present disclosure may take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium may be any device that may, for example, tangibly contain, store, communicate, or transmit a program for use by or in conjunction with any processor. The medium may be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable media are also available.

[0136] The above embodiments, implementations and aspects have been described to allow easy understanding of the present invention and do not limit the present invention. On the contrary, the present invention is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, and the scope should be given the broadest interpretation to cover all such modifications and equivalent structures permitted under the law.

Claims

1. A method, include: determining a motion model for inter prediction of a neighboring block that shares at least a portion of a boundary with a current block to be predicted; determining a translation motion vector for inter-frame prediction of the current block; generating a warping model for the inter prediction of the current block using the motion model and the translational motion vector; as well as An inter-frame prediction process is performed on the current block using the warping model.

2. The method according to claim 1, in, The motion model is one of a first translational motion vector or a first warping model.

3. The method according to claim 1, in, Generating the warpage model comprises: Motion continuity of pixels across the boundary is maintained.

4. The method according to claim 3, in, The motion continuity exists when the per-pixel motion vectors of the neighboring blocks are continuous across the boundary between the neighboring blocks and the current block.

5. The method according to any one of claims 1 to 4, in, Generating the warpage model comprises: establishing a common coordinate system for the motion model and the translation motion vector; and The common coordinate system is used to solve for parameters of the warping model.

6. The method according to any one of claims 1 to 4, in, Generating the warpage model comprises: The parameters of the warping model are solved using the motion model and the second translational motion vector.

7. The method according to any one of claims 1 to 6, in, The inter-frame prediction process for the current block includes: Determine a prediction block for the current block using the warping model and a reference frame; generating a residual block including differences between pixels of the current block and pixels of the prediction block; and The residual is encoded into a coded bitstream.

8. The method according to any one of claims 1 to 6, in, The inter-frame prediction process for the current block includes: Determine a prediction block for the current block using the warping model and a reference frame; decoding a residual block from the encoded bitstream; and The current block is reconstructed by adding pixels of the residual block to pixels of the prediction block.

9. The method according to any one of claims 1 to 8, in, The warping model includes a six-parameter equation representing at least one of rotation, scaling, shear, or stretching.

10. The method according to any one of claims 1 to 8, in, The boundary includes a vertical line between the adjacent block and the current block, the vertical line having a horizontal coordinate and for the vertical coordinate The plurality of values ​​of and generating the warpage model comprises: Generate a first equation that makes a first coefficient of the vertical coordinate y of the motion model equal to the vertical coordinate y of the warping model The first coefficient of Generate a second equation that makes the second coefficient of the vertical coordinate y of the motion model equal to the vertical coordinate y of the warping model The second coefficient of Generate a third equation, the third equation makes the horizontal coordinates of the motion model The first coefficient and the first translation parameter are equal to the horizontal coordinate of the warping model The first coefficient and the first translation parameter of ; generating a fourth equation that makes the second coefficient and the second translation parameter of the horizontal coordinate x of the motion model equal to the second coefficient and the second translation parameter of the horizontal coordinate x of the warping model; generating a fifth equation using a defined pixel position in the current block, the fifth equation making a predicted horizontal pixel coordinate obtained using the translational motion vector equal to a predicted horizontal pixel coordinate obtained using the warping model; generating a sixth equation using the defined pixel position in the current block, the sixth equation making a predicted vertical pixel coordinate obtained using the translational motion vector equal to a predicted vertical pixel coordinate obtained using the warping model; and Solving the first equation, the second equation, the third equation, the fourth equation, the fifth equation and the sixth equation to obtain the vertical coordinate of the warping model The first coefficient of the warping model, the vertical coordinate of the warping model The second coefficient of the warping model, the horizontal coordinate of the warping model The first coefficient of the warping model, the first translation parameter of the warping model, the horizontal coordinate of the warping model The second coefficient and the second translation parameter of the warping model.

11. The method according to any one of claims 1 to 10, in, The motion model includes a first warping model, and generating the warping model includes modifying the first warping model using the translational motion vector.

12. The method according to any one of claims 1 to 10, in, The motion model includes a first translational motion vector, and generating the warping model includes: determining a six-parameter warping model using the first translational motion vector; and The six parameter warping model is modified using the translational motion vector.

13. The method according to any one of claims 1 to 12, in: The prediction mode of the current block includes a composite prediction mode; The method comprises: determining another translational motion vector for the inter-frame prediction of the current block; and generating another warping model for the inter prediction of the current block using the motion model and the another translational motion vector; and Performing the inter prediction process on the current block includes performing the inter prediction process using the warp model and the another warp model.

14. The method according to any one of claims 1 to 12, in: The prediction mode of the current block includes a composite prediction mode having a first portion of the current block predicted using intra prediction and a second portion of the current block predicted using the translational motion vector; and Performing the inter prediction process on the current block includes using the warping model to perform the inter prediction process on only the second portion of the current block.

15. A device, include: A processor configured to execute the method according to any one of claims 1 to 14.