Method and apparatus for selecting and grouping warped samples - Patents.com

Abstract

A system and method are provided for encoding and decoding using a warp motion mode. The method includes receiving a bitstream including a coding picture, obtaining a plurality of motion vectors pointing to positions in neighboring blocks adjacent to a current block in the coding picture, the plurality of motion vectors including at least one temporal motion vector, obtaining motion samples for a warp model based on the plurality of motion vectors, and decoding the current block by using the warp model and the motion samples including the at least one temporal motion vector for prediction. Some embodiments may include a method for deriving the warp model using temporal motion information, and may further provide warp sample selection and grouping. Some embodiments may provide an improvement of the local warp motion mode.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to and claims the benefit of U.S. Provisional Application No. 63 / 392,381, filed July 26, 2022, and U.S. Patent Application No. 17 / 980,331, filed November 3, 2022, the entire contents of which are incorporated by reference.

[0002] [Technical field] TECHNICAL FIELD Embodiments of the present disclosure relate to advanced image and video coding techniques, and more particularly to improvements in local warp motion modes. [Background technology]

[0003] AV1 (AOMedia Video 1) is an open video coding format designed for video transmission over the Internet. It was developed as a successor to VP9 by AOMedia (Alliance for Open Media), a consortium founded in 2015 that includes the semiconductor industry, video-on-demand providers, video content producers, software developers, and web browser vendors. Many of the components of the AV1 project were derived from previous research efforts by members of the consortium. Individual contributors started experimental technology platforms several years ago. Xiph / Mozilla's Daala already released its code in 2010, Google's experimental VP9 evolution project V10 was announced on September 12, 2014, and Cisco's Thor on August 11, 2015. Built on the VP9 code base, AV1 incorporates further technologies, some of which were developed in these experimental formats. The first version 0.1.0 of the AV1 reference codec was released on April 7, 2016. The consortium announced the release of the AV1 Bitstream Specification on March 28, 2018, along with encoders and decoders based on the reference software. Validated version 1.0.0 of the specification was released on June 25, 2018. Validated version 1.0.0 with errata 1 of the specification was released on January 8, 2019. The AV1 Bitstream Specification includes the reference video codec.

[0004] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (version 1), 2014 (version 2), 2015 (version 3) and 2016 (version 4). They also investigated the potential need for standardization of future video coding technologies that may perform significantly better than HEVC in compression capabilities. In October 2017, they issued a Joint Call for Proposals on Video Compression with Capability beyond HEVC (CfP). By February 15, 2018, a total of 22 CfP responses had been submitted for standard dynamic range (SDR), 12 for high dynamic range (HDR), and 12 for the 360° video category. In April 2018, all received CfP responses were evaluated at the 122 MPEG / 10th JVET (Joint Video Exploration Team or Joint Video Expert Team) meeting. As a result of this meeting, the JVET formally started standardization of the next generation of video coding beyond HEVC. The new standard is called VVC (Versatile Video Coding). Summary of the Invention

[0005] According to an embodiment of the present disclosure, a method performed by a decoder is provided, the method including the steps of receiving a bitstream including a coding picture, obtaining a plurality of motion vectors pointing to positions in neighboring blocks adjacent to a current block in the coding picture, the plurality of motion vectors including at least one temporal motion vector, obtaining motion samples for a warping model based on the plurality of motion vectors, and decoding the current block by using the warping model and the motion samples including the at least one temporal motion vector for prediction.

[0006] According to one or more embodiments of the present disclosure, the at least one temporal motion vector includes a temporal motion vector at a position of a spatial neighboring block of the current block that has not yet been constructed by the decoder.

[0007] According to one or more embodiments of the present disclosure, the at least one temporal motion vector includes a temporal motion vector at a position of a spatial neighboring block of the current block previously constructed by the decoder.

[0008] According to one or more embodiments of the present disclosure, the method further includes applying a motion offset to the at least one temporal motion vector.

[0009] According to one or more embodiments of the present disclosure, the method further includes obtaining the warp model using a regression / least squared error method on a plurality of motion vectors, including at least one temporal motion vector.

[0010] According to one or more embodiments of the present disclosure, the multiple motion vectors include at least one temporal motion vector and at least one spatial motion vector.

[0011] According to one or more embodiments of the present disclosure, the multiple motion vectors include at least one temporal motion vector and no spatial motion vector.

[0012] According to one or more embodiments of the present disclosure, the decoding step further includes the steps of building a list of neighboring blocks of the plurality of groups and selecting a group from the list from among the neighboring blocks of the plurality of groups, and the obtaining a plurality of motion vectors includes the step of obtaining a plurality of motion vectors from the neighboring blocks of the selected group.

[0013] According to one or more embodiments of the present disclosure, the selecting step includes selecting a group from the list based on an index signaled in the bitstream, the index indicating which group from the list should be used to obtain the warp model.

[0014] According to one or more embodiments of the present disclosure, the decoding step includes decoding the current block by performing warped motion compensation.

[0015] According to an embodiment of the present disclosure, a system is provided, the system including at least one memory configured to store program code, and at least one processor configured to receive a bitstream including a coding picture, access the computer program code, and operate as instructed by the computer program code, the computer program code including: a motion vector acquisition code configured to cause the at least one processor to obtain a plurality of motion vectors pointing to positions in neighboring blocks adjacent to a current block in the coding picture, the plurality of motion vectors including at least one temporal motion vector, a motion sample acquisition code configured to cause the at least one processor to obtain motion samples for a warp model based on the plurality of motion vectors, and a decoding code configured to cause the at least one processor to decode the current block by using the warp model and the motion samples including the at least one temporal motion vector for prediction.

[0016] According to one or more embodiments of the present disclosure, the at least one temporal motion vector includes a temporal motion vector at a position of a spatial neighboring block of the current block that has not yet been constructed by the at least one processor.

[0017] According to one or more embodiments of the present disclosure, the at least one temporal motion vector includes a temporal motion vector at a position of a spatial neighboring block of the current block previously constructed by the at least one processor.

[0018] According to one or more embodiments of the present disclosure, the computer program code further includes motion offset code configured to cause the at least one processor to apply a motion offset to the at least one temporal motion vector.

[0019] According to one or more embodiments of the present disclosure, the computer program code further includes warp model acquisition code configured to cause the at least one processor to acquire a warp model by using a regression / least squared error method on a plurality of motion vectors including at least one temporal motion vector.

[0020] According to one or more embodiments of the present disclosure, the multiple motion vectors include at least one temporal motion vector and at least one spatial motion vector.

[0021] According to one or more embodiments of the present disclosure, the multiple motion vectors include at least one temporal motion vector and no spatial motion vector.

[0022] According to one or more embodiments of the present disclosure, the computer program code further includes a list construction code configured to cause the at least one processor to construct a list of neighboring blocks of the plurality of groups; and a selection code configured to cause the at least one processor to select a group from the list among the neighboring blocks of the plurality of groups, and the motion vector acquisition code configured to cause the at least one processor to obtain a plurality of motion vectors from the neighboring blocks of the selected group.

[0023] According to one or more embodiments of the present disclosure, the selection code is configured to cause the at least one processor to select a group from the list based on an index signaled in the bitstream, the index indicating which group from the list should be used to obtain the warp model.

[0024] According to an embodiment of the present disclosure, a non-transitory computer-readable medium storing computer code is provided, which, when executed by at least one processor, causes the at least one processor to implement a decoder that performs the following steps: receiving a bitstream including a coding picture, obtaining a plurality of motion vectors pointing to positions in neighboring blocks adjacent to a current block in the coding picture, the plurality of motion vectors including at least one temporal motion vector, obtaining motion samples for a warping model based on the plurality of motion vectors, and decoding the current block using the warping model and the motion samples including the at least one temporal motion vector for prediction. [Brief description of the drawings]

[0025] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Diagram 2]FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Diagram 3] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 5A] FIG. 1 is a schematic diagram of a partition tree in VP9 according to one embodiment. [Figure 5B] FIG. 2 is a schematic diagram of a partition tree in AV1 according to one embodiment. [Figure 6A] FIG. 2 is a schematic diagram of block partitioning for a quad-tree plus binary tree (QTBT) structure according to one embodiment. [Figure 6B] FIG. 6B is a schematic diagram of a corresponding tree representation of the block-partitioned QTBT structure shown in FIG. 6A. [Figure 7A] FIG. 13 is a schematic diagram of vertical center-side ternary tree partitioning according to one embodiment. [Figure 7B] FIG. 13 is a schematic diagram of horizontal center-side ternary tree partitioning according to one embodiment. [Figure 8] FIG. 2 is a schematic diagram of search points for merge mode with motion vector difference (MMVD) of two reference frames according to one embodiment; [Figure 9] 4 is a schematic diagram of exemplary motion samples used to derive model parameters for a block using local warp motion prediction, according to one embodiment; [Figure 10] 2 is a schematic diagram of an example of a warped sample collection scan order, according to one embodiment; [Figure 11] FIG. 13 is a schematic diagram of motion vectors within a block using warp extension mode according to one embodiment. [Figure 12] FIG. 1 is a schematic diagram of an MMVD search process for a current frame using two reference frames according to one embodiment. [Figure 13]4 is a schematic diagram of a current block and neighboring blocks used to create a warped model, according to one embodiment; [Figure 14] 4 is a schematic diagram of a current block and neighboring blocks used to create a warped model, according to one embodiment; [Figure 15] 4 is a schematic diagram of a current block and neighboring blocks used to create a warped model, according to one embodiment; [Figure 16] 4 is a schematic diagram of a current block and neighboring blocks used to create a warped model, according to one embodiment; [Figure 17] FIG. 2 is a block diagram of computer code according to an embodiment. [Figure 18] FIG. 1 illustrates a computer system suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The following detailed description of the exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements.

[0027] The above disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the disclosure or may be acquired from practice of the implementations. Moreover, one or more features or components of an embodiment may be incorporated or combined with other embodiments (or one or more features of the other embodiments). Furthermore, in the flowcharts and descriptions of the operations provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) simultaneously, and the order of one or more operations may be switched.

[0028] It is apparent that the systems and / or methods described herein may be implemented in different forms of hardware, software, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. It is understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0029] The features described below may be used separately or combined in any order. Furthermore, the embodiments may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0030] No element, act, or instruction used herein should be construed as critical or essential unless expressly stated. Also, as used herein, the singular is intended to include one or more items and may be used interchangeably with "one or more." When one item is intended, "one" or similar terms are used. Also, as used herein, terms such as "has," "have," "having," "include," "including," and the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless otherwise specified. Furthermore, phrases such as "at least one of A and B" or "at least one of A or B" should be understood as including only A, only B, or both A and B.

[0031] FIG. 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 includes at least two terminals 110, 120 interconnected via a network 150. For one-way transmission of data, the first terminal 110 may encode video data at a local location for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the other terminal's encoded video data (coding video data) from the network 150, decode the encoded data, and display the restored video data. One-way data transmission may be common in media provisioning applications, etc.

[0032] 1 shows a second pair of terminals 130, 140 provided to support bidirectional transmission of encoded video that may occur, for example, during a video conference. For the bidirectional transmission of data, each terminal 130, 140 may encode video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 may also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the reconstructed video data on a local display device.

[0033] In FIG. 1, the terminals 110-140 may be depicted as servers, personal computers, and smartphones, and / or any other type of terminal. For example, the terminals 110-140 may be laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. The network 150 represents any number of networks that convey encoded video data between the terminals 110-140, including, for example, wired and / or wireless communication networks. The communication network 150 may exchange data in circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of the network 150 is not important to the operation of the present disclosure, unless otherwise described herein below.

[0034] 2 shows an arrangement of video encoders and decoders in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter is similarly applicable to other video-enabled applications including, for example, videoconferencing, digital TV, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.), etc.

[0035] As shown in FIG. 2, the streaming system 200 may include a capture subsystem 213 that may include a video source 201 and an encoder 203. For example, the video source 201 may be a digital camera and may be configured to create an uncompressed video sample stream 202. The uncompressed video sample stream 202 may provide a high amount of data when compared to an encoded video bitstream and may be processed by an encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream 204 may include a lower amount of data when compared to the sample stream and may be stored on a streaming server 205 for future use. One or more streaming clients 206 may access the streaming server 205 to obtain a video bitstream 209, which may be a copy 208 of the encoded video data 204.

[0036] In an embodiment, the streaming server 205 may also function as a Media-Aware Network Element (MANE). For example, the streaming server 205 may be configured to prune the encoded video bitstreams 204 in order to tailor potentially different bitstreams to one or more of the streaming clients 206. In an embodiment, a MANE may be provided separately from the streaming server 205 in the streaming system 200.

[0037] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video bitstream 209, which is an input copy of the encoded video bitstream 204, and generate an output video sample stream 210 that may be rendered on a display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to a particular video encoding / compression standard. Examples of these standards include, but are not limited to, ITU-T Recommendation H.265. The developing video encoding standard is informally known as Versatile Video Coding (VVC). Embodiments of the present disclosure may be used in the context of VVC.

[0038] FIG. 3 illustrates an exemplary functional block diagram of a video decoder 210 attached to a display 212 according to one embodiment of the disclosure.

[0039] The video decoder 210 may include a channel 312, a receiver 310, a buffer memory 315, an entropy decoder / parser 320, a scaler / inverse transform unit 351, an intra prediction unit 352, a motion compensated prediction unit 353, an aggregator 355, a loop filter unit 356, a reference picture memory 357, and a current picture memory. In at least one embodiment, the video decoder 210 may include an integrated circuit, a series of integrated circuits, and / or other electronic circuitry. The video decoder 210 may also be embodied partially or entirely in software running on one or more CPUs having associated memory.

[0040] In this and other embodiments, the receiver 310 may receive one or more coded video sequences to be decoded by the decoder 210, one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel 312, which may be a hardware / software link to a storage device that stores the coded video data. The receiver 310 may receive the coded video data along with other data (e.g., coded audio data and / or auxiliary data streams), which may be forwarded to respective usage entities (not shown). The receiver 310 may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory 315 may be coupled between the receiver 310 and the entropy decoder / parser 320 (hereinafter the "parser"). If the receiver 310 is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory 315 may not be used or may be small. For use in a best effort packet network such as the Internet, a buffer memory 315 may be required and can be relatively large and adaptively sized.

[0041] The video decoder 210 may include a parser 320 for reconstructing symbols 321 from the entropy-coded video sequence. These categories of symbols may include, for example, information used to manage the operation of the decoder 210 and potentially information for controlling a rendering device, such as a display 212, which may be coupled to the decoder as shown in FIG. 2. The rendering device control information may be, for example, in the form of a Supplementary Enhancement Information (SEI) (SEI message) or a Video Usability Information (VUI) parameter set fragment (not shown). The parser 320 may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may follow any video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 320 may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the group. The subgroups may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transformation unit (TU), a prediction unit (PU), etc. Parser 320 may also extract information from the coded video sequence, such as transform coefficients, quantization parameter values, motion vectors, etc.

[0042] Parser 320 may perform an entropy decoding / parsing operation on the video sequence received from buffer memory 315 to generate symbols 321 .

[0043] The reconstruction of symbols 321 may involve several different units, depending on the type of coded video picture or portion thereof (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors. Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by parser 320. The flow of such subgroup control information between parser 320 and the following units is not shown for clarity.

[0044] In addition to the functional blocks described above, the decoder 210 can be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0045] One unit is a scalar / inverse transform unit 462. The scalar / inverse transform unit 351 may receive quantized transform coefficients as symbols 321 from the parser 320 along with control information (including which transform to use, block size, quantization factor, quantization scaling matrix, etc.). The scalar / inverse transform unit 351 may output a block containing sample values ​​that may be input to an aggregator 355.

[0046] In some cases, the output samples of the scalar / inverse transform unit 351 may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by the intra-picture prediction unit 352. In some cases, the intra-picture prediction unit 352 generates blocks of the same size and shape of the block being reconstructed using surrounding already reconstructed information retrieved from the (partially reconstructed) current picture from the current picture memory 358. In some cases, the aggregator 355 adds, sample by sample, the prediction information generated by the intra-picture prediction unit 352 to the output sample information provided by the scalar / inverse transform unit 351.

[0047] In other cases, the output samples of the scalar / inverse transform unit 351 may relate to an inter-coded and potentially motion-compensated block. In such a case, the motion compensation prediction unit 353 may access the reference picture memory 357 to retrieve samples used for prediction. After motion compensating the retrieved samples according to the symbols 321 related to the block, these samples may be added by the aggregator 355 to the output of the scalar / inverse transform unit 351 (in this case, called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory 357 from which the motion compensation prediction unit 353 retrieves the prediction samples may be controlled by a motion vector. The motion vector may be made available to the motion compensation prediction unit 353, for example, in the form of a symbol 321, which may have X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory 357 when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0048] The output samples of aggregator 355 may be subjected to various loop filtering techniques in loop filter unit 356. Video compression techniques may include in-loop filter techniques, controlled by parameters contained in the coded video bitstream and made available to loop filter unit 356 as symbols 321 from parser 320, that may be responsive to meta-information obtained during decoding of previous parts of the coded pictures or coded video sequences (in decoding order), as well as to previously reconstructed loop filtered sample values.

[0049] The output of the loop filter unit 356 may be a sample stream that may be output to a rendering device, such as the display 212, and may also be stored in a reference picture memory 357 for use in future inter-picture prediction.

[0050] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 320), the current reference picture can become part of reference picture memory 357, and a new current picture memory can be reallocated before beginning reconstruction of a subsequent coded picture.

[0051] The video decoder 210 may perform decoding operations according to a given video compression technique, which may be documented in a standard such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to the syntax of the video compression technique or standard as specified in the video compression technique or standard, and in particular as specified in its profile document. Also, for compliance with some video compression techniques or standards, the complexity of the coded video sequence may be within a range defined by a level of the video compression technique or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (e.g., measured in megasamples per second), the maximum reference picture size, etc. In some cases, the limits set by the level can be further limited through a Hypothetical Reference Decoder (HRD) specification and metadata about HRD buffer management conveyed in the coded video sequence.

[0052] In one embodiment, the receiver 310 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder 210 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0053] FIG. 4 illustrates an example functional block diagram of a video encoder 203 associated with a video source 201 according to one embodiment of the present disclosure.

[0054] The video encoder 203 may include, for example, an encoder, which may be a source coder 430, a coding engine 432, a (local) decoder 433, a reference picture memory 434, a predictor 435, a transmitter 440, an entropy coder 445, a controller 450, and a channel 460.

[0055] The encoder 203 may receive video samples from a video source 201 (not part of the encoder), which may capture video images to be encoded by the encoder 203 .

[0056] The video source 201 may provide a source video sequence to be encoded by the encoder 203 in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media presentation system, the video source 201 may be a storage device that stores pre-prepared videos. In a video conferencing system, the video source 203 may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples, depending on the sampling structure, color space, etc. in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.

[0057] According to one embodiment, the encoder 203 may encode and compress pictures of a source video sequence into an encoded video sequence 443 in real-time or under any other time constraint required by an application. Achieving an appropriate encoding rate is one function of the controller 450. The controller 450 may also control and be operatively coupled to other functional units, as described below. Coupling is not shown for clarity. Parameters set by the controller 450 may include rate control related parameters (picture skip, quantization, lambda value for rate distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can readily recognize other functions of the controller 450 that may be relevant to a video encoder 203 optimized for a particular system design.

[0058] Some video encoders operate in what those skilled in the art will easily recognize as a "coding loop". As a very simplified description, the coding loop can consist of the coding part of the source coder 430 (responsible for generating symbols based on the input picture to be coded and the reference pictures) and a (local) decoder 433 embedded in the encoder 203. The decoder 433 reconstructs the symbols to generate sample data similar to what a (remote) decoder would generate when the compression between the symbols and the coded video bitstream is lossless for a particular video compression technique. That reconstructed sample stream may be input to a reference picture memory 434. Since the decoding of the symbol stream results in bit-wise accurate results independent of the location of the decoder (local or remote), the contents of the reference picture memory are also bit-wise accurate between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronization (including the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is known to those skilled in the art.

[0059] The operation of the "local" decoder 433 may be the same as the "remote" decoder 210, which has already been described in detail above in connection with Figure 3. However, the entropy decoding portion of the decoder 210, including the channel 312, the receiver 310, the buffer memory 315 and the parser 320, may not be fully implemented in the local decoder 433, since symbols are available and the encoding / decoding of the symbols into the encoded video sequence by the entropy coder 445 and the parser 320 may be lossless.

[0060] An observation that can be made at this point is that any decoder techniques, other than analysis / entropy decoding, present in the decoder must be present in substantially the same functional form in the corresponding encoder. For this reason, the subject matter of the disclosure focuses on the decoder operation. A description of the encoder techniques can be omitted, as they may be the inverse of the decoder techniques described generically. Only in certain areas is more detailed description necessary and is provided below.

[0061] As part of its operation, the source coder 430 may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence designated as “reference frames.” In this manner, the coding engine 432 codes differences between pixel blocks of the input frame and pixel blocks of reference frames that may be selected as predictive references for the input frame.

[0062] The local decoder 433 may decode the encoded video data of the frames that may be designated as reference frames based on the symbols generated by the source coder 430. The operation of the encoding engine 432 may advantageously be a lossy process. If the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence may be a replica of the source video sequence, typically with some errors. The local decoder 433 may replicate the decoding process that may be performed by the video decoder on the reference frames and store the reconstructed reference frames in the reference picture memory 434. In this way, the encoder 203 may locally store copies of the reconstructed reference frames that have common content as the reconstructed reference frames (without transmission errors) obtained by the far-end video decoder.

[0063] The predictor 435 may perform a prediction search for the coding engine 432. That is, for a new frame to be coded, the predictor 435 may search the reference picture memory 434 for sample data (as candidate reference pixel blocks) or specific metadata (reference picture motion vectors, block shapes, etc.). These may serve as suitable prediction references for the new picture. The predictor 435 may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, the input picture determined by the search results obtained by the predictor 435 may have prediction references drawn from multiple reference pictures stored in the reference picture memory 434.

[0064] Controller 450 may manage the encoding operations of source coder 430, including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0065] The output of all the above functional units may be subjected to entropy coding in entropy coder 445. The entropy coder converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as, for example, Huffman coding, variable length coding, arithmetic coding, etc.

[0066] The transmitter 440 may buffer the encoded video sequence produced by the entropy coder 445 and prepare it for transmission over a communication channel 460, which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter 440 may merge the encoded video data from the coder 430 with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (not shown).

[0067] The controller 450 may manage the operation of the encoder 203. During encoding, the controller 450 may assign each encoded picture a particular encoding picture type, which may affect the encoding technique that may be applied to each picture. For example, pictures may often be assigned as an intra picture (I picture), a predicted picture (P picture), or a bidirectionally predicted picture (B picture).

[0068] An Intra picture (I-picture) may be one that can be coded and decoded without using other frames in a sequence as a source of prediction. Some video codecs allow different types of Intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will recognize these variations of I-pictures and their respective uses and characteristics.

[0069] A predictive picture (P-picture) may be one that can be encoded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0070] Bidirectionally predicted pictures (B-pictures) may be those that can be coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0071] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8 or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective picture of the block. For example, blocks of an I-picture may be non-predictively coded or may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P-picture may be non-predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be non-predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0072] Video encoder 203 may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In doing so, video encoder 203 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0073] In one embodiment, the transmitter 440 may transmit additional data along with the coded video. The source coder 430 may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other types of redundant data such as redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0074] [Block partitioning in VP9 and AV1]

[0075] As mentioned above, AV1 is an open video coding format designed for video transmission over the Internet, developed as a successor to VP9. Referring to FIG. 5A, VP9 uses a four-way partition tree starting from the 64×64 level down to the 4×4 level, with some further restrictions for blocks 8×8 and below (as shown in the top half of FIG. 5A). The partitions designated as R may be called recursive in that the same partition tree may be repeated at lower scales until the partition reaches the lowest 4×4 level.

[0076] As shown in Figure 5B, AV1 not only extends the partition tree to a 10-way structure, but also increases the maximum size (called a superblock in VP9 / AV1 terminology) to start at 128x128. This may include 4:1 / 1:4 rectangular partitions, which were not present in VP9. None of the rectangular partitions may be further subdivided. Additionally, AV1 adds more flexibility to the use of partitions below the 8x8 level, in the sense that 2x2 chroma inter prediction is allowed in certain cases.

[0077] [Block Partitioning in HEVC]

[0078] In HEVC, a coding tree unit (CTU) may be partitioned into coding units (CU) by using a quad-tree (QT) structure called a coding tree to adapt to different local characteristics. The decision on whether to code a picture region using inter-picture (temporal) prediction or intra-picture (spatial) prediction may be made at the CU level. Each CU may be further partitioned into one, two or four prediction units (PU) according to a PU partition type. Within one PU, the same prediction process may be applied, and related information may be transmitted to the decoder for each PU. After obtaining the residual block by applying a prediction process based on the PU partition type, the CU may be partitioned into transform units (TU) according to another QT structure such as the coding tree for CU. One of the important features of the HEVC structure is to have multiple partition concepts, including CU, PU and TU. In HEVC, a CU or TU can only be square in shape, while a PU may be square or rectangular in shape for an inter-prediction block. In HEVC, a coding block may be further divided into four square sub-blocks, and a transform is performed on each sub-block, i.e., TU. Each TU may be further divided recursively (using QT partitioning) into smaller TUs, which are called Residual Quad-Tree (RQT).

[0079] At picture boundaries, HEVC employs implicit quadtree partitioning so that blocks maintain the quadtree partitioning until their size fits the picture boundary.

[0080] [Block partitioning in VVC]

[0081] 1. Block partitioning structure using quad-tree (QT) and binary tree (BT)

[0082] The QTBT block structure removes the concept of multiple partition types. That is, the QTBT block structure removes the separation of the concepts of CU, PU, ​​and TU, and supports more flexibility for CU partition shapes. In the QTBT block structure, a CU may have a square or rectangular shape. Referring to FIG. 6A-6B, a coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf node is further partitioned by a binary tree structure. There are two partition types in BT partitioning: symmetric horizontal partitioning and symmetric vertical partitioning. The BT leaf node may be called a CU, and its segmentation is used for prediction and transform processing without further partitioning. This means that the CU, PU, ​​and TU have the same block size in the QTBT coding block structure. In the joint exploration model (JEM), a CU may be composed of coding blocks (CB) of different color components. For example, in the case of a prediction (P, prediction) slice and a binary (B, binary) slice in a 4:2:0 chroma format, one CU may include one luma CB and two chroma CBs. A CU may also be composed of or include a single component CB. For example, one CU includes only one luma CB or only two chroma CBs in the case of an I slice.

[0083] The following parameters may be defined for the QTBT partitioning scheme: (a) CTU size: The root node size of the quadtree, which may be the same concept as in HEVC (b) MinQTSize: The minimum allowable quadtree leaf node size. (c) MaxBTSize: Maximum allowable size of binary tree root node (d) MaxBTDepth: Maximum allowable binary tree depth (e) MinBTSize: The minimum allowable size of a binary tree leaf node.

[0084] In one example of a QTBT partitioning structure, the CTU size may be set to 128×128 luma samples with two corresponding 64×64 blocks of chroma samples, MinQTSize may be set to 16×16, MaxBTSize may be set to 64×64, MinBTSize may be set to 4×4 (for both width and height), and MaxBTDepth may be set to 4. Quad-tree partitioning may be first applied to the CTU to generate a QT leaf node. The QT leaf node may have a size from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the leaf QT node is 128×128, it may not be further partitioned by the binary tree since the size exceeds MaxBTSize (i.e., 64×64). Otherwise, the leaf QT node may be further partitioned by the binary tree. Thus, the QT leaf node is also the root node of the BT and has a BT depth of zero. When the BT depth reaches MaxBTDepth (i.e., 4), further splitting may not be considered. When a BT node has a width equal to MinBTSize (i.e., 4), further horizontal splitting may not be considered. Similarly, when a BT node has a height equal to MinBTSize, further vertical splitting may not be considered. BT leaf nodes may be further processed by prediction and transform processing without further partitioning. In JEM, for example, the maximum CTU size may be 256x256 luma samples.

[0085] FIG. 6A shows an example of block partitioning by using QTBT, and FIG. 6B shows the corresponding tree representation. Solid lines indicate QT partitioning, and dotted lines indicate BT partitioning. At each partition (i.e., non-leaf) node of BT, one flag may be signaled to indicate which partition type (i.e., horizontal or vertical) is used, with 0 indicating horizontal partitioning and 1 indicating vertical partitioning. In the case of QT partitioning, it may not be necessary to indicate the partition type, since QT partitioning may always partition a block both horizontally and vertically to generate four sub-blocks with equal size.

[0086] The QTBT scheme may support the flexibility for luma and chroma to have separate QTBT structures. Currently, for P slices and B slices, the luma CTB and chroma CTB in one CTU share the same QTBT structure. However, for I slices, the luma CTB is partitioned into CUs by a QTBT structure, and the chroma CTB is partitioned into chroma CUs by another QTBT structure. This means that a CU in an I slice is composed of a coding block of a luma component or a coding block of two chroma components, and a CU in a P slice or B slice is composed of coding blocks of all three color components.

[0087] In HEVC, to reduce memory access for motion compensation, inter prediction of small blocks is restricted, bi-prediction is not supported for 4x8 and 8x4 blocks, and inter prediction is not supported for 4x4 blocks. QTBT implemented in JEM-7.0 removes these restrictions.

[0088] 2. Block partitioning using a ternary tree (TT)

[0089] In VVC, a multi-type-tree (MTT) structure is included, which further adds a vertical centered ternary tree and a horizontal centered ternary tree in addition to the QTBT, as shown in Figures 7A-7B. Figure 7A shows the vertical centered ternary tree partitioning, and Figure 7B shows the horizontal centered ternary tree partitioning.

[0090] The main advantages of ternary tree partitioning include, but are not limited to, (a) it provides a complement to quadtree and binary tree partitioning, i.e., while quadtrees and binary trees always split along block centers, ternary tree partitioning can capture objects located at block centers, and (b) the width and height of the proposed ternary tree partitions may always be powers of two, and therefore no further transformations may be required.

[0091] The two-level tree design is primarily motivated by reduced complexity. In theory, the complexity of traversing a tree is T D where T denotes the number of split types and D is the depth of the tree.

[0092] [Merge mode with motion vector differences (MMVD)]

[0093] In merge mode, the implicitly derived motion information is directly used for generating the prediction samples of the current CU. A merge mode with motion vector differences (MMVD) is introduced in VVC. The MMVD flag is signaled immediately after sending the skip and merge flags to specify whether the MMVD mode is used for a CU.

[0094] In MMVD, after a merge candidate is selected, it may be further refined by signaled motion vector difference (MVD) information. The MVD information may include a merge candidate flag, an index to specify the magnitude of motion, and an index to indicate the motion direction. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the motion vector (MV) base. A merge candidate flag may be signaled to specify which one of the first two candidates is used.

[0095] A distance index may be used, which may specify motion magnitude information, indicating a predefined offset from the starting point. Figure 8 shows MMVD search points of two reference frames according to some embodiments. As shown in Figure 8, the offset may be added to either the horizontal or vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 1 below. [Table 1]

[0096] A direction index may be used and may represent the direction of the MVD relative to the starting point. The direction index may represent one of four directions, as shown in Table 2 below. The meaning of the MVD code may change according to the information of the starting MV. When the starting MV is a uni-predictive MV or a bi-predictive MV and both lists point to the same side of the current picture (i.e., the picture order counts (POC) of the two references are both greater than or less than the POC of the current picture), the code in Table 2 specifies the code of the MV offset added to the starting MV. When the start MV is a bi-predictive MV, the two MVs point to different sides of the current picture (i.e., the POC of one reference is larger than that of the current picture and the POC of the other reference is smaller than that of the current picture), and the difference in the POC in list 0 (L0) is larger than that in list 1 (L1), the code in Table 2 specifies the sign of the MV offset added to the L0 MV component of the start MV, and the code of the L1 MV has the opposite value. Otherwise, when the difference in the POC in L1 is larger than that of L0, the code in Table 2 specifies the sign of the MV offset added to the L1 MV component of the start MV, and the code of the L0 MV has the opposite value.

[0097] The MVD may be scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling may be required. Otherwise, if the difference in POC in L0 is larger than that of L1, the MVD of L1 is scaled. If the POC difference of L1 is larger than that of L0, the MVD of L0 may be scaled in the same way. If the starting MV is uni-predicted, the MVD may be added to the available MV. [Table 2]

[0098] [Symmetric MVD coding]

[0099] In VVC, in addition to signaling MVD for normal unidirectional and bidirectional prediction modes, a symmetric MVD mode for signaling bidirectional MVD may be applied. In the symmetric MVD mode, motion information including both L0 and L1 reference picture indexes and the MVD of L1 may be derived or may not be explicitly signaled.

[0100] The decoding process for the symmetric MVD mode may be as follows.

[0101] At the slice level, the variables BiDirPredFlag, RefIdxSymL0 and RefIdxSymL1 may be derived as follows: If mvd_l1_zero_flag is 1, then BiDirPredFlag is set equal to 0. Otherwise, if the closest reference picture in L0 and the closest reference picture in L1 form a forward and backward pair of reference pictures or a backward and forward pair of reference pictures, then BiDirPredFlag is set to 1 and both the L0 reference picture and the L1 reference picture are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0.

[0102] At the CU level, if the CU is bi-predictively coded and BiDirPredFlag is equal to 1, a symmetric mode flag may be explicitly signaled indicating whether symmetric mode is used or not. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 may be explicitly signaled. The reference indexes of L0 and L1 may be set equal to a pair of reference pictures, respectively. MVD1 may be set equal to (-MVD0).

[0103] [Intermode coding in CWG-B018]

[0104] In AV1, for each coding block in an inter frame, if the mode of the current block is an inter coding mode rather than a skip mode, another flag may be signaled to indicate whether a single or mixed reference mode is used for the current block. In a single reference mode, the predictive block may be generated by one motion vector. In a mixed reference mode, the predictive block may be generated by a weighted average of two predictive blocks derived from two motion vectors.

[0105] For the single reference case, the following modes may be signaled: (a) NEARMV - Use one of the motion vector predictors (MVPs) in the list indicated by the Dynamic Reference List (DRL) index. (b) NEWMV - Apply a difference to the MVP using one of the MVPs in the list signaled by the DRL index as a reference. (c) GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0106] For the mixed reference case, the following modes may be signaled: (a) NEAR_NEARMV - Use one of the MVPs in the list signaled by the DRL index (b) NEAR_NEWMV - use one of the MVPs in the list signaled by the DRL index as a reference and send a delta MV for the second MV (c) NEW_NEARMV - send a differential MV for the first MV using one of the MVPs in the list signaled by the DRL index as a reference (d) NEW_NEWMV - use one of the MVPs in the list signaled by the DRL index as a reference and send a delta MV for both MVs. (e) GLOBAL_GLOBALMV - use MV from each reference based on frame-level global motion parameters

[0107] [Motion Vector Differential Coding in AV1]

[0108] AV1 allows for 1 / 8 pixel motion vector precision (or accuracy), and the following syntax may be used to signal motion vector differences in reference frames L0 or L1.

[0109] According to an embodiment, the syntax mv_joint specifies which components of the motion vector difference are non-zero. A value of 0 may indicate that there is no non-zero MVD along either the horizontal or vertical direction. A value of 1 may indicate that there is non-zero MVD only along the horizontal direction. A value of 2 may indicate that there is non-zero MVD only along the vertical direction. A value of 3 may indicate that there is non-zero MVD along both the horizontal and vertical directions.

[0110] According to an embodiment, the syntax element mv_sign specifies whether the motion vector difference is positive or negative.

[0111] According to an embodiment, the syntax element mv_class specifies the class of the motion vector difference, where a higher class means that the motion vector difference has a larger magnitude, as shown in Table 3 below. [Table 3]

[0112] According to an embodiment, the syntax element mv_bit specifies the integer part of the offset between the motion vector difference and the starting magnitude of each MV class.

[0113] According to an embodiment, the syntax element mv_fr specifies the first two fractional bits of the motion vector difference.

[0114] According to an embodiment, the syntax mv_hp specifies the third fractional bit of the motion vector difference.

[0115] [Adaptive MVD Resolution in CWG-B092]

[0116] For the NEW_NEARMV and NEAR_NEWMV modes, the accuracy of the MVD may depend on the associated class and the size of the MVD.

[0117] First, fractional MVDs may be allowed only if the size of the MVD is less than or equal to one pixel.

[0118] Then, when the value of the associated MV class is greater than or equal to MV_CLASS_1, only one MVD value may be allowed, and the MVD value in each MV class may be derived as 4, 8, 16, 32, 64 for MV class 1 (MV_CLASS_1), MV class 2 (MV_CLASS_2), MV class 3 (MV_CLASS_3), MV class 4 (MV_CLASS_4) or MV class 5 (MV_CLASS_5).

[0119] The allowable MVD values ​​for each MV class are shown in Table 4 below. [Table 4]

[0120] In some embodiments, one context may be used to signal mv_joint or mv_class if the current block is coded using NEW_NEARMV or NEAR_NEWMV modes, and another context may be used to signal mv_joint or mv_class if the current block is not coded using NEW_NEARMV or NEAR_NEWMV modes.

[0121] [Joint MVD Coding (JMVD) in CWG-B092]

[0122] A new inter-coding mode named JOINT_NEWMV may be applied to indicate whether the MVDs for the two reference lists are signaled together. When the inter-prediction mode is equal to the JOINT_NEWMV mode, the MVDs for the reference lists L0 and L1 are signaled together. Thus, only one MVD named joint_mvd may be signaled and transmitted to the decoder, and the differential MVs for the reference lists L0 and L1 may be derived from joint_mvd.

[0123] The JOINT_NEWMV mode may be signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV and GLOBAL_GLOBALMV modes. No further context may be added.

[0124] When JOINT_NEWMV mode is signaled and the POC distances between two reference frames and the current frame are different, the MVD may be scaled for reference list L0 or reference list L1 based on the POC distances. Specifically, the distance between reference frame list L0 and the current frame is denoted as td0, and the distance between reference frame list L1 and the current frame is denoted as td1. If td0 is greater than or equal to td1, joint_mvd may be used directly for reference list L0, and mvd for reference list L1 may be derived from joint_mvd based on the following equation (1): derived_mvd=td1 / td0*joint_mvd Formula (1)

[0125] Otherwise, if td1 is greater than or equal to td0, then joint_mvd for reference list L1 may be used directly, and mvd for reference list L0 may be derived from joint_mvd based on equation (2) below. derived_mvd=td0 / td1*joint_mvd Formula (2)

[0126] [Improvements to Adaptive MVD in CWG-C011]

[0127] A new inter-coding mode named AMVDMV may be added for the single reference case. When the AMVDMV mode is selected, this may indicate that AMVD is applied to the signal MVD.

[0128] One flag named amvd_flag may be added under JOINT_NEWMV mode to indicate whether AMVD is applied to the joint MVD coding mode named joint AMVD coding. When adaptive MVD resolution is applied to the joint MVD coding mode named joint AMVD coding, the MVDs for two reference frames may be signaled together, and the precision of the MVD may be implicitly determined by the size of the MVD. Otherwise, the MVDs for two (or more) reference frames are signaled together, and MVD coding is applied.

[0129] [Adaptive motion vector resolution (AMVR) in CWG-C012 and CWG-C020]

[0130] AMVR was first proposed in CWG-C012 and supports a total of seven MV precisions (i.e., 8, 4, 2, 1, 1 / 2, 1 / 4, 1 / 8). For each prediction block, the AVM encoder explores all supported precision values ​​and signals the best precision to the decoder.

[0131] To reduce the runtime of the encoder, two precision sets may be supported. Each precision set may contain four predefined precisions. The precision set may be adaptively selected at the frame level based on the maximum precision value of the frame. Similar to AV1, the maximum precision may be signaled in the frame header. Table 5 below summarizes the supported precision values ​​based on the frame-level maximum precision. [Table 5]

[0132] In AVM software (similar to AV1), a frame level flag may be present to indicate whether the MV of the frame includes sub-pel precision or not. AMVR may be enabled only if the value of the cur_frame_force_integer_mv flag is 0. In AMVR, if the precision of the block is less than maximum precision, the motion model and the interpolation filter may not be signaled. If the precision of the block is less than maximum precision, the motion mode may be estimated as translational motion and the interpolation filter may be estimated as a REGULAR interpolation filter. Similarly, if the precision of the block is either 4-pel or 8-pel, the inter-intra mode may not be signaled and may be estimated to be 0.

[0133] [Warp motion mode in AV1]

[0134] Motion compensation typically assumes a translational motion model between the reference and target blocks, whereas warped motion utilizes an affine model.

[0135] The affine motion model can be expressed by the following equation (3):

number

[0136] 1. Global Warp Motion Compensation

[0137] In global warp motion compensation, global motion information including a global motion type and some motion parameters may be signaled for each inter reference frame. The global motion types and the associated parameter numbers are listed in Table 6 below. [Table 6]

[0138] After signaling the reference frame index, if global motion is selected, the global motion type and parameters associated with the given reference frame for the current coding block may be used.

[0139] 2. Local Warp Motion Compensation

[0140] For inter-coding blocks, local warping motion may be allowed when the following conditions are met: (1) the current block uses single-reference prediction, (2) the width or height of the coding block is greater than or equal to 8, and (3) at least one of the adjacent neighboring blocks uses the same reference frame as the current block.

[0141] If a local warp motion is used for the current block, the affine model parameters may be estimated by a mean square minimization of the difference between the reference projection and the modeled projection based on the MVs of the current block and its adjacent neighboring blocks. To estimate the parameters of the local warp motion, if the neighboring block uses the same reference frame as the current block, a projection sample pair of the center sample in the neighboring block and its corresponding sample in the reference frame may be obtained. Then, three additional samples may be created by shifting the center position by a quarter sample in one or both dimensions. These additional samples may also be considered as projection sample pairs to ensure the stability of the model parameter estimation process.

[0142] The MVs of neighboring blocks used to derive motion parameters may be referred to as motion samples. The motion samples may be selected from neighboring blocks that use the same reference frame as the current block. The warped motion prediction mode may be enabled only for blocks that use a single reference frame. For example, referring to FIG. 9, the MVs of neighboring blocks B0, B1, and B2 are referred to as MV0, MV1, and MV2, respectively. FIG. 6 illustrates exemplary motion samples used to derive model parameters of a block using local warped motion prediction, according to some embodiments. FIG. 9 illustrates exemplary motion samples used to derive model parameters of a block using local warped motion prediction, according to some embodiments. The current block 600 may be predicted using uni-prediction with reference frame Ref0. The neighboring block B0 may be predicted using mixed prediction with reference frames Ref0 and Ref1. The neighboring block B1 may be predicted using uni-prediction with reference frame Ref0. The neighboring block B2 may be predicted using mixed prediction with reference frames Ref0 and Ref2. The motion vector MV0Ref0 of neighboring block B0, the motion vector MV1Ref0 of neighboring block B1, and the motion vector MV2Ref0 of neighboring block B2 may be used as motion samples to derive the affine motion parameters of the current block 600.

[0143] 3. Sample collection process of local warp motion in AV1 and AVM

[0144] In local warp motion, samples and motion vectors of neighboring blocks may be collected in a stack to calculate a warp model using a regression / least squared error method. In AV1 and AVM, a group of sample positions in a reference picture and its reference positions pointed to by its MV may be collected. Up to eight pairs of positions may be stored in the stack for regression / least squared error warp model calculation.

[0145] FIG. 10 is a schematic diagram of an example of a warp sample collection scanning order. As shown in FIG. 10, only spatial neighboring blocks may be scanned. If the neighboring block has the same reference picture as the current block 610 (e.g., if the neighboring block is composite, one of the reference pictures is the same as the reference picture of the current block), the center position of the neighboring block and the center position of the reference block of the neighboring block pointed to by the MV of the neighboring block may be collected. Referring to FIG. 10, the scanning order may be defined as follows: (1) the upper row neighboring block 611 is scanned, (2) the left column neighboring block 612 is scanned, (3) the upper left neighboring block 613 is scanned, and (4) the upper right neighboring block 614 is scanned.

[0146] [Local Warp Extension Mode and Local Warp Difference Mode from CWG-C050]

[0147] In addition to translational motion, AVM also supports warp motion compensation. Two types of warp motion models are supported: global warp model and local warp model. The global warp model is associated with each reference frame, and each of the four non-translational parameters has 12-bit precision, while the translational motion vectors are coded with 15-bit precision. A coding block may choose to use it directly (provided with a reference frame index). The global warp model captures frame-level scaling and rotation, and thus focuses primarily on inflexible motion across the entire frame.

[0148] Additionally, local warping models at the coding block level are also supported. In the local warping mode, also known as WARPED_CAUSAL, the warping parameters of the current block may be derived by fitting a model to nearby motion vectors using least squares.

[0149] 1. Warp Extension Mode

[0150] The new warp motion mode is called WARP_EXTEND. In WARP_EXTEND mode, neighboring block motions are smoothly extended into the current block, but with some ability to change the warp parameters. This allows complex warping motions to be expressed and spread across multiple blocks, while minimizing blocking artifacts.

[0151] To achieve this, the WARP_EXTEND mode applied to the NEWMV block constructs a new warp model based on two constraints: (1) the per-pixel motion vectors generated by the new warp model must be contiguous with the per-pixel motion vectors in neighboring blocks, and (2) the central pixel of the current block must have a per-pixel motion vector that matches the motion vector signaled for the block as a whole.

[0152] For example, referring to Figure 11, if the neighboring block 622 to the left of the current block 620 is warped, a warp model that matches the motion vectors may be used. Figure 11 illustrates motion vectors within a block using warp extension mode, according to some embodiments.

[0153] The above two constraints for constructing a new warp model may imply certain equations involving the warp parameters of the neighboring blocks and the current block, which may then be solved to compute the warp model for the current block.

[0154] For example, if (A,...,F) represent adjacent warped models and (A',...,F') represents a new warped model, then the first constraint, at each point along the common edge, is:

number

[0155] Points along the edge have different values ​​of y, but they all have the same value of x. This means that the coefficient of y must be the same on both sides (i.e. B'=B and D'=D). The coefficient of x, on the other hand, provides two equations for the other coefficients, shown below in Equations 5-8.

number

[0156] The second constraint specifies that the motion vector of the center of the block must be equal to the one signaled using the NEWMV mechanism. This provides two further equations, resulting in a system of six equations in six variables with a unique solution.

[0157] These equations can be solved efficiently in both software and hardware. The solution can be solved using basic addition, subtraction, multiplication, and division by powers of 2. Therefore, this mode is significantly less complex than the least-squares based local warping mode.

[0158] According to an embodiment, there may be several neighboring blocks that can be extended from. In such a case, it is useful to have some way to select which block to extend from. This problem is similar to the problem faced in motion vector prediction. In particular, there may be several possible motion vectors from nearby blocks, and the one to use must be selected as the basis for NEWMV coding.

[0159] A solution to the above problem may include an extension to handle the need for WARP_EXTEND. This may be done by tracking the source of each motion vector prediction. WARP_EXTEND may then be enabled only if the selected motion vector prediction is taken from a directly neighboring block. That block may then be used as a single "neighboring block" in the remainder of the algorithm.

[0160] In some cases, the warp model of the neighbors will be very good as is, without needing further modification. To code this case more cheaply, WARP_EXTEND may be used for the NEARMV block. The selection of the neighbors may be the same as in NEWMV, except that the selection in NEWMV requires that the neighbors be warped rather than simply translated. However, if this is true and WARP_EXTEND is selected, the warp model parameters of the neighbors may be copied to the current block.

[0161] 2. Warp Diff Mode

[0162] In some embodiments, a motion mode called WARP_DELTA may be provided, in which the warp model of a block is coded as a difference from a predicted warp model, similar to how a motion vector is coded as a difference from a predicted motion vector. Predictions may be sourced either from a global motion model (if present) or from neighboring blocks.

[0163] To avoid having multiple ways to code the same prediction warp model, the following restrictions may be applied: If the mode is NEARMV or NEWMV, the same neighbor selection logic as described for WARP_EXTEND may be used. If this results in a warped neighboring block, then that block's model (without applying the rest of the WARP_EXTEND logic) may be used as prediction. Otherwise, the global warp model may be used as the base.

[0164] The difference for each of the non-translational parameters may then be coded. Finally, the translational part of the model may be adjusted so that the per-pixel motion vector of the center of the block matches the global motion vector of the block.

[0165] This tool (i.e., WARP_DELTA) uses more bits to encode than the other warp modes because it involves explicitly coding the delta for each warp parameter. Therefore, WARP_DELTA may be disabled for blocks smaller than 16x16. However, the decoding logic is quite simple and can represent more complex motions not possible with the other warp modes.

[0166] [Merge with Motion Vector Difference (MMVD)]

[0167] Merge with Motion Vector Difference (MMVD) may be used for either skip mode or merge mode using the motion vector representation method. FIG. 12 shows the MMDV search process for a current frame 630 including a current block 632 using two reference frames 634 and 636 (e.g., the two reference frames shown in FIG. 8). MMVD reuses merge candidates in VVC. A candidate may be selected from among the merge candidates and further extended by the motion vector representation method. MMVD provides a new motion vector representation using simple signaling. The representation method may include a starting point, a motion magnitude, and a motion direction. The MMVD technique may use the merge candidate list of VVC. However, only candidates that are the default merge type (MRG_TYPE_DEFAULT_N) may be considered for the extension of MMVD. A base candidate index (IDX) may define a starting point. The base candidate IDX may indicate the best candidate among the candidates in the list, as shown in Table 7 below. [Table 7]

[0168] If the number of base candidates is equal to 1, the base candidate IDX may not be signaled.

[0169] A distance index may be used to represent the magnitude of the motion information. The distance index may indicate a predetermined distance from the starting point information. The predetermined distance may be as shown in Table 8 below. [Table 8]

[0170] A direction index may be used to represent the direction of the MVD relative to the starting point. The direction index may represent four directions as shown in Table 9 below. [Table 9]

[0171] The MMVD flag may be signaled immediately after sending the skip and merge flags. If the skip and merge flags are true, the MMVD flag may be parsed. If the MMVD flag is equal to 1, the MMVD syntax may be parsed. But if it is not 1, the AFFINE flag may be parsed. If the AFFINE flag is equal to 1, the AFFINE mode may be used. But if it is not 1, the skip / merge index may be parsed for the VTM's skip / merge mode.

[0172] [Issues in related technologies]

[0173] In the related art, only one group of samples is used to calculate the warp model, and only spatially adjacent blocks are considered, which is suboptimal because samples collected from different groups may result in different warp models that may be better than the group in the related art.

[0174] Exemplary embodiments

[0175] The embodiments of the present disclosure may be used separately or combined in any order. Moreover, each of the embodiments of the present disclosure (e.g., the method, the encoder, and the decoder) may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium. Hereinafter, the term "block" may be interpreted as a prediction block, a coding block, or a coding unit (CU, coding unit).

[0176] According to an embodiment, the temporal motion information may be used to derive a better warping model.

[0177] According to an embodiment, temporal motion vectors (MVs) may be used in addition to or as a replacement for spatial MVs during warp motion sample collection.

[0178] According to one or more embodiments, the temporal MV may be used in the spatial neighboring blocks of the current block that have not yet been constructed. In this case, the embodiments of the present disclosure may determine the center position of the neighboring block and its reference position pointed to by the temporal MV, assuming that the neighboring block size is a temporal MV storage grid (e.g., 8×8). Exemplary temporal block positions T1-T5 where the temporal MV may be used are shown in FIG. 13. Referring to FIG. 13, a current block 610, an upper row neighboring block(s) 611, a left column neighboring block(s) 612, an upper left neighboring block 613, an upper right neighboring block 614, a lower left neighboring block 615, and temporal block positions T1-T5 are shown. According to an embodiment, time block position T1 may correspond to the lower right neighboring block, time block position T2 may correspond to the neighboring block immediately above the lower right neighboring block and to the right of the current block 610, time block position T3 may correspond to the neighboring block immediately to the left of the lower right neighboring block and below the current block 610, time block position T4 may correspond to the neighboring block immediately below the upper right neighboring block 614 and to the right of the current block 610, and time block position T5 may correspond to the neighboring block immediately to the right of the lower left neighboring block 615 and below the current block 610. According to an embodiment, the scanning order of the time neighboring blocks is not fixed.

[0179] According to one or more embodiments, the temporal MV may be used at the position where the spatial neighboring block is reconstructed, but this spatial neighboring block may not have a valid MV (e.g., this block is coded using intra mode). With reference to FIG. 13, such an exemplary position may correspond to one or more spatial neighboring blocks, such as one or more of the upper row neighboring block(s) 611, the left column neighboring block(s) 612, the top left neighboring block 613, the top right neighboring block 614, and the bottom left neighboring block 615. In this case, the embodiments of the present disclosure may use either the actual block size of the spatial neighboring block or the size of the temporal MV storage grid (e.g., 8×8) to calculate the center position within the neighboring blocks of the current block 610 in the current picture and its reference position pointed to by the temporal MV. According to the embodiment, the scanning order of the temporal neighboring blocks may not be fixed.

[0180] According to one or more embodiments, a temporal MV is used at a predefined relative position to the current block plus a motion offset (or displacement) that is signaled or derived from the MV used by a neighboring block.

[0181] According to an embodiment, a subset of the available neighboring blocks is used in a regression / least squared error method to compute the warp model.

[0182] 14, only positions in the neighboring block 611 above the current block 610 are used in the warp model calculation. For example, the positions (or the motion vectors of such positions) may be used in a regression / least squared error method to calculate the warp model.

[0183] 14, in accordance with one or more embodiments, only positions in the neighboring block 611 above the current block 610 and the positions of the upper right neighboring block 614 are used in the warp model calculation. For example, the positions (or the motion vectors of such positions) may be used in a regression / least squared error method to calculate the warp model.

[0184] 14, in accordance with one or more embodiments, only the positions of the neighboring block 611 above the current block 610, the position of the upper right neighboring block 614, and the position of the upper left neighboring block 613 are used in the warp model calculation. For example, the positions (or the motion vectors of such positions) may be used in a regression / least squared error method to calculate the warp model.

[0185] According to an embodiment, spatial / temporal neighboring blocks surrounding one or more corners of the current block are used to collect samples and MVs used in the warp model calculation.

[0186] 15, in accordance with one or more embodiments, spatial neighboring blocks S and temporal neighboring blocks T surrounding corners P1, P2, P3, and P4 of a current block 610 are used to compute a warp model. For example, samples / MVs of such blocks may be collected and used in the warp model computation.

[0187] 16, in accordance with one or more embodiments, spatial neighboring blocks S and temporal neighboring blocks T surrounding a number of samples (e.g., samples at corners P1, P2, and P4, or samples at corners P1, P3, and P4) are used to calculate a warp model. For example, samples / MVs of such blocks may be collected and used in the warp model calculation.

[0188] According to an embodiment, referring to Figure 16, a middle one of the spatially neighboring blocks S or a middle one of the temporally neighboring blocks T, or a subset of these blocks, is used for the warp model calculation. For example, samples / MVs of such blocks may be collected and used in the warp model calculation.

[0189] According to an embodiment, the spatial / temporal neighboring blocks of multiple groups are collected into a group list. This group list may be constructed at both the encoder side and the decoder side (e.g., by the encoder and the decoder) and may be used to compute a warp model. The list capacity (e.g., the maximum number of groups) may be predefined or signaled in a high-level syntax. Each group in the list may be used to compute a different warp model individually. An index indicating which group is used to compute the warp model finally used in the construction (e.g., encoding or decoding of the current block) may be signaled in the bitstream. The groups may be constructed using the temporal neighboring blocks, the spatial / temporal neighboring blocks surrounding one or more corners, and the spatial / temporal neighboring blocks at intermediate positions.

[0190] According to one or more embodiments, the group list may be a fixed group list. The construction rules for the group list may be predefined at both the encoder side and the decoder side. For example, the first group of the group list may include or consist of neighboring blocks in the AV1 design as shown in FIG. 10, the second group of the group list may include or consist of spatial neighboring blocks S and temporal neighboring blocks T surrounding corners P1, P3, P4 shown in FIG. 15, the third group of the group list may include or consist of spatial neighboring blocks S and temporal neighboring blocks T surrounding corners P2, P3, P4, and the fourth group of the group list may include upper neighboring blocks of the current block (e.g., upper row neighboring block 611 and upper right neighboring block 614 in FIG. 14), and the index of the group that gives the best warp model may be signaled in the bitstream. If all of the groups do not contain valid samples used to calculate the warp model. the index and the warp motion mode indicator may not be signaled. For example, the encoder may not signal an index and a warp motion mode indicator in the bitstream based on determining that none of the groups contain valid samples to be used to compute the warp model, and the decoder may determine that none of the groups contain valid samples to be used to compute the warp model based on not receiving an index and a warp motion mode indicator in the bitstream.

[0191] According to one or more embodiments, the group list may be a dynamic group list. That is, during construction of the group list, if one of the groups does not contain valid samples for calculating the warp model, this group is not inserted into the list. If all given groups have already been examined and the list is still shorter than the maximum number of groups in the group list, the list may be padded with zeros. If the list is full (e.g., the list size is equal to the maximum number of groups) before all given groups have been examined, the list construction is terminated.

[0192] According to one or more embodiments, during construction of the group list, a tested group that is identical or similar to a group already in the group list is not inserted. For example, the encoder and / or decoder may compare the tested group to one or more of the groups already in the group list for similarity, and based on the results of the comparison meeting a predetermined similarity criterion, the encoder and / or decoder may not insert the tested group into the group list.

[0193] According to one or more embodiments, during construction of the group list, an examined group having elements that are identical or similar to elements in a group already in the group list is not inserted. For example, if a first group in the group list has two current picture positions (-1,-1) and (-1,-2), and the currently examined group is composed of current picture positions (-1,-2) and (-1,-1), then this group is not inserted. For example, the encoder and / or decoder may compare one or more elements (e.g., picture positions) of the examined group with one or more elements (e.g., picture positions) of groups already in the group list for similarity, and based on the result of the comparison meeting a predetermined similarity criterion, the encoder and / or decoder may not insert the examined group into the group list.

[0194] According to one or more embodiments, apart from the above group list construction method, extended warp models from neighboring blocks (not limited to the block pointed to by the MVP index) can also be used as candidates in the group list. If an extended warp model from a neighboring block is used, the encoder and / or decoder may decide to skip the regression / least squares calculation. Instead, exemption projection or simply copying the warp model from the neighboring block is used. According to an embodiment, the neighboring blocks are not limited to adjacent spatial neighboring blocks, warp models from temporal neighboring blocks or non-adjacent spatial blocks can also be used.

[0195] According to an embodiment, before calculating the warp model, the collected MVs of neighboring blocks (i.e., motion samples) can be refined. In one example, an MV smoothing filter may be applied across both horizontal and vertical components of the collected MVs. In one example, motion vector differences (e.g., MMVDs) can be signaled to correct the collected MVs. That is, for example, the decoder may correct the collected MVs based on motion vector differences that may be signaled in the bitstream by the encoder.

[0196] According to an embodiment, when multiple groups (or sets) of motion samples are available and a selection of a group index is signaled (e.g., by an encoder), the groups of motion samples may be reordered by the decoder before signaling the group index at the encoder, or may be reordered by the encoder after analyzing the group index at the decoder.

[0197] According to one or more embodiments, the groups are reordered based on the regression error when the derived warp model is applied to the motion samples.

[0198] According to one or more embodiments, the groups are reordered based on how many motion samples are derived from the local warp coding block or based on how many motion samples are derived from adjacent blocks that are coded using the signaled MVD.

[0199] According to an embodiment, at least one processor and a memory storing computer program instructions (i.e., computer code) may be provided that, when executed by the at least one processor, may implement an encoder or a decoder and may perform any number of functions described in this disclosure, including functions performed by an encoder or a decoder.

[0200] For example, with reference to FIG. 17, at least one processor may implement an encoder 702 or a decoder 704. The computer program instructions may include, for example, encoding (or decoding) code 710 configured to cause at least one processor to code (or decode) at least one picture received in accordance with an embodiment of the present disclosure. The encoding (or decoding) code 710 may include, for example, a motion vector obtainment code 720, a motion sample / warp model obtainment code 730, a warp motion compensation code 740, a motion offset code 750, a list construction code 760, a selection code 770, and a refinement code 780. Although exemplary code for performing certain functions is described herein, it is understood that embodiments of the present disclosure may include other code for performing any other functions described in the present disclosure.

[0201] The motion vector acquisition code 720 may be configured to cause at least one processor to acquire a plurality of motion vectors pointing to positions in neighboring blocks (e.g., spatial and / or temporal neighboring blocks) adjacent to the current block according to an embodiment of the present disclosure. The motion vectors may include at least one temporal motion vector.

[0202] The motion sample / warp model acquisition code 730 may be configured to cause at least one processor to acquire a motion sample and / or a warp model based on the motion vectors and / or warp models of neighboring blocks in accordance with an embodiment of the present disclosure.

[0203] The warp motion compensation code 740 may be configured to cause at least one processor to decode the current block in accordance with an embodiment of the present disclosure, for example, by performing warp motion compensation using the warp model and the motion samples.

[0204] The motion offset code 750 may be configured to cause at least one processor to apply a motion offset to at least one temporal motion vector in accordance with an embodiment of the present disclosure.

[0205] The list building code 760 may be configured to cause at least one processor to build a list (e.g., a candidate list) according to an embodiment of the present disclosure. The list may include, for example, multiple groups of neighboring blocks, warp models of the neighboring blocks, and / or motion vectors (also referred to as motion samples) of the multiple groups of neighboring blocks.

[0206] Selection code 770 may be configured to cause the at least one processor to select from the list an entry to be used for prediction of, for example, a current block, in accordance with an embodiment of the present disclosure.

[0207] The refinement code 780 may be configured to cause the at least one processor to refine the obtained motion vectors before calculating a warp model based on the refined motion vectors in accordance with an embodiment of the present disclosure.

[0208] When an encoder 702 is provided, the computer program instructions may further include a transmission code 790 that may be configured to cause at least one processor to transmit a bitstream including, for example, coded pictures to a decoder (e.g., decoder 704) in accordance with an embodiment of this disclosure. The encoder 702 may provide in the bitstream one or more of syntax elements such as those described in this disclosure in accordance with an embodiment of this disclosure.

[0209] The techniques of the embodiments of the present invention can be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, FIG. 18 illustrates a computer system 900 suitable for implementing embodiments of the disclosed subject matter.

[0210] Computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code that includes instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly or through interpretation, microcode execution, etc.

[0211] The instructions may be executed in various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0212] 18 for computer system 900 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 900.

[0213] The computer system 900 may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, through tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), and olfactory input (not shown). Human interface input devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from a still image camera, etc.), and video (2D video, 3D video including stereoscopic vision, etc.).

[0214] The human interface input devices may include one or more of a keyboard 901, a mouse 902, a trackpad 903, a touch screen 910, a data glove, a joystick 905, a microphone 906, a scanner 907, and a camera 908 (only one of each is shown).

[0215] The computer system 900 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 910, data gloves, or joystick 905, although there may be haptic feedback devices that do not function as input devices). For example, such devices may be audio output devices (speakers 909, headphones (not shown), etc.), visual output devices (e.g., screens 910, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may output two-dimensional visual output or more than three-dimensional output through such means as stereoscopic image output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0216] The computer system 900 may also include human-accessible storage devices and their associated media, such as optical media or similar media 921, including CD / DVD ROM / RW 920 with CDs / DVDs, thumb drives 922, removable hard drives or solid state drives 93, legacy magnetic media such as tapes and floppy disks (not shown), special ROM / ASIC / PLD based devices such as security dongles (not shown).

[0217] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0218] The computer system 900 may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, WLAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV and terrestrial broadcast TV, vehicular and industrial networks including CANBus, etc. Certain networks typically require an external network interface adapter (such as a USB port on the computer system 900) attached to a particular general-purpose data port or peripheral bus 949, while others are typically integrated into the core of the computer system 900 by attachment to a system bus (such as an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system) as described below. Using any of these networks, the computer system 900 may communicate with other entities. Such communications may be one-way receive-only (e.g., broadcast TV), one-way transmit-only (e.g., from a particular CANbus to a particular CANbus device), or bidirectional, to other computer systems, for example, using local or wide area digital networks. Such communications may include communications to cloud computing environment 955. As noted above, specific protocols and protocol stacks may be used in each of these networks and network interfaces.

[0219] The above-mentioned human interface devices, human accessible storage devices and network interface 954 may be attached to the core 940 of the computer system 900 .

[0220] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 943, hardware accelerators for specific tasks 944, etc. These devices may be connected through a system bus 948, along with read only memory (ROM) 945, random access memory (RAM) 946, and internal mass storage 947, such as an internal non-user accessible hard drive, SSD, etc. In some computer systems, the system bus 948 is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 948 or through a peripheral bus 949. Peripheral bus architectures include PCI, USB, etc. A graphics adapter 950 may be included in the core 940.

[0221] The CPU 941, GPU 942, FPGA 943 and accelerator 944 can execute certain instructions, which in combination can constitute the above-mentioned computer code. The computer code can be stored in ROM 945 or RAM 946. Also, temporary data can be stored in RAM 946 while permanent data can be stored, for example, in internal mass storage 947. Fast storage and retrieval from any of the memory devices can be enabled through the use of cache memory, which can be closely associated with one or more of the CPU 941, GPU 942, mass storage 947, ROM 945, RAM 946, etc.

[0222] The computer-readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0223] By way of example and not limitation, the architecture and computer system 900 having the core 940 in particular can provide functionality as a result of the processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer readable media. Such computer readable media can be user accessible mass storage as introduced above, as well as media associated with the core 940's specific storage of a non-transitory nature, such as the core internal mass storage 947 or ROM 945. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core 940. The computer readable media can include one or more memory devices or chips according to specific needs. The software can cause the core 940 and in particular the processor therein (including CPU, GPU, FPGA, etc.) to execute certain processes or certain parts of certain processes described herein, including defining data structures stored in RAM 946 and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 944) that may operate in place of or in conjunction with software to perform particular processes, or particular portions of particular processes, described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0224] While this disclosure describes several non-limiting exemplary embodiments, there are modifications, permutations, and various substitute equivalents that are within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise various systems and methods that, although not explicitly shown or described herein, embody the principles of this disclosure and are therefore within the spirit and scope of this disclosure.

Claims

1. A method performed by a decoder, comprising: receiving a bitstream including coding pictures; obtaining a plurality of motion vectors pointing to positions in neighboring blocks adjacent to a current block in the coding picture, the plurality of motion vectors including at least one temporal motion vector; obtaining motion samples for a warp model based on the plurality of motion vectors; decoding the current block by using the warp model and the motion samples including the at least one temporal motion vector for prediction; The method includes:

2. The method of claim 1 , wherein the at least one temporal motion vector comprises a temporal motion vector at a location of a spatial neighboring block of the current block that has not yet been constructed by the decoder.

3. The method of claim 1 , wherein the at least one temporal motion vector comprises a temporal motion vector at a position of a spatial neighboring block of the current block previously constructed by the decoder.

4. The method of claim 1 , further comprising the step of applying a motion offset to the at least one temporal motion vector.

5. The method of claim 1 , further comprising: obtaining the warp model using a regression / least squared error method on the plurality of motion vectors, including the at least one temporal motion vector.

6. The method of claim 1 , wherein the plurality of motion vectors comprises the at least one temporal motion vector and at least one spatial motion vector.

7. The method of claim 1 , wherein the plurality of motion vectors includes the at least one temporal motion vector and does not include a spatial motion vector.

8. The step of decoding comprises: constructing a list of adjacent blocks for a number of groups; selecting a group from the list among adjacent blocks of the plurality of groups; Further comprising: The method of claim 1 , wherein obtaining the plurality of motion vectors comprises obtaining the plurality of motion vectors from neighboring blocks of a selected group of the motion vectors.

9. the selecting step includes selecting the group from the list based on an index signaled in the bitstream; The method of claim 8 , wherein the index indicates which group from the list should be used to obtain the warp model.

10. The method of claim 1 , wherein the decoding step comprises decoding the current block by performing warped motion compensation.

11. 1. A system comprising: at least one memory configured to store computer program code; at least one processor configured to execute the method according to any one of claims 1 to 10 according to said computer program code; A system including:

12. A computer program which, when executed by at least one processor, causes the at least one processor to carry out a method according to any one of claims 1 to 10.

13. A method performed by an encoder, comprising: obtaining a plurality of motion vectors pointing to positions in neighboring blocks adjacent to a current block in a coding picture, the plurality of motion vectors including at least one temporal motion vector; obtaining motion samples for a warp model based on the plurality of motion vectors; encoding the current block by using the warp model and the motion samples including the at least one temporal motion vector; transmitting a bitstream including the coded pictures; The method includes:

Images