Method, system and computer program for warp extension and warp delta signaling with backup candidates

Abstract

Various embodiments described herein include methods and systems for coding video. In one aspect, the method includes obtaining encoded video data including a plurality of blocks and obtaining a motion vector predictor (MVP) candidate block from an MVP list based on an MVP index. The method further includes determining whether the MVP candidate block is suitable for the warp extension mode in accordance with a determination that a block of the plurality of blocks is designated for the warp extension mode. The method also includes identifying a backup MVP candidate block suitable for the warp extension mode in accordance with a determination that the MVP candidate block is not suitable for the warp extension mode. The method further includes obtaining a warp model from the backup MVP candidate block and performing a warp extension operation on the block using the warp model.

Description

[Technical Field]

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 427,412, filed November 22, 2022, entitled "Improvements on Warp Extend and Warp Delta Signaling with Backup Candidates," and is a continuation of and claims priority to U.S. Patent Application No. 18 / 124,764, filed March 22, 2023, entitled "Systems and Methods for Improvements Warp Extend and Warp Delta Signaling with Backup Candidates," all of which are hereby incorporated by reference in their entireties.

[0002] Technical Field The disclosed embodiments relate generally to video coding and include, but are not limited to, systems and methods for warp extension and warp delta signaling. [Background technology]

[0003] Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video teleconferencing devices, video streaming devices, etc. The electronic devices transmit, receive, or otherwise communicate digital video data over communication networks and / or store the digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video coding may be used to compress the video data according to one or more video coding standards before the video data is communicated or stored.

[0004] Several video codec standards have been developed, including AV1 (AOMedia Video 1), VVC (Versatile Video Coding), JEM (Joint Exploration test Model), HEVC / H.265 (High-Efficiency Video Coding), AVC / H.264 (Advanced Video Coding), and MPEG (Moving Picture Expert Group) coding. Video coding generally uses prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit the redundancy inherent in video data. Video coding aims to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality.

[0005] HEVC, also known as H.265, is a video compression standard designed as part of the MPEG-H project. ITU-T and ISO / IEC published the HEVC / H.265 standard in 2013 (Version 1), 2014 (Version 2), 2015 (Version 3), and 2016 (Version 4). VVC (Versatile Video Coding), also known as H.266, is a video compression standard intended as the successor to HEVC. ITU-T and ISO / IEC published the VVC / H.266 standard in 2020 (Version 1) and 2022 (Version 2). AV1 is an open video coding format designed as a replacement for HEVC. Validated version 1.0.0, including errata 1, was released on January 8, 2019. Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, a video stream can be encoded into a bitstream with compression and then transmitted to a decoder that can decode / decompress the video stream in preparation for viewing or further processing. Compression of the video stream can exploit spatial and temporal correlations in the video signal through spatial and / or motion compensated prediction. Motion compensated prediction can include inter-prediction. Inter-prediction can use one or more motion vectors to generate blocks to be coded using previously coded and decoded pixels. A decoder receiving the coded signal can recreate the blocks.

[0007] Not all motion across an image is translational. Therefore, translational motion patterns cannot accurately describe more complex motions such as rotation, zoom, and shear. To overcome this drawback, a warped motion mode using an affine model can be used. However, it is important to determine when (e.g., on which block) warped motion is allowed. In addition, when warped motion is allowed but the motion vector predictor (MVP) candidate block is not suitable for the warped model, it is important to obtain a suitable backup candidate block. [Means for solving the problem]

[0008] According to some embodiments, a method for decoding video is provided, the method including: (i) obtaining encoded video data including a plurality of blocks; (ii) obtaining a motion vector predictor (MVP) candidate block from an MVP list based on an MVP index; (iii) determining whether the MVP candidate block is suitable for the warp extension mode according to a determination that a block of the plurality of blocks is designated for the warp extension mode; (iv) identifying a backup MVP candidate block suitable for the warp extension mode according to a determination that the MVP candidate block is not suitable for the warp extension mode; (v) obtaining a warp model from the backup MVP candidate block; and (vi) performing a warp extension operation on the block using the warp model.

[0009] According to some embodiments, a computing system, such as a streaming system, a server system, a personal computer system, or other electronic device, is provided. The computing system includes control circuitry and a memory that stores one or more sets of instructions. The one or more sets of instructions include instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and / or a decoder component.

[0010] According to some embodiments, a non-transitory computer-readable storage medium is provided that stores one or more sets of instructions for execution by a computing system, the one or more sets of instructions including instructions for performing any of the methods described herein.

[0011] Accordingly, methods, as well as devices and systems, for coding video are disclosed that may complement or replace conventional methods, devices and systems for video coding.

[0012] The features and advantages described herein are not necessarily all-inclusive, and in particular, some additional features and advantages will become apparent to those skilled in the art upon consideration of the drawings, specification, and claims provided in this disclosure. Furthermore, it should be noted that the language used herein has been chosen primarily for ease of reading and educational purposes, and not necessarily to delineate or limit the subject matter described herein. [Brief explanation of the drawings]

[0013] In order that the present disclosure may be more fully understood, a more particular description may be made by reference to features of various embodiments, some of which are illustrated in the accompanying drawings. However, the accompanying drawings are merely illustrative of relevant features of the present disclosure, and therefore, the description is not necessarily to be considered limiting, as it may allow for other useful features as those skilled in the art will appreciate upon reading the present disclosure.

[0014] [Figure 1] 1 is a block diagram illustrating an exemplary communication system according to some embodiments.

[0015] [Figure 2A] FIG. 2 is a block diagram illustrating exemplary elements of an encoder component according to some embodiments.

[0016] [Figure 2B] FIG. 2 is a block diagram illustrating exemplary elements of a decoder component according to some embodiments.

[0017] [Figure 3] FIG. 1 is a block diagram illustrating an exemplary server system according to some embodiments.

[0018] [Figure 4A] FIG. 2 illustrates an exemplary coding tree structure according to some embodiments. [Figure 4B]FIG. 2 illustrates an exemplary coding tree structure according to some embodiments. [Figure 4C] FIG. 2 illustrates an exemplary coding tree structure according to some embodiments. [Figure 4D] FIG. 2 illustrates an exemplary coding tree structure according to some embodiments.

[0019] [Figure 5A] FIG. 10 illustrates exemplary motion samples used to derive model parameters for a block using local warp motion prediction, according to some embodiments.

[0020] [Figure 5B] FIG. 10 illustrates motion vectors within a block using warp extension mode, according to some embodiments.

[0021] [Figure 5C] FIG. 1 illustrates a current block with two spatial neighbors, according to some embodiments.

[0022] [Figure 5D] FIG. 1 illustrates a current block with four spatial neighbors, according to some embodiments.

[0023] [Figure 5E] FIG. 1 illustrates a current block with nine spatial neighbors, according to some embodiments.

[0024] [Figure 6] 1 is a flow diagram illustrating an exemplary method for coding video, according to some embodiments.

[0025] According to common practice, the various features illustrated in the drawings are not necessarily drawn to scale and like reference numerals may be used to refer to like features throughout the specification and drawings. DETAILED DESCRIPTION OF THE INVENTION

[0026] This disclosure describes, among other things, when (e.g., on which blocks) to enable warp motion mode and how to obtain appropriate candidate blocks for a motion model. Allowing warp extension mode (or warp delta mode) and context derivation for warp extension mode may require an MVP index-pointed MVP candidate. However, requiring an MVP index-pointed MVP candidate during the analysis process requires building an entire MVP list (DRL list). Building an MVP list can significantly delay the analysis process. The methods and systems described herein include enabling warp extension mode and / or warp delta mode without requiring an MVP list to be built. For example, this determination may be based on the motion modes of neighboring blocks. The methods and systems described herein may further include identifying a suitable candidate block for the allowed warp extension mode and / or warp delta mode (e.g., when an MVP candidate block is not suitable). An MVP candidate block is a candidate block pointed to by an MVP index.

[0027] Exemplary Systems and Devices 1 is a block diagram illustrating a communication system 100 according to some embodiments. Communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 through 120-m) that are communicatively coupled to one another via one or more networks. In some embodiments, communication system 100 is a streaming system for use with video-enabled applications, such as video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0028] Source device 102 includes a video source 104 (e.g., a camera component or media storage) and an encoder component 106. In some embodiments, video source 104 is a digital camera (e.g., configured to generate an uncompressed video sample stream). Encoder component 106 generates one or more encoded video bitstreams from the video stream. The video stream from video source 104 may be a high amount of data compared to encoded video bitstream 108 generated by encoder component 106. Because encoded video bitstream 108 is a low amount of data (less data) compared to the video stream from the video source, encoded video bitstream 108 requires less bandwidth to transmit and less storage space to store compared to the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video data over network 110).

[0029] The one or more networks 110 represent any number of networks that convey information between the source device 102, the server system 112, and / or the electronic device 120, including, for example, wired and / or wireless communication networks. The one or more networks 110 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet.

[0030] One or more networks 110 include a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, server system 112 is or includes a streaming server (e.g., configured to store and / or distribute video content, such as an encoded video stream from source device 102). Server system 112 includes a coder component 114 (e.g., configured to encode and / or decode video data). In some embodiments, coder component 114 includes an encoder component and / or a decoder component. In various embodiments, coder component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, coder component 114 is configured to decode encoded video bitstream 108 and re-encode the video data using a different encoding standard and / or methodology to generate encoded video data 116. In some embodiments, server system 112 is configured to generate multiple video formats and / or encodings from encoded video bitstream 108.

[0031] In some embodiments, server system 112 functions as a Media-Aware Network Element (MANE). For example, server system 112 may be configured to prune encoded video bitstream 108 to adapt potentially different bitstreams to one or more of electronic devices 120. In some embodiments, a MANE is provided separately from server system 112.

[0032] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, decoder component 122 is configured to decode encoded video data 116 to generate an output video stream that can be rendered on a display or other type of rendering device. In some embodiments, one or more of electronic devices 120 does not include a display component (e.g., is communicatively coupled to an external display device and / or includes media storage). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0033] Source device 120 and / or multiple electronic devices 102 may be referred to as “terminal devices” or “user devices.” In some embodiments, one or more of source device 120 and / or electronic devices 110 are instances of a server system, a personal computer, a portable device (e.g., a smartphone, tablet, or laptop), a wearable device, a videoconferencing device, and / or other types of electronic devices.

[0034] In an exemplary operation of communication system 100, source device 102 transmits encoded video bitstream 108 to server system 112. For example, source device 102 may code a stream of images captured by the source device. Server system 112 may receive encoded video bitstream 108 and decode and / or encode encoded video bitstream 108 using coder component 114. For example, server system 112 may apply encoding to the video data that is more optimal for network transmission and / or storage. Server system 112 may transmit encoded video data 116 (e.g., one or more coded video bitstreams) to one or more of electronic devices 120. Each electronic device 120 may decode encoded video data 116 to recover and optionally display video pictures.

[0035] In some embodiments, the transmission is a one-way data transmission. One-way data transmission may be used in media serving applications, etc. In some embodiments, the transmission is a two-way data transmission. Two-way data transmission may be used in video conferencing applications, etc. In some embodiments, the coded video bitstream 108 and / or the coded video data 116 are encoded and / or decoded according to any of the video coding / compression standards described herein, such as HEVC, VVC, and / or AV1.

[0036] FIG. 2A is a block diagram illustrating exemplary elements of the encoder component 106 according to some embodiments. The encoder component 106 receives a source video sequence from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives a video sequence from a remote video source (e.g., a video source that is a component of a different device than the encoder component 106). The video source 104 may provide the source video sequence in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 Y CrCb or RGB), and any suitable sampling structure (e.g., Y CrCb 4:2:0 or Y CrCb 4:4:4). In some embodiments, the video source 104 is a storage device that stores previously captured / prepared video. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that, when viewed sequentially, impart motion. The pictures themselves may be organized as a spatial array of pixels, where each picture may contain one or more samples, depending on the sampling structure, color space, etc. in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following discussion will focus on samples.

[0037] The encoder component 106 is configured to encode and / or compress pictures of a source video sequence into a coded video sequence 216 in real time or under other time constraints required by the application. Implementing an appropriate coding rate is one function of the controller 204. In some embodiments, the controller 204 controls and is operatively coupled to other functional units, as described below. Parameters set by the controller 204 may include rate control-related parameters (e.g., picture skip, quantizer, and / or lambda values ​​for rate-distortion optimization techniques), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. Those skilled in the art will readily identify other functions of the controller 204 as they may be relevant to the encoder component 106 being optimized for a particular system design.

[0038] In some embodiments, the encoder component 106 is configured to operate in a coding loop. In a simplified example, the coding loop includes a source coder 202 (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder 210. The decoder 210 reconstructs the symbols to create sample data in a manner similar to a (remote) decoder (when the compression between the symbols and the coded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory 208. Because decoding the symbol stream yields bit-accurate results regardless of the location of the decoder (local or remote), the contents in the reference picture memory 208 are also bit-accurate between the local and remote encoders. In this way, the prediction portion of the encoder interprets the same sample values ​​as reference picture samples as the decoder interprets them when using prediction during decoding. This principle of reference picture synchrony (and the resulting drift if synchrony cannot be maintained, e.g., due to channel error) is known to those skilled in the art.

[0039] The operation of decoder 210 may be the same as that of a remote decoder, such as decoder component 122, which is described in more detail below in connection with Figure 2B. However, with brief reference to Figure 2B, because symbols are available and the encoding / decoding of symbols into a coded video sequence by entropy coder 214 and parser 254 may be lossless, the entropy decoding portion of decoder component 122, including buffer memory 252 and parser 254, may not be implemented entirely within local decoder 210.

[0040] It can be observed in this respect that any decoder technology other than analysis / entropy decoding present in the decoder must exist in substantially the same functional form in the corresponding encoder. For this reason, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder technology can be omitted, since it is the opposite of the decoder technology, which is described comprehensively. Only in certain areas will a more detailed description be required, which will be provided below.

[0041] As part of its operation, source coder 202 may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence designated as reference frames. In this manner, coding engine 212 codes differences between pixel blocks of the input frame and pixel blocks of reference frames that may be selected as prediction references for the input frame. Controller 204 may manage the coding operations of source coder 202, including, for example, setting parameters and subgroup parameters used to encode the video data.

[0042] The decoder 210 decodes coded video data of frames that may be designated as reference frames based on symbols created by the source coder 202. The operation of the coding engine 212 may advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 2A ), the reconstructed video sequence may be a replica of the source video sequence, with some errors. The decoder 210 may replicate the decoding process that may be performed by a remote video decoder on the reference frames and store the reconstructed reference frames in the reference picture memory 208. In this way, the encoder component 106 locally stores copies of reconstructed reference frames that have common content as the reconstructed reference frames (without transmission errors) that will be obtained by the remote video decoder.

[0043] The predictor 206 may perform the prediction search of the coding engine 212. That is, for a new frame to be coded, the predictor 206 may search the reference picture memory 208 for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors and block shapes, that can serve as suitable prediction references for the new picture. The predictor 206 may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor 206, the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory 208.

[0044] The output of all the aforementioned functional units may be subject to entropy coding in entropy coder 214. Entropy coder 214 converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable length coding, and / or arithmetic coding).

[0045] In some embodiments, the output of the entropy coder 214 is coupled to a transmitter. The transmitter may be configured to buffer coded video sequences created by the entropy coder 214 and prepare them for transmission over a communication channel 218, which may be a hardware / software link to a storage device that stores the coded video data. The transmitter may be configured to merge the coded video data from the source coder 202 with other data to be transmitted, such as coded audio data and / or an auxiliary data stream (source not shown). In some embodiments, the transmitter may transmit additional data along with the coded video. The source coder 202 may include such data as part of the coded video sequence. The additional data may include other forms of redundant data, such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, etc.

[0046] The controller 204 may manage the operation of the encoder component 106. During coding, the controller 204 may assign a particular coding picture type to each coded picture, which may affect the coding technique applied to the respective picture. For example, a picture may be assigned as an intra picture (I picture), a predicted picture (P picture), or a bidirectionally predicted picture (B picture). An intra picture may be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will be aware of these variations of I pictures and their respective uses and characteristics, and therefore, they will not be repeated here. A predicted picture may be coded and decoded using intra prediction or inter prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block. Bidirectionally predicted pictures can be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0047] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block-by-block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be non-predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be non-predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0048] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is partitioned into blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector called a motion vector. A motion vector points to a reference block within the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0049] Encoder component 106 may perform coding operations in accordance with a given video coding technique or standard, such as any of those described herein. In doing so, encoder component 106 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0050] 2B is a block diagram illustrating exemplary elements of a decoder component 122 according to some embodiments. The decoder component 122 of FIG. 2B is coupled to a channel 218 and a display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to a loop filter unit 256 and configured to transmit data to the display 124 (e.g., via a wired or wireless connection).

[0051] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more coded video sequences to be decoded by decoder component 122. In some embodiments, the decoding of each coded video sequence is independent of the other coded video sequences. Each coded video sequence may be received from channel 218, which may be a hardware / software link to a storage device that stores the coded video data. The receiver may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be transferred to each using an entity (not shown). The receiver may separate the coded video sequence from the other data. In some embodiments, the receiver receives additional (redundant) data along with the coded video. The additional data may be included as part of the coded video sequence. The additional data may be used by decoder component 122 to decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, and the like.

[0052] In some embodiments, the decoder component 122 includes a buffer memory 252, a parser 254 (sometimes referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra-picture prediction unit 262, a motion compensated prediction unit 260, an aggregator 268, a loop filter unit 256, a reference picture memory 266, and a current picture memory 264. In some embodiments, the decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuitry. In some embodiments, the decoder component 122 is implemented at least partially in software.

[0053] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to combat network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, in addition to buffer memory 252 internal to decoder component 122 (e.g., configured to handle playback timing), a separate buffer memory is provided external to decoder component 122 (e.g., to combat network jitter). When receiving data from a storage / forwarding device with sufficient bandwidth and controllability or from an isosynchronous network, buffer memory 252 may not be required or can be small. For use over a best-effort packet network such as the Internet, buffer memory 252 may be required, may be relatively large, may advantageously be adaptively sized, and may be implemented, at least in part, in an operating system or similar element (not shown) external to decoder component 122.

[0054] Parser 254 is configured to reconstruct symbols 270 from the coded video sequence. The symbols may include, for example, information used to manage the operation of decoder component 122 and / or information for controlling a rendering device such as display 124. The rendering device control information may be in the form of, for example, a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). Parser 254 parses (entropy decodes) the coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 254 may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. Parser 254 may also extract information from the coded video sequence, such as transform coefficients, quantization parameter values, motion vectors, etc.

[0055] The reconstruction of symbols 270 may involve multiple different units, depending on the type of coded video picture or portion thereof (such as inter and intra pictures, inter and intra blocks) and other factors. Which units are involved and how they are involved may be controlled by subgroup control information parsed by parser 254 from the coded video sequence. The flow of such subgroup control information between parser 254 and the underlying units is not shown for clarity.

[0056] In addition to the functional blocks already mentioned, the decoder component 122 can be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the conceptual subdivision into lower-level functional units is maintained.

[0057] The scaler / inverse transform unit 258 receives the quantized transform coefficients and control information (such as which transform to use, block size, quantization coefficients and / or quantization scaling matrix) from the parser 254 as symbols 270. The scaler / inverse transform unit 258 may output blocks containing sample values ​​that may be input to the aggregator 268.

[0058] In some cases, the output samples of the scaler / inverse transform unit 258 relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit 262. The intra-picture prediction unit 262 may generate blocks of the same size and shape as the block being reconstructed using surrounding reconstructed information fetched from the current (partially reconstructed) picture from the current picture memory 264. The aggregator 268 may add, on a sample-by-sample basis, the prediction information generated by the intra-picture prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258.

[0059] In other cases, the output samples of the scalar / inverse transform unit 258 relate to an inter-coded, potentially motion-compensated, block. In such cases, the motion-compensated prediction unit 260 may access the reference picture memory 266 to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols 270 associated with the block, these samples may be added by the aggregator 268 to the output of the scalar / inverse transform unit 258 (in this case, referred to as residual samples or residual signals), thus generating output sample information. The addresses in the reference picture memory 266 from which the motion-compensated prediction unit 260 fetches the prediction samples may be controlled by motion vectors. The motion vectors may be available to the motion-compensated prediction unit 260 in the form of symbols 270, which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory 266 when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0060] The output samples of aggregator 268 may be subjected to various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video bitstream and made available to loop filter unit 256 as symbols 270 from parser 254, but may also be responsive to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, or may be responsive to previously reconstructed loop-filtered sample values.

[0061] The output of the loop filter unit 256 may be a sample stream that can be output to a rendering device such as the display 124, as well as stored in the reference picture memory 266 for use in future inter-picture prediction.

[0062] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 254), the current reference picture can become part of reference picture memory 266, and a new current picture memory can be reallocated before beginning reconstruction of the next coded picture.

[0063] Decoder component 122 may perform decoding operations according to a given video compression technique, which may be documented in a standard, such as any of the standards described herein. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that it conforms to the syntax of the video compression technique or standard as specified in the video compression technique document or standard, particularly the profile document therein. Also, to comply with some video compression techniques or standards, the complexity of the coded video sequence may be within a range defined by a level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further constrained in some cases through metadata for Hypothetical Reference Decoder (HRD) specifications and HRD buffer management signaled in the coded video sequence.

[0064] 3 is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes a control circuit 302, one or more network interfaces 304, a memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuit 302 includes one or more processors (e.g., a CPU, a GPU, and / or a DPU). In some embodiments, the control circuit includes one or more field programmable gate arrays (FPGAs), hardware accelerators, and / or one or more integrated circuits (e.g., application specific integrated circuits).

[0065] The network interface 304 may be configured to interface with one or more communications networks (e.g., wireless, wired, and / or optical networks). Communications networks may be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of communications networks include cellular networks, including Ethernet, WLAN, GSM, 3G, 4G, 5G, LTE, and the like; TV wired or wireless wide-area digital networks, including cable TV, satellite TV, and terrestrial broadcast TV; vehicular and industrial networks, including CANbus; and the like. Such communications may be unidirectional, receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., CANbus to a specific CANbus device), or bidirectional (e.g., to other computer systems using local or wide-area digital networks). Such communications may include communications to one or more cloud computing networks.

[0066] The user interface 306 includes one or more output devices 308 and / or one or more input devices 310. The input devices 310 may include one or more of a keyboard, a mouse, a trackpad, a touchscreen, a data glove, a joystick, a microphone, a scanner, a camera, etc. The output devices 308 may include one or more of an audio output device (e.g., a speaker), a visual output device (e.g., a display or monitor), etc.

[0067] Memory 314 may include high-speed random-access memory (such as DRAM, SRAM, DDR RAM, and / or other random-access solid-state memory devices) and / or non-volatile memory (such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices). Memory 314 optionally includes one or more storage devices located remotely from control circuitry 302. Memory 314, or alternatively, a non-volatile solid-state memory device within memory 314, comprises a non-transitory computer-readable storage medium. In some embodiments, memory 314, or the non-transitory computer-readable storage medium of memory 314, stores the following programs, modules, instructions, and data structures, or a subset or superset thereof: • An operating system 316, which contains procedures for handling various basic system services and performing hardware-dependent tasks; • A network communications module 318 used to connect the server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections); A coding module 320 for performing various functions related to encoding and / or decoding data, such as video data. In some embodiments, the coding module 320 is an instance of the coder component 114. The coding module 320 may include, but is not limited to, one or more of the following: a decoding module 322 for performing various functions related to decoding of encoded data, such as those described above with respect to the decoder component 122; and an encoding module 340 for performing various functions on the encoded data, such as those described above with respect to the encoder component 106; A picture memory 352 for storing pictures and picture data, e.g., for use with coding module 320. In some embodiments, picture memory 352 includes one or more of reference picture memory 208, buffer memory 252, current picture memory 264, and reference picture memory 266.

[0068] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions described above with respect to the parser 254), a transform module 326 (e.g., configured to perform various functions described above with respect to the scalar / inverse transform unit 258), a prediction module 328 (e.g., configured to perform various functions described above with respect to the motion compensation prediction unit 260 and / or the intra-picture prediction unit 262), and a filter module 330 (e.g., configured to perform various functions described above with respect to the loop filter unit 256).

[0069] In some embodiments, encoding module 340 includes a code module 342 (e.g., configured to perform various functions described above with respect to source coder 202, coding engine 212, and / or entropy coder 214) and a prediction module 344 (e.g., configured to perform various functions described above with respect to predictor 206). In some embodiments, decoding module 322 and / or encoding module 340 include a subset of the modules shown in Figure 3. For example, a shared prediction module is used by both decoding module 322 and encoding module 340.

[0070] Each of the above-identified modules stored in memory 314 corresponds to a set of instructions for performing functions described herein. The above-identified modules (e.g., sets of instructions) need not be implemented as separate software programs, procedures, or modules; thus, various subsets of these modules may be combined or otherwise rearranged in various embodiments. For example, coding module 320 optionally does not include separate decoding and encoding modules, but rather uses the same set of modules to perform both sets of functionality. In some embodiments, memory 314 stores a subset of the above-identified modules and data structures. In some embodiments, memory 314 stores additional modules and data structures not described above, such as an audio processing module.

[0071] In some embodiments, the server system 112 includes a web or Hypertext Transfer Protocol (HTTP) server, a File Transfer Protocol (FTP) server, and web pages and applications implemented using Common Gateway Interface (CGI) scripts, the PHP Hypertext Preprocessor (PHP), Active Server Pages (ASP), Hypertext Markup Language (HTML), Extensible Markup Language (XML), Java, JavaScript, Asynchronous JavaScript and XML (AJAX), XHP, Javelin, Wireless Universal Resource Files (WURFL), etc.

[0072] While FIG. 3 illustrates a server system 112 according to some embodiments, FIG. 3 is intended as a functional illustration of various features that may be present in one or more server systems, rather than a structural schematic of the embodiments described herein. In practice, those skilled in the art will recognize that items shown separately may be combined and some items may be separated. For example, some items shown separately in FIG. 3 may be implemented on a single server, and a single item may be implemented by one or more servers. The actual number of servers used to implement server system 112 and how functionality is allocated among them will vary from implementation to implementation and will, optionally, depend in part on the amount of data traffic the server system handles during peak usage periods as well as during average usage periods.

[0073] Exemplary Coating Approaches 4A-4D show exemplary coding tree structures according to some embodiments. As shown in the first coding tree structure (400) in FIG. 4A, some coding approaches (e.g., VP9) use a four-way partition tree from a 64x64 level down to a 4x4 level, with some additional restrictions on 8x8 blocks. In FIG. 4A, the partitions shown as R can be referred to as recursive in that the same partition tree is repeated at lower scales until the lowest 4x4 level is reached.

[0074] As shown in the second coding tree structure (402) of FIG. 4B, some coding approaches (e.g., AV1) expand the partition tree to a 10-way structure and increase the maximum size (e.g., referred to as a superblock in VP9 / AV1 terminology) to start at 128x128. The second coding tree structure includes a 4:1 / 1:4 rectangular partition not present in the first coding tree structure. The partition type with three subpartitions in the second row of FIG. 4B is called a T-shaped partition. The rectangular partitions in this tree structure cannot be further subdivided. In addition to the coding block size, a coding tree depth can be defined to indicate the division depth from the root node. For example, the coding tree depth of the root node, e.g., 128x128, is set to 0, and after the tree block is further divided, the coding tree depth increases by 1.

[0075] As an example, instead of enforcing a fixed transform unit size as in VP9, ​​AV1 allows for the partitioning of luma coding blocks into transform units of multiple sizes that can be represented by recursive partitions down to a maximum of two levels. To incorporate AV1's extended coding block partitions, transform sizes from 4x4 to 64x64 square, 2:1 / 1:2, and 4:1 / 1:4 are supported. For chroma blocks, only the largest possible transform unit is allowed.

[0076] As an example, a CTU may be divided into CUs by using a quad-tree structure, denoted as a coding tree, to accommodate various local characteristics, as in HEVC. In some embodiments, the decision of whether to code a picture area using inter-picture (temporal) prediction or intra-picture (spatial) prediction is made at the CU level. Depending on the PU partition type, each CU may be further divided into one, two, or four PUs. Within one PU, the same prediction process is applied, and related information is transmitted to the decoder on a PU-by-PU basis. After obtaining a residual block by applying a prediction process based on the PU partition type, the CU may be partitioned into TUs according to another quad-tree structure, such as the coding tree of the CU. One important feature of the HEVC structure is the concept of multiple partitions, including CUs, PUs, and TUs. In HEVC, a CU or TU can only be square, while a PU may be square or rectangular in shape for inter-predicted blocks. In HEVC, one coding block may be further divided into four square sub-blocks, and a transform is performed on each sub-block (TU). Each TU can be further divided recursively (using quadtree partitioning) into smaller TUs, which are called residual quadtrees (RQTs). At picture boundaries such as HEVC, implicit quadtree partitioning may be used so that blocks maintain their quadtree partitioning until their size fits the picture boundary.

[0077] A quadtree with nested multitype trees using binary and ternary split segmentation structures, such as VVC, may replace the concept of multiple partition unit types, eliminating the separation of CU, PU, ​​and TU concepts except as needed for CUs with sizes too large for the maximum transform length, and supporting more flexibility for CU partition shapes. In the coding tree structure, CUs can have either square or rectangular shapes. ACTUs are initially partitioned using a quadtree (also called a quadtree) structure. The quadtree leaf nodes can be further partitioned using a multitype tree structure. As shown in the third coding tree structure (404) in FIG. 4C, the multitype tree structure includes four split types. For example, the multitype tree structure includes vertical binary split (SPLIT_BT_VER), horizontal binary split (SPLIT_BT_HOR), vertical ternary split (SPLIT_TT_VER), and horizontal ternary split (SPLIT_TT_HOR). The leaf nodes of the multitype tree are called CUs, and this segmentation is used for prediction and transform processing without further partitioning, unless the CU is too large for the maximum transform length. This means that in most cases, the CU, PU, ​​and TU have the same block size in a quad-tree with a nested multitype tree coding block structure. An exception occurs when the supported maximum transform length is smaller than the width or height of the color components of the CU. An example of block partitioning for one CTU (406) is shown in Figure 4D, which shows an example of a quad-tree with a nested multitype tree coding block structure.

[0078] As in VVC, the maximum supported luma transform size is 64 x 64, and the maximum supported chroma transform size is 32 x 32. When the width or height of the CB is larger than the maximum transform width or height, the CB is automatically split horizontally and / or vertically to meet the transform size limit in that direction.

[0079] The coding tree scheme supports the ability for luma and chroma to have separate block tree structures, as in VTM7. In some cases, for P slices and B slices, the luma and chroma CTBs in one CTU share the same coding tree structure. However, for I slices, luma and chroma can have separate block tree structures. When the separate block tree mode is applied, the luma CTB is partitioned into CUs by one coding tree structure, and the chroma CTB is divided into chroma CUs by another coding tree structure. This means that a CU in an I slice can contain or consist of a coding block for the luma component or a coding block for two chroma components, and a CU in a P slice or B slice always contains or consists of coding blocks for all three color components unless the video is monochrome.

[0080] To support extended coding block partitions, like AV1, multiple transform sizes (e.g., ranging from 4 points to 64 points in each dimension) and transform shapes (e.g., square or rectangular with width / height ratios of 2:1 / 1:2 and 4:1 / 1:4) may be used.

[0081] In some embodiments, for each coded block in an inter frame, if the mode of the current block is an inter-coding mode rather than a skip mode, a separate flag is signaled to indicate whether a single reference mode or a mixed reference mode is used for the current block. In a single reference mode, a predictive block may be generated by one motion vector. In a mixed reference mode, a predictive block is generated by a weighted average of two predictive blocks derived from two motion vectors. The modes that may be signaled for the single reference case are detailed in Table 1 below. [Table 1] The modes that may be signaled for the decoding reference case are detailed in Table 2 below. [Table 2]

[0082] Some standards, such as AV1, allow for a motion vector accuracy (or precision) of 1 / 8 pixel. Syntax may be used to signal the motion vector difference in reference frame list zero (L0) or list 1 (L1), as follows: For example, the syntax mv_joint specifies which components of the motion vector difference are non-zero. A value of 0 for the syntax mv_joint indicates that there are no non-zero MVDs in either the horizontal or vertical direction, a value of 1 indicates that there are non-zero MVDs only in the horizontal direction, a value of 2 indicates that there are non-zero MVDs only in the vertical direction, and a value of 3 indicates that there are non-zero MVDs in both the horizontal and vertical directions. The syntax mv_sign specifies whether the motion vector difference is positive or negative. The syntax mv_class specifies the class of the motion vector difference. As shown in Table 3 below, a higher class means a larger magnitude of the motion vector difference. The syntax mv_bit specifies the integer part of the offset between the motion vector difference and the starting magnitude of each MV class. The syntax mv_fr specifies the first two fractional bits of the motion vector differential. The syntax mv_hp specifies the third fractional bit of the motion vector differential. [Table 3]

[0083] For NEW_NEARMV and NEAR_NEWMV modes (shown in Table 2 above), the MVD precision depends on the associated class and MVD magnitude. For example, fractional MVD is only allowed if the MVD magnitude is 1 pixel or less. In addition, only one MVD value is allowed when the associated MV class value is MV_CLASS_1 or greater, and the MVD value for each MV class is derived as 4, 8, 16, 32, or 64 for MV class 1 (MV_CLASS_1), MV class 2 (MV_CLASS_2), MV class 3 (MV_CLASS_3), MV class 4 (MV_CLASS_4), or MV class 5 (MV_CLASS_5). The allowed MVD values ​​for each MV class are shown in Table 4 below. [Table 4]

[0084] In some embodiments, one context is used to signal mv_joint or mv_class if the current block is coded using NEW_NEARMV or NEAR_NEWMV mode, and another context is used to signal mv_joint or mv_class if the current block is not coded using NEW_NEARMV or NEAR_NEWMV mode.

[0085] The inter-coding mode JOINT_NEWMV may be applied to indicate whether the MVDs of two reference lists are jointly signaled. When the inter-prediction mode is equal to JOINT_NEWMV mode, the MVDs of reference L0 and reference L1 are jointly signaled. In this way, only one MVD named joint_mvd may be signaled and transmitted to the decoder, and the delta MVs of reference L0 and reference L1 may be derived from joint_mvd. The JOINT_NEWMV mode is signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes.

[0086] When JOINT_NEWMV mode is signaled and the picture order count (POC) distances between two reference frames and the current frame are different, the MVD is scaled for reference L0 or reference L1 based on the POC distance. For example, let td0 be the distance between reference frame L0 and the current frame, and td1 be the distance between reference frame L1 and the current frame. If td0 is greater than or equal to td1, joint_mvd is used directly for reference L0, and the mvd of reference L1 is derived from joint_mvd based on Equation 1 below.

number

number

[0087] An inter-coding mode AMVDMV may be added to the single-reference case. When the AMVDMV mode is selected, this indicates that adaptive MVD resolution (AMVD) is applied to the signal MVD. A flag, such as amvd_flag, may be added under the JOINT_NEWMV mode to indicate whether AMVD is applied to the joint MVD coding mode. When adaptive MVD resolution is applied to the joint MVD coding mode, called joint AMVD coding, the MVDs for two reference frames are jointly signaled, and the accuracy of the MVD is implicitly determined by the size of the MVD. The MVDs for two (or more) reference frames are jointly signaled, and MVD coding is applied.

[0088] AMVR, first proposed in CWG-C012, supports a total of seven MV precisions (8, 4, 2, 1, 1 / 2, 1 / 4, and 1 / 8). For each prediction block, the AVM encoder searches all supported precision values ​​and signals the highest precision to the decoder. To reduce encoder runtime, two precision sets are supported. Each precision set contains four predefined precisions. The precision set is adaptively selected at the frame level based on the frame's maximum precision value. Similar to AV1, the maximum precision is signaled in the frame header. Table 5 summarizes the supported precision values ​​based on the frame-level maximum precision. [Table 5]

[0089] In the AOM Video Model (AVM), like AV1, there is a frame-level flag that indicates whether the MV of a frame contains sub-pel precision. AMVR is enabled only if the value of the cur_frame_force_integer_mv flag is 0. In AMVR, if the block precision is less than the maximum precision, the motion model and interpolation filter are not signaled. If the block precision is less than the maximum precision, the motion mode is estimated to translation motion and the interpolation filter is estimated to be the REGULAR interpolation filter. Similarly, if the block precision is either 4-pel or 8-pel, the inter-intra mode is not signaled and is assumed to be 0.

[0090] Motion compensation typically assumes a translational motion model between the reference block and the target block. However, warped motion utilizes an affine motion model. The affine motion model can be expressed as Equation 3 below:

number

[0091] In global warp motion compensation, global motion information is signaled for each inter reference frame, including a global motion type and some motion parameters. The global motion types and the associated number of parameters are listed in Table 6. [Table 6]

[0092] After signaling the reference frame index, if global motion is selected, the global motion type and parameters associated with the given reference frame are used for the current coding block.

[0093] In local warp motion compensation, local warp motion is allowed for inter-coding blocks when the following conditions are met: First, the current block must use single reference prediction. The width or height of the coding block must be equal to or greater than 8. Finally, at least one of the adjacent neighboring blocks must use the same reference frame as the current block.

[0094] When local warp motion is used for the current block, the affine model parameters are estimated by mean-squared minimization of the difference between the reference and modeled projections based on the MVs of the current block and its adjacent neighboring blocks. To estimate the parameters of the local warp motion, if the neighboring block uses the same reference frame as the current block, a projection sample pair is obtained: the center sample of the neighboring block and its corresponding sample in the reference frame. Then, three extra samples are created by shifting the center position by a quarter sample in one or both dimensions. These extra samples may also be considered as projection sample pairs to ensure the stability of the model parameter estimation process.

[0095] The MVs of neighboring blocks used to derive the motion parameters are called motion samples. The motion samples are selected from neighboring blocks that use the same reference frame as the current block. Note that warped motion prediction mode is only enabled for blocks that use a single reference frame.

[0096] 5A shows exemplary motion samples used to derive model parameters for a block using local warp motion prediction, according to some embodiments. As shown in FIG. 5A, the MVs of neighboring blocks B0, B1, and B2 are referred to as MV0, MV1, and MV2, respectively. The current block is predicted using single prediction with reference frame Ref0. For example, neighboring block B0 is predicted using mixed prediction with reference frames Ref0 and Ref1, neighboring block B1 is predicted using single prediction with reference frame Ref0, and neighboring block B2 is predicted using mixed prediction with reference frames Ref0 and Ref2. The motion vector MV0 of B0 is Ref0 , B1's motion vector MV1 Ref0 and B2's motion vector MV2 Ref0 may be used as the motion sample for deriving the affine motion parameters of the current block.

[0097] In addition to translational motion, AVM also supports warp motion compensation. Two types of warp motion models are supported: global warp model and local warp model. For example, a global warp model is associated with each reference frame. In this case, each of the four non-translational parameters has 12-bit precision, and translational motion vectors are coded with 15-bit precision. A coding block may choose to use it directly (a reference frame index is provided). The global warp model captures frame-level scaling and rotation. Therefore, the global warp model primarily focuses on rigid motion across the frame. A local warp model at the coding block level is also supported. In the local warp mode, also known as WARPED_CAUSAL, the warp parameters for the current block are derived by fitting the model to nearby motion vectors using a least-squares method.

[0098] In the warp motion mode WARP_EXTEND, the motion of neighboring blocks is smoothly extended into the current block, with some ability to modify the warp parameters. This allows complex warping motions to be expressed distributed across multiple blocks while minimizing blocking artifacts. To achieve this, the WARP_EXTEND mode applied to the NEWMV block constructs a new warp model based on two constraints: the per-pixel motion vectors generated by the new warp model must be contiguous with the per-pixel motion vectors in neighboring blocks, and the pixel at the center of the current block must have a per-pixel motion vector that matches the motion vector signaled for the entire block. Figure 5B illustrates motion vectors within a block using the warp extension mode, according to some embodiments. As shown in Figure 5B, if a neighboring block, for example, on the left side of the current block, is warped, a model that matches the motion vectors shown in Figure 5B is used as the warp model.

[0099] The two constraints for constructing a new warp model imply specific equations involving the warp parameters of the neighboring blocks and the current block. These equations can then be solved to calculate the warp model for the current block. For example, if (A,...,F) represents the neighboring warp model and (A',...,F') represents the new warp model, the first constraint is as follows for each point along the common edge:

number

number

[0100] The second constraint specifies that the motion vector of the block center must be equal to the one signaled using the NEWMV mechanism. This gives two more equations, resulting in a system of six equations in six variables with a unique solution. These equations can be solved efficiently in both software and hardware. The solution can be solved using basic addition, subtraction, multiplication, and division by powers of two. Therefore, this mode is significantly less complex than the least-squares-based local warp mode.

[0101] Note that there may be multiple neighboring blocks to extend from. Therefore, a method is needed to select which block to extend from. This problem arises in motion vector prediction as well. In particular, there may be several possible motion vectors from nearby blocks, one of which should be selected as the base for NEWMV coding. This solution can be extended to handle the need for WARP_EXTEND. This is done by tracking the source of each motion vector prediction. WARP_EXTEND is then enabled only if the selected motion vector prediction is obtained from a directly adjacent block. That block is then used as the single "neighboring block" for the rest of the algorithm.

[0102] Note that the neighbor's warp model may be very good as is, without needing any further modification. To make this case cheaper to code, WARP_EXTEND may be used for the NEARMV block. The neighbor selection is the same as for NEWMV, except that the NEWMV selection requires the neighbor to be warped (not just moved by a translational motion). However, if this condition is true and WARP_EXTEND is selected, the neighbor's warp model parameters are copied to the current block.

[0103] In some embodiments, the motion mode WARP_DELTA may be used. In this mode, the warp model of a block is coded as a delta from a predicted warp model, similar to how motion vectors are coded as deltas from predicted motion vectors. The prediction can come from either the global motion model (if present) or from neighboring blocks.

[0104] Constraints may be applied to avoid having multiple ways of coding the same prediction warp model. For example, if the mode is NEARMV or NEWMV, the same neighborhood selection logic as described for WARP_EXTEND is used. This ensures that if a neighboring block is warped, its model is used as the prediction (without applying the rest of the WARP_EXTEND logic). Otherwise, the global warp model is used as the base. Other constraints may also be applied. This example is not intended to limit the scope of the embodiments. Deltas for each of the non-translational parameters may then be coded. Finally, the translational part of the model is adjusted so that the motion vector for each pixel at the center of the block matches the motion vector for the entire block.

[0105] This tool (WARP_DELTA) uses more bits to encode than the other warp modes because it explicitly codes the delta for each warp parameter. Therefore, WARP_DELTA can be disabled for blocks smaller than 16x16. However, the decoding logic is very simple, and therefore it can express more complex motion than the other warp modes can.

[0106] FIG. 5C illustrates a current block with two spatial neighbors according to some embodiments. FIG. 5D illustrates a current block with four spatial neighbors according to some embodiments. FIG. 5E illustrates a current block with nine spatial neighbors according to some embodiments. In some embodiments, the numbering of neighboring blocks indicates the scan order. In some embodiments, warp extension mode is not always allowed at the syntax level. To check whether warp extension mode is allowed for the current block, an MVP list (dynamic reference list) needs to be constructed. MVP candidates indicated by MVP indexes are checked. For example, a candidate may be denoted as S. In this example, the position of S is checked, and if S is not from the top-left spatial neighbor (block 2 in FIG. 5C) or the top-left spatial neighbor (block 1 in FIG. 5C), warp extension is not allowed. Otherwise (if S is located at 1 or 2), the mode of the current block is checked. If the current block is NEWMV, warp extension is allowed; otherwise (for example, the current block is NEWMV), the motion mode of S is checked. If S is a warp (eg, a regular warp, an extended warp, or a delta warp), then warp extension is allowed; otherwise, warp extension is not allowed.

[0107] Similar to warp extension mode, in some embodiments, warp delta mode is also not always allowed. Allowing warp delta mode also depends on the MVP candidate. For example, the MVP candidate (S) pointed to by the MVP index is checked. If the current block uses NEWMV, warp delta mode is allowed. Otherwise, if the current block uses NEWMV, it is determined whether S is from a spatial neighborhood (e.g., it is determined that S is not from TMPV, scaled SMVP, or Ref MV bank, etc.). In this example, if S is spatial and S is a warp mode (e.g., normal warp, extended warp, or delta warp), warp delta mode is allowed; otherwise, warp delta mode is not allowed.

[0108] In some embodiments, two contexts are required to code and / or analyze the warp extension mode. For example, the first context is derived based on the motion mode of the MVP candidate (S) pointed to by the MVP index. If the mode of the current block is NEWMV, the first context is equal to 0. If the current block is NEWMV and S is a warp mode, the first context is equal to 1. If the current block is NEWMV and S is a global motion, the first context is equal to 2. If the current block is NEWMV and S is a translational motion, the first context is equal to 3. As an example, the second context depends on the current block boundary size adjacent to S.

[0109] The allowance of warp extension mode and warp delta mode, and the context derivation of warp extension mode each require MVP candidates pointed to by MVP index. Thus, during the analysis process, the entire MVP list (DRL list) needs to be constructed, which significantly delays the analysis process.

[0110] In some embodiments, when normal warp mode is allowed (eg, no MVP list needs to be constructed during the analysis phase), warp extension mode or warp delta mode is allowed.

[0111] In some embodiments, whether warp extension mode or warp delta mode is allowed depends on the availability and mode or motion mode of spatial neighboring blocks and the mode of the current block. In some embodiments, the number of neighboring blocks scanned and the locations of the neighboring blocks are predefined. In one example, spatial neighboring blocks are scanned and if one or more of the neighboring blocks are available and located within the same tile of the current block, warp extension mode or warp delta mode is allowed. In another example, spatial neighboring blocks are scanned and if one or more of the neighboring blocks are available, located within the same tile of the current block, and are in inter mode (e.g., not intra), warp extension or warp delta mode is allowed. In another example, spatial neighboring blocks are scanned and if one or more of the neighboring blocks are available, located within the same tile of the current block, coded by inter mode (e.g., not intra), and are in warp mode, warp extension or warp delta mode is allowed. In another example, spatial neighboring blocks are scanned, and if one or more of the neighboring blocks are available, located within the same tile of the current block, and coded by inter mode (e.g., not intra), the current block mode is further checked. If the current block is in NEWMV mode, warp extension mode or warp delta mode is allowed; otherwise, if the current block is in NEWMV mode and one of the spatial neighboring blocks is in warp mode, warp extension mode or warp delta mode of the current block is allowed. In another example, if the current block is in NEWMV mode, warp extension mode or warp delta mode is allowed; otherwise, the availability of spatial neighbors is checked, for example, as described in the previous example. In another example, whether warp extension mode is allowed and whether warp delta is allowed are determined by two different processes. In another example, scanning the neighboring blocks only includes the two upper and left neighbors of the current block, i.e., the spatial neighboring blocks located at (-1, 0) and (0, -1) as shown in FIG. 5C.In another example, the scanning of neighboring blocks includes only the four top and left neighbors of the current block, i.e., spatial neighboring blocks located at (-1,0), (0,-1), (height -1,-1), and (-1, width -1), as shown in Figure 5D. In another example, when scanning neighboring blocks, the positions of the neighboring blocks may depend on the block shape, and / or aspect ratio, and / or block size.

[0112] In some embodiments, one context is used during coding / analysis of warp extension mode. In some embodiments, this context is derived depending on the block size. For example, there are a total of 22 possible block sizes, and the context ranges from 0 to 21.

[0113] In some embodiments, one context is used during coding / parsing of the warp extension mode. In some embodiments, the context depends on the status and / or mode of spatial neighboring blocks. In some embodiments, the number of neighboring blocks scanned and the scan order are predefined. In one example, the context depends on the number of neighboring blocks using a warp mode. In another example, the context depends on the number of neighboring blocks using global motion. In another example, the context depends on the number of neighboring blocks using translational motion. In another example, the context depends on the number of neighboring blocks that have the same reference picture as the current block and that use warp mode, global motion, and / or translational motion. In another example, two or more contexts are used during coding and / or parsing of the warp extension mode. In some embodiments, the particular context depends on the status and / or mode of spatial neighboring blocks at a predefined location (e.g., block 2 in FIG. 5C ). In another example, as shown in FIG. 5C , the first context is derived from the status of block 1. For example, if the current block uses NEARMV mode, the context value is equal to 0. If block 1 is not available, is not in the same tile, is not coded using inter mode, or does not have the same reference frame as the current block, the context value is equal to 1. If block 1 uses warp mode, the context value is equal to 2. If block 1 uses global motion, the context value is equal to 3. If block 1 uses translational motion, the context value is equal to 4. In some embodiments, the same rules apply to a second context value that is derived based on the status of block 2.

[0114] 6 is a flow diagram illustrating a method 600 of video decoding according to some embodiments. Method 600 may be performed on a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and memory storing instructions for execution by the control circuitry. In some embodiments, method 600 is performed by executing instructions stored in a memory (e.g., memory 314) of the computing system. The term block as used herein may be interpreted as a prediction block, a coding block, or a coding unit (CU).

[0115] The system obtains coded video data including a plurality of blocks (602). In some embodiments, the system obtains an MVP candidate block from a motion vector predictor (MVP) list based on an MVP index. In accordance with a determination that a block of the plurality of blocks is designated for the warp extension mode, the system determines whether the MVP candidate block is suitable for the warp extension mode (604). In accordance with a determination that the MVP candidate block is not suitable for the warp extension mode, the system identifies a backup candidate block (606). In some embodiments, in accordance with a determination that the MVP candidate block is not suitable for the warp extension mode, the system scans a set of spatial neighboring blocks to identify a backup candidate block (608). In some embodiments, the system identifies a second backup candidate block, where the second backup candidate block is one of the MVP candidate block, an adjacent spatial neighboring block of the block, or a non-adjacent spatial neighboring block (610). The system obtains a warp model from the backup candidate block (612). In some embodiments, the warp model is derived from the backup candidate block and the second backup candidate block (614). The system performs a warp extension operation on the block using the warp model (616). In some embodiments, following a determination that the MVP candidate block is suitable for the warp extension mode, the system obtains a second warp model from the MVP candidate block and performs a warp extension scan on the block using the second warp model (618).

[0116] In some embodiments, warp extension mode is allowed, but if the MVP point candidate block is not suitable for warp extension (e.g., does not exist, is not in the same tile, and / or does not use a warp model when the current block is NEAR MVS), a backup candidate block is used as (or to derive) the base warp model for warp extension mode (and / or warp delta mode).

[0117] In some embodiments, the backup block used as the base warp model for warp extension is from one of the adjacent spatial neighboring blocks (e.g., block 1 or 2 in FIG. 5C). In one example, if the candidate block indicated by the MVP index is not suitable for warp extension, the neighboring blocks are scanned in a predefined order (e.g., the order shown in FIG. 5E). In this example, the first available (e.g., available and in the same tile) inter-coded block is used as the base warp model for the warp extension mode of the current block. In another example, spatial candidates are scanned, and the first available inter block having the same reference frame as the current block is used as the base for the warp extension mode of the current block. In another example, spatial candidates are scanned, and the inter block coded with the first warp mode is used as the base for the warp extension mode of the current block. In another example, spatial candidates are scanned, and the inter block coded with the first warp mode having the same reference picture as the current block is used as the base for the warp extension mode of the current block. In another example, spatial candidates are scanned, and if the current block is coded in NEWMV mode, the first available inter block with the same reference frame as the current block is used as the base for the warp extension mode of the current block. In this example, if the current block is coded in NEWMV mode, the first inter block coded in the first warp mode with the same reference picture as the current block is used as the base for the warp extension mode of the current block. In another example, if a base warp model has not yet been identified after scanning all available candidates, the global warp model is used as the base warp model.

[0118] In another example, scanning using an adaptive order of neighboring blocks is used. The scanning is performed from blocks close to the MVP point candidate (on both sides) toward blocks further away from the MVP point candidate. For example, as shown in FIG. 5E, if the MVP point candidate is located at 3, which is not coded or available using warp mode, the scanning order can be 8, 7, 1, 4, 5, 9, 2, 6 (or 7, 8, 4, 1, 9, 5, 2, 6). The first available or suitable block (e.g., coded in warp mode) is used as the base for warp extension mode. If the MVP candidate is non-adjacent or non-spatial, all blocks are scanned in a predefined order.

[0119] In another example, only a portion of the neighboring blocks are scanned, for example, if the MVP point candidates are spatially adjacent and from the left side of the current block, only 7, 3, 8, 1, and 5 are scanned. If the MVP point candidates are spatially adjacent and from the top side of the current block, only 7, 4, 9, 2, and 6 are scanned.

[0120] In another example, only a portion of the neighboring blocks are scanned, e.g., if the MVP point candidate is from the left side of the current block, only 7, 3, 8, 1, and 5 are scanned. If the MVP point candidate is from the top side of the current block, only 7, 4, 9, 2, and 6 are scanned.

[0121] In some embodiments, the search order of neighboring blocks (when the MVP point candidate does not provide a base warp model) may depend on the block shape or aspect ratio: for example, if the block is a vertical block (block height is greater than block width), the spatial candidate located on the left side is scanned first, and if the block is a horizontal block (block width is greater than block height), the spatial candidate located on the top side is scanned first.

[0122] In some embodiments, the backup used as the base for warp extension is from one of non-adjacent spatial neighboring blocks, history neighbors, or temporal co-located blocks. The first available, or available and coded in the warp mode candidate, is used as the base for warp extension mode. In some embodiments, the check order of non-adjacent spatial neighboring blocks, history neighbors, or temporal co-located blocks is predefined. In one example, the predefined check order is non-adjacent spatial neighboring blocks, then history neighbors, then temporal co-located blocks. In some embodiments, the check order of non-adjacent spatial neighboring blocks is adaptive using a method similar to that described above.

[0123] In some embodiments, there are multiple blocks used as a base warp model for warp extension, and these multiple blocks are from multiple adjacent spatial neighborhood blocks. In some embodiments, the multiple blocks include MVP point candidates if the MVP point candidates provide a valid warp base model. In some embodiments, the multiple blocks are from neighborhood blocks that provide a valid warp base model. In some embodiments, the warp base model is calculated using warp base models associated with the multiple blocks. In some embodiments, the multiple blocks are derived by scanning adjacent or non-adjacent neighborhood blocks in a predefined scan order or an adaptive scan order and checking whether a valid warp model is provided.

[0124] 6 depicts some logical stages in a particular order, but order-independent stages may be reordered and other stages may be combined or separated. Some reordering or other groupings not specifically mentioned will be apparent to those skilled in the art, and thus the ordering and groupings presented herein are not intended to be exhaustive. Furthermore, it should be recognized that the various stages may be implemented in hardware, firmware, software, or any combination thereof.

[0125] We now turn to some exemplary embodiments.

[0126] (A1) In one aspect, some embodiments include a method (e.g., method 600) for decoding video. In some embodiments, the method is implemented in a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is implemented in a coding module (e.g., coding module 320). In some embodiments, the method is implemented in a parser (e.g., parser 254). The method includes: (i) obtaining encoded video data including a plurality of blocks; (ii) obtaining a motion vector predictor (MVP) candidate block from an MVP list based on an MVP index; (iii) determining whether the MVP candidate block is suitable for the warp extension mode according to a determination that a block of the plurality of blocks is designated for the warp extension mode; (iv) identifying a backup candidate block suitable for the warp extension mode according to a determination that the MVP candidate block is not suitable for the warp extension mode; (v) obtaining a warp model from the backup candidate block; and (vi) performing a warp extension operation on the block using the warp model.

[0127] (A2) In some embodiments of A1, the method further includes: (i) obtaining a second warp model from the MVP candidate block according to a determination that the MVP candidate block is suitable for the warp extension mode; and (ii) performing a warp extension operation on the block using the second warp model.

[0128] (A3) In some embodiments of A1 or A2, obtaining the warp model from the backup candidate block includes deriving the warp model from the backup candidate block.

[0129] (A4) In some embodiments of A1 or A2, obtaining the warp model from the backup candidate block includes using the backup candidate block as the warp model.

[0130] (A5) In some embodiments of any of A1-A4, the backup candidate block is identified from a set of spatial neighboring blocks of the block.

[0131] (A6) In some embodiments of any of A1-A5, the MVP candidate block is determined to be unsuitable based on one or more of: (i) the MVP candidate block does not exist; (ii) the MVP candidate block is not in the same tile as the block; and (iii) the MVP candidate block does not use a warp model.

[0132] (A7) In some embodiments of any of A1 to A6, the method further includes: (i) determining whether the MVP candidate block is suitable for warp delta mode in accordance with a determination that a block of the plurality of blocks is designated for warp delta mode; (ii) obtaining a warp model from the backup candidate block in accordance with a determination that the MVP candidate block is not suitable for warp delta mode; and (iii) performing a warp delta operation on the block using the warp model.

[0133] (A8) In some embodiments of any of A1 to A7, the method further includes, in accordance with a determination that the MVP candidate block is not suitable for the warp extension mode, scanning the set of spatial neighboring blocks to identify a backup candidate block.

[0134] (A9) In some embodiments of A8, in accordance with determining that the first spatial neighboring block is designated as inter-coded, identify the first spatial neighboring block as a backup candidate block, e.g., the first spatial neighboring block is available (has the same reference frame) and is inter-coded.

[0135] (A10) In some embodiments of A8, identify the first spatial neighboring block as a backup candidate block in accordance with determining that the first spatial neighboring block is designated as warp mode coding and inter coding.

[0136] (A11) In some embodiments of any of A8 to A10, identifying the first spatial neighboring block as a backup candidate block according to determining that the first spatial neighboring block has the same reference picture as the block.

[0137] (A12) In some embodiments of any of A8-A11, the scan stops once a backup candidate block is identified, e.g., once a suitable candidate block is identified or once the complete set of spatial neighboring blocks has been analyzed.

[0138] (A13) In some embodiments of any of A8 to A12, the scanning order for scanning is based on the relative positions of the MVP candidate blocks. For example, if the MVP point candidate is located at 3, the scanning order can be 8, 7, 1, 4, 5, 9, 2, 6 (or 7, 8, 4, 1, 9, 5, 2, 6).

[0139] (A14) In some embodiments of any of A8 to A13, the scan order for the scan is a predefined order.

[0140] (A15) In some embodiments of any of A8-A14, the set of spatial neighboring blocks is identified based on the relative position of the MVP candidate block. For example, if the MVP point candidate is the neighboring space from the left side of the block, only 7, 3, 8, 1, and 5 are scanned. If the MVP point candidate is the neighboring space from above the block, only 7, 4, 9, 2, and 6 are scanned.

[0141] (A16) In some embodiments of any of A8 to A15, the scan order for the scan is based on one or more characteristics of the blocks. In some embodiments, the one or more characteristics include block shape and / or aspect ratio. For example, if the block is a vertical block (block height is greater than block width), the spatial candidate located on the left is scanned first, and if the block is a horizontal block (block width is greater than block height), the spatial candidate located on the top is scanned first.

[0142] (A17) In some embodiments of any of A1 to A16, the method further includes a step of (i) determining whether the block is coded for a nearby motion vector (NEARMV) mode or a new motion vector (NEWMV) mode; (ii) following a determination that the block is coded for a NEWMV mode, a backup candidate block is identified further according to a determination that the backup candidate block is designated as being inter-coded and has the same reference frame as the block; and (iii) following a determination that the block is coded for a NEARMV mode, a backup candidate block is identified further according to a determination that the backup candidate block is designated as being inter-coded in warp mode and has the same reference picture as the block.

[0143] (A18) In some embodiments of any of A1 to A17, the method further includes: (i) determining whether the MVP candidate block is suitable for the warp extension mode in accordance with a determination that a second block of the plurality of blocks is designated for the warp extension mode; and (ii) performing a warp extension operation on the second block using the global warp model in accordance with a determination that the MVP candidate block is not suitable for the warp extension mode and a determination that a backup candidate block has not been identified.

[0144] (A19) In some embodiments of any of A1-A18, the backup candidate block is a non-adjacent spatial neighbor of the block. In some embodiments, identifying the backup candidate block includes scanning a set of adjacent spatial neighbors of the block, and, according to a determination that the set of adjacent spatial neighbors does not include the backup candidate block, scanning a set of non-adjacent spatial neighbors. In some embodiments, the set of non-adjacent spatial neighbors is scanned in a predefined order. In some embodiments, the order of scanning the non-adjacent spatial neighbors is adaptive (e.g., based on the position of the MVP candidate block).

[0145] (A20) In some embodiments of any of A1 to A19, the backup candidate block is a historical neighboring block of the block.

[0146] (A21) In some embodiments of any of A1-A20, the backup candidate block is a temporary co-located block of the block. In some embodiments, identifying the backup candidate block includes scanning, for the block, a set of contiguous spatial neighboring blocks, then a set of non-contiguous spatial neighboring blocks, then a set of historical neighboring blocks, and then a set of temporary co-located blocks until the backup candidate block is identified.

[0147] (A22) In some embodiments of any of A1 to A21, the method further includes identifying a second backup candidate block, wherein the second backup candidate block is one of an MVP candidate block, an adjacent spatial neighboring block of the block, or a non-adjacent spatial neighboring block.

[0148] (A23) In some embodiments of A22, the warp model is derived from the backup candidate block and the second backup candidate block, e.g., the warp model is a collection of warp-based models of the backup candidate block and the second backup candidate block, respectively.

[0149] (A24) In some embodiments of A22 or A23, a second back-up candidate block is identified according to a determination that the second back-up candidate block provides a valid warp base model.

[0150] The methods described herein may be used separately or combined in any order. Each of the methods may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In some embodiments, the processing circuitry executes a program stored on a non-transitory computer-readable medium.

[0151] In another aspect, some embodiments include a computing system (e.g., server system 112) including control circuitry (e.g., control circuitry 302) and a memory (e.g., memory 314) coupled to the control circuitry, the memory storing one or more sets of instructions configured to be executed by the control circuitry, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1-A24 above).

[0152] In yet another aspect, some embodiments include a non-transitory computer-readable storage medium storing one or more sets of instructions for execution by control circuitry of a computing system, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1-A24 above).

[0153] It will be understood that although terms such as "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another.

[0154] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the claims. When used in the description of the embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that as used herein, the term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items. It will be further understood that as used herein, the terms "comprises" and / or "comprising" specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0155] As used herein, the term "if" may be interpreted to mean "when" or "assuming" or "response to determining" or "in accordance with a determination that" or "in response to detecting" a stated condition precedent is true, depending on the context. Similarly, the phrases "determine [the stated condition precedent is true]" or "if [the stated condition precedent is true]" or "if [the stated condition precedent is true]" may be interpreted to mean "determine" or "in response to determining" or "in accordance with a determination that" or "detect" or "in response to detecting" a stated condition precedent is true, depending on the context.

[0156] The foregoing description has been described with reference to specific embodiments for purposes of explanation. However, the exemplary discussions above are not intended to be exhaustive or to limit the scope of the claims to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments have been chosen and described in order to best explain the principles of operation and practical applications, thereby enabling those skilled in the art.

Claims

1. 1. A method of video decoding executed in a computing system having a memory and one or more processors, the method comprising: obtaining encoded video data comprising a plurality of blocks; obtaining a motion vector predictor (MVP) candidate block from an MVP list based on an MVP index; determining whether the MVP candidate block is suitable for the warp extension mode in accordance with a determination that a block of the plurality of blocks is designated for the warp extension mode; identifying a backup MVP candidate block suitable for the warp extension mode in accordance with a determination that the MVP candidate block is not suitable for the warp extension mode; obtaining a warp model from the backup MVP candidate block; performing a warp extension operation on the block using the warp model; A method comprising:

2. obtaining a second warp model from the MVP candidate block according to determining that the MVP candidate block is suitable for the warp extension mode; performing a warp extend operation on the block using the second warp model; The method of claim 1 further comprising:

3. obtaining the warp model from the backup MVP candidate block includes deriving the warp model from the backup MVP candidate block; The method of claim 1.

4. obtaining the warp model from the backup MVP candidate block includes using the backup MVP candidate block as the warp model; The method of claim 1.

5. the backup MVP candidate block is identified from a set of spatial neighboring blocks of the block; The method of claim 1.

6. The MVP candidate block is The MVP candidate block does not exist; the MVP candidate block is not within the same tile as the block; The MVP candidate block does not use a warp model; is determined to be unsuitable based on one or more of the following: The method of claim 1.

7. determining whether the MVP candidate block is suitable for the warp delta mode in accordance with a determination that the block of the plurality of blocks is designated for the warp delta mode; obtaining the warp model from the backup MVP candidate block according to a determination that the MVP candidate block is not suitable for the warp delta mode; performing a warp delta operation on the block using the warp model; The method of claim 1 further comprising:

8. scanning a set of spatial neighboring blocks in a predefined order to identify a first available inter-coded block as the backup MVP candidate block according to a determination that the MVP candidate block is not suitable for the warp extension mode; The method of claim 1 further comprising:

9. determining whether the block is coded for a nearby motion vector (NEARMV) mode or a new motion vector (NEWMV) mode; In accordance with a determination that the block is coded for the NEWMV mode, the backup MVP candidate block is identified further in accordance with a determination that the backup MVP candidate block is designated as inter-coded and has the same reference frame as the block; In accordance with a determination that the block is coded for the NEAR MVP mode, the backup MVP candidate block is identified further in accordance with a determination that the backup MVP candidate block is designated as being inter-coded in warp mode and has the same reference picture as the block. The method of claim 1.

10. determining whether the MVP candidate block is suitable for the warp extension mode in accordance with a determination that a second block of the plurality of blocks is designated for the warp extension mode; performing a warp extension operation on the second block using a global warp model in accordance with a determination that the MVP candidate block is not suitable for the warp extension mode and a determination that a backup MVP candidate block has not been identified; The method of claim 1 further comprising:

11. The backup MVP candidate block is a non-adjacent spatial neighbor of the block. The method of claim 1.

12. The backup MVP candidate block is a historical neighboring block of the block. The method of claim 1.

13. the backup MVP candidate block is a temporary co-located block of the block; The method of claim 1.

14. further comprising identifying a second backup MVP candidate block, the second backup MVP candidate block being one of the MVP candidate block, an adjacent spatial neighboring block of the block, or a non-adjacent spatial neighboring block. The method of claim 1.

15. the warp model is derived from the backup MVP candidate block and the second backup MVP candidate block; 15. The method of claim 14.

16. 1. A computing system comprising: a control circuit; Memory and one or more sets of instructions stored in the memory and configured for execution by the control circuitry; 16. A computing system comprising: a processor configured to: generate a programmable logic circuit for executing a programmable logic circuit;

17. A computer program which, when executed by a computing device, causes the computing device to perform the method of any one of claims 1 to 15.

Images