System and method for twisted sample selection and grouping

Abstract

Various embodiments herein include methods and systems of video encoding and decoding. In one aspect, a method includes obtaining video data comprising a plurality of blocks, the plurality of blocks including a first block, where the first block is to be encoded in a warp mode or a motion mode. The method also includes identifying two or more twist groups for the first block and determining a respective cost associated with each of the two or more twist groups. The method further includes selecting a twist group from the two or more twist groups based on the respective costs, and encoding the first block and a syntax element indicating the twist mode or the motion mode according to the selected twist group irrespective of whether a valid motion vector is available.

Description

Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 401,477, filed on August 26, 2022, entitled “Method and Apparatus for Warp Samples Selection and Grouping Improvements,” and is a continuation of and claims priority to U.S. Patent Application No. 18 / 143,520, filed on May 4, 2023, entitled “Systems and Methods for Warp Sample Selection and Grouping,” all of which are hereby incorporated by reference in their entirety. Technical Field

[0002] Embodiments of the present disclosure relate generally to video encoding and decoding, including but not limited to systems and methods for warped sample selection and grouping. Background Art

[0003] Digital video is supported by various electronic devices such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc. Electronic devices transmit and receive digital video data over a communication network or otherwise transmit digital video data, and / or store digital video data on a storage device. Due to the limited bandwidth capacity of a communication network and the limited memory resources of a storage device, video encoding may be used to compress the video data according to one or more video encoding standards before transmitting or storing the video data.

[0004] A variety of video codec standards have been developed. For example, video coding standards include the Alliance for Open Media Video 1 (AV1), Versatile Video Coding (VVC), Joint Exploration Test Model (JEM), High Efficiency Video Coding (HEVC / H.265), Advanced Video Coding (AVC / H.264), and Moving Picture Experts Group (MPEG) coding. Video coding typically uses prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that exploit the redundancy inherent in video data. Video coding aims to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation of video quality.

[0005] HEVC (also known as H.265) is a video compression standard designed as part of the MPEG-H project. ITU-T and ISO / IEC published the HEVC / H.265 standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). Versatile Video Coding (VVC), also known as H.266, is a video compression standard intended as a successor to HEVC. ITU-T and ISO / IEC published the VVC / H.266 standard in 2020 (version 1) and 2022 (version 2). AV1 is an open video coding format designed as a replacement for HEVC. On January 8, 2019, Confirmed Version 1.0.0 of the specification with Errata 1 was released. Summary of the invention

[0006] As described above, a video stream may be encoded into a code stream that involves compression and then transmitted to a decoder, which may decode / decompress the video stream for viewing or further processing. Compression of a video stream may exploit spatial and temporal correlations of a video signal through spatial and / or motion compensated prediction. Motion compensated prediction may include inter-frame prediction. Inter-frame prediction may use one or more motion vectors to generate a block to be encoded using previously encoded and decoded pixels. A decoder that receives the encoded signal may recreate the block.

[0007] Not all motion on an image is translational motion. Therefore, the translational motion model cannot accurately describe more complex motions, such as rotation, zoom, and shear. To overcome this shortcoming, the warp motion model can be used, which uses an affine model. However, it is important to determine when (e.g., on which blocks) to allow warp motion and which warp samples to use for the warp model.

[0008] According to some embodiments, a method for encoding a video is provided. The method includes: (i) obtaining video data including a plurality of blocks, the plurality of blocks including a first block; (ii) identifying two or more warp groups for the first block; (iii) determining a corresponding cost associated with each of the two or more warp groups; (iv) selecting a warp group from the two or more warp groups based on the corresponding costs; and (v) encoding the first block according to the selected warp group.

[0009] According to some embodiments, a method for decoding a video is provided, the method comprising: (i) obtaining video data including a plurality of blocks from a bitstream, the plurality of blocks including a first block; (ii) identifying a selected warp group from two or more warp groups; and (iii) decoding the first block according to the selected warp group.

[0010] According to some embodiments, a computing system is provided, such as a streaming system, a server system, a personal computer system, or other electronic device. The computing system includes a control circuit and a memory storing one or more groups of instructions. One or more groups of instructions include instructions for performing any method described herein. In some embodiments, the computing system includes an encoder component and / or a decoder component.

[0011] According to some embodiments, a non-volatile computer-readable storage medium is provided. The non-volatile computer-readable storage medium stores one or more sets of instructions for execution by a computing system. The one or more sets of instructions include instructions for executing any of the methods described herein.

[0012] Therefore, a device and system having a method for encoding / decoding a video is disclosed. Such methods, devices and systems can supplement or replace conventional methods, apparatuses and systems for video encoding / decoding.

[0013] The features and advantages described in the specification are not necessarily all-inclusive, and in particular, some additional features and advantages will be apparent to those of ordinary skill in the art in view of the drawings, descriptions, and claims provided in this disclosure. In addition, it should be noted that the language used in the specification is primarily selected for readability and instructional purposes, and is not necessarily selected to describe or limit the subject matter described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to understand the present disclosure in more detail, a more specific description may be made with reference to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the accompanying drawings illustrate only the relevant features of the present disclosure and are therefore not necessarily considered to be limiting, as those skilled in the art will understand after reading the present disclosure that the description may allow for other effective features.

[0015] Figure 1 is a block diagram illustrating an example communication system in accordance with some embodiments.

[0016] Figure 2A is a block diagram illustrating example elements of an encoder component according to some embodiments.

[0017] Figure 2B is a block diagram illustrating example elements of a decoder component according to some embodiments.

[0018] Figure 3 is a block diagram illustrating an example server system in accordance with some embodiments.

[0019] FIG. 4A to FIG. 4D An example coding tree structure according to some embodiments is illustrated.

[0020] Figure 5AIllustrated are MMDV search points for two reference frames according to some embodiments.

[0021] Figure 5B Illustrated are example motion samples for deriving model parameters for a block using warp motion prediction according to some embodiments.

[0022] Figure 5C Illustrated are example motion vectors in a block using a warp motion mode in accordance with some embodiments.

[0023] Figure 5D A current block with four spatial neighboring blocks is illustrated according to some embodiments.

[0024] Figure 5E A current block with ten neighboring blocks is illustrated according to some embodiments.

[0025] Fig. 6A is a flow chart illustrating an example method of encoding a video according to some embodiments.

[0026] Figure 6B is a flow chart illustrating an example method of decoding a video according to some embodiments.

[0027] In accordance with common practice, the various features illustrated in the drawings are not necessarily drawn to scale, and like reference numerals may be used to denote like features throughout the specification and drawings. DETAILED DESCRIPTION

[0028] The present disclosure describes, inter alia, improvements to warp motion modes in video encoding and decoding, such as by considering multiple warp groups and selecting the warp group with the lowest loss (highest accuracy) during encoding / decoding. In some cases, coding efficiency is improved by signaling (e.g., via one or more flags) whether multiple warp modes are enabled and / or which warp group or groups are selected. In some cases, coding efficiency is improved by deriving (e.g., using context information) the selected warp group to be used during encoding / decoding. Example Systems and Devices

[0029] Figure 1 1 is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic device 120-1 to electronic device 120-m) communicatively coupled to each other via one or more networks. In some embodiments, the communication system 100 is a streaming system used, for example, with video-enabled applications such as video conferencing applications, digital television applications, and media storage and / or distribution applications.

[0030] Source device 102 includes a video source 104 (e.g., a camera component or a media storage device) and an encoder component 106. In some embodiments, video source 104 is a digital camera (e.g., configured to create an uncompressed video sample stream). Encoder component 106 generates one or more encoded video streams from the video stream. The video stream from video source 104 can have a high data volume compared to the encoded video stream 108 generated by encoder component 106. Because the encoded video stream 108 has a lower data volume (less data) than the video stream from the video source, the encoded video stream 108 requires less bandwidth to transmit and requires less storage space to store than the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video data to one or more networks 110).

[0031] One or more networks 110 represent any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (wired) and / or wireless communication networks. One or more networks 110 can exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks, and / or the Internet.

[0032] One or more networks 110 include a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is or includes a streaming server (e.g., configured to store and / or distribute video content such as an encoded video stream from the source device 102). The server system 112 includes an encoder component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the encoder component 114 includes an encoder component and / or a decoder component. In various embodiments, the encoder component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the encoder component 114 is configured to decode the encoded video stream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings from the encoded video stream 108.

[0033] In some embodiments, the server system 112 functions as a media aware network element (MANE). For example, the server system 112 can be configured to prune the encoded video stream 108 to cut a potentially different stream for one or more of the electronic devices 120. In some embodiments, the MANE is provided separately from the server system 112.

[0034] The electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode the encoded video data 116 to generate an output video stream that can be rendered on a display or other type of rendering device. In some embodiments, one or more of the electronic devices 120 do not include a display component (e.g., communicatively coupled to an external display device and / or include a media storage device). In some embodiments, the electronic device 120 is a streaming client. In some embodiments, the electronic device 120 is configured to access the server system 112 to obtain the encoded video data 116.

[0035] The source device and / or the plurality of electronic devices 120 are sometimes referred to as “end devices” or “user devices.” In some embodiments, one or more of the source device 102 and / or the electronic devices 120 are examples of server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing devices, and / or other types of electronic devices.

[0036] In an example operation of the communication system 100, the source device 102 transmits an encoded video stream 108 to the server system 112. For example, the source device 102 can encode a picture stream captured by the source device. The server system 112 receives the encoded video stream 108 and can decode and / or encode the encoded video stream 108 using an encoder component 114. For example, the server system 112 can apply a coding that is more optimized for network transmission and / or storage to the video data. The server system 112 can transmit the encoded video data 116 (e.g., one or more encoded video streams) to one or more of the electronic devices 120. Each electronic device 120 can decode the encoded video data 116 to recover and optionally display the video picture.

[0037] In some embodiments, the transmission discussed above is a unidirectional data transmission. Unidirectional data transmission is sometimes used for media service applications, etc. In some embodiments, the transmission discussed above is a bidirectional data transmission. Bidirectional data transmission is sometimes used for video conferencing applications, etc. In some embodiments, the encoded video code stream 108 and / or the encoded video data 116 are encoded and / or decoded according to any of the video encoding / compression standards described herein, such as HEVC, VVC, and / or AV1.

[0038] Figure 2A1 is a block diagram illustrating example elements of an encoder component 106 according to some embodiments. The encoder component 106 receives a source video sequence from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., a transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives the video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder component 106). The video source 104 can provide the source video sequence in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8 bits, 10 bits, or 12 bits), any color space (e.g., BT.601 Y CrCb or RGB), and any suitable sampling structure (e.g., Y CrCb 4:2:0 or Y CrCb 4:4:4). In some embodiments, the video source 104 is a storage device that stores previously captured / prepared videos. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. The video data can be provided as a plurality of separate pictures that produce motion when viewed in sequence. The picture itself can be organized as a spatial array of pixels, where each pixel can include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples can be easily understood by those of ordinary skill in the art. The following description focuses on examples.

[0039] The encoder component 106 is configured to encode and / or compress the pictures of the source video sequence into an encoded video sequence 216 in real time or under other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller 204. In some embodiments, the controller 204 controls other functional units as described below and is functionally coupled to the other functional units. The parameters set by the controller 204 may include parameters related to rate control (e.g., picture skipping, λ value of quantizer and / or rate distortion optimization techniques), picture size, picture group (GOP) layout, maximum motion vector search range, etc. Other functions of the controller 204 can be easily identified by those of ordinary skill in the art as they may be related to the encoder component 106 optimized for a particular system design.

[0040] In some embodiments, the encoder component 106 is configured to operate in an encoding loop. In a simplified example, the encoding loop includes a source encoder 202 (e.g., responsible for creating symbols, such as a symbol stream, based on the input picture to be encoded and the reference picture) and a (local) decoder 210. The decoder 210 reconstructs the symbols to create sample data in a similar manner to the (remote) decoder (when the compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to the reference picture memory 208. Since the decoding of the symbol stream leads to bit-accurate results independent of the decoder location (local or remote), the contents of the reference picture memory 208 are also bit-accurate between the local encoder and the remote encoder. In this way, the prediction part of the encoder interprets the same sample values ​​as the sample values ​​interpreted by the decoder when using prediction during decoding as reference picture samples. The principles of reference picture synchronization (and the resulting drift if the synchronization cannot be maintained, for example due to channel errors) are known to those of ordinary skill in the art.

[0041] The operation of the decoder 210 may be the same as the operation of the remote decoder, such as the decoder component 122, which will be described below in conjunction with Figure 2B However, a brief reference Figure 2B , since symbols are available and the entropy encoder 214 and parser 254 can be lossless in encoding / decoding symbols for the encoded video sequence, the entropy decoding portion of the decoder component 122, including the buffer memory 252 and the parser 254, may not be fully implemented in the local decoder 210.

[0042] At this point it can be observed that, in addition to parsing / entropy decoding present in a decoder, any decoder techniques must also be present in substantially the same functional form in the corresponding encoder. Thus, the disclosed subject matter focuses on decoder operation. The descriptions of encoder techniques may be abbreviated, as they are the inverse of the fully described decoder techniques. A more detailed description is only required in certain areas and is provided below.

[0043] As part of its operation, source encoder 202 may perform motion compensated predictive coding, which predictively encodes an input frame with reference to one or more previously encoded frames in a video sequence designated as reference frames. In this manner, encoding engine 212 encodes the difference between a pixel block of an input frame and a pixel module of a reference frame that may be selected as a predictive reference for the input frame. Controller 204 may manage the encoding operations of source encoder 202, including, for example, the setting of parameters and subgroup parameters for encoding video data.

[0044] The decoder 210 decodes the encoded video data of the frame that can be designated as the reference frame based on the symbols created by the source encoder 202. The operation of the encoding engine 212 can advantageously be a lossy process. Figure 2A When the encoded video data is decoded at a remote video decoder (not shown), the reconstructed video sequence may be a copy of the source video sequence with some errors. The decoder 210 replicates the decoding process that may be performed on the reference frame by the remote video decoder and may cause the reconstructed reference frame to be stored in the reference picture memory 208. In this way, the encoder component 106 locally stores a copy of the reconstructed reference frame with common content as the reconstructed reference frame to be obtained by the remote video decoder (without transmission errors).

[0045] The predictor 206 may perform a prediction search for the encoding engine 212. That is, for a new frame to be encoded, the predictor 206 may search the reference picture memory 208 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may be used as appropriate prediction references for the new picture. The predictor 206 may operate on a block of samples, pixel block by pixel block, to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor 206, the input picture may have a prediction reference drawn from multiple reference pictures stored in the reference picture memory 208.

[0046] The outputs of all of the above functional units may be entropy encoded in entropy encoder 214. Entropy encoder 214 converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those of ordinary skill in the art (e.g., Huffman coding, variable length coding, and / or arithmetic coding).

[0047] In some embodiments, the output of the entropy encoder 214 is coupled to a transmitter. The transmitter can be configured to buffer the coded video sequences created by the entropy encoder 214 to prepare them for transmission via a communication channel 218, which can be a hardware / software link to a storage device storing the coded video data. The transmitter can be configured to merge the coded video data from the source encoder 202 with other data to be transmitted, such as coded audio data and / or an auxiliary data stream (source not shown). In some embodiments, the transmitter can send additional data along with the coded video. The source encoder 202 can include such data as part of the coded video sequence. The additional data can include time / space / SNR enhancement layers, other forms of redundant data, such as redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, etc.

[0048] The controller 204 can manage the operation of the encoder component 106. During encoding, the controller 204 can assign a specific encoding picture type to each encoded picture, which may affect the encoding technology applied to the corresponding picture. For example, a picture can be designated as an intra picture (I picture), a predicted picture (P picture), or a bidirectional predicted picture (B picture). An intra picture can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh (IDR) pictures. Those of ordinary skill in the art know these variants of I pictures and their respective applications and features, so they are not repeated here. Predicted pictures can be encoded and decoded using intra prediction or inter prediction, using at most one motion vector and reference index to predict the sample values ​​of each block. Bidirectional predicted pictures can be encoded and decoded using intra prediction or inter prediction, using at most two motion vectors and reference indexes to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and related metadata to reconstruct a single block.

[0049] The source picture may typically be spatially subdivided into a number of blocks of samples (e.g., each block of samples is 4×4, 8×8, 4×8, or 16×16) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded or predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be non-predictively coded by spatial prediction or by temporal prediction with reference to one previously coded reference picture. Blocks of a B picture may be non-predictively coded by spatial prediction or by temporal prediction with reference to one or two previously coded reference pictures.

[0050] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-frame prediction (often abbreviated as intra-frame prediction) exploits spatial correlations in a given picture, while inter-frame prediction exploits (temporal or other) correlations between pictures. In one example, a particular picture in encoding / decoding (called the current picture) is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture can be encoded with a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case of using multiple reference pictures, can have a third dimension that identifies the reference picture.

[0051] The encoder component 106 may perform encoding operations according to a predetermined video encoding technique or standard, such as any described herein. In its operation, the encoder component 106 may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0052] Figure 2B is a block diagram illustrating example elements of decoder component 122 according to some embodiments. Figure 2B The decoder component 122 in is coupled to the channel 218 and the display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to the loop filter unit 256 and configured to transmit data to the display 124 (eg, via a wired or wireless connection).

[0053] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver can be configured to receive one or more encoded video sequences to be decoded by decoder component 122. In some embodiments, the decoding of each encoded video sequence is independent of other encoded video sequences. Each encoded video sequence can be received from channel 218, which can be a hardware / software link to a storage device storing encoded video data. The receiver can receive encoded video data with other data (e.g., encoded audio data and / or auxiliary data streams), which can be forwarded to their respective use entities (not depicted). The receiver can separate the encoded video sequence from the other data. In some embodiments, the receiver receives additional (redundant) data together with the encoded video. The additional data can be included as part of one or more encoded video sequences. Decoder component 122 can use the additional data to decode the data and / or more accurately reconstruct the original video data. The additional data can be in the form of, for example, time, space or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0054] According to some embodiments, decoder component 122 includes a buffer memory 252, a parser 254 (sometimes also referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra prediction unit 262, a motion compensated prediction unit 260, an aggregator 268, a loop filter unit 256, a reference picture memory 266, and a current picture memory 264. In some embodiments, decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuits. In some embodiments, decoder component 122 is implemented at least partially in software.

[0055] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to combat network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, in addition to buffer memory 252 internal to decoder component 122 (e.g., which is configured to handle playout timing), a separate buffer memory is provided external to decoder component 122 (e.g., to combat network jitter). When receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, buffer memory 252 may not be required or may be very small. For use on a best-effort packet network such as the Internet, buffer memory 252 may be necessary, which buffer memory 252 may be relatively large and may advantageously have an adaptive size and may be implemented at least in part in an operating system or similar element (not depicted) external to decoder component 122.

[0056] The parser 254 is configured to reconstruct the symbols 270 from the encoded video sequence. The symbols may include, for example, information for managing the operation of the decoder component 122, and / or information for controlling a rendering device such as the display 124. The control information for one or more rendering devices may be in the form of, for example, a supplemental enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not depicted). The parser 254 parses (entropy decodes) the encoded video sequence. The encoding of the encoded video sequence may be in accordance with a video encoding technique or standard, and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 254 may extract a set of subgroup parameters for at least one subgroup of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the set. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser 254 may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0057] The reconstruction of the symbol 270 may involve a number of different units, depending on the type of the coded video picture or portion thereof (such as: inter- and intra-pictures, inter- and intra-blocks) and other factors. Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by the parser 254. For clarity, such subgroup control information flow between the parser 254 and the multiple units is not depicted below.

[0058] In addition to the functional blocks already mentioned, decoder component 122 can be conceptually subdivided into a plurality of functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with each other. However, in order to describe the disclosed subject matter, the conceptual subdivision of the following functional units is maintained.

[0059] The sealer / inverse transform unit 258 receives the quantized transform coefficients and control information (such as the transform to be used, block size, quantization factor, and / or quantization scaling matrix) from the parser 254 as one or more symbols 270. The sealer / inverse transform unit 258 may output a block including sample values ​​that may be input into the aggregator 268.

[0060] In some cases, the output samples of the scaler / inverse transform unit 258 belong to intra-coded blocks; that is, blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information can be provided by the intra-frame prediction unit 262. The intra-frame picture prediction unit 262 can use the surrounding already reconstructed information obtained from the current (partially reconstructed) picture from the current picture memory 264 to generate a block of the same size and shape as the block under reconstruction. The aggregator 268 can add the prediction information that has been generated by the intra-frame prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258 on a per-sample basis.

[0061] In other cases, the output samples of the scaler / inverse transform unit 258 belong to an inter-frame coded and possibly motion compensated block. In this case, the motion compensated prediction unit 260 can access the reference picture memory 266 to obtain samples for prediction. After the obtained samples are motion compensated according to the symbol 270 belonging to the block, these samples can be added to the output of the scaler / inverse transform unit 258 (in this case, called residual samples or residual signals) by the aggregator 268 to generate output sample information. The address at which the motion compensated prediction unit 260 obtains the predicted samples from the reference picture memory 266 can be controlled by a motion vector. The motion vector can be used for the motion compensated prediction unit 260 in the form of a symbol 270, which can have, for example, X, Y and reference picture components. Motion compensation can also include interpolation of sample values ​​obtained from the reference picture memory 266 when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0062] The output samples of aggregator 268 may be subjected to various loop filtering techniques in loop filter unit 256. The video compression techniques may include in-loop filter techniques that are controlled by parameters included in the coded video bitstream and available to loop filter unit 256 as symbols 270 from parser 254, but may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of a coded picture or coded video sequence, and to previously reconstructed and loop filtered sample values.

[0063] The output of the loop filter unit 256 may be a sample stream that may be output to a rendering device such as the display 124 , as well as stored in the reference picture memory 266 for use in future inter-frame prediction.

[0064] Certain coded pictures, once fully reconstructed, may be used as reference pictures for future predictions. Once a coded picture is fully reconstructed and the coded picture has been identified as a reference picture (e.g., by parser 254), the current reference picture may become part of reference picture memory 266, and a new current picture memory may be reallocated before starting reconstruction of a subsequent coded picture.

[0065] The decoder component 122 may perform decoding operations according to a predetermined video compression technique that may be recorded in a standard, such as any of the standards described herein. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that it adheres to the syntax of the video compression technique or standard, as specified in the video compression technology document or standard and in particular in the profile document therein. Moreover, in order to conform to certain video compression techniques or standards, the complexity of the encoded video sequence may be within a range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the hypothetical reference decoder (HRD) specifications and metadata of the HRD buffer management signaled in the encoded video sequence.

[0066] Figure 3306, and a plurality of communication buses 312 for interconnecting these components. In some embodiments, the control circuit 302 includes one or more processors (e.g., CPUs, GPUs, and / or DPUs). In some embodiments, the control circuit includes one or more field programmable gate arrays (FPGAs), hardware accelerators, and / or one or more integrated circuits (e.g., application specific integrated circuits).

[0067] One or more network interfaces 304 may be configured to interface with one or more communication networks (e.g., wireless, wired, and / or optical networks). Communication networks may be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of communication networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial networks including CANBus, etc. Such communications may be one-way receive-only (e.g., broadcast television), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way (e.g., to other computer systems using local area digital networks or wide area digital networks). Such communications may include communications to one or more cloud computing networks.

[0068] The user interface 306 includes one or more output devices 308 and / or one or more input devices 310. The one or more input devices 310 may include one or more of the following: a keyboard, a mouse, a trackpad, a touch screen, a data glove, a joystick, a microphone, a scanner, a camera, etc. The one or more output devices 308 may include one or more of the following: an audio output device (e.g., a speaker), a visual output device (e.g., a display or viewer), etc.

[0069] The memory 314 may include high-speed random access memory (such as DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices) and / or non-volatile memory (such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices). The memory 314 optionally includes one or more storage devices remote from the control circuit 302. Alternatively, the memory 314 or one or more non-volatile solid-state storage devices within the memory 314 include a non-volatile computer-readable storage medium. In some embodiments, the memory 314 or the non-volatile computer-readable storage medium of the memory 314 stores the following programs, modules, instructions, and data structures, or a subset or superset thereof: An operating system 316, which includes programs for handling various basic system services and for performing hardware-related tasks; • A network communications module 318 for connecting the server system 112 to other computing devices via one or more network interfaces 304 (eg, via wired and / or wireless connections); An encoding module 320 for performing various functions related to encoding and / or decoding data (such as video data). In some embodiments, the encoding module 320 is an instance of the encoder component 114. The encoding module 320 includes, but is not limited to, one or more of the following: o a decoding module 322 for performing various functions related to decoding encoded data, such as those previously described with respect to the decoder component 122; and o an encoding module 340 for performing various functions related to encoding data, such as those previously described with respect to encoder component 106; and Picture memory 352 for storing pictures and picture data, for example for use by encoding module 320. In some embodiments, picture memory 352 includes one or more of: reference picture memory 208, buffer memory 252, current picture memory 264, and reference picture memory 266.

[0070] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform the various functions previously described with respect to the parser 254), a transform module 326 (e.g., configured to perform the various functions previously described with respect to the scaler / inverse transform unit 258), a prediction module 328 (e.g., configured to perform the various functions previously described with respect to the motion compensated prediction unit 260 and / or the intra-frame prediction unit 262), and a filter module 330 (e.g., configured to perform the various functions previously described with respect to the loop filter unit 256).

[0071] In some embodiments, the encoding module 340 includes an encoding module 342 (e.g., configured to perform various functions previously described with respect to the source encoder 202, the encoding engine 212, and / or the entropy encoder 214) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 include Figure 3 A subset of the modules shown. For example, both the decoding module 322 and the encoding module 340 use a shared prediction module.

[0072] Each of the above identified modules stored in the memory 314 corresponds to a set of instructions for performing the functions described herein. The above identified modules (e.g., a set of instructions) do not need to be implemented as separate software programs, programs or modules, and therefore various subsets of these modules can be combined or otherwise rearranged in various embodiments. For example, the encoding module 320 optionally does not include separate decoding and encoding modules, but uses a set of modules to perform two groups of functions. In some embodiments, the memory 314 stores a subset of the above identified modules and data structures. In some embodiments, the memory 314 stores additional modules and data structures not described above, such as audio processing modules.

[0073] In some embodiments, the server system 112 includes a web or hypertext transfer protocol (HTTP) server, a file transfer protocol (FTP) server, and web pages and applications implemented using common gateway interface (CGI) scripts, PHP hypertext preprocessor (PHP), active server pages (ASP), hypertext markup language (HTML), extensible markup language (XML), Java, JavaScript, asynchronous JavaScript and XML (AJAX), XHP, Javelin, Wireless Universal Resource File (WURFL), etc.

[0074] although Figure 3 The server system 112 is shown in accordance with some embodiments, but Figure 3 It is intended more as a functional description of various features that may be present in one or more server systems, rather than as a schematic diagram of the structures of the embodiments described herein. In practice, and as recognized by one of ordinary skill in the art, items shown separately may be combined and some items may be separated. For example, Figure 3 Some of the items shown individually in the figure can be implemented on a single server, and a single item can be implemented by one or more servers. The actual number of servers used to implement server system 112, and how features are distributed among them, will vary from one implementation to another, and optionally, will depend in part on the amount of data traffic that the server system handles during peak usage periods and during average usage periods. Example encoding method

[0075] FIG. 4A to FIG. 4D An example coding tree structure according to some embodiments is illustrated. Figure 4A As shown in the first coding tree structure (400) in FIG, some coding methods (e.g., VP9) use a 4-way partition tree starting from the 64×64 level down to the 4×4 level with some additional restrictions on blocks of 8×8. Figure 4AThe partitioning designated as R in can be called a recursive partitioning because the same partition tree is repeated at a lower scale until the lowest 4×4 level is reached.

[0076] like Figure 4B As shown in the second coding tree structure (402) in , some coding methods (e.g., AV1) expand the partition tree to a 10-way structure and increase the maximum size (e.g., superblock in VP9 / AV1 parlance) to start at 128×128. The second coding tree structure includes 4:1 / 1:4 rectangular partitions that were not in the first coding tree structure. Figure 4B The partition type with 3 subpartitions in the second row of is called T-type partition. The rectangular partitions in this tree structure cannot be further subdivided. In addition to the coding block size, the coding tree depth can be defined to indicate the partition depth starting from the root node. For example, the coding tree depth of the root node (e.g., 128×128) is set to 0, and the coding tree depth increases by 1 after the tree block is further split once.

[0077] For example, AV1 does not enforce a fixed transform unit size like VP9, ​​but allows the partitioning of luma coding blocks into transform units of various sizes, which can be represented by recursive partitioning down to up to 2 levels. To incorporate AV1's extended coding block partitioning, square, 2:1 / 1:2, and 4:1 / 1:4 transform sizes from 4×4 to 64×64 are supported. For chroma blocks, only the largest possible transform units are allowed.

[0078] For example, a CTU can be partitioned into CUs using a quadtree structure represented as a coding tree to accommodate various local characteristics such as in HEVC. In some embodiments, a decision is made at the CU level as to whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode a picture area. Each CU can be further partitioned into one, two, or four PUs according to the PU partition type. Within a PU, the same prediction process is applied, and relevant information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying a prediction process based on the PU partition type, the CU can be partitioned into TUs according to another quadtree structure (such as a coding tree for a CU). A key feature of the HEVC structure is that it has multiple partition concepts including CUs, PUs, and TUs. In HEVC, a CU or TU can only be square, while for inter-frame prediction blocks, PUs can be square or rectangular. In HEVC, a coding block can be further partitioned into four square sub-blocks, and a transform is performed on each sub-block (TU). Each TU can be further recursively partitioned (using quadtree partitioning) into smaller TUs, which is called a residual quadtree (RQT). At picture boundaries, such as in HEVC, an implicit quadtree partitioning may be employed so that a block will remain quadtree partitioned until the size fits within a picture boundary.

[0079] For example, in VVC, a quadtree with nested multi-type trees using binary and ternary segmentation structures can replace the concept of multiple partition unit types. For example, it eliminates the distinction between CU, PU, ​​and TU concepts (except for CUs with a maximum transform length that is too large), and supports greater flexibility in the shape of CU partitions. In the coding tree structure, a CU can have a square or rectangular shape. CTU is first partitioned by a quaternary tree (also called a quad-tree) structure. The quad-tree leaf nodes can be further partitioned by a multi-type tree structure. Figure 4C As shown in the third coding tree structure (404) in, the multi-type tree structure includes four types of segmentation. For example, the multi-type tree structure includes vertical binary segmentation (SPLIT_BT_VER), horizontal binary segmentation (SPLIT_BT_HOR), vertical ternary segmentation (SPLIT_TT_VER) and horizontal ternary segmentation (SPLIT_TT_HOR). The multi-type tree leaf node is called a CU, and unless the CU is too large for the maximum transform length, the segmentation is used for prediction and transform processing without any further partitioning. This means that in most cases, CUs, PUs, and TUs have the same block size in a quadtree with a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is less than the width or height of the color component of the CU. Figure 4D An example of block partitioning of a CTU (406) is shown in , which illustrates an example quadtree with a nested multi-type tree coding block structure.

[0080] For example, in VVC, the maximum supported luma transform size may be 64 × 64, and the maximum supported chroma transform size may be 32 × 32. When the width or height of the CB is greater than the maximum transform width or height, the CB is automatically split in the horizontal and / or vertical direction to meet the transform size limit in that direction.

[0081] The coding tree scheme supports the ability of luma and chroma to have separate block tree structures, such as in VTM7. In some cases, for P and B slices, the luma and chroma CTBs in one CTU share the same coding tree structure. However, for I slices, luma and chroma can have separate block tree structures. When a separate block tree mode is applied, the luma CTB is partitioned into CUs by one coding tree structure, and the chroma CTB is partitioned into chroma CUs by another coding tree structure. This means that the CU in the I slice may include or consist of a coding block of a luma component or a coding block of two chroma components, and the CU in a P or B slice may always include or consist of coding blocks of all three color components, unless the video is monochrome.

[0082] To support extended coding block partitioning, such as in AV1, multiple transform sizes (e.g., ranging from 4 to 64 points for each dimension) and transform shapes (e.g., square or rectangular with width / height ratios of 2:1 / 1:2 and 4:1 / 1:4) may be utilized.

[0083] In merge mode, implicitly derived motion information can be directly used for prediction sample generation of the current CU. Merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag can be signaled immediately after sending the skip flag and the merge flag to specify whether to use the MMVD mode for the CU. In MMVD, after selecting a merge candidate, the merge candidate can be further refined by signaling MVD information. The MVD information may include a merge candidate flag, an index specifying the motion amplitude, and an index for indicating the direction of motion. In merge mode, one of the first two merge candidate flags in the merge list can be used as the MV basis. The merge candidate flag can be signaled to specify which flag to use.

[0084] The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. Figure 5A FIG. 4 illustrates MMDV search points for two reference frames according to some embodiments. Figure 5A As shown, the offset can be added to the horizontal component or the vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 1 below. Table 1 - Distance index and corresponding offset

[0085] The direction index indicates the direction of the MVD relative to the starting point. The direction index can represent one of four directions, as shown in Table 2 below. The meaning of the MVD symbol can vary depending on the information of the starting MV. When the starting MV is a unidirectional prediction MV or a bidirectional prediction MV, and both lists point to the same side of the current picture (for example, the POCs of both references are greater than the POC of the current picture, or the POCs of both references are less than the POC of the current picture), the symbol in Table 2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectional prediction MV, the two MVs point to different sides of the current picture (for example, the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and the difference in POC in List 0 (L0) is greater than the difference in POC in List 1 (L1), the symbol in Table 2 specifies the sign of the MV offset added to the L0 MV component of the starting MV, and the symbol for L1 MV has an opposite value. If the difference of POC in L1 is greater than L0, the sign in Table 2 specifies the sign of the MV offset added to the L1 MV component of the starting MV, and the sign for L0 MV has the opposite value.

[0086] In some embodiments, the MVD is scaled according to the difference in POC in each direction. For example, if the difference in POC in both lists is the same, no scaling is required. If the difference in POC in L0 is greater than the difference in POC in L1, then the MVD of L1 is scaled. If the difference in POC of L1 is greater than that of L0, then the MVD of L0 is scaled in the same manner. If the starting MV is unidirectionally predicted, then the MVD is added to the available MVs. Direction Index 00 01 10 11 x-axis + - not applicable not applicable y-axis not applicable not applicable + - Table 2 - Sign of MV offset specified by direction index

[0087] For example, in VVC, in addition to normal unidirectional prediction and bidirectional prediction mode MVD signaling, a symmetric MVD mode of bidirectional MVD signaling can also be applied. In the symmetric MVD mode, motion information including reference picture indexes of both L0 and L1 and MVD of L1 can be derived (not signaled).

[0088] The decoding process of the symmetric MVD mode can be as follows. First, at the slice level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived. For example, if mvd_l1_zero_flag is 1, BiDirPredFlag is set to 0. If the nearest reference picture in L0 and the nearest reference picture in L1 form a forward reference picture and a backward reference picture pair or a backward reference picture and a forward reference picture pair, BiDirPredFlag is set to 1, and both the L0 and L1 reference pictures are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. Secondly, at the CU level, if the CU is bidirectionally predicted and BiDirPredFlag is equal to 1, the symmetric mode flag indicating whether the symmetric mode is used is explicitly signaled. When the symmetric mode flag is true (e.g., equal to 1), only mvp_l0_flag, mvp_l1_flag, and MVD0 are explicitly signaled. The reference indices of L0 and L1 are set equal to the pair of reference pictures respectively. Finally, MVD1 is set equal to (-MVD0).

[0089] In some embodiments, for each coded block in an inter frame, if the mode of the current block is not skip mode but inter coding mode, another flag is signaled to indicate whether single reference mode or composite reference mode is used for the current block. The prediction block can be generated by one motion vector in single reference mode. In composite reference mode, the prediction block is generated by weighted averaging two prediction blocks derived from two motion vectors. The modes that can be signaled for the single reference case are detailed in Table 3 below. Table 3 Single reference mode The modes that may be signaled for the composite reference case are detailed in Table 4 below. Table 4 - Composite reference mode

[0090] Some standards (such as AV1) allow motion vector precision (or accuracy) of 1 / 8 pixel. As shown below, syntax can be used to signal the motion vector difference in reference frame list zero (L0) or list one (L1). For example, the syntax mv_joint specifies which components of the motion vector difference are not zero. The syntax mv_joint value of 0 indicates that there is no non-zero MVD along either the horizontal or vertical direction, the value of 1 indicates that there is only a non-zero MVD along the horizontal direction, the value of 2 indicates that there is only a non-zero MVD along the vertical direction, and the value of 3 indicates that there is a non-zero MVD along both the horizontal and vertical directions. The syntax mv_sign specifies whether the motion vector difference is positive or negative. The syntax mv_class specifies the category of the motion vector difference. As shown in Table 5 below, a higher category means that the motion vector difference has a larger amplitude. The syntax mv_bit specifies the integer part of the offset between the motion vector difference and the starting amplitude of each MV category. The syntax mv_fr specifies the first two fractional bits of the motion vector difference. The syntax mv_hp specifies the third fractional bit of the motion vector difference. MV Category MVD Amplitude MV_CLASS_0 (0，2] MV_CLASS_1 (2，4] MV_CLASS_2 (4，8] MV_CLASS_3 (8，16] MV_CLASS_4 (16，32] MV_CLASS_5 (32，64] MV_CLASS_6 (64，128] MV_CLASS_7 (128，256] MV_CLASS_8 (256，512] MV_CLASS_9 (512，1024] MV_CLASS_10 (1024，2048] Table 5 - Motion vector categories and magnitudes

[0091] For NEW_NEARMV and NEAR_NEWMV modes (shown in Table 4 above), the precision of the MVD depends on the associated MVD class and magnitude. For example, fractional MVD is allowed only when the MVD magnitude is equal to or less than one pixel. Additionally, only one MVD value is allowed when the value of the associated MV class is equal to or greater than MV_CLASS_1, and the MVD values ​​in each MV class are derived as 4, 8, 16, 32, 64 for MV classes 1 (MV_CLASS_1), 2 (MV_CLASS_2), 3 (MV_CLASS_3), 4 (MV_CLASS_4), or 5 (MV_CLASS_5). The allowed MVD values ​​in each MV class are shown in Table 6 below. MV Category MVD Amplitude MV_CLASS_0 (0，1]，{2} MV_CLASS_1 {4} MV_CLASS2 {8} MV_CLASS_3 {16} MV_CLASS_4 {32} MV_CLASS_5 {64} MV_CLASS_6 {128} MV_CLASS_7 {256} MV_CLASS_8 {512} MV_CLASS_9 {1024} MV_CLASS_10 {2048} Table 6 - Permitted MVD per MV category

[0092] In some embodiments, if the current block is encoded using NEW_NEARMV or NEAR_NEWMV mode, one context is used to signal mv_joint or mv_class. If the current block is not encoded using NEW_NEARMV or NEAR_NEWMV mode, another context is used to signal mv_joint or mv_class.

[0093] The inter - frame coding mode JOINT_NEWMV can be applied to indicate whether the MVDs of two reference lists are signaled jointly. If the inter - frame prediction mode is equal to the JOINT_NEWMV mode, the MVDs of reference L0 and reference L1 are signaled jointly. In this way, only one MVD (named joint_mvd) can be signaled and transmitted to the decoder, and the incremental MVs for reference L0 and reference L1 can be derived from joint_mvd. The JOINT_NEWMV mode is signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes.

[0094] When the JOINT_NEWMV mode is signaled and the Picture Order Count (POC) distances between the two reference frames and the current frame are different, the MVD is scaled for reference L0 or reference L1 based on the POC distance. For example, the distance between reference frame L0 and the current frame is marked as td0, and the distance between reference frame L1 and the current frame is marked as tdl. If td0 is equal to or greater than tdl, joint_mvd is directly used for reference L0, and the mvd of reference L1 is derived from joint_mvd based on the following equation 1. Equation 1 - MVD derived from the joint MVD when td0>td1 If td1 is equal to or greater than td0, joint_mvd is directly used for reference L1, and the mvd of reference L0 is derived from joint_mvd based on the following equation 2. Equation 2 - MVD derived from the joint MVD when td0<td1

[0095] The inter - frame coding mode AMVDMV can be added to the single - reference case. When the AMVDMV mode is selected, it indicates that adaptive MVD resolution (AMVD) is applied to the signaled MVD. For example, a flag amvd_flag can be added in the JOINT_NEWMV mode to indicate whether AMVD is applied to the joint MVD coding mode. When the adaptive MVD resolution is applied to the joint MVD coding mode (named joint AMVD coding), the MVDs of the two reference frames are signaled jointly and the precision of the MVD is implicitly determined by the MVD magnitude. The MVDs of two (or more than two) reference frames are signaled jointly and MVD coding is applied.

[0096] In AMVR, which was originally proposed in CWG-C012, a total of seven MV precisions are supported (8, 4, 2, 1, 1 / 2, 1 / 4, 1 / 8). For each prediction block, the AVM encoder searches all supported precision values ​​and signals the best precision to the decoder. In order to reduce the encoder runtime, two precision sets are supported. Each precision set contains 4 predefined precisions. Based on the maximum precision value of the frame, the precision set is adaptively selected at the frame level. Similar to AV1, the maximum precision is signaled in the frame header. Table 7 summarizes the supported precision values ​​based on the maximum precision at the frame level. Maximum frame-level accuracy Supported MV accuracy 1 / 8 1 / 8、1 / 2、1、4 1 / 4 1 / 4、1、4、8 Table 7 - MV accuracy at each frame level

[0097] In the AOM video model (AVM) similar to AV1, there is a frame level flag to indicate whether the MV of a frame contains sub-pixel precision. AMVR is enabled only when the value of the cur_frame_force_integer_mv flag is 0. In AMVR, if the precision of the block is lower than the maximum precision, the motion model and interpolation filter are not signaled. If the precision of the block is lower than the maximum precision, the motion mode is inferred as translational motion and the interpolation filter is inferred as REGULAR interpolation filter. Similarly, if the precision of the block is 4 pixels or 8 pixels, the inter-intra mode is not signaled and is inferred to be 0.

[0098] Motion compensation usually assumes a translational motion model between a reference block and a target block. However, warping motion utilizes an affine model. The affine motion model can be expressed by the following equation 3. Equation 3 - Affine motion model where [x, y] are the coordinates of the original pixel, and [x', y'] are the warped coordinates of the reference block. According to Equation 3, up to six parameters are needed to specify the warp motion: a3 and b3 specify the MV of the translation; a1 and b2 specify the scale along the MV; and a2 and b1 specify the rotation.

[0099] In global warp motion compensation, global motion information is signaled for each inter-frame reference frame, which includes a global motion type and several motion parameters. The global motion type and the associated number of parameters are listed in Table 8. Global motion type Number of parameters Body (zero motion) 0 Pan 2 Rotation 4 Zoom 4 General Affine 6 Table 8 - Global motion types and associated parameter numbers

[0100] After signaling the reference frame index, if global motion is selected, the global motion type and parameters associated with the given reference frame are used for the current coding block.

[0101] In local warp motion compensation, local warp motion is allowed for inter-coded blocks when the following conditions are met. First, the current block must use single reference prediction. The width or height of the coded block must be greater than or equal to eight. Finally, at least one adjacent neighboring block must use the same reference frame as the current block.

[0102] If local warp motion is used for the current block, the affine model parameters are estimated by minimizing the mean square of the difference between the reference projection and the modeling projection based on the MV of the current block and its adjacent neighboring blocks. To estimate the parameters of the local warp motion, if the neighboring block uses the same reference frame as the current block, a projection sample pair of the center sample in the neighboring block and its corresponding sample in the reference frame is obtained. Subsequently, three additional samples are created by shifting the center position by a quarter of a sample in one or two dimensions. These additional samples can also be considered as projection sample pairs to ensure the stability of the model parameter estimation process.

[0103] The MVs of neighboring blocks used to derive motion parameters are called motion samples. Motion samples are selected from neighboring blocks that use the same reference frame as the current block. Note that warp motion prediction mode is enabled only for blocks that use a single reference frame.

[0104] Figure 5B 1 illustrates example motion samples for deriving model parameters for a block using local warp motion prediction according to some embodiments. Figure 5B As shown, the MVs of neighboring blocks B0, B1, and B2 are referred to as MV0, MV1, and MV2, respectively. The current block is predicted by reference frame Ref0 using unidirectional prediction. For example, neighboring block B0 is predicted by reference frames Ref0 and Ref1 using composite prediction; neighboring block B1 is predicted by reference frame Ref0 using unidirectional prediction; and neighboring block B2 is predicted by reference frames Ref0 and Ref2 using composite prediction. The motion vector MV0Ref0 of B0, MV1Ref0 of B1, and MV2Ref0 of B2 can be used as motion samples for deriving affine motion parameters of the current block.

[0105] In local warp motion, in order to calculate the warp model using regression / least square error methods, samples and motion vectors of neighboring blocks can be collected into a stack. For example, in AV1 and AVM, a set of sample positions and their reference positions in the reference picture pointed to by their MVs are collected. For example, up to 8 pairs of positions are stored in the stack for regression / least square error warp model calculation.

[0106] In warp motion mode, the motion of neighboring blocks is smoothly extended into the current block, but with some ability to modify the warp parameters. This allows complex warp motion to be reflected and diffused over multiple blocks while minimizing blocking artifacts. For example, to achieve this, the warp motion mode builds a new warp model based on two constraints: the per-pixel motion vectors generated by the new warp model should be continuous with the per-pixel motion vectors in the neighboring blocks, and the pixel at the center of the current block should have a per-pixel motion vector that matches the signaled motion vector of the block as a whole. Figure 5C FIG. 4 illustrates motion vectors in a block using a warp stretching pattern according to some embodiments. Figure 5C As shown, for example, if the adjacent block to the left of the current block is distorted, it meets Figure 5C The model of motion vectors illustrated in is used as the distortion model.

[0107] The two constraints used to construct the new warp model imply certain equations involving the warp parameters of the neighboring blocks and the current block. These equations can then be solved to compute the warp model for the current block. For example, if (A, ..., F) represents the warp model of the neighbor and (A', ..., F') represents the new warp model, then at each point along the common edge, the first constraint is as follows: Equation 4 - First constraint for warp modeling Note that points along the edge have different y values, but they all have the same x value. This means that the y coefficients must be the same on both sides (e.g., B'=B and D'=D). At the same time, the x coefficients provide an equation related to the other coefficients, which are defined by the following equation 5: B′=B D′=D A′x+E′=Ax+E C′x+F′=Cx+F Equation 5 - x coefficient for distortion modeling Wherein, in Equation 5, x is the horizontal position of the vertical column of pixels and is therefore effectively a constant.

[0108] The second constraint specifies that the motion vector at the center of the block must be equal to the signaled motion vector. This provides two more equations, resulting in a set of six equations with six variables that have a unique solution. These equations can be solved efficiently in software and hardware. They can be solved using basic addition, subtraction, multiplication, and power-of-two division. In this way, the pattern is much simpler than the local warping pattern based on the least squares method.

[0109] Note that there may be multiple neighboring blocks to extend from. Therefore, there needs to be a way to choose which block to extend from. The same problem is encountered in motion vector prediction. Specifically, there may be several possible motion vectors from nearby blocks, and one should be selected as the basis. The solution to this can be extended to the processing needs. This is done by tracing the source of each motion vector prediction. Then, the warp motion mode is enabled only when the selected motion vector prediction is taken from the directly neighboring block. This block is then used as a single "neighboring block" in the rest of the algorithm. Note that sometimes the warp model of the neighbors is good as is and does not require any further modifications.

[0110] As used herein, the term block may be interpreted as a prediction block, a coding block, or a coding unit (CU). Figure 5D As shown, the spatial neighboring blocks can be scanned. When the neighboring block has the same reference picture as the current block (for example, if the neighboring block is composite, one of the reference pictures is the same as the reference picture of the current block), the center position of the neighboring block and the center position of the reference block of the neighboring block pointed to by the MV of the neighboring block are collected. Figure 5D Specifically, (1) scan the top row adjacent blocks; (2) scan the left column adjacent blocks; (3) scan the upper left block; and (4) scan the upper right block. In some embodiments, Figure 5D The adjacent blocks shown correspond to a single warp group.

[0111] In some embodiments, multiple warp groupings are considered for generating the warp model. For example, multiple groups of spatial and / or temporal neighboring blocks and MVs are collected / evaluated and one of these groups is used to generate the warp model, rather than using one set of neighboring blocks to derive the warp model.

[0112] Figure 5E 1 illustrates a current block with multiple spatial and / or temporal neighboring blocks according to some embodiments. Figure 5E In the example of , a block marked with "S" indicates a spatially neighboring block, a block marked with "T" indicates a temporally neighboring block, and a block marked with "S / T" indicates a spatially and / or temporally neighboring block. In some embodiments, using Figure 5E For example, the first warp group may be associated with Figure 5D In this example, the second warp group can correspond to the adjacent blocks around Figure 5E The third warp group may correspond to the spatial / temporal neighboring blocks around the corners P1, P3, and P4. For example, for Figure 5EFor blocks marked as "S / T" in , the spatial neighbors are scanned first, and if the spatial neighbors are not available, the temporal neighbors are scanned next. In some embodiments, for the temporal neighbors, the block size is assumed to be 8×8.

[0113] In AV1 and AOM designs, there is a parsing dependency of the warp mode / motion mode. That is, the decoder needs to collect neighboring blocks and MVs first, and if there is no valid MV, the warp mode is not signaled and implied. The problem is that the parsing process is delayed because the neighboring block MVs need to be constructed first. Additionally, constructing all warp groups on the decoder side increases decoding complexity.

[0114] Fig. 6A 6 is a flow chart illustrating a method 600 of encoding a video according to some embodiments. The method 600 may be performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuit and a memory storing instructions for execution by the control circuit. In some embodiments, the method 600 is performed by executing instructions stored in a memory (e.g., memory 314) of the computing system.

[0115] The system obtains (602) video data including a plurality of blocks, the plurality of blocks including a first block. In some embodiments, the first block is to be encoded in a warp mode or a motion mode. The system identifies (604) two or more warp groups for the first block. The system determines (606) a corresponding cost associated with each of the two or more warp groups. The system selects (608) a warp group from the two or more warp groups based on the corresponding costs. The system encodes (610) the first block according to the selected warp group. In some embodiments, encoding the first block includes encoding the first block and a syntax element indicating the warp mode or the motion mode according to the selected warp group without regard to whether a valid motion vector is available. In some embodiments, the system encodes the first block according to the selected warp group based on (e.g., in response to) a determination that a valid motion vector is not available for deriving a warp model (e.g., no valid motion vector is available for deriving a warp model).

[0116] In some embodiments, the dependency between the MV of the neighboring blocks and the parsing of the warp mode / motion mode syntax elements is eliminated (e.g., to reduce pipeline delays in parsing warp mode / motion mode syntax elements). In some embodiments, during parsing / signaling of the syntax elements of the warp mode / motion mode syntax elements, if there are no valid MVs for warp model derivation, the syntax elements of the warp mode / motion mode are parsed / signaled. In an example, if there are no valid neighboring block MVs, syntax element 0 is signaled at the encoder and parsed at the decoder.

[0117] In some embodiments, a fixed order of warps is constructed at the encoder side. In some embodiments, the fixed order of warps includes a first set of spatial neighbors (e.g., Figure 5D In some embodiments, the fixed-order warp group includes a second set of spatial / temporal neighboring blocks surrounding the current block (eg, scanning Figure 5E In some embodiments, the fixed order warp groups include a third set of spatial / temporal neighboring blocks around the current block (e.g., scanning Figure 5E In some embodiments, the fixed-order warp groups include a fourth set of spatial / temporal neighboring blocks (e.g., Figure 5D Blocks 1 and 4 in ).

[0118] In some embodiments, during the construction of the warp groups (as described above), temporal neighboring blocks and corresponding MVs (e.g., Figure 5E marked with "T" in the figure).

[0119] In some embodiments, a flag is signaled in a high-level syntax (e.g., SPS, PPS, PH, etc.) to indicate whether to use multiple sets of warp model directions. For example, if the flag is true (or false), multiple sets are collected as described above, and one of the sets is used to derive the warp model. If the flag is false (or true), only one fixed set (e.g., the current design) is collected and used for warp model derivation.

[0120] In some embodiments, the number of these groups is signaled in a high-level syntax, the groups are constructed accordingly on the encoder side, and the group indices are parsed accordingly on the decoder side.

[0121] Figure 6B 6 is a flow chart illustrating a method 650 of decoding a video according to some embodiments. The method 650 may be performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuit and a memory storing instructions for execution by the control circuit. In some embodiments, the method 650 is performed by executing instructions stored in a memory (e.g., memory 314) of the computing system.

[0122] The system obtains (652) video data including a plurality of blocks, the plurality of blocks including a first block. The system identifies (654) a selected warp group from two or more warp groups. The system decodes (656) the first block according to the selected warp group.

[0123] In some embodiments, at the decoder component, a first group is constructed, and if there is no valid MV in the group, the warp mode is not signaled and is implied as a non-warp mode. Otherwise, if there is a valid MV in the group, the warp mode is parsed, and if the mode uses warping, the warp group index is further parsed. If the warp group index is not equal to 0, the corresponding group is further constructed at the decoder component. In this way, in the worst case, at most two groups need to be constructed at the decoder component.

[0124] In some embodiments, the dependency on the first group (described in the previous paragraph) is eliminated, and a fixed order of groups is constructed at the encoder side (e.g., as described above with reference to method 600). In some embodiments, at the decoder component, the warp mode / motion mode syntax element is parsed, the warp mode is selected, and the warp group index is parsed. Based on the warp group index, the corresponding group is constructed at the decoder component. Thus, in the worst case, at most one group needs to be constructed at the decoder component.

[0125] In some embodiments, the order of these groups is implicitly derived using coded information such as the availability of motion information from particular spatial / temporal neighboring blocks. In some embodiments, the order of these groups is signaled, for example, in a high-level syntax or at the block level. For example, the index of a group selected from a set of groups in a predefined order is signaled.

[0126] although Fig. 6A and Figure 6B The various logical stages are illustrated in a particular order, but stages that are not order-dependent may be reordered, and other stages may be combined or split. Certain reordering or other groupings not specifically mentioned will be apparent to one of ordinary skill in the art, and thus the ordering and groupings presented herein are not exhaustive. Furthermore, it should be appreciated that the various stages may be implemented in hardware, firmware, software, or any combination thereof.

[0127] We now turn to some example embodiments.

[0128] (A1) In one aspect, some embodiments include a method of video encoding (e.g., method 600). In some embodiments, the method is performed in a computing system (e.g., server system 112) having a memory and a control circuit. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). In some embodiments, the method is performed at an entropy encoder (e.g., entropy encoder 214). The method includes: (i) obtaining video data including a plurality of blocks, the plurality of blocks including a first block; (ii) identifying two or more warp groups for the first block; (iii) determining a corresponding cost associated with each of the two or more warp groups; (iv) selecting a warp group from the two or more warp groups based on the corresponding costs; and (v) encoding the first block according to the selected warp group. For example, the two or more warp groups include Figure 5D The groups and corresponding adjacent blocks shown are Figure 5E In some embodiments, determining the corresponding cost includes determining a coding loss corresponding to using a warp model derived from the warp group (e.g., and selecting the warp group with the lowest coding loss). In some embodiments, the first block is to be encoded in the warp mode or motion mode. In some embodiments, encoding the first block includes encoding the first block and a syntax element indicating the warp mode or motion mode according to the selected warp group, without considering whether a valid motion vector is available.

[0129] (A2) In some embodiments of A1, encoding the first block according to the selected warp group includes generating a warp model for the first block using the selected warp group.

[0130] (A3) In some embodiments of A2, generating the warp model includes: (i) scanning spatial neighboring blocks of the first block; (ii) based on a determination that the spatial neighboring blocks are available, generating the warp model based on motion vectors of the spatial neighboring blocks; and (iii) based on a determination that the spatial neighboring blocks are not available: (a) scanning temporal neighboring blocks of the first block, the temporal neighboring blocks having positions corresponding to the spatial neighboring blocks; and (b) generating the warp model based on motion vectors of the temporal neighboring blocks. For example, the selected warp group includes one or more neighboring blocks represented as space, one or more neighboring blocks represented as time, and / or one or more neighboring blocks represented as space / time. In this example, if the spatial neighboring blocks of the block represented as space / time are not available, the temporal neighboring blocks are scanned / used.

[0131] (A4) In some embodiments of any one of A1-A3, a first warp group of the two or more warp groups includes a first set of spatial neighboring blocks of the first block. For example, the first set of spatial neighboring blocks are neighboring blocks to the left and / or top of the first block.

[0132] (A5) In some embodiments of A4, a second warp group of the two or more warp groups includes a second set of spatial neighbors of the first block, the second set being different from the first set. For example, the second set includes one or more spatial neighbors not included in the first set. In some embodiments, the first spatial neighbor is included in each of the first set and the second set, while the second spatial neighbor is included in only one of the second set and the first set.

[0133] (A6) In some embodiments of A4 or A5, a second warp group in the two or more warp groups includes one or more spatial neighboring blocks of the first block and at least one temporal neighboring block of the first block. In some embodiments, each warp group in the two or more warp groups does not include a temporal neighboring block.

[0134] (A7) In some embodiments of any one of A1-A6, the two or more twist groups include four different twist groups.

[0135] (A8) In some embodiments of any one of A1-A7, each of the two or more warp groups has a corresponding scan order. In some embodiments, the respective scan orders are different from each other.

[0136] (A9) In some embodiments of any of A1-A8, the method further comprises signaling the selected warp group. For example, the selected warp group is explicitly signaled using a bit or a flag. In some embodiments, the two or more warp groups have a predefined order, and the index of the selected warp group is signaled. As another example, the selected warp group is signaled implicitly (e.g., using coding information, such as the availability of motion information from a particular spatial / temporal neighboring block).

[0137] (A10) In some embodiments of A9, the selected warp group is signaled in a high-level syntax, for example, a flag of the selected warp group is set in a sequence parameter set (SPS), a picture parameter set (PPS), or a picture header (PH) to signal the selected warp group.

[0138] (A11) In some embodiments of any of A1-A10, the method further includes determining whether to enable multiple sets of warp modes, wherein the two or more warp groups for the first block are identified based on a determination to enable the multiple sets of warp modes. In some embodiments, the method includes abandoning identifying two or more warp groups based on a determination to disable the multiple sets of warp modes.

[0139] (A12) In some embodiments of A11, multiple sets of warp modes are signaled in a high-level syntax, for example, flags of multiple sets of warp modes are set in a sequence parameter set (SPS), a picture parameter set (PPS), or a picture header (PH) to indicate the selected warp groups.

[0140] (A13) In some embodiments of A11 or A12, the method further includes encoding the first block according to a default warp group based on the determination to disable the multiple groups of warp modes.

[0141] (A14) In some embodiments of any of A1-A13, the method further comprises identifying a plurality of warp groups to determine associated costs, wherein the two or more warp groups are identified based on the number of warp groups identified. In some embodiments, the number of warp groups is represented in a high-level syntax. In some embodiments, the encoder identifies / constructs only N warp groups, where N is signaled in the high-level syntax. For example, the maximum number of warp groups may be 4, 6, 8, or 10. In some embodiments, only a subset of the maximum number of warp groups is considered (e.g., based on a flag in the high-level syntax). For example, warp groups 1-N may be considered, where N is in the range of 2 to 10. In some embodiments, a selected subset of warp groups may be considered (e.g., warp groups 1-4 may be considered, or warp groups 5-9 may be considered). In some embodiments, the warp groups considered are signaled (e.g., by signaling a starting warp group number, an ending warp group number, and / or a range of warp groups considered).

[0142] (B1) On the other hand, some embodiments include a method of video decoding (e.g., method 650). In some embodiments, the method is performed in a computing system (e.g., server system 112) including a memory and a control circuit. In some embodiments, the method is performed at an encoding module (e.g., encoding module 320). In some embodiments, the method is performed at a parser (e.g., parser 254). The method includes: (i) obtaining video data including a plurality of blocks (including a first block) from a bitstream; (ii) identifying a selected warp group from two or more warp groups; and (iii) decoding the first block according to the selected warp group. In some embodiments, the first block is specified to be encoded in a warp mode or a motion mode. In some embodiments, the method includes decoding a syntax element indicating a warp mode or a motion mode, regardless of whether a valid motion vector is available.

[0143] (B2) In some embodiments of B1, decoding the first block according to the selected warp group includes generating a warp model for the first block using the selected warp group.

[0144] (B3) In some embodiments of B1 or B2, the method further includes identifying a first warp group consisting of two or more warp groups, wherein the selected warp group is identified based on determining that the first warp group includes a valid motion vector. For example, the selected warp group can be the first warp group or another warp group in the two or more warp groups.

[0145] (B4) In some embodiments of B3, the method further includes, based on determining that the first warping group does not include a valid motion vector, forgoing signaling the warping mode of the first block.

[0146] (B5) In some embodiments of any of B1-B4, the selected warp group is identified based on a warp group index from the codestream. In some embodiments, the warp group index is signaled in a high-level syntax (e.g., in an SPS, PPS, or PH). In some embodiments, the warp group index is signaled at a block level. For example, the index signal of the selected group is signaled according to a set of predefined group orders.

[0147] (B6) In some embodiments of any of B1-B5, the selected warp group is identified based on a ranking of the two or more warp groups, and wherein the ranking is determined based on context information. For example, the context information includes the availability of motion information from a particular spatial / temporal neighboring block.

[0148] (B7) In some embodiments of any of B1-B6, the method further comprises identifying a plurality of warp groups to determine associated costs, wherein two or more warp combinations are identified based on a number of warp groups identified. In some embodiments, the number of warp groups is signaled in a high-level syntax.

[0149] (B8) In some embodiments of any of B1-B7, the method further includes determining whether the multi-group warp mode is enabled, wherein the selected group is identified from the two or more warp groups based on determining that the multi-group warp mode is enabled. In some embodiments, the method includes abandoning identifying the two or more warp groups based on determining that the multi-group warp mode is disabled. For example, when the multi-group warp mode is disabled, a default warp group is used.

[0150] (B9) In some embodiments of any one of B1-B8, the codestream corresponds to a video encoded according to any one of A1-A14.

[0151] The methods described herein can be used alone or in any order. Each method can be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In some embodiments, the processing circuit executes a program stored in a non-transient computer readable medium.

[0152] On the other hand, some embodiments include a computing system (e.g., server system 112) that includes a control circuit (e.g., control circuit 302) and a memory (e.g., memory 314) coupled to the control circuit, the memory storing one or more sets of instructions configured to be executed by the control circuit, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1 to A14, and B1 to B9 above).

[0153] On the other hand, some embodiments include a non-volatile computer-readable storage medium storing one or more sets of instructions for execution by control circuitry of a computing system, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1 to A14, and B1 to B9 above).

[0154] It will be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element.

[0155] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the claims. As used in the description of the embodiments and the appended claims, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that when used in this specification, the terms "comprises" and "comprising" indicate the presence of stated features, wholes, steps, operations, elements, and / or parts, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, parts, and / or groups thereof.

[0156] As used herein, the term "if" may be interpreted to mean "when" or "at the time of" or "in response to determining" or "upon determining" or "in response to detecting" the stated antecedent is true, depending on the context. Similarly, the phrase "if it is determined that [the stated antecedent is true]" or "if [the stated antecedent is true]" or "when [the stated antecedent is true]" may be interpreted to mean "at the time of determining" or "in response to determining" or "upon determining" or "upon detecting" or "in response to detecting" the stated antecedent is true, depending on the context.

[0157] For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the precise forms disclosed. In view of the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best explain the operating principles and practical applications so that others skilled in the art can understand.

Claims

1. A method for video encoding, executed in a computing system, the computing system comprising a memory and one or more processors, characterized in that: The method comprises: Obtaining video data including a plurality of blocks, the plurality of blocks including a first block, wherein the first block is to be encoded in a warp mode or a motion mode; identifying two or more warp groups for the first block; determining a respective cost associated with each of the two or more warp groups; selecting a warp group from the two or more warp groups based on the corresponding costs; and The first block and a syntax element indicating the warping mode or the motion mode are encoded according to the selected warping group, regardless of whether a valid motion vector is available.

2. The method according to claim 1, characterized in that Encoding the first block according to the selected warp set includes generating a warp model for the first block using the selected warp set.

3. The method according to claim 2, characterized in that Generating the distortion model comprises: Scanning spatially adjacent blocks of the first block; According to the determination that the spatial neighboring blocks are available, generating the warp model based on the motion vectors of the spatial neighboring blocks; and According to the determination that the spatially adjacent blocks cannot be used: scanning temporal neighboring blocks of the first block, the temporal neighboring blocks having positions corresponding to the spatial neighboring blocks; and The warp model is generated based on motion vectors of the temporal neighboring blocks.

4. The method according to claim 1, characterized in that: A first warping group of the two or more warping groups includes a first set of spatially neighboring blocks of the first block.

5. The method according to claim 4, characterized in that A second warping group of the two or more warping groups includes a second set of spatially neighboring blocks of the first block, the second set being different from the first set.

6. The method according to claim 4, characterized in that A second warp group of the two or more warp groups includes one or more spatial neighboring blocks of the first block and one or more temporal neighboring blocks of the first block.

7. The method according to claim 1, characterized in that The two or more twist groups include four different twist groups.

8. The method according to claim 1, characterized in that Each of the two or more warp groups has a corresponding scanning order.

9. The method according to claim 1, characterized in that: Further comprising signaling the selected warp group.

10. The method according to claim 9, characterized in that The selected warp group is signaled in a high-level syntax.

11. The method according to claim 1, characterized in that: Further comprising determining whether multiple sets of warp modes are enabled, wherein the two or more warp sets for the first block are identified based on a determination to enable the multiple sets of warp modes.

12. The method according to claim 11, characterized in that The sets of warping modes are signaled in a high-level syntax.

13. The method according to claim 11, characterized in that Further comprising encoding the first block according to a default warp group based on the determination to disable the plurality of warp modes.

14. The method according to claim 1, characterized in that Further included is identifying a number of warp groups for determining an associated cost, wherein the two or more warp groups are identified according to the identified number of warp groups.

15. A computing system, characterized in that: include: Control circuit; as well as A memory storing one or more programs configured to be executed by the control circuit, the one or more programs further comprising instructions for: Obtaining video data including a plurality of blocks, the plurality of blocks including a first block, wherein the first block is to be encoded in a warp mode or a motion mode; identifying two or more warp groups for the first block; determining a respective cost associated with each of the two or more warp groups; selecting a warp group from the two or more warp groups based on the corresponding costs; and The first block and a syntax element indicating the warping mode or the motion mode are encoded according to the selected warping group, regardless of whether a valid motion vector is available.

16. The computing system according to claim 15, characterized in that: Encoding the first block according to the selected warp set includes generating a warp model for the first block using the selected warp set.

17. The computing system according to claim 16, characterized in that: Generating the distortion model comprises: Scanning spatially adjacent blocks of the first block; According to the determination that the spatial neighboring blocks are available, generating the warp model based on the motion vectors of the spatial neighboring blocks; and According to the determination that the spatially adjacent blocks cannot be used: scanning temporal neighboring blocks of the first block, the temporal neighboring blocks having positions corresponding to the spatial neighboring blocks; and The warp model is generated based on motion vectors of the temporal neighboring blocks.

18. The computing system according to claim 15, characterized in that: Each of the two or more warp groups has a corresponding scanning order.

19. A non-volatile computer-readable storage medium storing one or more programs for execution by control circuitry of a computing system, the one or more programs comprising instructions for: Obtaining video data including a plurality of blocks, the plurality of blocks including a first block, wherein the first block is to be encoded in a warp mode or a motion mode; identifying two or more warp groups for the first block; determining a respective cost associated with each of the two or more warp groups; selecting a warp group from the two or more warp groups based on the corresponding costs; and The first block and a syntax element indicating the warping mode or the motion mode are encoded according to the selected warping group, regardless of whether a valid motion vector is available.

20. The non-volatile computer-readable storage medium of claim 19, wherein: Each of the two or more warp groups has a corresponding scanning order.

Images