Method, apparatus, and computer program for improved warp difference signal transmission

Abstract

A system and method for encoding and decoding using a warped motion mode is provided, the method includes the steps of receiving a bitstream including a coding picture, obtaining warp information signaled in the bitstream, the warp information including a warp type indicating a number of parameters to be corrected by a difference value, the number of parameters defining a warp motion for predicting a current block of the coding picture, obtaining at least one difference value from the bitstream, the difference being a difference of the at least one parameter defining the warp motion for predicting the current block, obtaining a warp model based on the warp information and the at least one difference value, and predicting the current block by performing warped motion compensation using the warp model.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims the priority of U.S. Provisional Application No. 63 / 392,388 filed on July 26, 2022 and U.S. Patent Application No. 17 / 981,842 filed on November 7, 2022, and incorporates the entire contents of these by reference.

[0002] [Technical Field] Embodiments of the present disclosure relate to advanced image and video coding (encoding) techniques, and more particularly, to improvements in local warp motion modes.

Background Art

[0003] AV1 (AOMedia Video 1) is an open video coding format designed for video transmission over the internet. It was developed as a successor to VP9 by AOMedia (Alliance for Open Media), a consortium founded in 2015 that includes the semiconductor industry, video-on-demand providers, video content producers, software developers, and web browser vendors. Many components of the AV1 project were derived from previous research efforts by the group's members. Individual contributors had launched experimental technology platforms several years prior. Xiph / Mozilla's Daala had already released its code in 2010, Google's experimental VP9 evolution project, V10, was announced on September 12, 2014, and Cisco's Thor was released on August 11, 2015. Built upon the VP9 codebase, AV1 incorporates further technologies, some of which were developed in these experimental formats. The first version of the AV1 reference codec, 0.1.0, was released on April 7, 2016. The organization announced the release of the AV1 bitstream specification on March 28, 2018, along with reference software-based encoders and decoders. A verified version 1.0.0 of the specification was released on June 25, 2018. A verified version 1.0.0 with errata 1 was released on January 8, 2019. The AV1 bitstream specification includes a reference video codec.

[0004] The ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). They also investigated the potential need for standardization of future video coding technologies that could offer significantly better performance than HEVC in terms of compression capabilities. In October 2017, they issued a Joint Call for Proposals on Video Compression with Capability Beyond HEVC (CfP). By February 15, 2018, a total of 22 CfP responses had been submitted for Standard Dynamic Range (SDR), 12 for High Dynamic Range (HDR), and 12 for the 360° video category. In April 2018, all received CfP responses were evaluated at the 122nd MPEG / 10th JVET (Joint Video Exploration Team or Joint Video Expert Team) meeting. As a result of this meeting, JVET officially began standardizing a next-generation video coding standard beyond HEVC. The new standard is called VVC (Versatile Video Coding). [Overview of the project]

[0005] Embodiments of the present disclosure provide a method performed by a decoder. The method includes the steps of: receiving a bitstream containing a coding picture; obtaining warp information signaled in the bitstream, the warp information including a warp type indicating the number of parameters to be corrected by a difference value, the number of parameters defining a warp motion for predicting the current block of the coding picture; obtaining from the bitstream at least one difference value which is the difference of at least one parameter defining the warp motion for predicting the current block; obtaining a warp model based on the warp information and at least one difference value; and predicting the current block by performing warp motion compensation using the warp model.

[0006] According to one or more embodiments of the present disclosure, the warp information includes a warp type, the warp type is a rotation and zoom type including four parameters that define the warp motion, the at least one difference value obtained is two difference values ​​for correcting two of the four parameters, and the step of obtaining the warp model includes the step of correcting two parameters based on the two difference values.

[0007] According to one or more embodiments of the present disclosure, warp information includes a warp type, the warp type is an affine type including six parameters that define warp motion, at least one difference value obtained is four difference values ​​to correct four of the six parameters, and the step of obtaining a warp model includes the step of correcting the four parameters based on the four difference values.

[0008] According to one or more embodiments of the present disclosure, the warp information includes a number of parameters defining the warp motion that are to be corrected by differential values, the number of at least one differential values ​​obtained is equal to the number of parameters of the warp information that are to be corrected by differential values.

[0009] According to one or more embodiments of the present disclosure, the step of obtaining an adaptive step size of at least one differential value further includes the step of obtaining a warp model, the step of obtaining a warp model based on the adaptive step size.

[0010] According to one or more embodiments of the present disclosure, the step of obtaining an adaptive step size includes the steps of obtaining a scaling factor and calculating an adaptive step size based on a predetermined base step size and scaling factor.

[0011] According to one or more embodiments of the present invention, the step of obtaining a scaling factor includes the step of obtaining an index indicating a scaling factor from among a plurality of scaling factors stored in a lookup table, and the step of obtaining a scaling factor based on the index and the lookup table.

[0012] According to one or more embodiments of the present invention, the step of obtaining a scaling factor includes the step of deriving a scaling factor based on the magnitude of the translational motion vector of the current block.

[0013] According to one or more embodiments of the present disclosure, the step of obtaining a scaling factor includes the step of deriving a scaling factor based on a shear parameter.

[0014] According to one or more embodiments of the present disclosure, the step of obtaining an adaptive step size includes obtaining an adaptive step size based on one or more of the block size, prediction mode, step size used by at least one adjacent block of the current block, predicted warp parameters, and at least one of the adjacent blocks of the current block being coded by a warp model or a translation model.

[0015] Embodiments of the present disclosure provide a system. The system includes at least one memory configured to store computer program code, and at least one processor configured to receive a bitstream containing a coding picture, access the computer program code, and operate as instructed by the computer program code. The computer program code includes a first acquisition code configured to cause at least one processor to acquire warp information signaled in the bitstream, the warp information including a warp type indicating the number of parameters to be corrected by a difference value, the number of parameters defining warp motion for predicting the current block of the coding picture, a second acquisition code configured to cause at least one processor to acquire at least one difference value from the bitstream, which is the difference of at least one parameter defining warp motion for predicting the current block, a third acquisition code configured to cause at least one processor to acquire a warp model based on the warp information and at least one difference value, and a warp motion compensation code configured to cause at least one processor to predict the current block by performing warp motion compensation using the warp model.

[0016] According to one or more embodiments of the present disclosure, the warp information includes a warp type, the warp type is a rotation and zoom type including four parameters that define the warp motion, the at least one difference value to be acquired is two difference values ​​to correct two of the four parameters, and the third acquisition code is configured to cause at least one processor to correct the two parameters based on the two difference values.

[0017] According to one or more embodiments of the present disclosure, the warp information includes a warp type, the warp type is an affine type including six parameters that define the warp motion, the at least one difference value to be acquired is four difference values ​​to correct four of the six parameters, and the third acquisition code is configured to cause at least one processor to correct the four parameters based on the four difference values.

[0018] According to one or more embodiments of the present disclosure, the warp information includes a number of parameters defining the warp motion that are to be corrected by differential values, the number of at least one differential values ​​obtained is equal to the number of parameters of the warp information that are to be corrected by differential values.

[0019] According to one or more embodiments of the present disclosure, the computer program code further includes a fourth acquisition code configured to cause at least one processor to acquire an adaptive step size of at least one differential value, the third acquisition code configured to cause at least one processor to acquire a warp model based on the adaptive step size.

[0020] According to one or more embodiments of the present disclosure, the fourth acquisition code is configured to cause at least one processor to acquire a scaling factor and to calculate an adaptive step size based on a predetermined base step size and scaling factor.

[0021] According to one or more embodiments of the present invention, the fourth acquisition code is configured to cause at least one processor to acquire a scaling factor by acquiring an index indicating a scaling factor from among a plurality of scaling factors stored in a lookup table, and acquiring the scaling factor based on the index and the lookup table.

[0022] According to one or more embodiments of the present invention, the fourth acquisition code is configured to cause at least one processor to acquire a scaling factor by deriving the scaling factor based on the magnitude of the translational motion vector of the current block.

[0023] According to one or more embodiments of the present disclosure, the fourth acquisition code is configured to cause at least one processor to acquire a scaling factor by deriving the scaling factor based on a shear deformation parameter indicating a shear deformation motion.

[0024] According to an embodiment of the present disclosure, a non-transitory computer-readable medium storing computer code is provided. The computer code, when executed by at least one processor, causes the at least one processor to receive a bitstream including a coding picture, obtain warp information signaled in the bitstream, the warp information including a warp type indicating the number of parameters to be corrected by a difference value, the number of parameters defining a warp motion for predicting a current block of the coding picture, obtain at least one difference value that is a difference of at least one parameter defining a warp motion for predicting the current block from the bitstream, obtain a warp model based on the warp information and the at least one difference value, and predict the current block by performing warp motion compensation using the warp model.

Brief Description of the Drawings

[0025] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1] It is a schematic diagram of a simplified block diagram of a communication system according to an embodiment. [Figure 2] It is a schematic diagram of a simplified block diagram of a communication system according to an embodiment. [Figure 3] Schematic diagram of a simplified block diagram of a decoder according to an embodiment. [Figure 4] Schematic diagram of a simplified block diagram of an encoder according to an embodiment. [Figure 5A] Schematic diagram of a partition tree in VP9 according to an embodiment. [Figure 5B] Schematic diagram of a partition tree in AV1 according to an embodiment. [Figure 6A] Schematic diagram of block partition splitting of a quad-tree plus binary tree (QTBT) structure according to an embodiment. [Figure 6B] Schematic diagram of the corresponding tree representation of the QTBT structure with block partition splitting shown in FIG. 6A. [Figure 7A] Schematic diagram of vertical center-side ternary tree partition splitting according to an embodiment. [Figure 7B] Schematic diagram of horizontal center-side ternary tree partition splitting according to an embodiment. [Figure 8] Schematic diagram of search points of a merge mode with motion vector difference (MMVD) using the motion vector difference between two reference frames according to an embodiment. [Figure 9] Schematic diagram of exemplary motion samples used to derive block model parameters using local warp motion prediction according to an embodiment. [Figure 10] Schematic diagram of an example of a warp sample collection scanning order according to an embodiment. [Figure 11] Schematic diagram of motion vectors within a block using the warp extension mode according to an embodiment. [Figure 12] Schematic diagram of the MMVD search process for the current frame using two reference frames according to an embodiment. [Figure 13] Block diagram of computer code according to an embodiment. [Figure 14]This is a diagram of a computer system suitable for implementing an embodiment of the present disclosure. [Modes for carrying out the invention]

[0026] A detailed description of exemplary embodiments follows with reference to the accompanying drawings. The same reference numerals in different drawings may identify the same or similar elements.

[0027] The above disclosure provides examples and explanations, but is not intended to be exhaustive or to limit implementations to the exact forms disclosed. Modifications and alterations are possible in light of this disclosure, or may be obtained from the implementation. Furthermore, one or more features or components of one embodiment may be incorporated into or combined with other embodiments (or one or more features of other embodiments). Furthermore, it should be understood that in the flowcharts and descriptions of operations provided below, one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) simultaneously, and the order of one or more operations may be changed.

[0028] It is evident that the systems and / or methods described herein may be implemented in different forms, such as hardware, software, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited to the implementation. It is understood that the software and hardware may be designed to implement the systems and / or methods based on the descriptions herein.

[0029] The features described below may be used separately or in any order. Furthermore, embodiments may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium.

[0030] Any elements, actions, or instructions used herein should not be construed as important or essential unless expressly stated otherwise. Furthermore, where used herein, the singular is intended to include one or more items and may be used interchangeably with “one or more.” When one item is intended, “one” or a similar term is used. Also, where used herein, terms such as “has,” “have,” “having,” “include,” and “including” are intended to be open-ended terms. Furthermore, the phrase “based on” is intended to mean “based at least partially” unless otherwise specified. Additionally, expressions such as “at least one of A and B” or “at least one of A or B” should be understood as including A only, B only, or both A and B.

[0031] Figure 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 includes at least two terminals 110, 120 interconnected via a network 150. For one-way data transmission, the first terminal 110 may encode local location video data for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive encoded video data from the other terminal via the network 150, decode the encoded data, and display the restored video data. One-way data transmission may be common in media delivery applications and the like.

[0032] Figure 1 shows a second pair of terminals 130, 140 provided to support the bidirectional transmission of encoded video that may occur, for example, during a video conference. For bidirectional data transmission, each terminal 130, 140 may encode video data captured at its local location for transmission to other terminals via the network 150. Each terminal 130, 140 may also receive encoded video data transmitted by other terminals, decode the encoded data, and display the restored video data on a local display device.

[0033] In Figure 1, terminals 110-140 may be represented as servers, personal computers, smartphones, and / or any other type of terminal. For example, terminals 110-140 may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks that transmit encoded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched channels and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of network 150 are not important to the operation of this disclosure unless described below herein.

[0034] Figure 2 shows an example of an application of the subject matter of the disclosure, illustrating the arrangement of a video encoder and decoder in a streaming environment. The subject matter of the disclosure is similarly applicable to other video-enabled applications, including, for example, video conferencing, digital TV, and the storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).

[0035] As shown in Figure 2, the streaming system 200 may include a capture subsystem 213 which can include a video source 201 and an encoder 203. For example, the video source 201 may be a digital camera and may be configured to produce an uncompressed video sample stream 202. The uncompressed video sample stream 202 may provide a higher data volume compared to the encoded video bitstream and can be processed by the encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the subject matter of disclosure, as will be described in more detail below. The encoded video bitstream 204 may contain a lower data volume compared to the sample stream and can be stored in the streaming server 205 for future use. One or more streaming clients 206 can access the streaming server 205 to obtain a video bitstream 209 which may become a copy 208 of the encoded video data 204.

[0036] In an embodiment, the streaming server 205 may also function as a Media-Aware Network Element (MANE). For example, the streaming server 205 may be configured to prune an encoded video bitstream 204 to align potentially different bitstreams with one or more of the streaming clients 206. In an embodiment, the MANE may be provided separately from the streaming server 205 in the streaming system 200.

[0037] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 can, for example, decode a video bitstream 209, which is an input copy of an encoded video bitstream 204, and generate an output video sample stream 210 that can be rendered on the display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204,209 can be encoded according to a specific video encoding / compression standard. Examples of these standards include, but are not limited to, ITU-T Recommendation H.265. Video encoding standards under development are informally known as VVC (Versatile Video Coding). Embodiments of this disclosure may be used in the context of VVC.

[0038] Figure 3 shows an exemplary functional block diagram of a video decoder 210 mounted on a display 212 according to one embodiment of the present disclosure.

[0039] The video decoder 210 may include a channel 312, a receiver 310, a buffer memory 315, an entropy decoder / parser 320, a scaler / inverse unit 351, an intra-prediction unit 352, a motion compensation prediction unit 353, an aggregator 355, a loop filter unit 356, a reference picture memory 357, and a current picture memory. In at least one embodiment, the video decoder 210 may include an integrated circuit, a series of integrated circuits, and / or other electronic circuits. The video decoder 210 may also be partially or entirely embodied in software running on one or more CPUs having associated memory.

[0040] In this embodiment and other embodiments, the receiver 310 may receive one or more encoded video sequences to be decoded by the decoder 210, or it may receive one encoded video sequence at a time, and the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequences may be received from channel 312, which may be a hardware / software link to a storage device that stores encoded video data. The receiver 310 may receive the encoded video data together with other data (e.g., encoded audio data and / or auxiliary data streams), which may be transferred to their respective user entities (not shown). The receiver 310 may isolate the encoded video sequences from other data. To prevent network jitter, a buffer memory 315 may be coupled between the receiver 310 and the entropy decoder / parser 320 (hereinafter referred to as the "parser"). If the receiver 310 is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory 315 may not be used or may be reduced in size. For use in best-effort packet networks such as the internet, a buffer memory of 315 may be required, which can be relatively large and adapted to different sizes.

[0041] The video decoder 210 may include a parser 320 for reconstructing symbols 321 from an entropy-encoded video sequence. These symbol categories may include, for example, information used to manage the operation of the decoder 210, and potentially information for controlling a rendering device, such as a display 212, which may be coupled to the decoder as shown in Figure 2. The rendering device control information may be in the form of, for example, Supplementary Enhancement Information (SEI) (SEI messages) or Video Usability Information (VUI) parameter set fragments (not shown). The parser 320 may parse / entropy-decode the received encoded video sequence. The encoding of the encoded video sequence may follow video encoding techniques or standards, and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser 320 may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to a group. Subgroups may include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transformation units (TU), prediction units (PU), etc. Parser 320 may also extract information from the coded video sequence, such as transformation coefficients, quantization parameter values, and motion vectors.

[0042] The parser 320 may perform an entropy decoding / analysis operation on the video sequence received from the buffer memory 315 in order to generate the symbol 321.

[0043] The reconstruction of symbol 321 may involve multiple different units, depending on the type of encoded video picture or portion thereof (e.g., interpicture and intrapicture, interblock and intrablock) and other factors. Which units are involved and how they are involved can be controlled by subgroup control information analyzed from the encoded video sequence by parser 320. The flow of such subgroup control information between parser 320 and the following multiple units is not illustrated for clarity.

[0044] In addition to the functional blocks described above, the decoder 210 can be conceptually subdivided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be integrated at least partially. However, for the purpose of illustrating the subject of disclosure, it is appropriate to conceptually subdivide it into the following functional units.

[0045] One unit is the scaler / inverse unit 462. The scaler / inverse unit 351 may receive quantized transformation coefficients as symbol 321 from the parser 320, along with control information (including which transformation to use, block size, quantization coefficients, quantization scaling matrix, etc.). The scaler / inverse unit 351 can output a block containing sample values ​​that can be input to the aggregator 355.

[0046] In some cases, the output samples of the scaler / inverse unit 351 may relate to intra-encoded blocks, i.e., blocks that do not use predictive information from previously reconstructed pictures but can use predictive information from previously reconstructed portions of the current picture. Such predictive information can be provided by the intra-picture predictive unit 352. In some cases, the intra-picture predictive unit 352 generates blocks of the same size and shape as the block being reconstructed, using already reconstructed surrounding information retrieved from the (partially reconstructed) current picture from the current picture memory 358. In some cases, the aggregator 355 adds the predictive information generated by the intra-picture predictive unit 352 to the output sample information provided by the scaler / inverse unit 351, sample by sample.

[0047] In other cases, the output samples of the scaler / inverse unit 351 may be associated with intercoded and potentially motion-compensated blocks. In such cases, the motion-compensated prediction unit 353 may access the reference picture memory 357 to retrieve samples to be used for prediction. After motion-compensating the retrieved samples according to the symbols 321 associated with the blocks, these samples can be added by the aggregator 355 to the output of the scaler / inverse unit 351 (in this case, called residual samples or residual signals) to generate output sample information. The address in the reference picture memory 357 from which the motion-compensated prediction unit 353 retrieves the prediction samples can be controlled by a motion vector. The motion vector may be available to the motion-compensated prediction unit 353 in the form of a symbol 321, which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory 357 when the precise motion vectors of the subsamples are used, motion vector prediction mechanisms, etc.

[0048] The output samples of the aggregator 355 can undergo various loop filtering techniques within the loop filter unit 356. The video compression technique may include an in-loop filtering technique, which is controlled by parameters included in the encoded video bitstream and made available to the loop filter unit 356 as symbols 321 from the parser 320, and can respond to metadata obtained during the decoding of earlier parts (in decoding order) of the encoded picture or encoded video sequence, as well as to previously reconstructed and loop-filtered sample values.

[0049] The output of the loop filter unit 356 can be a sample stream, which can be output to a rendering device such as the display 212, and can also be stored in the reference picture memory 357 for use in future interpicture prediction.

[0050] A particular encoded picture, once fully reconstructed, can be used as a reference picture for future predictions. Once an encoded picture is fully reconstructed and identified as a reference picture (e.g., by parser 320), the current reference picture can become part of reference picture memory 357, and new current picture memory can be reallocated before starting the reconstruction of subsequent encoded pictures.

[0051] The video decoder 210 may perform decoding operations according to a predetermined video compression technique that may be documented in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that it conforms to the syntax of the video compression technique or standard, as specified in the video compression technique or standard, and particularly in its profile document. Also, for compliance with certain video compression techniques or standards, the complexity of the encoded video sequence may be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level can be further limited through the Hypothetical Reference Decoder (HRD) specification and metadata about HRD buffer management transmitted in the encoded video sequence.

[0052] In one embodiment, the receiver 310 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder 210 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, time, space, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0053] Figure 4 shows an exemplary functional block diagram of a video encoder 203 associated with a video source 201 according to one embodiment of the present disclosure.

[0054] The video encoder 203 may include, for example, an encoder which is a source coder 430, an encoding engine 432, a (local) decoder 433, a reference picture memory 434, a predictor 435, a transmitter 440, an entropy coder 445, a controller 450, and a channel 460.

[0055] The encoder 203 may receive video samples from a video source 201 (which is not part of the encoder), and the video source 201 may capture video images to be encoded by the encoder 203.

[0056] The video source 201 may provide a source video sequence to be encoded by the encoder 203 in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media delivery system, the video source 201 may also be a storage device that stores pre-prepared video. In a video conferencing system, the video source 203 may also be a camera that captures local image information as a video sequence. The video data may be provided as a series of individual pictures that convey motion when viewed sequentially. The picture itself may be composed of a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.

[0057] According to one embodiment, the encoder 203 may encode and compress the pictures of the source video sequence into an encoded video sequence 443 in real time or under any other time constraints required by the application. Achieving an appropriate encoding speed is one function of the controller 450. The controller 450 may also control other functional units, as described below, and may be functionally coupled to these units. The couplings are not shown for clarity. Parameters set by the controller 450 may include rate control-related parameters (such as picture skipping, quantization, lambda values ​​for rate distortion optimization techniques), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art will readily recognize other functions of the controller 450 that may be relevant to a video encoder 203 optimized for a particular system design.

[0058] Some video encoders operate in what is readily recognizable to those skilled in the art as an "encoding loop." In very simplified terms, the encoding loop can consist of an encoding portion of a source coder 430 (responsible for generating symbols based on the input picture and reference picture to be encoded) and a (local) decoder 433 embedded in the encoder 203. The decoder 433 reconstructs the symbols to generate sample data similar to what a (remote) decoder would generate, provided that the compression between the symbols and the encoded video bitstream is reversible in a given video compression technique. The reconstructed sample stream may be fed into a reference picture memory 434. Since decoding the symbol stream yields bitwise-accurate results independent of the decoder's location (local or remote), the contents of the reference picture memory are also bitwise-accurate between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder "sees" when using predictions during decoding. This basic principle of the synchronization of reference pictures (including, for example, drift resulting from the inability to maintain synchronization due to channel errors) is known to those skilled in the art.

[0059] The operation of the "local" decoder 433 can be the same as that of the "remote" decoder 210, which has already been described in detail above in relation to Figure 3. However, since symbols are available and the encoding / decoding of symbols to the encoded video sequence by the entropy coder 445 and parser 320 can be reversible, the entropy decoding portion of the decoder 210, including channel 312, receiver 310, buffer memory 315 and parser 320, does not have to be fully implemented in the local decoder 433.

[0060] A consideration that can be made at this point is that any decoder technique present within the decoder, excluding analysis / entropy decoding, must exist in substantially the same functional form within the corresponding encoder. Therefore, the subject of this disclosure will focus on decoder operation. A description of encoder techniques can be omitted, as it may be the inverse of a comprehensive description of decoder techniques. More detailed explanations are necessary only in specific areas and are provided below.

[0061] As part of its operation, the source coder 430 may perform motion-compensated predictive coding, which predictively codes the input frame by referencing one or more previously coded frames from a video sequence designated as “reference frames”. In this way, the coding engine 432 codes the difference between the pixel blocks of the input frame and the pixel blocks of the reference frame that may be selected as a predictive reference for the input frame.

[0062] The local decoder 433 may decode the encoded video data of a frame that may be designated as a reference frame based on the symbols generated by the source coder 430. The operation of the encoding engine 432 may, advantageously, be lossy. If the encoded video data can be decoded by a video decoder (not shown in Figure 4), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local decoder 433 may replicate the decoding process that may be performed by the video decoder on the reference frame and store the reconstructed reference frame in the reference picture memory 434. Thus, the encoder 203 may locally store a copy of the reconstructed reference frame having common content as a reconstructed reference frame (without transmission errors) to be acquired by the far-end video decoder.

[0063] The predictor 435 may perform a predictive search for the coding engine 432. That is, for a new frame to be coded, the predictor 435 may search the reference picture memory 434 for sample data (as candidate reference pixel blocks) or specific metadata (reference picture motion vector, block shape, etc.). These may function as appropriate predictive references for the new picture. The predictor 435 may operate sample block-by-pixel block to find appropriate predictive references. In some cases, the input picture determined by the search results obtained by the predictor 435 may have predictive references drawn from multiple reference pictures stored in the reference picture memory 434.

[0064] The controller 450 may manage the encoding operation of the source coder 430, including, for example, setting parameters and subgroup parameters used to encode video data.

[0065] The outputs of all the above functional units may undergo entropy coding in the entropy coder 445. The entropy coder converts the symbols generated by the various functional units into coded video sequences by lossless compression of the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0066] Transmitter 440 may buffer the encoded video sequence generated by entropy coder 445 and prepare it for transmission over communication channel 460, which may be a hardware / software link to a storage device that stores the encoded video data. Transmitter 440 may merge the encoded video data from coder 430 with other data to be transmitted (e.g., encoded audio data and / or auxiliary data streams (not shown)).

[0067] The controller 450 may manage the operation of the encoder 203. During encoding, the controller 450 may assign a specific encoded picture type to each encoded picture. This encoded picture type may influence the encoding technique that can be applied to each picture. For example, a picture may often be assigned as an intra-picture (I-picture), a predictive picture (P-picture), or a bidirectional predictive picture (B-picture).

[0068] An intra-picture (I-picture) may be one that can be encoded and decoded without using other frames in the sequence as a source for prediction. Some video codecs enable different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will recognize these variations of I-pictures, as well as their respective uses and characteristics.

[0069] The prediction picture (P-picture) may be encoded and decoded using intra-prediction or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0070] A bidirectional predictive picture (B-picture) may be encoded and decoded using intra-prediction or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multiple predictive picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0071] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block may be coded. Blocks may be coded predictively by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, blocks of picture I may be coded unpredictably, or they may be coded predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded unpredictably by referencing one previously coded reference picture, via spatial prediction or temporal prediction. Blocks of picture B may be coded unpredictably by referencing one or two previously coded reference pictures, via spatial prediction or temporal prediction.

[0072] The video encoder 203 may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In these operations, the video encoder 203 may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to the syntax specified by the video coding technique or standard being used.

[0073] In one embodiment, the transmitter 440 may transmit additional data along with the encoded video. The source coder 430 may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, and the like.

[0074] [Block partitioning in VP9 and AV1]

[0075] As described above, AV1 is an open video coding format designed for video transmission over the internet, developed as a successor to VP9. Referring to Figure 5A, VP9 uses a four-way partition tree starting at 64x64 levels down to 4x4 levels, with some further limitations on blocks below 8x8 (as shown in the upper half of Figure 5A). Partitions designated as R may be called recursive in that the same partition tree may be repeated at lower scales until the partition reaches the lowest 4x4 level.

[0076] As shown in Figure 5B, AV1 not only extends the partition tree to a 10-way structure but also increases the maximum size (called a superblock in VP9 / AV1 terminology) to start at 128x128. This may include 4:1 / 1:4 rectangular partitions that were not present in VP9. None of the rectangular partitions may be further subdivided. Furthermore, AV1 adds more flexibility to the use of partitions below the 8x8 level, in the sense that 2x2 chromainter prediction becomes possible in certain cases.

[0077] [Block partitioning in HEVC]

[0078] In HEVC, a coding tree unit (CTU) may be divided into coding units (CU) by using a quad-tree (QT) structure called a coding tree to adapt to various local characteristics. The decision of whether to code a picture region using interpicture (time) prediction or intrapicture (spatial) prediction may be made at the CU level. Each CU can be further divided into one, two, or four prediction units (PU) according to the PU partitioning type. Within a single PU, the same prediction process may be applied, and relevant information may be transmitted to the decoder for each PU. After obtaining residual blocks by applying the prediction process based on the PU partitioning type, the CU can be partitioned into transform units (TU) according to other QT structures such as coding trees for the CU. One of the key features of the HEVC structure is that it has multiple partition concepts, including CUs, PUs, and TUs. In HEVC, CUs or TUs can only be square in shape, while PUs can be square or rectangular in shape for inter-prediction blocks. In HEVC, a single coding block may be further divided into four square subblocks, and a transformation is performed on each subblock, i.e., the TU. Each TU may be further recursively divided into smaller TUs (using QT partitioning), which is called a Residual Quad-Tree (RQT).

[0079] At picture boundaries, HEVC employs implicit quadtree partitioning so that blocks maintain their quadtree partitioning until their size fits the picture boundary.

[0080] [Block partitioning in VVC]

[0081] 1. Block partitioning structure using quad-tree (QT) + BT (binary tree)

[0082] The QTBT block structure eliminates the concept of multiple partition types. That is, the QTBT block structure eliminates the separation of the concepts of CU, PU, ​​and TU, and supports more flexibility for the CU partition shape. In the QTBT block structure, the CU may have a square or rectangular shape. Referring to Figures 6A and 6B, the coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a binary tree structure. There are two types of BT partitioning: symmetric horizontal partitioning and symmetric vertical partitioning. The BT leaf nodes may also be called CUs, and their segmentation is used for prediction and transformation processing without further partitioning. This means that CU, PU, ​​and TU have the same block size in the QTBT coding block structure. In the joint exploration model (JEM), the CU may consist of coding blocks (CBs) of different color components. For example, in the case of prediction (P) and binary (B) slices of a 4:2:0 chroma format, one CU may contain one luma CB and two chroma CBs. A CU may also consist of a single-component CB or contain a single-component CB. For example, in the case of an I slice, one CU may contain only one luma CB or only two chroma CBs.

[0083] The following parameters may be defined for the QTBT partitioning scheme. (a) CTU size: The size of the root node of a quadtree, which can be considered the same concept as HEVC. (b) MinQTSize: Minimum allowable quadtree leaf node size (c) MaxBTSize: Maximum allowable binary tree root node size (d) MaxBTDepth: Maximum allowable binary tree depth (e) MinBTSize: Minimum allowable binary tree leaf node size

[0084] In one example of a QTBT partitioning structure, the CTU size may be set to a 128x128 chroma sample with two corresponding 64x64 blocks of chroma samples, MinQTSize may be set to 16x16, MaxBTSize may be set to 64x64, MinBTSize (for both width and height) may be set to 4x4, and MaxBTDepth may be set to 4. The quadtree partitioning may be applied to the CTU first to generate QT leaf nodes. QT leaf nodes may have sizes ranging from 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If a leaf QT node is 128x128, it does not need to be further partitioned by a binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, the leaf QT node may be further partitioned by a binary tree. Thus, the QT leaf node is also the root node of the BT and has a BT depth of zero. When the BT depth reaches MaxBTDepth (i.e., 4), further partitioning may not be considered. When a BT node has a width equal to MinBTSize (i.e., 4), further horizontal partitioning may not be considered. Similarly, when a BT node has a height equal to MinBTSize, further vertical partitioning may not be considered. The leaf nodes of the BT may be further processed by prediction and transformation processes without further partitioning. In JEM, for example, the maximum CTU size may be 256 × 256 luma samples.

[0085] Figure 6A shows an example of block partitioning using QTBT, and Figure 6B shows the corresponding tree representation. Solid lines represent QT partitions, and dotted lines represent BT partitions. At each BT partition (i.e., non-leaf) node, a flag may be signaled to indicate which partition type (i.e., horizontal or vertical) is used, where 0 indicates a horizontal partition and 1 indicates a vertical partition. In the case of QT partitions, it may not be necessary to indicate the partition type, as a QT partition can always divide a block both horizontally and vertically to produce four subblocks of equal size.

[0086] The QTBT method may support flexibility for luma and chroma so that they have separate QTBT structures. Currently, for P-slice and B-slice, the luma CTB and chroma CTB within one CTU share the same QTBT structure. However, for I-slice, the luma CTB is partitioned into CUs by the QTBT structure, and the chroma CTB is partitioned into chroma CUs by the other QTBT structure. This means that the CUs in I-slice consist of coding blocks for the luma component or coding blocks for the two chroma components, and the CUs in P-slice or B-slice consist of coding blocks for all three color components.

[0087] In HEVC, inter-prediction for small blocks is limited to reduce motion-compensated memory access, bi-prediction is not supported for 4x8 and 8x4 blocks, and inter-prediction is not supported for 4x4 blocks. These limitations are removed in QTBT, which is implemented in JEM-7.0.

[0088] 2. Block partitioning using a ternary tree (TT)

[0089] In VVC, a multi-type-tree (MTT) structure is included, as shown in Figures 7A and 7B. This adds vertical central ternary trees and horizontal central ternary trees to the QTBT. Figure 7A shows a vertical central ternary tree partition, and Figure 7B shows a horizontal central ternary tree partition.

[0090] The main advantages of ternary tree partitioning include, but are not limited to, (a) providing a complement to quadtree and binary tree partitioning, i.e., while quadtrees and binary trees always partition along the center of a block, ternary tree partitioning can capture objects located at the center of a block, and (b) the width and height of the proposed ternary tree partitions may always be powers of 2, and therefore no further transformation may be required.

[0091] Two-level tree design is primarily motivated by complexity reduction. Theoretically, the complexity of traversing a tree is T D Here, T represents the number of partition types and D is the depth of the tree.

[0092] [Merge mode with motion vector differences (MMVD)]

[0093] In merge mode, implicitly derived motion information is used directly for predicting samples for the current CU. A merge mode with motion vector differences (MMVD) is introduced in VVC. The MMVD flag is transmitted immediately after the skip and merge flags are sent to specify whether MMVD mode is used for the CU.

[0094] In MMVD, after a merge candidate is selected, this may be further refined by signaled motion vector difference (MVD) information. The MVD information may include a merge candidate flag, an index to specify the magnitude of the motion, and an index to indicate the direction of the motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the motion vector (MV) base. A merge candidate flag may be signaled to specify which of the first two candidates is used.

[0095] A distance index may be used, or motion magnitude information may be specified, indicating a predetermined offset from the starting point. Figure 8 shows the MMVD search points of two reference frames according to several embodiments. As shown in Figure 8, the offset may be added to either the horizontal or vertical component of the starting MV. The relationship between the distance index and the predetermined offset is specified in Table 1 below. [Table 1]

[0096] A direction index may be used, which may represent the direction of the MVD relative to the starting point. The direction index may represent one of four directions, as shown in Table 2 below. The meaning of the MVD code may change according to the information of the starting MV. When the starting MV is a single-prediction MV or a bi-prediction MV, and both lists point to the same side of the current picture (i.e., the picture order counts (POCs) of the two references are either greater than or less than the POC of the current picture), the code in Table 2 specifies the code of the MV offset added to the starting MV. If the starting MV is a bipredicted MV, and the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and the difference in POCs in List 0 (L0) is greater than the difference in POCs in List 1 (L1), then the sign in Table 2 specifies the sign of the MV offset added to the L0 MV component of the starting MV, and the sign of the L1 MV has the opposite value. Otherwise, if the difference in POCs in L1 is greater than that of L0, then the sign in Table 2 specifies the sign of the MV offset added to the L1 MV component of the starting MV, and the sign of the L0 MV has the opposite value.

[0097] The MVD may be scaled according to the difference in POCs in each direction. If the difference in POCs in both lists is the same, scaling may not be necessary. Otherwise, if the difference in POCs in L0 is greater than the difference in L1, the MVD of L1 is scaled. If the difference in POCs in L1 is greater than that of L0, the MVD of L0 may be scaled in the same way. If the initial MV is single-predicted, the MVD may be added to the available MVs. [Table 2]

[0098] [Symmetric MVD coding]

[0099] In VVC, in addition to the signal transmission of MVD in the normal unidirectional and bidirectional prediction modes, a symmetric MVD mode may be applied for the signal transmission of bidirectional MVD. In symmetric MVD mode, motion information including the reference picture indices of both L0 and L1 and the MVD of L1 may be derived, and may not be explicitly signaled.

[0100] The decoding process for symmetric MVD mode may also be as follows:

[0101] At the slice level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 may be derived as follows: If mvd_l1_zero_flag is 1, BiDirPredFlag is set to equal 0. Otherwise, if the nearest reference picture in L0 and the nearest reference picture in L1 form a forward-backward or backward-backward reference picture pair, BiDirPredFlag is set to 1, and both the L0 and L1 reference pictures are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0.

[0102] At the CU level, if the CU is bipredictively coded and BiDirPredFlag is equal to 1, a symmetric mode flag indicating whether symmetric mode is used may be explicitly signaled. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 may be explicitly signaled. The reference indices of L0 and L1 may be set to equal the pair of reference pictures, respectively. MVD1 may be set to equal (-MVD0).

[0103] [Intermode coding in CWG-B018]

[0104] In AV1, for each coding block within an interframe, if the current block's mode is intercoding mode and not skip mode, other flags may be signaled to indicate whether single-reference mode or composite-reference mode is used for the current block. In single-reference mode, the prediction block may be generated by a single motion vector. In composite-reference mode, the prediction block may be generated by a weighted average of two prediction blocks derived from two motion vectors.

[0105] For a single reference, the following modes may be transmitted: (a) NEARMV - Uses one of the motion vector predictors (MVPs) in the list indicated by the Dynamic Reference List (DRL) index. (b) NEWMV - Use one of the MVPs in the list signaled by the DRL index as a reference and apply the difference to the MVP. (c)GLOBALMV - Uses motion vectors based on frame-level global motion parameters

[0106] In the case of composite references, the following modes may be transmitted as signals: (a) NEAR_NEARMV - Uses one of the MVPs in the list signaled by the DRL index. (b) NEAR_NEWMV - Uses one of the MVPs in the list signaled by the DRL index as a reference and sends a differential MV for the second MV. (c) NEW_NEARMV - Uses one of the MVPs in the list signaled by the DRL index as a reference and sends a differential MV for the first MV. (d) NEW_NEWMV - Uses one of the MVPs in the list signaled by the DRL index as a reference and sends the differential MV for both MVs. (e)GLOBAL_GLOBALMV - Uses MV from each reference based on frame-level global motion parameters.

[0107] [Motion vector difference coding in AV1] AV1 enables motion vector accuracy (or precision) of 1 / 8 pixel, and the following syntax

[0108] This may be used to signal the motion vector difference in reference frame L0 or L1.

[0109] According to the embodiment, the syntax mv_joint specifies which component of the motion vector difference is non-zero. A value of 0 may indicate that no non-zero MVD exists along either the horizontal or vertical direction. A value of 1 may indicate that a non-zero MVD exists along the horizontal direction only. A value of 2 may indicate that a non-zero MVD exists along the vertical direction only. A value of 3 may indicate that a non-zero MVD exists along both the horizontal and vertical directions.

[0110] According to one embodiment, the syntax element mv_sign specifies whether the motion vector difference is positive or negative.

[0111] According to the embodiment, the syntax element mv_class specifies the class of the motion vector difference. As shown in Table 3 below, a higher class means that the motion vector difference has a larger magnitude. [Table 3]

[0112] According to one embodiment, the syntax element mv_bit specifies the integer part of the offset between the motion vector difference and the magnitude of the start of each MV class.

[0113] According to one embodiment, the syntax element mv_fr specifies the first two fractional bits of the motion vector difference.

[0114] According to one embodiment, the syntax mv_hp specifies the third fractional bit of the motion vector difference.

[0115] [Adaptive MVD resolution in CWG-B092]

[0116] For NEW_NEARMV and NEAR_NEWMV modes, the precision of the MVD may depend on the relevant class and the size of the MVD.

[0117] First, fractional MVDs may only be permitted if the size of the MVD is 1 pixel or less.

[0118] Next, when the value of the relevant MV class is MV_CLASS_1 or greater, only one MVD value may be allowed, and the MVD values ​​for each MV class may be derived as 4, 8, 16, 32, and 64 for MV class 1 (MV_CLASS_1), MV class 2 (MV_CLASS_2), MV class 3 (MV_CLASS_3), MV class 4 (MV_CLASS_4), or MV class 5 (MV_CLASS_5).

[0119] The permissible MVD values ​​for each MV class are shown in Table 4 below. [Table 4]

[0120] In some embodiments, if the current block is coded using NEW_NEARMV or NEAR_NEWMV mode, one context may be used to signal mv_joint or mv_class. If the current block is not coded using NEW_NEARMV or NEAR_NEWMV mode, another context may be used to signal mv_joint or mv_class.

[0121] [Joint MVD Coding (JMVD) in CWG-B092]

[0122] A new intercoding mode named JOINT_NEWMV may be applied to indicate whether the MVDs for two reference lists are signaled together. If the inter-prediction mode is equal to the JOINT_NEWMV mode, the MVDs for reference lists L0 and L1 are signaled together. Thus, only one MVD named joint_mvd may be signaled and transmitted to the decoder, and the difference MV for reference lists L0 and L1 may be derived from joint_mvd.

[0123] The JOINT_NEWMV mode may be signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes. No further context is required.

[0124] When the JOINT_NEWMV mode is signaled and the POC distances between two reference frames and the current frame are different, the MVD may be scaled with respect to reference list L0 or reference list L1 based on the POC distance. Specifically, the distance between reference frame list L0 and the current frame is shown as td0, and the distance between reference frame list L1 and the current frame is shown as td1. If td0 is greater than or equal to td1, joint_mvd may be used directly for reference list L0, and the mvd for reference L1 may be derived from joint_mvd based on the following equation (1). derived_mvd=td1 / td0*joint_mvd Formula (1)

[0125] Otherwise, if td1 is greater than or equal to td0, joint_mvd may be used directly for the reference list L1, and mvd for the reference list L0 may be derived from joint_mvd based on the following equation (2). derived_mvd=td0 / td1*joint_mvd Formula (2)

[0126] [Improvements to Adaptive MVD in CWG-C011]

[0127] A new intercoding mode named AMVDMV may be added for single references. When the AMVDMV mode is selected, this may indicate that AMVD is applied to the signal MVD.

[0128] To indicate whether AMVD is applied to a joint MVD coding mode, a flag named amvd_flag may be added under the JOINT_NEWMV mode. When adaptive MVD resolution is applied to a joint MVD coding mode named joint AMVD coding, the MVDs for two reference frames may be signaled together, and the precision of the MVD may be implicitly determined by the size of the MVD. Otherwise, the MVDs for two (or more) reference frames are signaled together, and MVD coding is applied.

[0129] [Adaptive motion vector resolution (AMVR) in CWG-C012 and CWG-C020]

[0130] AMVR was first proposed in CWG-C012 and supports a total of seven MV accuracies (i.e., 8, 4, 2, 1, 1 / 2, 1 / 4, and 1 / 8). For each prediction block, the AVM encoder searches all supported accuracy values ​​and signals the best accuracy to the decoder.

[0131] To reduce encoder execution time, two precision sets may be supported. Each precision set may contain four predetermined precisions. The precision set may be adaptively selected at the frame level based on the maximum precision value of the frame. Similar to AV1, the maximum precision may be signaled in the frame header. Table 5 below summarizes the precision values ​​supported based on the frame-level maximum precision. [Table 5]

[0132] In AVM software (similar to AV1), a frame-level flag may exist to indicate whether the frame's MV includes sub-PEL precision. AMVR may only be enabled if the value of the cur_frame_force_integer_mv flag is 0. In AMVR, if the block precision is lower than the maximum precision, the motion model and interpolation filter may not be transmitted. If the block precision is lower than the maximum precision, the motion mode may be estimated as translational motion, and the interpolation filter may be estimated as a REGULAR interpolation filter. Similarly, if the block precision is either 4 PEL or 8 PEL, the inter-intra mode may not be transmitted and may be estimated as 0.

[0133] [Warp movement mode in AV1]

[0134] Motion compensation typically assumes a translational motion model between a reference block and a target block. However, warped motion utilizes an affine model.

[0135] The affine motion model can be expressed by the following equation (3).

number

[0136] 1. Global warp motion compensation

[0137] In global warp motion compensation, global motion information, including the global motion type and several motion parameters, may be signaled for each interreference frame. The number of global motion types and associated parameters are listed in Table 6 below. [Table 6]

[0138] If a global motion is selected after the reference frame index has been signaled, the global motion type and parameters associated with the given reference frame may now be used for the coding block.

[0139] 2. Local warp motion compensation

[0140] Local warp movement may be permitted for an interconnecting block if the following conditions are met: (1) The current block uses single reference prediction. (2) The width or height of the interconnecting block is 8 or greater. (3) At least one of the adjacent neighboring blocks uses the same reference frame as the current block.

[0141] When local warp motion is used for the current block, the affine model parameters may be estimated by minimizing the mean square of the difference between the reference projection and the modeled projection, based on the MV of the current block and its adjacent neighboring blocks. To estimate the parameters of local warp motion, if the adjacent block uses the same reference frame as the current block, projected sample pairs may be obtained between the central sample in the adjacent block and its corresponding sample in the reference frame. Subsequently, three additional samples may be created by shifting the central position by a quarter sample in one or both dimensions. These additional samples may also be considered projected sample pairs to ensure stability in the model parameter estimation process.

[0142] The motion samples of adjacent blocks used to derive motion parameters may be called motion samples. Motion samples may be selected from adjacent blocks that use the same reference frame as the current block. The warp motion prediction mode may be enabled only for blocks that use a single reference frame. For example, referring to Figure 9, the motion samples of adjacent blocks B0, B1, and B2 are called MV0, MV1, and MV2, respectively. Figure 6 shows exemplary motion samples used to derive model parameters of blocks using local warp motion prediction in several embodiments. Figure 9 shows exemplary motion samples used to derive model parameters of blocks using local warp motion prediction in several embodiments. The current block 600 may be predicted using a single prediction with reference frame Ref0. Adjacent block B0 may be predicted using a composite prediction with reference frames Ref0 and Ref1. Adjacent block B1 may be predicted using a single prediction with reference frame Ref0. Adjacent block B2 may be predicted using a composite prediction with reference frames Ref0 and Ref2. The motion vectors MV0Ref0 of adjacent block B0, MV1Ref0 of adjacent block B1, and MV2Ref0 of adjacent block B2 may now be used as motion samples to derive the affine motion parameters of block 600.

[0143] 3. Sample acquisition process for local warp motion in AV1 and AVM

[0144] For local warp motion, samples and motion vectors from adjacent blocks may be collected in a stack to compute a warp model using regression / least squares error. For AV1 and AVM, the sample positions of one group in a reference picture and their reference positions pointed to by its MV may be collected. Up to eight pairs of positions may be stored in the stack for regression / least squares error warp model computation.

[0145] Figure 10 is a schematic diagram of an example of a warp sample acquisition scan sequence. As shown in Figure 10, only spatially adjacent blocks may be scanned. If an adjacent block has the same reference picture as the current block 610 (for example, if the adjacent block is composite, one of the reference pictures is the same as the reference picture of the current block), the center position of the adjacent block and the center position of the reference block of the adjacent block pointed to by the MV of the adjacent block may be acquired. Referring to Figure 10, the scan sequence may be defined as follows: (1) the adjacent block 611 in the top row is scanned, (2) the adjacent block 612 in the left column is scanned, (3) the upper left adjacent block 613 is scanned, and (4) the upper right adjacent block 614 is scanned.

[0146] [Local warp extension mode and local warp difference mode from CWG-C050]

[0147] In addition to translational motion, AVM also supports warp motion compensation. Two types of warp motion models are supported: a global warp model and a local warp model. The global warp model is associated with each reference frame, with each of the four non-translational parameters having 12-bit precision, and the translational motion vector being coded with 15-bit precision. The coding block may choose to use it directly (a reference frame index is provided). The global warp model captures frame-level scaling and rotation, and therefore primarily focuses on inflexible motion across the entire frame.

[0148] Furthermore, local warp models at the coding block level are also supported. In local warp mode, also known as WARPED_CAUSAL, the warp parameters of the current block may be derived by fitting the model to nearby motion vectors using least squares.

[0149] 1. Warp Expansion Mode

[0150] The new warp movement mode is called WARP_EXTEND. In WARP_EXTEND mode, the movement of adjacent blocks smoothly extends to the current block, but retains some ability to modify warp parameters. This allows for the representation of complex warping movements that spread across multiple blocks while minimizing blocking artifacts.

[0151] To achieve this, the WARP_EXTEND mode applied to the NEWMV block constructs a new warp model based on two constraints: (1) the per-pixel motion vectors generated by the new warp model must be continuum with the per-pixel motion vectors in adjacent blocks, and (2) the pixel at the center of the current block must have a per-pixel motion vector that matches the motion vectors signaled for the block as a whole.

[0152] For example, referring to Figure 11, if the adjacent block 622 to the left of block 620 is currently warped, a warp model that fits the motion vector may be used. Figure 11 shows motion vectors within a block using a warp extension mode according to several embodiments.

[0153] The two constraints above for constructing a new warp model may mean specific equations containing the warp parameters of the adjacent block and the current block. These equations may then be solved to compute the warp model for the current block.

[0154] For example, if (A,...,F) represents adjacent warp models and (A',...,F') represents a new warp model, the first constraint is given by equation (4) below at each point along a common edge.

number

[0155] Points along the edge have different values ​​for y, but they all have the same value for x. This means that the coefficient of y must be the same on both sides (i.e., B'=B and D'=D). On the other hand, the coefficient of x provides two equations with respect to the other coefficients, as shown in equations 5-8 below.

number

[0156] The second constraint specifies that the motion vector of the center of the block must be equal to that transmitted using the NEWMV mechanism. This gives rise to two further equations, resulting in a system of six equations in six variables, each with a unique solution.

[0157] These equations can be efficiently solved in both software and hardware. The solutions can be found using basic addition, subtraction, multiplication, and division by powers of two. Therefore, this mode is not significantly more complex than least-squares-based local warp modes.

[0158] In some embodiments, there may be multiple adjacent blocks that can serve as the source for extension. In such cases, it is useful to have several ways of selecting which block to extend from. This problem is similar to the problems faced in motion vector prediction. In particular, there may be several possible motion vectors from nearby blocks, and one of them must be selected as the base for NEWMV coding.

[0159] A solution to the above problem may include extending the algorithm to handle the need for WARP_EXTEND. This may be done by tracking the source of each motion vector prediction. WARP_EXTEND may then be enabled only if the selected motion vector prediction is taken from a directly adjacent block. That block may then be used as a single "adjacent block" for the rest of the algorithm.

[0160] In some cases, the adjacent warp model will be perfectly fine as is, without requiring any further modifications. To code this more cheaply, WARP_EXTEND may be used in the NEARMV block. The adjacent selection may be the same as in NEWMV, except that the selection in NEWMV requires the adjacent to be warped rather than simply translated. However, if this is true and WARP_EXTEND is selected, the adjacent warp model parameters may now be copied into the block.

[0161] 2. Warp Difference Mode

[0162] In some embodiments, a motion mode called WARP_DELTA may be provided. In this mode, the block's warp model is coded as a difference from the predicted warp model, similar to how the motion vector is coded as a difference from the predicted motion vector. The prediction may be supplied from either the global motion model (if any) or the adjacent block.

[0163] To avoid having multiple ways to encode the same predictive warp model, the following restrictions may apply: If the mode is NEARMV or NEWMV, the same adjacency selection logic described for WARP_EXTEND may be used. If this results in a warped adjacency block, the model of that block (without applying the rest of the WARP_EXTEND logic) may be used as the predictor. Otherwise, the global warp model may be used as the base.

[0164] Next, the difference for each of the non-translational parameters may be coded. Finally, the translational portion of the model may be adjusted so that the motion vector for each pixel at the center of the block matches the overall motion vector of the block.

[0165] This tool (i.e., WARP_DELTA) uses more bits to encode than other warp modes because it involves explicitly coding the difference for each warp parameter. Therefore, WARP_DELTA may be disabled for blocks smaller than 16x16. However, the decoding logic is extremely simple and can represent more complex movements that are not possible with other warp modes.

[0166] [Merge with Motion Vector Difference (MMVD)]

[0167] Merge with Motion Vector Difference (MMVD) may be used for either skip mode or merge mode using a motion vector representation method. Figure 12 shows the MMDV search process for the current frame 630, including the current block 632, using two reference frames 634 and 636 (for example, the two reference frames shown in Figure 8). MMVD reuses merge candidates within the VVC. Candidates are selected from the merge candidates and may be further extended by a motion vector representation method. MMVD provides a new motion vector representation using simple signal transmission. The representation method may include the starting point, the magnitude of the motion, and the direction of the motion. The MMVD technique may use the merge candidate list of the VVC. However, only candidates of the default merge type (MRG_TYPE_DEFAULT_N) may be considered for MMVD extension. A base candidate index (IDX) may define the starting point. The base candidate IDX may indicate the best candidate among the candidates in the list, as shown in Table 7 below. [Table 7]

[0168] If the number of base candidates is equal to 1, base candidate IDX does not need to be signaled.

[0169] A distance index representing the magnitude of the movement may be used. The distance index may indicate a predetermined distance from the starting point information. The predetermined distance may be as shown in Table 8 below. [Table 8]

[0170] A direction index may be used to represent the direction of the MVD relative to the starting point. The direction index may represent four directions, as shown in Table 9 below. [Table 9]

[0171] The MMVD flag may be signaled immediately after the skip and merge flags have been transmitted. If the skip and merge flags are true, the MMVD flag may be parsed. If the MMVD flag is equal to 1, the MMVD syntax may be parsed. However, if it is not 1, the AFFINE flag may be parsed. If the AFFINE flag is equal to 1, the AFFINE mode may be used. However, if it is not 1, the skip / merge index may be parsed for the VTM's skip / merge mode.

[0172] [Challenges in related technologies]

[0173] In the current warp difference signaling of CWG-C050, only two difference values ​​are allowed for rotation and zoom warp. Furthermore, the difference values ​​are signaled using a fixed step length (also called "size"). These are suboptimal for warp model correction.

[0174] [Example Embodiments]

[0175] The embodiments of this disclosure may be used separately or combined in any order. Furthermore, each of the embodiments of this disclosure (e.g., methods, encoders, and decoders) may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium. Hereinafter, the term “block” may be interpreted as a prediction block, coding block, or coding unit (CU).

[0176] According to one embodiment, a warp type (or the number of parameters to be corrected by the difference value) is first transmitted within the bitstream, and depending on the warp type, the corresponding difference value is transmitted within the bitstream. According to another embodiment, at least one syntax element that explicitly indicates the warp type to be used (indicating the number of parameters to be corrected by the difference value) may be transmitted within the bitstream by the encoder, retrieved from the bitstream by the decoder, and used by the decoder to decode at least one coding picture of the bitstream by obtaining a warp model and performing warp motion compensation based on the warp model. For example, the current block may be predicted by performing warp motion compensation using the warp model.

[0177] According to one or more embodiments, two different warp types may be signaled within the bitstream. The two warp types are (1) a rotational and zoom type having four parameters (including translational parameters), and (2) an affine type having six parameters. For example, the four parameters of the rotational and zoom type may be represented as [a, -b, e; b, a, f], and the six parameters of the affine type may be represented as [a, b, c; d, e, f]. Two difference values ​​may be signaled (e.g., within the bitstream) to correct parameters a and b, respectively, based on the rotational and zoom type being used. Four difference values ​​may be signaled (e.g., within the bitstream) to correct parameters a, b, c, and d, respectively, based on the affine type being used. According to embodiments, a decoder (or encoder) may correct the parameters based on the signaled difference values ​​and perform warp motion compensation based on the corrected parameters. According to embodiments, the parameters may also be motion parameters that define the warp motion of a warp model having a warp type. According to one embodiment, the decoder (or encoder) may, for example, obtain a warp model (e.g., corrected motion parameters) of the current block based on the transmitted difference value, and perform warp motion compensation on the current block based on the obtained warp model.

[0178] In one or more embodiments, a syntax element is signaled indicating how many difference values ​​will be used. For example, if the value of the syntax element is equal to 0, no difference values ​​are signaled; if the value of the syntax element is equal to 1, only the difference value correcting parameter a is signaled; if the syntax element is equal to 2, difference values ​​correcting parameters a and b (e.g., two difference values ​​for each) are signaled; if the syntax element is equal to 3, difference values ​​correcting parameters a, b, and c (e.g., three difference values ​​for each) are signaled; and if the syntax element is equal to 4, difference values ​​correcting parameters a, b, c, and d (e.g., four difference values ​​for each) are signaled. In some embodiments, parameters e and f may be derived (e.g., by a decoder) using parameters a, b, c, d, and MV. Thus, it may not be necessary to further signal the difference values ​​to be obtained. According to one embodiment, the decoder (or encoder) may correct warp-type parameters (e.g., motion parameters) based on the transmitted difference values, derive other warp-type parameters (e.g., motion parameters) from the corrected parameters, and perform warp motion compensation based on the corrected parameters and the derived parameters. According to another embodiment, the encoder may transmit no difference values, or one or more difference values, to the decoder (e.g., within a bitstream) based on the values ​​of the syntax elements.

[0179] According to one embodiment, an adaptive step size can be used instead of a fixed step size for the difference value.

[0180] According to one or more embodiments, the step size is scaled from a predetermined base step size, and the scaling factor can be a fraction or an integer. The scaling factor can be signaled per block. According to embodiments, the encoder may signal the scaling factor corresponding to a block in the bitstream (e.g., the current block) to the decoder, which may obtain the scaling factor from the bitstream, scale the step size of the difference value based on the scaling factor and the predetermined base step size, and perform warp motion compensation (e.g., of the current block) based on the difference value and the scaled step size of the difference value. According to embodiments, the encoder may perform the same or similar functions as the decoder described above.

[0181] According to one embodiment, supported / allowed scaling factors are stored in a predetermined lookup table. Instead of directly signaling the scaling factors, the entry index in the table is signaled to the bitstream. According to one embodiment, the index may identify a particular scaling factor stored in the predetermined lookup table. According to one embodiment, the encoder may select the index to be signaled based on the predetermined lookup table and the retrieved scaling factor, and signal the index in the bitstream. According to one embodiment, the decoder may retrieve the index from the bitstream and retrieve the scaling factor based on the index and the predetermined lookup table. According to one embodiment, the supported scaling factors in the lookup table do not have to be evenly distributed, and the scaling factors may have finer granularity the closer the index to which the scaling factor is associated is to an index of zero, and coarser granularity the further the index to which the scaling factor is associated is from an index of zero. For example, the difference between scaling factors corresponding to adjacent indices may be smaller the closer to an index of zero and larger the further away from an index of zero. In one example, the supported scaling factors in the lookup table are [1,2,4,8,16,32].

[0182] According to one or more embodiments, the scaling factor can be derived depending on the magnitude of the current block's translation MV (or parameters, e.g., parameters e and f in a 6-parameter warp model). According to embodiments, the encoder and / or decoder may derive the scaling factor.

[0183] According to one or more embodiments, the scaling factor can be derived linearly from the warp model parameters and based on shear parameters (e.g., alpha, beta, gamma, and theta) that control the range of the warp. According to embodiments, the encoder and / or decoder may derive shear parameters from the warp model parameters and derive the scaling factor based on the shear parameters.

[0184] According to one or more embodiments, the scaling factor can be updated during encoding based on coding information known to both the encoder and the decoder, such as the step size used by the coding block size. According to embodiments, the encoder may update the scaling factor.

[0185] According to one or more embodiments, the step size may depend on other coding information, including, but not limited to, the block size, the prediction mode (e.g., whether NEW mode or NEAR mode is applied), the step size used by the adjacent block, one or more of the predicted warp parameters, and whether the adjacent block is coded by a warp model or a translation model. According to embodiments, the step size may be obtained by an encoder and / or decoder based on other coding information.

[0186] According to one or more embodiments, the step sizes may be unevenly distributed. For example, the step sizes may have finer granularity closer to zero and coarser granularity further from zero. For example, the difference between adjacent step sizes may be smaller closer to zero and larger further from zero.

[0187] According to the embodiment, multiple classes of difference values ​​may be predefined. The class of difference values ​​(depending on the magnitude of the difference value) is first transmitted. Different classes may have different step sizes (precision) of the difference values. According to the embodiment, the class of difference values ​​may be transmitted in the bitstream by an encoder. According to the embodiment, a decoder may obtain the class of difference values ​​from the bitstream and perform warp motion compensation (e.g., of the current block) based on the difference value of the class.

[0188] According to the embodiment, the signal transmission of the sign of the difference value can be concealed. That is, on the decoder side, a warp predictor having position and a negative difference value is constructed, and from among the warp predictors, the warp predictor having the best smoothness with adjacent blocks is determined and may be used as the final warp predictor for performing warp motion compensation.

[0189] According to the embodiment, the step size may be predicted from coding information known to both the encoder and the decoder, including but not limited to the step size used by adjacent blocks, regardless of whether adjacent blocks are coded by a warp model or a translation model, or it may be signaled using a context derived from the coding information. According to the embodiment, the encoder and / or decoder may predict the step size based on the coding information.

[0190] According to the embodiment, at least one processor and memory for storing computer program instructions (i.e., computer code) may be provided. When executed by at least one processor, the computer program instructions may implement an encoder or decoder and perform any number of functions described in this disclosure, including functions performed by the encoder or decoder.

[0191] Referring to Figure 13, for example, at least one processor may implement an encoder 702 or a decoder 704. Computer program instructions may include, for example, an encoding (or decoding) code 710 configured to cause at least one processor to code (or decode) at least one picture received in accordance with an embodiment of the present disclosure. The encoding (or decoding) code 710 may include, for example, a first acquisition code 720, a second acquisition code 730, a third acquisition code 740, a fourth acquisition code 750, a warp motion compensation code 760, an update code 770, a class code 775, a warp predictor code 780, and a transmit code 790. While exemplary code for performing a particular function is described herein, it will be understood that embodiments of the present disclosure may include other code for performing any other function described herein.

[0192] The first acquisition code 720 may be configured to cause at least one processor to acquire warp information in accordance with embodiments of the present disclosure. For example, the warp information may include a warp type indicating the number of parameters to be corrected by a differential value, the number of parameters defining, for example, a warp movement for predicting the current block of a coding picture in accordance with embodiments of the present disclosure.

[0193] The second acquisition code 730 may be configured to cause at least one processor to acquire, in accordance with embodiments of the present disclosure, at least one difference value which is, for example, the difference of at least one parameter that defines a warp movement for predicting the current block.

[0194] A third acquisition code 740 may be configured to cause at least one processor to acquire a warp model based on warp information and at least one difference value, in accordance with embodiments of the present disclosure. In one example, the warp model may be acquired by correcting at least one parameter based on at least one difference value, in accordance with embodiments of the present disclosure. In another example, the warp model may be acquired further based on the step size (e.g., adaptive step size) of at least one difference value, in accordance with embodiments of the present disclosure.

[0195] A fourth acquisition code 750 may be configured to cause at least one processor to acquire at least one differential step size (e.g., an adaptive step size) according to embodiments of the present disclosure. In one example, the step size may be acquired according to embodiments of the present disclosure by explicitly signaling (e.g., within a bitstream) or by deriving the step size. In one example, the step size may be an adaptive step size acquired according to embodiments of the present disclosure based on a scaling factor and a predetermined base step size.

[0196] The warp motion compensation code 760 may be configured to cause at least one processor to predict the current block by, for example, performing warp motion compensation using a warp model, according to embodiments of the present disclosure.

[0197] The update code 770 may be configured to cause at least one processor to update the scaling factor during encoding based on coding information known to both the encoder and the decoder, such as the step size used by the coding block size, according to embodiments of the present disclosure.

[0198] Class code 775 may be configured to cause at least one processor to signal the class of difference values ​​(e.g., in a bitstream) or to retrieve the class of difference values ​​from a bitstream in order to perform warp motion compensation (e.g., of the current block) based on the class difference values, according to embodiments of the present disclosure.

[0199] When decoder 704 is provided, warp predictor code 780 may be configured to cause at least one processor to construct warp predictors having position and negative difference values ​​in accordance with embodiments of the present disclosure, to determine from among the warp predictors the warp predictor having the best smoothness with respect to adjacent blocks, and to use said warp predictor as the final warp predictor for performing warp motion compensation.

[0200] When encoder 702 is provided, the computer program instructions may further include, for example, transmit code 790 which may be configured to cause at least one processor to transmit a bitstream containing a coding picture to a decoder (e.g., decoder 704) according to embodiments of the present disclosure. Encoder 702 may, according to embodiments of the present disclosure, provide information relating to warp motion in the bitstream (e.g., syntax elements), such as the information described herein.

[0201] The technology of embodiments of the present invention is implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 14 shows a computer system 900 suitable for implementing embodiments of the subject matter disclosed.

[0202] Computer software can be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms, and can create code that includes instructions that can be executed directly or through interpretation, microcode execution, etc., by a computer's central processing unit (CPU), graphics processing unit (GPU), etc.

[0203] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game devices, and Internet of Things devices.

[0204] The components of computer system 900 shown in Figure 14 are essentially illustrative and are not intended to imply any limitations on the scope or functionality of computer software implementing embodiments of this disclosure. Furthermore, the configuration of the components should not be construed as having any dependence or requirement on any one or combination of components shown in the exemplary embodiments of computer system 900.

[0205] The computer system 900 may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, through tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, applause, etc.), visual input (gestures, etc.), or olfactory input (not shown). The human interface input devices can also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (voices, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), and video (2D video, 3D video including stereoscopic images, etc.).

[0206] The human interface input device may include one or more of the following: keyboard 901, mouse 902, trackpad 903, touch screen 910, data glove, joystick 905, microphone 906, scanner 907, and camera 908 (only one of each is shown).

[0207] The computer system 900 may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touch screen 910, data glove, or joystick 905, although there may also be tactile feedback devices that do not function as input devices). For example, such devices may include audio output devices (speakers 909, headphones (not shown), etc.), visual output devices (screens 910 including CRT screens, LCD screens, plasma screens, OLED screens, etc., with or without touch screen input functionality and with or without tactile feedback functionality, some of which can output two-dimensional or more-than-three-dimensional visual output through means such as stereoscopic image output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0208] The computer system 900 may also include human-accessible storage devices and their associated media, such as optical media or similar media 921 including CD / DVD ROM / RW 920 having CD / DVD, thumb drives 922, removable hard drives or solid-state drives 93, legacy magnetic media such as tapes and floppy disks (not shown), and special ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0209] Those skilled in the art should also understand that, when used in connection with the subject matter currently disclosed, the term “computer-readable medium” does not encompass a transmission medium, carrier wave, or other transient signal.

[0210] The computer system 900 may also include interfaces to one or more communication networks. These networks can be, for example, wireless, wired, or optical. Furthermore, these networks can be local, wide-area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet® and wireless LAN; cellular networks including GSM®, 3G, 4G, 5G, LTE, etc.; wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial TV; and vehicle and industrial networks including CANBus, etc. Certain networks generally require an external network interface adapter (e.g., a USB port on the computer system 900) attached to a specific general-purpose data port or peripheral bus 949, while others are generally integrated into the core of the computer system 900 by attachment to a system bus (e.g., an Ethernet® interface to a PC computer system or a cellular network interface to a smartphone computer system), as described later. Using any of these networks, the computer system 900 can communicate with other entities. Such communications can be unidirectional, receive-only (e.g., broadcast TV), unidirectional, transmit-only (e.g., from a specific CANbus to a specific CANbus device), or bidirectional, for example, using a local or wide-area digital network to other computer systems. Such communications may include communications to a cloud computing environment 955. As described above, specific protocols and protocol stacks can be used in each of these networks and network interfaces.

[0211] The above-mentioned human interface device, human-accessible storage device, and network interface 954 can be mounted on the core 940 of the computer system 900.

[0212] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) 943, hardware accelerators 944 for specific tasks, etc. These devices may be connected via a system bus 948 along with read-only memory (ROM) 945, random access memory (RAM) 946, internal non-user-accessible hard drives, SSDs, or other internal mass storage 947. In some computer systems, the system bus 948 is accessible in the form of one or more physical plugs to allow for expansion with further CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus 948 or via a peripheral bus 949. The architecture of the peripheral bus includes PCI, USB, etc. A graphics adapter 950 may be included in the core 940.

[0213] The CPU 941, GPU 942, FPGA 943, and accelerator 944 can execute specific instructions that, when combined, constitute the computer code described above. This computer code can be stored in ROM 945 or RAM 946. Temporary data can be stored in RAM 946, while permanent data can be stored, for example, in internal mass storage 947. High-speed storage and retrieval of any of the memory devices can be enabled through the use of cache memory that can be closely associated with one or more CPUs 941, GPUs 942, mass storage 947, ROM 945, RAM 946, etc.

[0214] A computer-readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.

[0215] As an example, and not an limitation, a computer system 900 having an architecture and in particular a core 940 can provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of the core 940 of a non-transient nature, such as core internal mass storage 947 or ROM 945. Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core 940. The computer-readable media can include one or more memory devices or chips according to specific needs. The software can cause the core 940 and in particular the processor (including CPU, GPU, FPGA, etc.) within it to execute specific processes or specific parts of specific processes as described herein, including defining data structures to be stored in RAM 946 and modifying such data structures according to processes defined by the software. Furthermore, or alternatively, a computer system may provide functionality as a result of being embodied in logic hardwired or otherwise in circuitry (e.g., in accelerator 944), which may operate in place of or in conjunction with software to execute specific processes or specific parts of specific processes described herein. References to software include logic, and vice versa, as appropriate. References to computer-readable media may include circuitry that stores software for execution (e.g., integrated circuits (ICs)), circuitry that embodies logic for execution, or both, where appropriate. This disclosure encompasses any appropriate combination of hardware and software.

[0216] While this disclosure describes several non-limiting exemplary embodiments, there are various modifications, substitutions, and equivalent alternatives that fall within the scope of this disclosure. Therefore, those skilled in the art will understand that various systems and methods, not expressly shown or described herein, embody the principles of this disclosure and thus fall within the spirit and scope of this disclosure.

Claims

1. A method performed by the decoder, The steps include receiving a bitstream containing a coding picture, A step of acquiring warp information transmitted within the bitstream, wherein the warp information includes a warp type indicating the number of parameters to be corrected by a difference value, and the number of parameters defines a warp movement for predicting the current block of the coding picture. The steps include obtaining from the bitstream at least one difference value which is the difference of at least one parameter that defines the warp movement for predicting the current block, A step of obtaining a warp model based on the warp information and the at least one difference value, The steps include predicting the current block by performing warp motion compensation using the warp model, and Includes, The warp information includes the number of parameters that define the warp motion, which should be corrected by the difference value. A method wherein the number of at least one difference values ​​obtained is equal to the number of parameters of the warp information to be corrected by the difference values.

2. The warp type is a rotation and zoom type that includes four parameters defining the warp motion, The at least one difference value obtained is two difference values ​​for correcting two of the four parameters, The method according to claim 1, wherein the step of obtaining the warp model includes the step of correcting the two parameters based on the two difference values.

3. The warp type is an affine type that includes six parameters defining the warp motion, The at least one difference value obtained is four difference values ​​for correcting four of the six parameters, The method according to claim 1, wherein the step of obtaining the warp model includes the step of correcting the four parameters based on the four difference values.

4. The step further includes obtaining the adaptive step size of at least one difference value, The method according to claim 1, wherein the step of obtaining the warp model includes the step of obtaining the warp model based on the adaptive step size.

5. The step of obtaining the aforementioned adaptive step size is: Steps to obtain the scaling factor, A step of calculating the adaptive step size based on a predetermined base step size and the scaling factor. The method according to claim 4, including the method described in claim 4.

6. The step of obtaining the scaling factor is: The steps include obtaining an index representing the scaling factor from among multiple scaling factors stored in a lookup table, A step of obtaining the scaling factor based on the index and the lookup table. The method according to claim 5, including the method described in claim 5.

7. The method according to claim 5, wherein the step of obtaining the scaling factor includes the step of deriving the scaling factor based on the magnitude of the translational motion vector of the current block.

8. The method according to claim 5, wherein the step of obtaining the scaling factor includes the step of deriving the scaling factor based on shear deformation parameters.

9. The method according to claim 4, wherein the step of obtaining the adaptive step size includes obtaining the adaptive step size based on one or more of the block size, prediction mode, step size used by at least one adjacent block of the current block, predicted warp parameters, and at least one of the adjacent blocks of the current block being coded by a warp model or a translation model.

10. At least one memory configured to store computer program code, A processor configured to receive a bitstream containing a coding picture, access the computer program code, and perform the method according to any one of claims 1 to 9 in accordance with the computer program code, and A device that includes this.

11. A computer program that causes at least one processor to perform the method according to any one of claims 1 to 9.

Images