Method, apparatus and computer program for improved warped difference signaling - Patents.com

Abstract

A system and method for encoding and decoding using a warped motion mode is provided, the method includes the steps of receiving a bitstream including a coding picture, obtaining warp information signaled in the bitstream, the warp information including a warp type indicating a number of parameters to be corrected by a difference value, the number of parameters defining a warp motion for predicting a current block of the coding picture, obtaining at least one difference value from the bitstream, the difference being a difference of the at least one parameter defining the warp motion for predicting the current block, obtaining a warp model based on the warp information and the at least one difference value, and predicting the current block by performing warped motion compensation using the warp model.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Application No. 63 / 392,388, filed July 26, 2022, and U.S. Patent Application No. 17 / 981,842, filed November 7, 2022, the entire contents of which are incorporated by reference.

[0002] [Technical field] TECHNICAL FIELD Embodiments of the present disclosure relate to advanced image and video coding techniques, and more particularly to improvements in local warp motion modes. [Background technology]

[0003] AV1 (AOMedia Video 1) is an open video coding format designed for video transmission over the Internet. It was developed as a successor to VP9 by AOMedia (Alliance for Open Media), a consortium founded in 2015 that includes the semiconductor industry, video-on-demand providers, video content producers, software developers, and web browser vendors. Many of the components of the AV1 project were derived from previous research efforts by members of the consortium. Individual contributors started experimental technology platforms several years ago. Xiph / Mozilla's Daala already released its code in 2010, Google's experimental VP9 evolution project V10 was announced on September 12, 2014, and Cisco's Thor on August 11, 2015. Built on the VP9 code base, AV1 incorporates further technologies, some of which were developed in these experimental formats. The first version 0.1.0 of the AV1 reference codec was released on April 7, 2016. The consortium announced the release of the AV1 Bitstream Specification on March 28, 2018, along with encoders and decoders based on the reference software. Validated version 1.0.0 of the specification was released on June 25, 2018. Validated version 1.0.0 with errata 1 of the specification was released on January 8, 2019. The AV1 Bitstream Specification includes the reference video codec.

[0004] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (version 1), 2014 (version 2), 2015 (version 3) and 2016 (version 4). They also investigated the potential need for standardization of future video coding technologies that may perform significantly better than HEVC in compression capabilities. In October 2017, they issued a Joint Call for Proposals on Video Compression with Capability beyond HEVC (CfP). By February 15, 2018, a total of 22 CfP responses had been submitted for standard dynamic range (SDR), 12 for high dynamic range (HDR), and 12 for the 360° video category. In April 2018, all received CfP responses were evaluated at the 122 MPEG / 10th JVET (Joint Video Exploration Team or Joint Video Expert Team) meeting. As a result of this meeting, the JVET formally started standardization of the next generation of video coding beyond HEVC. The new standard is called VVC (Versatile Video Coding). Summary of the Invention

[0005] According to an embodiment of the present disclosure, a method is provided for execution by a decoder, the method including the steps of receiving a bitstream including a coding picture, obtaining warp information signaled in the bitstream, the warp information including a warp type indicating a number of parameters to be corrected by a difference value, the number of parameters defining a warp motion for predicting a current block of the coding picture, obtaining at least one difference value from the bitstream, the difference being a difference of the at least one parameter defining the warp motion for predicting the current block, obtaining a warp model based on the warp information and the at least one difference value, and predicting the current block by performing warp motion compensation using the warp model.

[0006] According to one or more embodiments of the present disclosure, the warp information includes a warp type, the warp type being a rotation and zoom type including four parameters defining a warp movement, the at least one difference value obtained is two difference values ​​for correcting two of the four parameters, and the step of obtaining the warp model includes a step of correcting the two parameters based on the two difference values.

[0007] According to one or more embodiments of the present disclosure, the warp information includes a warp type, the warp type is an affine type including six parameters defining the warp movement, the at least one difference value obtained is four difference values ​​for correcting four of the six parameters, and the step of obtaining the warp model includes a step of correcting the four parameters based on the four difference values.

[0008] According to one or more embodiments of the present disclosure, the warp information includes a number of parameters defining the warp movement that are to be corrected by the differential value, and the number of the at least one differential value obtained is equal to the number of parameters of the warp information that are to be corrected by the differential value.

[0009] According to one or more embodiments of the present disclosure, the method further includes obtaining an adaptive step size for the at least one difference value, and obtaining the warp model includes obtaining the warp model based on the adaptive step size.

[0010] According to one or more embodiments of the present disclosure, obtaining an adaptive step size includes obtaining a scaling factor and calculating an adaptive step size based on a predetermined base step size and the scaling factor.

[0011] According to one or more embodiments of the invention, obtaining the scaling factor includes obtaining an index indicating the scaling factor from among a plurality of scaling factors stored in a lookup table, and obtaining the scaling factor based on the index and the lookup table.

[0012] According to one or more embodiments of the present invention, obtaining the scaling factor includes deriving the scaling factor based on a magnitude of a translational motion vector of the current block.

[0013] According to one or more embodiments of the present disclosure, obtaining the scaling factor includes deriving the scaling factor based on a shear parameter.

[0014] According to one or more embodiments of the present disclosure, obtaining the adaptive step size includes obtaining the adaptive step size based on at least one of the block size, the prediction mode, a step size used by at least one neighboring block of the current block, one or more of the predicted warp parameters, and whether the neighboring block of the current block was coded by a warp model or a translation model.

[0015] According to an embodiment of the present disclosure, a system is provided, the system including at least one memory configured to store computer program code, and at least one processor configured to receive a bitstream including a coding picture, access the computer program code, and operate as instructed by the computer program code, the computer program code including: a first acquisition code configured to cause the at least one processor to obtain warping information signaled in the bitstream, the warping information including a warp type indicating a number of parameters to be corrected by a difference value, the number of parameters defining a warping movement for predicting a current block of the coding picture; a second acquisition code configured to cause the at least one processor to obtain from the bitstream at least one difference value, the difference of the at least one parameter defining the warping movement for predicting the current block; a third acquisition code configured to cause the at least one processor to obtain a warping model based on the warping information and the at least one difference value; and a warping motion compensation code configured to cause the at least one processor to predict the current block by performing warping motion compensation using the warping model.

[0016] According to one or more embodiments of the present disclosure, the warp information includes a warp type, the warp type being a rotation and zoom type including four parameters defining the warp movement, the at least one difference value obtained is two difference values ​​for correcting two of the four parameters, and the third acquisition code is configured to cause the at least one processor to correct the two parameters based on the two difference values.

[0017] According to one or more embodiments of the present disclosure, the warp information includes a warp type, the warp type being an affine type including six parameters defining the warp movement, the at least one difference value obtained is four difference values ​​for correcting four of the six parameters, and the third acquisition code is configured to cause the at least one processor to correct the four parameters based on the four difference values.

[0018] According to one or more embodiments of the present disclosure, the warp information includes a number of parameters defining the warp movement that are to be corrected by the differential value, and the number of the at least one differential value obtained is equal to the number of parameters of the warp information that are to be corrected by the differential value.

[0019] According to one or more embodiments of the present disclosure, the computer program code further includes fourth obtaining code configured to cause the at least one processor to obtain an adaptive step size for the at least one difference value, and the third obtaining code configured to cause the at least one processor to obtain a warp model based on the adaptive step size.

[0020] According to one or more embodiments of the present disclosure, the fourth obtaining code is configured to cause the at least one processor to obtain a scaling factor and to calculate an adaptive step size based on a predetermined base step size and the scaling factor.

[0021] According to one or more embodiments of the invention, the fourth obtaining code is configured to cause the at least one processor to obtain the scaling factor by obtaining an index indicating the scaling factor from among a plurality of scaling factors stored in a lookup table, and obtaining the scaling factor based on the index and the lookup table.

[0022] According to one or more embodiments of the invention, the fourth obtaining code is configured to cause the at least one processor to obtain the scaling factor by deriving the scaling factor based on a magnitude of the translational motion vector of the current block.

[0023] According to one or more embodiments of the present disclosure, the fourth obtaining code is configured to cause the at least one processor to obtain the scaling factor by deriving the scaling factor based on a shear parameter indicative of the shear motion.

[0024] According to an embodiment of the present disclosure, a non-transitory computer-readable medium storing computer code is provided, which, when executed by at least one processor, is configured to cause the at least one processor to implement a decoder that performs the following steps: receiving a bitstream including a coding picture, obtaining warp information signaled in the bitstream, the warp information including a warp type indicating a number of parameters to be corrected by a difference value, the number of parameters defining a warp motion for predicting a current block of the coding picture, obtaining from the bitstream at least one difference value that is a difference of the at least one parameter defining the warp motion for predicting the current block, obtaining a warp model based on the warp information and the at least one difference value, and predicting the current block by performing warp motion compensation using the warp model. [Brief description of the drawings]

[0025] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Diagram 2] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Diagram 3] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 5A] FIG. 1 is a schematic diagram of a partition tree in VP9 according to one embodiment. [Figure 5B] FIG. 2 is a schematic diagram of a partition tree in AV1 according to one embodiment. [Figure 6A] FIG. 2 is a schematic diagram of block partitioning for a quad-tree plus binary tree (QTBT) structure according to one embodiment. [Figure 6B] FIG. 6B is a schematic diagram of a corresponding tree representation of the block-partitioned QTBT structure shown in FIG. 6A. [Figure 7A] FIG. 13 is a schematic diagram of vertical center-side ternary tree partitioning according to one embodiment. [Figure 7B] FIG. 13 is a schematic diagram of horizontal center-side ternary tree partitioning according to one embodiment. [Figure 8] FIG. 2 is a schematic diagram of search points for merge mode with motion vector difference (MMVD) of two reference frames according to one embodiment; [Figure 9] 4 is a schematic diagram of exemplary motion samples used to derive model parameters for a block using local warp motion prediction, according to one embodiment; [Figure 10] 2 is a schematic diagram of an example of a warped sample collection scan order, according to one embodiment; [Figure 11] FIG. 13 is a schematic diagram of motion vectors within a block using warp extension mode according to one embodiment. [Figure 12] FIG. 1 is a schematic diagram of an MMVD search process for a current frame using two reference frames according to one embodiment. [Figure 13] FIG. 2 is a block diagram of computer code according to an embodiment. [Figure 14]FIG. 1 illustrates a computer system suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The following detailed description of the exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements.

[0027] The above disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the disclosure or may be acquired from practice of the implementations. Moreover, one or more features or components of an embodiment may be incorporated or combined with other embodiments (or one or more features of the other embodiments). Furthermore, in the flowcharts and descriptions of the operations provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) simultaneously, and the order of one or more operations may be switched.

[0028] It is apparent that the systems and / or methods described herein may be implemented in different forms of hardware, software, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. It is understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0029] The features described below may be used separately or combined in any order. Furthermore, the embodiments may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0030] No element, act, or instruction used herein should be construed as critical or essential unless expressly stated. Also, as used herein, the singular is intended to include one or more items and may be used interchangeably with "one or more." When one item is intended, "one" or similar terms are used. Also, as used herein, terms such as "has," "have," "having," "include," "including," and the like are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless otherwise specified. Furthermore, phrases such as "at least one of A and B" or "at least one of A or B" should be understood as including only A, only B, or both A and B.

[0031] FIG. 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 includes at least two terminals 110, 120 interconnected via a network 150. For one-way transmission of data, the first terminal 110 may encode video data at a local location for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the other terminal's encoded video data (coding video data) from the network 150, decode the encoded data, and display the restored video data. One-way data transmission may be common in media provisioning applications, etc.

[0032] 1 shows a second pair of terminals 130, 140 provided to support bidirectional transmission of encoded video that may occur, for example, during a video conference. For the bidirectional transmission of data, each terminal 130, 140 may encode video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 may also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the reconstructed video data on a local display device.

[0033] In FIG. 1, the terminals 110-140 may be depicted as servers, personal computers, and smartphones, and / or any other type of terminal. For example, the terminals 110-140 may be laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. The network 150 represents any number of networks that convey encoded video data between the terminals 110-140, including, for example, wired and / or wireless communication networks. The communication network 150 may exchange data in circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of the network 150 is not important to the operation of the present disclosure, unless otherwise described herein below.

[0034] 2 shows an arrangement of video encoders and decoders in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter is similarly applicable to other video-enabled applications including, for example, videoconferencing, digital TV, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.), etc.

[0035] As shown in FIG. 2, the streaming system 200 may include a capture subsystem 213 that may include a video source 201 and an encoder 203. For example, the video source 201 may be a digital camera and may be configured to create an uncompressed video sample stream 202. The uncompressed video sample stream 202 may provide a high amount of data when compared to an encoded video bitstream and may be processed by an encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream 204 may include a lower amount of data when compared to the sample stream and may be stored on a streaming server 205 for future use. One or more streaming clients 206 may access the streaming server 205 to obtain a video bitstream 209, which may be a copy 208 of the encoded video data 204.

[0036] In an embodiment, the streaming server 205 may also function as a Media-Aware Network Element (MANE). For example, the streaming server 205 may be configured to prune the encoded video bitstreams 204 in order to tailor potentially different bitstreams to one or more of the streaming clients 206. In an embodiment, a MANE may be provided separately from the streaming server 205 in the streaming system 200.

[0037] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video bitstream 209, which is an input copy of the encoded video bitstream 204, and generate an output video sample stream 210 that may be rendered on a display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to a particular video encoding / compression standard. Examples of these standards include, but are not limited to, ITU-T Recommendation H.265. The developing video encoding standard is informally known as Versatile Video Coding (VVC). Embodiments of the present disclosure may be used in the context of VVC.

[0038] FIG. 3 illustrates an exemplary functional block diagram of a video decoder 210 attached to a display 212 according to one embodiment of the disclosure.

[0039] The video decoder 210 may include a channel 312, a receiver 310, a buffer memory 315, an entropy decoder / parser 320, a scaler / inverse transform unit 351, an intra prediction unit 352, a motion compensated prediction unit 353, an aggregator 355, a loop filter unit 356, a reference picture memory 357, and a current picture memory. In at least one embodiment, the video decoder 210 may include an integrated circuit, a series of integrated circuits, and / or other electronic circuitry. The video decoder 210 may also be embodied partially or entirely in software running on one or more CPUs having associated memory.

[0040] In this and other embodiments, the receiver 310 may receive one or more coded video sequences to be decoded by the decoder 210, one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel 312, which may be a hardware / software link to a storage device that stores the coded video data. The receiver 310 may receive the coded video data along with other data (e.g., coded audio data and / or auxiliary data streams), which may be forwarded to respective usage entities (not shown). The receiver 310 may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory 315 may be coupled between the receiver 310 and the entropy decoder / parser 320 (hereinafter the "parser"). If the receiver 310 is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory 315 may not be used or may be small. For use in a best effort packet network such as the Internet, a buffer memory 315 may be required and can be relatively large and adaptively sized.

[0041] The video decoder 210 may include a parser 320 for reconstructing symbols 321 from the entropy-coded video sequence. These categories of symbols may include, for example, information used to manage the operation of the decoder 210 and potentially information for controlling a rendering device, such as a display 212, which may be coupled to the decoder as shown in FIG. 2. The rendering device control information may be, for example, in the form of a Supplementary Enhancement Information (SEI) (SEI message) or a Video Usability Information (VUI) parameter set fragment (not shown). The parser 320 may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may follow any video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 320 may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the group. The subgroups may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transformation unit (TU), a prediction unit (PU), etc. Parser 320 may also extract information from the coded video sequence, such as transform coefficients, quantization parameter values, motion vectors, etc.

[0042] Parser 320 may perform an entropy decoding / parsing operation on the video sequence received from buffer memory 315 to generate symbols 321 .

[0043] The reconstruction of symbols 321 may involve several different units, depending on the type of coded video picture or portion thereof (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors. Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by parser 320. The flow of such subgroup control information between parser 320 and the following units is not shown for clarity.

[0044] In addition to the functional blocks described above, the decoder 210 can be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0045] One unit is a scalar / inverse transform unit 462. The scalar / inverse transform unit 351 may receive quantized transform coefficients as symbols 321 from the parser 320 along with control information (including which transform to use, block size, quantization factor, quantization scaling matrix, etc.). The scalar / inverse transform unit 351 may output a block containing sample values ​​that may be input to an aggregator 355.

[0046] In some cases, the output samples of the scalar / inverse transform unit 351 may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by the intra-picture prediction unit 352. In some cases, the intra-picture prediction unit 352 generates blocks of the same size and shape of the block being reconstructed using surrounding already reconstructed information retrieved from the (partially reconstructed) current picture from the current picture memory 358. In some cases, the aggregator 355 adds, sample by sample, the prediction information generated by the intra-picture prediction unit 352 to the output sample information provided by the scalar / inverse transform unit 351.

[0047] In other cases, the output samples of the scalar / inverse transform unit 351 may relate to an inter-coded and potentially motion-compensated block. In such a case, the motion compensation prediction unit 353 may access the reference picture memory 357 to retrieve samples used for prediction. After motion compensating the retrieved samples according to the symbols 321 related to the block, these samples may be added by the aggregator 355 to the output of the scalar / inverse transform unit 351 (in this case, called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory 357 from which the motion compensation prediction unit 353 retrieves the prediction samples may be controlled by a motion vector. The motion vector may be made available to the motion compensation prediction unit 353, for example, in the form of a symbol 321, which may have X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory 357 when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0048] The output samples of aggregator 355 may be subjected to various loop filtering techniques in loop filter unit 356. Video compression techniques may include in-loop filter techniques, controlled by parameters contained in the coded video bitstream and made available to loop filter unit 356 as symbols 321 from parser 320, that may be responsive to meta-information obtained during decoding of previous parts of the coded pictures or coded video sequences (in decoding order), as well as to previously reconstructed loop filtered sample values.

[0049] The output of the loop filter unit 356 may be a sample stream that may be output to a rendering device, such as the display 212, and may also be stored in a reference picture memory 357 for use in future inter-picture prediction.

[0050] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 320), the current reference picture can become part of reference picture memory 357, and a new current picture memory can be reallocated before beginning reconstruction of a subsequent coded picture.

[0051] The video decoder 210 may perform decoding operations according to a given video compression technique, which may be documented in a standard such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to the syntax of the video compression technique or standard as specified in the video compression technique or standard, and in particular as specified in its profile document. Also, for compliance with some video compression techniques or standards, the complexity of the coded video sequence may be within a range defined by a level of the video compression technique or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (e.g., measured in megasamples per second), the maximum reference picture size, etc. In some cases, the limits set by the level can be further limited through a Hypothetical Reference Decoder (HRD) specification and metadata about HRD buffer management conveyed in the coded video sequence.

[0052] In one embodiment, the receiver 310 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder 210 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0053] FIG. 4 illustrates an example functional block diagram of a video encoder 203 associated with a video source 201 according to one embodiment of the present disclosure.

[0054] The video encoder 203 may include, for example, an encoder, which may be a source coder 430, a coding engine 432, a (local) decoder 433, a reference picture memory 434, a predictor 435, a transmitter 440, an entropy coder 445, a controller 450, and a channel 460.

[0055] The encoder 203 may receive video samples from a video source 201 (not part of the encoder), which may capture video images to be encoded by the encoder 203 .

[0056] The video source 201 may provide a source video sequence to be encoded by the encoder 203 in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media presentation system, the video source 201 may be a storage device that stores pre-prepared videos. In a video conferencing system, the video source 203 may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples, depending on the sampling structure, color space, etc. in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.

[0057] According to one embodiment, the encoder 203 may encode and compress pictures of a source video sequence into an encoded video sequence 443 in real-time or under any other time constraint required by an application. Achieving an appropriate encoding rate is one function of the controller 450. The controller 450 may also control and be operatively coupled to other functional units, as described below. Coupling is not shown for clarity. Parameters set by the controller 450 may include rate control related parameters (picture skip, quantization, lambda value for rate distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can readily recognize other functions of the controller 450 that may be relevant to a video encoder 203 optimized for a particular system design.

[0058] Some video encoders operate in what those skilled in the art will easily recognize as a "coding loop". As a very simplified description, the coding loop can consist of the coding part of the source coder 430 (responsible for generating symbols based on the input picture to be coded and the reference pictures) and a (local) decoder 433 embedded in the encoder 203. The decoder 433 reconstructs the symbols to generate sample data similar to what a (remote) decoder would generate when the compression between the symbols and the coded video bitstream is lossless for a particular video compression technique. That reconstructed sample stream may be input to a reference picture memory 434. Since the decoding of the symbol stream results in bit-wise accurate results independent of the location of the decoder (local or remote), the contents of the reference picture memory are also bit-wise accurate between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronization (including the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is known to those skilled in the art.

[0059] The operation of the "local" decoder 433 may be the same as the "remote" decoder 210, which has already been described in detail above in connection with Figure 3. However, the entropy decoding portion of the decoder 210, including the channel 312, the receiver 310, the buffer memory 315 and the parser 320, may not be fully implemented in the local decoder 433, since symbols are available and the encoding / decoding of the symbols into the encoded video sequence by the entropy coder 445 and the parser 320 may be lossless.

[0060] An observation that can be made at this point is that any decoder techniques, other than analysis / entropy decoding, present in the decoder must be present in substantially the same functional form in the corresponding encoder. For this reason, the subject matter of the disclosure focuses on the decoder operation. A description of the encoder techniques can be omitted, as they may be the inverse of the decoder techniques described generically. Only in certain areas is more detailed description necessary and is provided below.

[0061] As part of its operation, the source coder 430 may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence designated as “reference frames.” In this manner, the coding engine 432 codes differences between pixel blocks of the input frame and pixel blocks of reference frames that may be selected as predictive references for the input frame.

[0062] The local decoder 433 may decode the encoded video data of the frames that may be designated as reference frames based on the symbols generated by the source coder 430. The operation of the encoding engine 432 may advantageously be a lossy process. If the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence may be a replica of the source video sequence, typically with some errors. The local decoder 433 may replicate the decoding process that may be performed by the video decoder on the reference frames and store the reconstructed reference frames in the reference picture memory 434. In this way, the encoder 203 may locally store copies of the reconstructed reference frames that have common content as the reconstructed reference frames (without transmission errors) obtained by the far-end video decoder.

[0063] The predictor 435 may perform a prediction search for the coding engine 432. That is, for a new frame to be coded, the predictor 435 may search the reference picture memory 434 for sample data (as candidate reference pixel blocks) or specific metadata (reference picture motion vectors, block shapes, etc.). These may serve as suitable prediction references for the new picture. The predictor 435 may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, the input picture determined by the search results obtained by the predictor 435 may have prediction references drawn from multiple reference pictures stored in the reference picture memory 434.

[0064] Controller 450 may manage the encoding operations of source coder 430, including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0065] The output of all the above functional units may be subjected to entropy coding in entropy coder 445. The entropy coder converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as, for example, Huffman coding, variable length coding, arithmetic coding, etc.

[0066] The transmitter 440 may buffer the encoded video sequence produced by the entropy coder 445 and prepare it for transmission over a communication channel 460, which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter 440 may merge the encoded video data from the coder 430 with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (not shown).

[0067] The controller 450 may manage the operation of the encoder 203. During encoding, the controller 450 may assign each encoded picture a particular encoding picture type, which may affect the encoding technique that may be applied to each picture. For example, pictures may often be assigned as an intra picture (I picture), a predicted picture (P picture), or a bidirectionally predicted picture (B picture).

[0068] An Intra picture (I-picture) may be one that can be coded and decoded without using other frames in a sequence as a source of prediction. Some video codecs allow different types of Intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will recognize these variations of I-pictures and their respective uses and characteristics.

[0069] A predictive picture (P-picture) may be one that can be encoded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0070] Bidirectionally predicted pictures (B-pictures) may be those that can be coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0071] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8 or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective picture of the block. For example, blocks of an I-picture may be non-predictively coded or may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P-picture may be non-predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be non-predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0072] Video encoder 203 may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In doing so, video encoder 203 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0073] In one embodiment, the transmitter 440 may transmit additional data along with the coded video. The source coder 430 may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other types of redundant data such as redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0074] [Block partitioning in VP9 and AV1]

[0075] As mentioned above, AV1 is an open video coding format designed for video transmission over the Internet, developed as a successor to VP9. Referring to FIG. 5A, VP9 uses a four-way partition tree starting from the 64×64 level down to the 4×4 level, with some further restrictions for blocks 8×8 and below (as shown in the top half of FIG. 5A). The partitions designated as R may be called recursive in that the same partition tree may be repeated at lower scales until the partition reaches the lowest 4×4 level.

[0076] As shown in Figure 5B, AV1 not only extends the partition tree to a 10-way structure, but also increases the maximum size (called a superblock in VP9 / AV1 terminology) to start at 128x128. This may include 4:1 / 1:4 rectangular partitions, which were not present in VP9. None of the rectangular partitions may be further subdivided. Additionally, AV1 adds more flexibility to the use of partitions below the 8x8 level, in the sense that 2x2 chroma inter prediction is allowed in certain cases.

[0077] [Block Partitioning in HEVC]

[0078] In HEVC, a coding tree unit (CTU) may be partitioned into coding units (CU) by using a quad-tree (QT) structure called a coding tree to adapt to different local characteristics. The decision on whether to code a picture region using inter-picture (temporal) prediction or intra-picture (spatial) prediction may be made at the CU level. Each CU may be further partitioned into one, two or four prediction units (PU) according to a PU partition type. Within one PU, the same prediction process may be applied, and related information may be transmitted to the decoder for each PU. After obtaining the residual block by applying a prediction process based on the PU partition type, the CU may be partitioned into transform units (TU) according to another QT structure such as the coding tree for CU. One of the important features of the HEVC structure is to have multiple partition concepts, including CU, PU and TU. In HEVC, a CU or TU can only be square in shape, while a PU may be square or rectangular in shape for an inter-prediction block. In HEVC, a coding block may be further divided into four square sub-blocks, and a transform is performed on each sub-block, i.e., TU. Each TU may be further divided recursively (using QT partitioning) into smaller TUs, which are called Residual Quad-Tree (RQT).

[0079] At picture boundaries, HEVC employs implicit quadtree partitioning so that blocks maintain the quadtree partitioning until their size fits the picture boundary.

[0080] [Block partitioning in VVC]

[0081] 1. Block partitioning structure using quad-tree (QT) and binary tree (BT)

[0082] The QTBT block structure removes the concept of multiple partition types. That is, the QTBT block structure removes the separation of the concepts of CU, PU, ​​and TU, and supports more flexibility for CU partition shapes. In the QTBT block structure, a CU may have a square or rectangular shape. Referring to FIG. 6A-6B, a coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf node is further partitioned by a binary tree structure. There are two partition types in BT partitioning: symmetric horizontal partitioning and symmetric vertical partitioning. The BT leaf node may be called a CU, and its segmentation is used for prediction and transform processing without further partitioning. This means that the CU, PU, ​​and TU have the same block size in the QTBT coding block structure. In the joint exploration model (JEM), a CU may be composed of coding blocks (CB) of different color components. For example, in the case of a prediction (P, prediction) slice and a binary (B, binary) slice in a 4:2:0 chroma format, one CU may include one luma CB and two chroma CBs. A CU may also be composed of or include a single component CB. For example, one CU includes only one luma CB or only two chroma CBs in the case of an I slice.

[0083] The following parameters may be defined for the QTBT partitioning scheme: (a) CTU size: The root node size of the quadtree, which may be the same concept as in HEVC (b) MinQTSize: The minimum allowable quadtree leaf node size. (c) MaxBTSize: Maximum allowable size of binary tree root node (d) MaxBTDepth: Maximum allowable binary tree depth (e) MinBTSize: The minimum allowable size of a binary tree leaf node.

[0084] In one example of a QTBT partitioning structure, the CTU size may be set to 128×128 luma samples with two corresponding 64×64 blocks of chroma samples, MinQTSize may be set to 16×16, MaxBTSize may be set to 64×64, MinBTSize may be set to 4×4 (for both width and height), and MaxBTDepth may be set to 4. Quad-tree partitioning may be first applied to the CTU to generate a QT leaf node. The QT leaf node may have a size from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the leaf QT node is 128×128, it may not be further partitioned by the binary tree since the size exceeds MaxBTSize (i.e., 64×64). Otherwise, the leaf QT node may be further partitioned by the binary tree. Thus, the QT leaf node is also the root node of the BT and has a BT depth of zero. When the BT depth reaches MaxBTDepth (i.e., 4), further splitting may not be considered. When a BT node has a width equal to MinBTSize (i.e., 4), further horizontal splitting may not be considered. Similarly, when a BT node has a height equal to MinBTSize, further vertical splitting may not be considered. BT leaf nodes may be further processed by prediction and transform processing without further partitioning. In JEM, for example, the maximum CTU size may be 256x256 luma samples.

[0085] FIG. 6A shows an example of block partitioning by using QTBT, and FIG. 6B shows the corresponding tree representation. Solid lines indicate QT partitioning, and dotted lines indicate BT partitioning. At each partition (i.e., non-leaf) node of BT, one flag may be signaled to indicate which partition type (i.e., horizontal or vertical) is used, with 0 indicating horizontal partitioning and 1 indicating vertical partitioning. In the case of QT partitioning, it may not be necessary to indicate the partition type, since QT partitioning may always partition a block both horizontally and vertically to generate four sub-blocks with equal size.

[0086] The QTBT scheme may support the flexibility for luma and chroma to have separate QTBT structures. Currently, for P slices and B slices, the luma CTB and chroma CTB in one CTU share the same QTBT structure. However, for I slices, the luma CTB is partitioned into CUs by a QTBT structure, and the chroma CTB is partitioned into chroma CUs by another QTBT structure. This means that a CU in an I slice is composed of a coding block of a luma component or a coding block of two chroma components, and a CU in a P slice or B slice is composed of coding blocks of all three color components.

[0087] In HEVC, to reduce memory access for motion compensation, inter prediction of small blocks is restricted, bi-prediction is not supported for 4x8 and 8x4 blocks, and inter prediction is not supported for 4x4 blocks. QTBT implemented in JEM-7.0 removes these restrictions.

[0088] 2. Block partitioning using a ternary tree (TT)

[0089] In VVC, a multi-type-tree (MTT) structure is included, which further adds a vertical centered ternary tree and a horizontal centered ternary tree in addition to the QTBT, as shown in Figures 7A-7B. Figure 7A shows the vertical centered ternary tree partitioning, and Figure 7B shows the horizontal centered ternary tree partitioning.

[0090] The main advantages of ternary tree partitioning include, but are not limited to, (a) it provides a complement to quadtree and binary tree partitioning, i.e., while quadtrees and binary trees always split along block centers, ternary tree partitioning can capture objects located at block centers, and (b) the width and height of the proposed ternary tree partitions may always be powers of two, and therefore no further transformations may be required.

[0091] The two-level tree design is primarily motivated by reduced complexity. In theory, the complexity of traversing a tree is T D where T denotes the number of split types and D is the depth of the tree.

[0092] [Merge mode with motion vector differences (MMVD)]

[0093] In merge mode, the implicitly derived motion information is directly used for generating the prediction samples of the current CU. A merge mode with motion vector differences (MMVD) is introduced in VVC. The MMVD flag is signaled immediately after sending the skip and merge flags to specify whether the MMVD mode is used for a CU.

[0094] In MMVD, after a merge candidate is selected, it may be further refined by signaled motion vector difference (MVD) information. The MVD information may include a merge candidate flag, an index to specify the magnitude of motion, and an index to indicate the motion direction. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the motion vector (MV) base. A merge candidate flag may be signaled to specify which one of the first two candidates is used.

[0095] A distance index may be used, which may specify motion magnitude information, indicating a predefined offset from the starting point. Figure 8 shows MMVD search points of two reference frames according to some embodiments. As shown in Figure 8, the offset may be added to either the horizontal or vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 1 below. [Table 1]

[0096] A direction index may be used and may represent the direction of the MVD relative to the starting point. The direction index may represent one of four directions, as shown in Table 2 below. The meaning of the MVD code may change according to the information of the starting MV. When the starting MV is a uni-predictive MV or a bi-predictive MV and both lists point to the same side of the current picture (i.e., the picture order counts (POC) of the two references are both greater than or less than the POC of the current picture), the code in Table 2 specifies the code of the MV offset added to the starting MV. When the start MV is a bi-predictive MV, the two MVs point to different sides of the current picture (i.e., the POC of one reference is larger than that of the current picture and the POC of the other reference is smaller than that of the current picture), and the difference in the POC in list 0 (L0) is larger than that in list 1 (L1), the code in Table 2 specifies the sign of the MV offset added to the L0 MV component of the start MV, and the code of the L1 MV has the opposite value. Otherwise, when the difference in the POC in L1 is larger than that of L0, the code in Table 2 specifies the sign of the MV offset added to the L1 MV component of the start MV, and the code of the L0 MV has the opposite value.

[0097] The MVD may be scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling may be required. Otherwise, if the difference in POC in L0 is larger than that of L1, the MVD of L1 is scaled. If the POC difference of L1 is larger than that of L0, the MVD of L0 may be scaled in the same way. If the starting MV is uni-predicted, the MVD may be added to the available MV. [Table 2]

[0098] [Symmetric MVD coding]

[0099] In VVC, in addition to signaling MVD for normal unidirectional and bidirectional prediction modes, a symmetric MVD mode for signaling bidirectional MVD may be applied. In the symmetric MVD mode, motion information including both L0 and L1 reference picture indexes and the MVD of L1 may be derived or may not be explicitly signaled.

[0100] The decoding process for the symmetric MVD mode may be as follows.

[0101] At the slice level, the variables BiDirPredFlag, RefIdxSymL0 and RefIdxSymL1 may be derived as follows: If mvd_l1_zero_flag is 1, then BiDirPredFlag is set equal to 0. Otherwise, if the closest reference picture in L0 and the closest reference picture in L1 form a forward and backward pair of reference pictures or a backward and forward pair of reference pictures, then BiDirPredFlag is set to 1 and both the L0 reference picture and the L1 reference picture are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0.

[0102] At the CU level, if the CU is bi-predictively coded and BiDirPredFlag is equal to 1, a symmetric mode flag may be explicitly signaled indicating whether symmetric mode is used or not. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 may be explicitly signaled. The reference indexes of L0 and L1 may be set equal to a pair of reference pictures, respectively. MVD1 may be set equal to (-MVD0).

[0103] [Intermode coding in CWG-B018]

[0104] In AV1, for each coding block in an inter frame, if the mode of the current block is an inter coding mode rather than a skip mode, another flag may be signaled to indicate whether a single or mixed reference mode is used for the current block. In a single reference mode, the predictive block may be generated by one motion vector. In a mixed reference mode, the predictive block may be generated by a weighted average of two predictive blocks derived from two motion vectors.

[0105] For the single reference case, the following modes may be signaled: (a) NEARMV - Use one of the motion vector predictors (MVPs) in the list indicated by the Dynamic Reference List (DRL) index. (b) NEWMV - Apply a difference to the MVP using one of the MVPs in the list signaled by the DRL index as a reference. (c) GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0106] For the mixed reference case, the following modes may be signaled: (a) NEAR_NEARMV - Use one of the MVPs in the list signaled by the DRL index (b) NEAR_NEWMV - use one of the MVPs in the list signaled by the DRL index as a reference and send a delta MV for the second MV (c) NEW_NEARMV - send a differential MV for the first MV using one of the MVPs in the list signaled by the DRL index as a reference (d) NEW_NEWMV - use one of the MVPs in the list signaled by the DRL index as a reference and send a delta MV for both MVs. (e) GLOBAL_GLOBALMV - use MV from each reference based on frame-level global motion parameters

[0107] [Motion Vector Differential Coding in AV1] AV1 allows for 1 / 8 pixel motion vector precision (or accuracy) and has the following syntax:

[0108] may be used to signal the motion vector difference in the reference frames L0 or L1.

[0109] According to an embodiment, the syntax mv_joint specifies which components of the motion vector difference are non-zero. A value of 0 may indicate that there is no non-zero MVD along either the horizontal or vertical direction. A value of 1 may indicate that there is non-zero MVD only along the horizontal direction. A value of 2 may indicate that there is non-zero MVD only along the vertical direction. A value of 3 may indicate that there is non-zero MVD along both the horizontal and vertical directions.

[0110] According to an embodiment, the syntax element mv_sign specifies whether the motion vector difference is positive or negative.

[0111] According to an embodiment, the syntax element mv_class specifies the class of the motion vector difference, where a higher class means that the motion vector difference has a larger magnitude, as shown in Table 3 below. [Table 3]

[0112] According to an embodiment, the syntax element mv_bit specifies the integer part of the offset between the motion vector difference and the starting magnitude of each MV class.

[0113] According to an embodiment, the syntax element mv_fr specifies the first two fractional bits of the motion vector difference.

[0114] According to an embodiment, the syntax mv_hp specifies the third fractional bit of the motion vector difference.

[0115] [Adaptive MVD Resolution in CWG-B092]

[0116] For the NEW_NEARMV and NEAR_NEWMV modes, the accuracy of the MVD may depend on the associated class and the size of the MVD.

[0117] First, fractional MVDs may be allowed only if the size of the MVD is less than or equal to one pixel.

[0118] Then, when the value of the associated MV class is greater than or equal to MV_CLASS_1, only one MVD value may be allowed, and the MVD value in each MV class may be derived as 4, 8, 16, 32, 64 for MV class 1 (MV_CLASS_1), MV class 2 (MV_CLASS_2), MV class 3 (MV_CLASS_3), MV class 4 (MV_CLASS_4) or MV class 5 (MV_CLASS_5).

[0119] The allowable MVD values ​​for each MV class are shown in Table 4 below. [Table 4]

[0120] In some embodiments, one context may be used to signal mv_joint or mv_class if the current block is coded using NEW_NEARMV or NEAR_NEWMV modes, and another context may be used to signal mv_joint or mv_class if the current block is not coded using NEW_NEARMV or NEAR_NEWMV modes.

[0121] [Joint MVD Coding (JMVD) in CWG-B092]

[0122] A new inter-coding mode named JOINT_NEWMV may be applied to indicate whether the MVDs for the two reference lists are signaled together. When the inter-prediction mode is equal to the JOINT_NEWMV mode, the MVDs for the reference lists L0 and L1 are signaled together. Thus, only one MVD named joint_mvd may be signaled and transmitted to the decoder, and the differential MVs for the reference lists L0 and L1 may be derived from joint_mvd.

[0123] The JOINT_NEWMV mode may be signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV and GLOBAL_GLOBALMV modes. No further context may be added.

[0124] When JOINT_NEWMV mode is signaled and the POC distances between two reference frames and the current frame are different, the MVD may be scaled for reference list L0 or reference list L1 based on the POC distances. Specifically, the distance between reference frame list L0 and the current frame is denoted as td0, and the distance between reference frame list L1 and the current frame is denoted as td1. If td0 is greater than or equal to td1, joint_mvd may be used directly for reference list L0, and mvd for reference list L1 may be derived from joint_mvd based on the following equation (1): derived_mvd=td1 / td0*joint_mvd Formula (1)

[0125] Otherwise, if td1 is greater than or equal to td0, then joint_mvd for reference list L1 may be used directly, and mvd for reference list L0 may be derived from joint_mvd based on equation (2) below. derived_mvd=td0 / td1*joint_mvd Formula (2)

[0126] [Improvements to Adaptive MVD in CWG-C011]

[0127] A new inter-coding mode named AMVDMV may be added for the single reference case. When the AMVDMV mode is selected, this may indicate that AMVD is applied to the signal MVD.

[0128] One flag named amvd_flag may be added under JOINT_NEWMV mode to indicate whether AMVD is applied to the joint MVD coding mode named joint AMVD coding. When adaptive MVD resolution is applied to the joint MVD coding mode named joint AMVD coding, the MVDs for two reference frames may be signaled together, and the precision of the MVD may be implicitly determined by the size of the MVD. Otherwise, the MVDs for two (or more) reference frames are signaled together, and MVD coding is applied.

[0129] [Adaptive motion vector resolution (AMVR) in CWG-C012 and CWG-C020]

[0130] AMVR was first proposed in CWG-C012 and supports a total of seven MV precisions (i.e., 8, 4, 2, 1, 1 / 2, 1 / 4, 1 / 8). For each prediction block, the AVM encoder explores all supported precision values ​​and signals the best precision to the decoder.

[0131] To reduce the runtime of the encoder, two precision sets may be supported. Each precision set may contain four predefined precisions. The precision set may be adaptively selected at the frame level based on the maximum precision value of the frame. Similar to AV1, the maximum precision may be signaled in the frame header. Table 5 below summarizes the supported precision values ​​based on the frame-level maximum precision. [Table 5]

[0132] In AVM software (similar to AV1), a frame level flag may be present to indicate whether the MV of the frame includes sub-pel precision or not. AMVR may be enabled only if the value of the cur_frame_force_integer_mv flag is 0. In AMVR, if the precision of the block is less than maximum precision, the motion model and the interpolation filter may not be signaled. If the precision of the block is less than maximum precision, the motion mode may be estimated as translational motion and the interpolation filter may be estimated as a REGULAR interpolation filter. Similarly, if the precision of the block is either 4-pel or 8-pel, the inter-intra mode may not be signaled and may be estimated to be 0.

[0133] [Warp motion mode in AV1]

[0134] Motion compensation typically assumes a translational motion model between the reference and target blocks, whereas warped motion utilizes an affine model.

[0135] The affine motion model can be expressed by the following equation (3):

number

[0136] 1. Global Warp Motion Compensation

[0137] In global warp motion compensation, global motion information including a global motion type and some motion parameters may be signaled for each inter reference frame. The global motion types and the associated parameter numbers are listed in Table 6 below. [Table 6]

[0138] After signaling the reference frame index, if global motion is selected, the global motion type and parameters associated with the given reference frame for the current coding block may be used.

[0139] 2. Local Warp Motion Compensation

[0140] For inter-coding blocks, local warping motion may be allowed when the following conditions are met: (1) the current block uses single-reference prediction, (2) the width or height of the coding block is greater than or equal to 8, and (3) at least one of the adjacent neighboring blocks uses the same reference frame as the current block.

[0141] If a local warp motion is used for the current block, the affine model parameters may be estimated by a mean square minimization of the difference between the reference projection and the modeled projection based on the MVs of the current block and its adjacent neighboring blocks. To estimate the parameters of the local warp motion, if the neighboring block uses the same reference frame as the current block, a projection sample pair of the center sample in the neighboring block and its corresponding sample in the reference frame may be obtained. Then, three additional samples may be created by shifting the center position by a quarter sample in one or both dimensions. These additional samples may also be considered as projection sample pairs to ensure the stability of the model parameter estimation process.

[0142] The MVs of neighboring blocks used to derive motion parameters may be referred to as motion samples. The motion samples may be selected from neighboring blocks that use the same reference frame as the current block. The warped motion prediction mode may be enabled only for blocks that use a single reference frame. For example, referring to FIG. 9, the MVs of neighboring blocks B0, B1, and B2 are referred to as MV0, MV1, and MV2, respectively. FIG. 6 illustrates exemplary motion samples used to derive model parameters of a block using local warped motion prediction, according to some embodiments. FIG. 9 illustrates exemplary motion samples used to derive model parameters of a block using local warped motion prediction, according to some embodiments. The current block 600 may be predicted using uni-prediction with reference frame Ref0. The neighboring block B0 may be predicted using mixed prediction with reference frames Ref0 and Ref1. The neighboring block B1 may be predicted using uni-prediction with reference frame Ref0. The neighboring block B2 may be predicted using mixed prediction with reference frames Ref0 and Ref2. The motion vector MV0Ref0 of neighboring block B0, the motion vector MV1Ref0 of neighboring block B1, and the motion vector MV2Ref0 of neighboring block B2 may be used as motion samples to derive the affine motion parameters of the current block 600.

[0143] 3. Sample collection process of local warp motion in AV1 and AVM

[0144] In local warp motion, samples and motion vectors of neighboring blocks may be collected in a stack to calculate a warp model using a regression / least squared error method. In AV1 and AVM, a group of sample positions in a reference picture and its reference positions pointed to by its MV may be collected. Up to eight pairs of positions may be stored in the stack for regression / least squared error warp model calculation.

[0145] FIG. 10 is a schematic diagram of an example of a warp sample collection scanning order. As shown in FIG. 10, only spatial neighboring blocks may be scanned. If the neighboring block has the same reference picture as the current block 610 (e.g., if the neighboring block is composite, one of the reference pictures is the same as the reference picture of the current block), the center position of the neighboring block and the center position of the reference block of the neighboring block pointed to by the MV of the neighboring block may be collected. Referring to FIG. 10, the scanning order may be defined as follows: (1) the upper row neighboring block 611 is scanned, (2) the left column neighboring block 612 is scanned, (3) the upper left neighboring block 613 is scanned, and (4) the upper right neighboring block 614 is scanned.

[0146] [Local Warp Extension Mode and Local Warp Difference Mode from CWG-C050]

[0147] In addition to translational motion, AVM also supports warp motion compensation. Two types of warp motion models are supported: global warp model and local warp model. The global warp model is associated with each reference frame, and each of the four non-translational parameters has 12-bit precision, while the translational motion vectors are coded with 15-bit precision. A coding block may choose to use it directly (provided with a reference frame index). The global warp model captures frame-level scaling and rotation, and thus focuses primarily on inflexible motion across the entire frame.

[0148] Additionally, local warping models at the coding block level are also supported. In the local warping mode, also known as WARPED_CAUSAL, the warping parameters of the current block may be derived by fitting a model to nearby motion vectors using least squares.

[0149] 1. Warp Extension Mode

[0150] The new warp motion mode is called WARP_EXTEND. In WARP_EXTEND mode, neighboring block motions are smoothly extended into the current block, but with some ability to change the warp parameters. This allows complex warping motions to be expressed and spread across multiple blocks, while minimizing blocking artifacts.

[0151] To achieve this, the WARP_EXTEND mode applied to the NEWMV block constructs a new warp model based on two constraints: (1) the per-pixel motion vectors generated by the new warp model must be contiguous with the per-pixel motion vectors in neighboring blocks, and (2) the central pixel of the current block must have a per-pixel motion vector that matches the motion vector signaled for the block as a whole.

[0152] For example, referring to Figure 11, if the neighboring block 622 to the left of the current block 620 is warped, a warp model that matches the motion vectors may be used. Figure 11 illustrates motion vectors within a block using warp extension mode, according to some embodiments.

[0153] The above two constraints for constructing a new warp model may imply certain equations involving the warp parameters of the neighboring blocks and the current block, which may then be solved to compute the warp model for the current block.

[0154] For example, if (A,...,F) represent adjacent warped models and (A',...,F') represents a new warped model, then the first constraint, at each point along the common edge, is:

number

[0155] Points along the edge have different values ​​of y, but they all have the same value of x. This means that the coefficient of y must be the same on both sides (i.e. B'=B and D'=D). The coefficient of x, on the other hand, provides two equations for the other coefficients, shown below in Equations 5-8.

number

[0156] The second constraint specifies that the motion vector of the center of the block must be equal to the one signaled using the NEWMV mechanism. This provides two further equations, resulting in a system of six equations in six variables with a unique solution.

[0157] These equations can be solved efficiently in both software and hardware. The solution can be solved using basic addition, subtraction, multiplication, and division by powers of 2. Therefore, this mode is significantly less complex than the least-squares based local warping mode.

[0158] According to an embodiment, there may be several neighboring blocks that can be extended from. In such a case, it is useful to have some way to select which block to extend from. This problem is similar to the problem faced in motion vector prediction. In particular, there may be several possible motion vectors from nearby blocks, and the one to use must be selected as the basis for NEWMV coding.

[0159] A solution to the above problem may include an extension to handle the need for WARP_EXTEND. This may be done by tracking the source of each motion vector prediction. WARP_EXTEND may then be enabled only if the selected motion vector prediction is taken from a directly neighboring block. That block may then be used as a single "neighboring block" in the remainder of the algorithm.

[0160] In some cases, the warp model of the neighbors will be very good as is, without needing further modification. To code this case more cheaply, WARP_EXTEND may be used for the NEARMV block. The selection of the neighbors may be the same as in NEWMV, except that the selection in NEWMV requires that the neighbors be warped rather than simply translated. However, if this is true and WARP_EXTEND is selected, the warp model parameters of the neighbors may be copied to the current block.

[0161] 2. Warp Diff Mode

[0162] In some embodiments, a motion mode called WARP_DELTA may be provided, in which the warp model of a block is coded as a difference from a predicted warp model, similar to how a motion vector is coded as a difference from a predicted motion vector. Predictions may be sourced either from a global motion model (if present) or from neighboring blocks.

[0163] To avoid having multiple ways to code the same prediction warp model, the following restrictions may be applied: If the mode is NEARMV or NEWMV, the same neighbor selection logic as described for WARP_EXTEND may be used. If this results in a warped neighboring block, then that block's model (without applying the rest of the WARP_EXTEND logic) may be used as prediction. Otherwise, the global warp model may be used as the base.

[0164] The difference for each of the non-translational parameters may then be coded. Finally, the translational part of the model may be adjusted so that the per-pixel motion vector of the center of the block matches the global motion vector of the block.

[0165] This tool (i.e., WARP_DELTA) uses more bits to encode than the other warp modes because it involves explicitly coding the delta for each warp parameter. Therefore, WARP_DELTA may be disabled for blocks smaller than 16x16. However, the decoding logic is quite simple and can represent more complex motions not possible with the other warp modes.

[0166] [Merge with Motion Vector Difference (MMVD)]

[0167] Merge with Motion Vector Difference (MMVD) may be used for either skip mode or merge mode using the motion vector representation method. FIG. 12 shows the MMDV search process for a current frame 630 including a current block 632 using two reference frames 634 and 636 (e.g., the two reference frames shown in FIG. 8). MMVD reuses merge candidates in VVC. A candidate may be selected from among the merge candidates and further extended by the motion vector representation method. MMVD provides a new motion vector representation using simple signaling. The representation method may include a starting point, a motion magnitude, and a motion direction. The MMVD technique may use the merge candidate list of VVC. However, only candidates that are the default merge type (MRG_TYPE_DEFAULT_N) may be considered for the extension of MMVD. A base candidate index (IDX) may define a starting point. The base candidate IDX may indicate the best candidate among the candidates in the list, as shown in Table 7 below. [Table 7]

[0168] If the number of base candidates is equal to 1, the base candidate IDX may not be signaled.

[0169] A distance index may be used to represent the magnitude of the motion information. The distance index may indicate a predetermined distance from the starting point information. The predetermined distance may be as shown in Table 8 below. [Table 8]

[0170] A direction index may be used to represent the direction of the MVD relative to the starting point. The direction index may represent four directions as shown in Table 9 below. [Table 9]

[0171] The MMVD flag may be signaled immediately after sending the skip and merge flags. If the skip and merge flags are true, the MMVD flag may be parsed. If the MMVD flag is equal to 1, the MMVD syntax may be parsed. But if it is not 1, the AFFINE flag may be parsed. If the AFFINE flag is equal to 1, the AFFINE mode may be used. But if it is not 1, the skip / merge index may be parsed for the VTM's skip / merge mode.

[0172] [Issues in related technologies]

[0173] CWG-C050's current warp difference signaling only allows two difference values ​​for rotation and zoom warps. Furthermore, the difference values ​​are signaled using a fixed step length (also called "size"), which is suboptimal for warp model corrections.

[0174] Exemplary embodiments

[0175] The embodiments of the present disclosure may be used separately or combined in any order. Moreover, each of the embodiments of the present disclosure (e.g., the method, the encoder, and the decoder) may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium. Hereinafter, the term "block" may be interpreted as a prediction block, a coding block, or a coding unit (CU, coding unit).

[0176] According to an embodiment, a warp type (or the number of parameters to be corrected by the difference value) is signaled in the bitstream first, and depending on the warp type, the corresponding difference value is signaled in the bitstream. According to an embodiment, at least one syntax element explicitly indicating the warp type to be used (indicating the number of parameters to be corrected by the difference value) may be signaled in the bitstream by the encoder, obtained from the bitstream by the decoder, and used by the decoder to obtain a warp model and perform warp motion compensation based on the warp model to decode at least one coding picture of the bitstream. For example, a current block may be predicted by performing warp motion compensation using the warp model.

[0177] According to one or more embodiments, two different warp types may be signaled in the bitstream. The two warp types are (1) a rotation and zoom type including four parameters (including a translation parameter) and (2) an affine type including six parameters. As an example, the four parameters of the rotation and zoom type may be represented as [a,-b,e;b,a,f], and the six parameters of the affine type may be represented as [a,b,c;d,e,f]. Based on the rotation and zoom type being used, two difference values ​​may be signaled (e.g., in the bitstream) to correct the parameters a and b, respectively. Based on the affine type being used, four difference values ​​may be signaled (e.g., in the bitstream) to correct the parameters a, b, c, and d, respectively. According to embodiments, a decoder (or an encoder) may correct the parameters based on the signaled difference values ​​and perform warp motion compensation based on the corrected parameters. According to embodiments, the parameters may be motion parameters that define the warp motion of a warp model having a warp type. According to an embodiment, the decoder (or encoder) may, for example, obtain a warp model (e.g., compensated motion parameters) for the current block based on the signaled difference values, and perform warp motion compensation on the current block based on the obtained warp model.

[0178] According to one or more embodiments, a syntax element is signaled indicating how many difference values ​​are to be used. For example, if the value of the syntax element is equal to 0, no difference values ​​are signaled, if the value of the syntax element is equal to 1, only the difference value correcting parameter a is signaled, if the syntax element is equal to 2, the difference values ​​correcting parameters a and b (e.g., two respective difference values) are signaled, if the syntax element is equal to 3, the difference values ​​correcting parameters a, b and c (e.g., three respective difference values) are signaled, and if the syntax element is equal to 4, the difference values ​​correcting parameters a, b, c and d (e.g., four respective difference values) are signaled. According to embodiments, parameters e and f may be derived (e.g., by a decoder) using parameters a, b, c, d and MV. Thus, they may not need to further signal the difference values ​​to be obtained. According to an embodiment, the decoder (or encoder) may correct parameters (e.g., motion parameters) of the warp type based on the signaled difference values, derive other parameters (e.g., motion parameters) of the warp type from the corrected parameters, and perform warp motion compensation based on the corrected and derived parameters. According to an embodiment, the encoder may signal to the decoder (e.g., in the bitstream) no difference values ​​or one or more of the difference values ​​based on the values ​​of the syntax elements.

[0179] According to an embodiment, instead of a fixed step size of the difference values, an adaptive step size can be used.

[0180] According to one or more embodiments, the step size is scaled from a predefined base step size, and the scaling factor can be a fraction or an integer. The scaling factor can be signaled on a block-by-block basis. According to an embodiment, the encoder may signal a scaling factor corresponding to a block (e.g., a current block) in the bitstream to a decoder, and the decoder may obtain the scaling factor from the bitstream, scale a step size of the difference value based on the scaling factor and the predefined base step size, and perform warped motion compensation (e.g., of the current block) based on the difference value and the scaled step size of the difference value. According to an embodiment, the encoder may perform the same or similar functions as the decoder described above.

[0181] According to one embodiment, the supported / allowed scaling factors are stored in a predefined look-up table. Instead of directly signaling the scaling factor, an entry index in the table is signaled in the bitstream. According to an embodiment, the index may identify a particular scaling factor stored in the predefined look-up table. According to an embodiment, the encoder may select an index to be signaled based on the predefined look-up table and the obtained scaling factor and signal the index in the bitstream. According to an embodiment, the decoder may obtain an index from the bitstream and obtain a scaling factor based on the index and the predefined look-up table. According to an embodiment, the supported scaling factors in the look-up table may not be evenly distributed, and the scaling factors may have a finer granularity the closer the associated index of the scaling factor is to the zero index, and the scaling factors may have a coarser granularity the further the associated index of the scaling factor is from the zero index. For example, the difference between the scaling factors corresponding to adjacent indexes may be smaller closer to the zero index and larger further away from the zero index. In one example, the supported scaling factors in the lookup table are [1,2,4,8,16,32].

[0182] According to one or more embodiments, the scaling factor can be derived depending on the magnitude of the translational MV (or parameters, e.g., parameters e and f in a six-parameter warp model) of the current block. According to an embodiment, the encoder and / or the decoder may derive the scaling factor.

[0183] According to one or more embodiments, the scaling factors may be derived based on shear parameters (e.g., alpha, beta, gamma, and theta) that are linearly derived from the warp model parameters and control the extent of the warp. According to embodiments, the encoder and / or decoder may derive the shear parameters from the warp model parameters and derive the scaling factors based on the shear parameters.

[0184] According to one or more embodiments, the scaling factor can be updated during encoding based on coding information known to both the encoder and the decoder, such as the step size used by the coding block size. According to an embodiment, the encoder may update the scaling factor.

[0185] According to one or more embodiments, the step size may depend on other coding information, including, but not limited to, the block size, the prediction mode (e.g., whether NEW or NEAR mode is applied), the step size used by the neighboring blocks, one or more of the predicted warp parameters, whether the neighboring blocks are coded by a warp model or a translation model, etc. According to embodiments, the step size may be obtained by the encoder and / or the decoder based on the other coding information.

[0186] According to one or more embodiments, the step sizes may be non-uniformly distributed. For example, the step sizes may have finer granularity closer to a zero step size and coarser granularity farther from the zero step size. For example, the difference between adjacent step sizes may be smaller closer to a zero step size and larger farther from the zero step size.

[0187] According to an embodiment, multiple classes of difference values ​​may be predefined first. The class of difference values ​​(which depends on the magnitude of the difference values) is signaled first. Different classes may have different step sizes (precision) of the difference values. According to an embodiment, the class of difference values ​​may be signaled in the bitstream by the encoder. According to an embodiment, the decoder may obtain the class of difference values ​​from the bitstream and perform warp motion compensation (e.g., of the current block) based on the class of difference values.

[0188] According to an embodiment, the signaling of the sign of the difference value can be hidden, i.e., at the decoder side, a warp predictor with the position and the negative difference value is constructed, and among the warp predictors, the warp predictor with the best smoothness with the neighboring blocks is determined and may be used as the final warp predictor to perform warp motion compensation.

[0189] According to embodiments, the step size may be predicted from coding information known to both the encoder and the decoder, including but not limited to step sizes used by neighboring blocks, whether the neighboring blocks are coded according to a warp model or a translation model, or may be signaled using context derived from the coding information. According to embodiments, the encoder and / or decoder may predict the step size based on the coding information.

[0190] According to an embodiment, at least one processor and a memory storing computer program instructions (i.e., computer code) may be provided that, when executed by the at least one processor, may implement an encoder or a decoder and may perform any number of functions described in this disclosure, including functions performed by an encoder or a decoder.

[0191] For example, with reference to FIG. 13, at least one processor may implement an encoder 702 or a decoder 704. The computer program instructions may include, for example, encoding (or decoding) code 710 configured to cause at least one processor to code (or decode) at least one picture received according to an embodiment of the present disclosure. The encoding (or decoding) code 710 may include, for example, a first acquisition code 720, a second acquisition code 730, a third acquisition code 740, a fourth acquisition code 750, a warped motion compensation code 760, an update code 770, a class code 775, a warped predictor code 780, and a transmission code 790. Although exemplary code for performing certain functions is described herein, it is understood that embodiments of the present disclosure may include other code for performing any other functions described in the present disclosure.

[0192] The first acquisition code 720 may be configured to cause the at least one processor to acquire warp information according to an embodiment of the present disclosure. For example, the warp information may include a warp type indicating a number of parameters to be corrected by the difference value, the number of parameters defining, for example, a warp movement for predicting a current block of a coding picture according to an embodiment of the present disclosure.

[0193] The second acquisition code 730 may be configured to cause the at least one processor to obtain at least one difference value that is, for example, a difference of at least one parameter defining a warp movement for predicting the current block according to an embodiment of the present disclosure.

[0194] The third acquisition code 740 may be configured to cause the at least one processor to acquire a warp model based on the warp information and the at least one difference value according to an embodiment of the present disclosure. In one example, the warp model may be acquired by correcting at least one parameter based on the at least one difference value according to an embodiment of the present disclosure. In one example, the warp model may be acquired further based on a step size (e.g., an adaptive step size) of the at least one difference value according to an embodiment of the present disclosure.

[0195] The fourth obtaining code 750 may be configured to cause the at least one processor to obtain a step size (e.g., an adaptive step size) of the at least one difference value in accordance with an embodiment of the present disclosure. In one example, the step size may be obtained by explicitly signaling (e.g., in the bitstream) or by deriving the step size in accordance with an embodiment of the present disclosure. In one example, the step size may be an adaptive step size obtained based on a scaling factor and a predetermined base step size in accordance with an embodiment of the present disclosure.

[0196] The warp motion compensation code 760 may be configured to cause the at least one processor to predict the current block by performing warp motion compensation using the warp model, for example, in accordance with an embodiment of this disclosure.

[0197] The update code 770 may be configured to cause at least one processor to update the scaling factor during encoding based on coding information known to both the encoder and the decoder, such as the step size used by the coding block size, in accordance with an embodiment of the present disclosure.

[0198] The class code 775 may be configured to cause at least one processor to signal (e.g., within a bitstream) or obtain a class of difference values ​​from a bitstream to perform warp motion compensation (e.g., of a current block) based on the class difference values ​​in accordance with an embodiment of the present disclosure.

[0199] When the decoder 704 is provided, the warp predictor code 780 may be configured to cause at least one processor to construct a warp predictor having a position and a negative difference value according to an embodiment of the present disclosure, determine from among the warp predictors a warp predictor having the best smoothness with adjacent blocks, and use the warp predictor as a final warp predictor for performing warp motion compensation.

[0200] When the encoder 702 is provided, the computer program instructions may further include a transmission code 790 that may be configured, for example, to cause at least one processor to transmit a bitstream including the coded pictures to a decoder (e.g., the decoder 704) in accordance with an embodiment of this disclosure. The encoder 702 may provide information (e.g., syntax elements) related to warping movements in the bitstream, such as information described in this disclosure, in accordance with an embodiment of this disclosure.

[0201] The techniques of the embodiments of the present invention can be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, FIG. 14 illustrates a computer system 900 suitable for implementing embodiments of the disclosed subject matter.

[0202] Computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code that includes instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly or through interpretation, microcode execution, etc.

[0203] The instructions may be executed in various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0204] 14 for computer system 900 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 900.

[0205] The computer system 900 may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, through tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), and olfactory input (not shown). Human interface input devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from a still image camera, etc.), and video (2D video, 3D video including stereoscopic vision, etc.).

[0206] The human interface input devices may include one or more of a keyboard 901, a mouse 902, a trackpad 903, a touch screen 910, a data glove, a joystick 905, a microphone 906, a scanner 907, and a camera 908 (only one of each is shown).

[0207] The computer system 900 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 910, data gloves, or joystick 905, although there may be haptic feedback devices that do not function as input devices). For example, such devices may be audio output devices (speakers 909, headphones (not shown), etc.), visual output devices (e.g., screens 910, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may output two-dimensional visual output or more than three-dimensional output through such means as stereoscopic image output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0208] The computer system 900 may also include human-accessible storage devices and their associated media, such as optical media or similar media 921, including CD / DVD ROM / RW 920 with CDs / DVDs, thumb drives 922, removable hard drives or solid state drives 93, legacy magnetic media such as tapes and floppy disks (not shown), special ROM / ASIC / PLD based devices such as security dongles (not shown).

[0209] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0210] The computer system 900 may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, WLAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV and terrestrial broadcast TV, vehicular and industrial networks including CANBus, etc. Certain networks typically require an external network interface adapter (such as a USB port on the computer system 900) attached to a particular general-purpose data port or peripheral bus 949, while others are typically integrated into the core of the computer system 900 by attachment to a system bus (such as an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system) as described below. Using any of these networks, the computer system 900 may communicate with other entities. Such communications may be one-way receive-only (e.g., broadcast TV), one-way transmit-only (e.g., from a particular CANbus to a particular CANbus device), or bidirectional, to other computer systems, for example, using local or wide area digital networks. Such communications may include communications to cloud computing environment 955. As noted above, specific protocols and protocol stacks may be used in each of these networks and network interfaces.

[0211] The above-mentioned human interface devices, human accessible storage devices and network interface 954 may be attached to the core 940 of the computer system 900 .

[0212] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 943, hardware accelerators for specific tasks 944, etc. These devices may be connected through a system bus 948, along with read only memory (ROM) 945, random access memory (RAM) 946, and internal mass storage 947, such as an internal non-user accessible hard drive, SSD, etc. In some computer systems, the system bus 948 is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 948 or through a peripheral bus 949. Peripheral bus architectures include PCI, USB, etc. A graphics adapter 950 may be included in the core 940.

[0213] The CPU 941, GPU 942, FPGA 943 and accelerator 944 can execute certain instructions, which in combination can constitute the above-mentioned computer code. The computer code can be stored in ROM 945 or RAM 946. Also, temporary data can be stored in RAM 946 while permanent data can be stored, for example, in internal mass storage 947. Fast storage and retrieval from any of the memory devices can be enabled through the use of cache memory, which can be closely associated with one or more of the CPU 941, GPU 942, mass storage 947, ROM 945, RAM 946, etc.

[0214] The computer-readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0215] By way of example and not limitation, the architecture and computer system 900 having the core 940 in particular can provide functionality as a result of the processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer readable media. Such computer readable media can be user accessible mass storage as introduced above, as well as media associated with the core 940's specific storage of a non-transitory nature, such as the core internal mass storage 947 or ROM 945. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core 940. The computer readable media can include one or more memory devices or chips according to specific needs. The software can cause the core 940 and in particular the processor therein (including CPU, GPU, FPGA, etc.) to execute certain processes or certain parts of certain processes described herein, including defining data structures stored in RAM 946 and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 944) that may operate in place of or in conjunction with software to perform particular processes, or particular portions of particular processes, described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0216] While this disclosure describes several non-limiting exemplary embodiments, there are modifications, permutations, and various substitute equivalents that are within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise various systems and methods that, although not explicitly shown or described herein, embody the principles of this disclosure and are therefore within the spirit and scope of this disclosure.

Claims

1. A method performed by a decoder, comprising: receiving a bitstream including coding pictures; obtaining warp information signaled in the bitstream, the warp information including a warp type indicating a number of parameters to be corrected by a difference value, the number of parameters defining a warp motion for predicting a current block of the coding picture; obtaining from the bitstream at least one difference value which is a difference of at least one parameter defining the warp motion for predicting the current block; obtaining a warped model based on the warp information and the at least one difference value; predicting the current block by performing warp motion compensation using the warp model; The method includes:

2. the warp type is a rotate and zoom type that includes four parameters that define the warp movement; the at least one difference value obtained is two difference values ​​for correcting two parameters of the four parameters; The method of claim 1 , wherein obtaining the warp model comprises correcting the two parameters based on the two difference values.

3. the warp type is an affine type that includes six parameters that define the warp motion; the at least one difference value obtained is four difference values ​​for correcting four parameters among the six parameters; The method of claim 1 , wherein obtaining the warp model comprises correcting the four parameters based on the four difference values.

4. the warp information includes a number of the parameters defining the warp movement to be corrected by the difference value; The method of claim 1 , wherein a number of the at least one difference value obtained is equal to a number of the parameters of the warp information to be corrected by the difference value.

5. obtaining an adaptive step size for the at least one difference value; The method of claim 1 , wherein obtaining the warp model comprises obtaining the warp model based on the adaptive step size.

6. The step of obtaining an adaptive step size includes: obtaining a scaling factor; calculating the adaptive step size based on a predetermined base step size and the scaling factor; The method of claim 5 , comprising:

7. The step of obtaining the scaling factor includes: obtaining an index indicating the scaling factor from among a plurality of scaling factors stored in a lookup table; obtaining the scaling factor based on the index and the lookup table; The method of claim 6, comprising:

8. The method of claim 6 , wherein obtaining the scaling factor comprises deriving the scaling factor based on a magnitude of a translational motion vector of the current block.

9. The method of claim 6 , wherein obtaining the scaling factor comprises deriving the scaling factor based on a shear parameter.

10. 6. The method of claim 5, wherein obtaining the adaptive step size comprises obtaining the adaptive step size based on at least one of a block size, a prediction mode, a step size used by at least one neighboring block of the current block, one or more predicted warp parameters, and whether the neighboring block of the current block was coded by a warp model or a translation model.

11. at least one memory configured to store computer program code; at least one processor configured to receive a bitstream containing coded pictures, to access said computer program code, and to perform the method of any one of claims 1 to 10 according to said computer program code; An apparatus comprising:

12. A computer program product causing at least one processor to carry out a method according to any one of claims 1 to 10.

13. A method performed by an encoder, comprising: obtaining warp information, the warp information including a warp type indicating a number of parameters to be corrected by a difference value, the number of parameters defining a warp motion for predicting a current block of a coding picture; obtaining at least one difference value which is a difference of at least one parameter defining said warping motion for predicting said current block; transmitting a bitstream including the coded picture, the warp information, and the at least one difference value; Including, The current block is predicted by performing warp motion compensation using a warp model obtained based on the warp information and the at least one difference value.

Images