Method and apparatus for selecting and grouping warp samples
The method improves video decoding efficiency by constructing a warp model using time and spatial motion vectors to enhance prediction accuracy in video coding technologies like AV1 and VVC.
Patent Information
- Application Number
- JP2026200086
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-11-03
- Filing Date
- 2026-09-03
- Publication Date
- 2026-11-19
AI Technical Summary
Existing video coding technologies, such as AV1 and VVC, face challenges in efficiently utilizing motion vectors for improved compression and prediction, particularly in handling local warp motion modes.
The method involves receiving a bitstream, obtaining motion vectors, constructing a warp model using time and spatial motion vectors, and applying a motion offset to enhance prediction accuracy.
This approach improves video decoding efficiency by leveraging a warp model with time motion vectors, enhancing compression capabilities and prediction accuracy.
Smart Images

Figure 2026182992000001_ABST
Abstract
Description
Technical Field
[0001] [Cross-Reference to Related Applications] This application claims priority to U.S. Provisional Application No. 63 / 392,381 filed on July 26, 2022 and U.S. Patent Application No. 17 / 980,331 filed on November 3, 2022, claims priority to these applications, and the entire contents of these applications are incorporated herein by reference.
[0002] [Technical Field] Embodiments of the present disclosure relate to advanced image and video coding technologies, and more specifically, to improvements to local warp motion modes.
Background Art
[0003] AV1 (AOMedia Video 1) is an open video coding format designed for video transmission over the internet. It was developed as a successor to VP9 by AOMedia (Alliance for Open Media), a consortium founded in 2015 that includes the semiconductor industry, video-on-demand providers, video content producers, software developers, and web browser vendors. Many components of the AV1 project were derived from previous research efforts by the group's members. Individual contributors had launched experimental technology platforms several years prior. Xiph / Mozilla's Daala had already released its code in 2010, Google's experimental VP9 evolution project, V10, was announced on September 12, 2014, and Cisco's Thor was released on August 11, 2015. Built upon the VP9 codebase, AV1 incorporates further technologies, some of which were developed in these experimental formats. The first version of the AV1 reference codec, 0.1.0, was released on April 7, 2016. The organization announced the release of the AV1 bitstream specification on March 28, 2018, along with reference software-based encoders and decoders. A verified version 1.0.0 of the specification was released on June 25, 2018. A verified version 1.0.0 with errata 1 was released on January 8, 2019. The AV1 bitstream specification includes a reference video codec.
[0004] The ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). They also investigated the potential need for standardization of future video coding technologies that could offer significantly better performance than HEVC in terms of compression capabilities. In October 2017, they issued a Joint Call for Proposals on Video Compression with Capability Beyond HEVC (CfP). By February 15, 2018, a total of 22 CfP responses had been submitted for Standard Dynamic Range (SDR), 12 for High Dynamic Range (HDR), and 12 for the 360° video category. In April 2018, all received CfP responses were evaluated at the 122nd MPEG / 10th JVET (Joint Video Exploration Team or Joint Video Expert Team) meeting. As a result of this meeting, JVET officially began standardizing a next-generation video coding standard beyond HEVC. The new standard is called VVC (Versatile Video Coding). [Overview of the Initiative]
[0005] Embodiments of the present disclosure provide a method performed by a decoder. The method includes: receiving a bitstream including a coding picture; obtaining a plurality of motion vectors pointing to positions in adjacent blocks adjacent to the current block in the coding picture, wherein the plurality of motion vectors include at least one time motion vector; obtaining a motion sample for a warp model based on the plurality of motion vectors; and decoding the current block by using the warp model and the motion sample including at least one time motion vector for prediction.
[0006] According to one or more embodiments of the present disclosure, at least one time motion vector includes a time motion vector at the position of a spatially adjacent block of the current block that has not yet been constructed by the decoder.
[0007] According to one or more embodiments of the present disclosure, at least one time motion vector includes a time motion vector at the position of a spatially adjacent block of the current block previously constructed by the decoder.
[0008] According to one or more embodiments of the present disclosure, the method further includes the step of applying a motion offset to at least one time motion vector.
[0009] According to one or more embodiments of the present disclosure, the method further includes the step of obtaining a warp model using a regression / least squares error method on a plurality of motion vectors, including at least one time motion vector.
[0010] According to one or more embodiments of the present disclosure, the plurality of motion vectors include at least one time motion vector and at least one spatial motion vector.
[0011] According to one or more embodiments of the present disclosure, the plurality of motion vectors include at least one time motion vector and do not include a spatial motion vector.
[0012] According to one or more embodiments of the present disclosure, the decoding step further includes the step of constructing a list of adjacent blocks of a plurality of groups, and the step of selecting a group from the adjacent blocks of the plurality of groups from the list, and the step of obtaining a plurality of motion vectors includes the step of obtaining a plurality of motion vectors from the adjacent blocks of the selected group.
[0013] According to one or more embodiments of the present disclosure, the selection step includes selecting a group from a list based on an index signaled in the bitstream, the index indicating which group from the list should be used to obtain the warp model.
[0014] According to one or more embodiments of the present disclosure, the decoding step includes decoding the current block by performing warp motion compensation.
[0015] Embodiments of the present disclosure provide a system, comprising at least one memory configured to store program code, and at least one processor configured to receive a bitstream containing a coding picture, access computer program code, and operate as instructed by the computer program code, wherein the computer program code comprises motion vector acquisition code configured to cause at least one processor to acquire a plurality of motion vectors pointing to a position in an adjacent block adjacent to the current block in the coding picture, the plurality of motion vectors comprising at least one time motion vector, motion sample acquisition code configured to cause at least one processor to acquire a motion sample for a warp model based on the plurality of motion vectors, and decoding code configured to cause at least one processor to decode the current block by using the warp model and the motion sample comprising at least one time motion vector for prediction.
[0016] According to one or more embodiments of the present disclosure, at least one time motion vector includes a time motion vector at the location of a spatially adjacent block of the current block that has not yet been constructed by at least one processor.
[0017] According to one or more embodiments of the present disclosure, at least one time motion vector includes a time motion vector at the position of a spatially adjacent block of the current block previously constructed by at least one processor.
[0018] According to one or more embodiments of the present disclosure, the computer program code further includes motion offset code configured to cause at least one processor to apply a motion offset to at least one time motion vector.
[0019] According to one or more embodiments of the present disclosure, the computer program code further includes warp model acquisition code configured to cause at least one processor to acquire a warp model by using a regression / least squares error method on a plurality of motion vectors, including at least one time motion vector.
[0020] According to one or more embodiments of the present disclosure, the plurality of motion vectors include at least one time motion vector and at least one spatial motion vector.
[0021] According to one or more embodiments of the present disclosure, the plurality of motion vectors include at least one time motion vector and do not include a spatial motion vector.
[0022] According to one or more embodiments of the present disclosure, the computer program code further includes list construction code configured to cause at least one processor to construct a list of adjacent blocks of a plurality of groups, and selection code configured to cause at least one processor to select a group from the list of adjacent blocks of a plurality of groups, wherein the motion vector acquisition code is configured to cause at least one processor to acquire a plurality of motion vectors from the adjacent blocks of the selected group.
[0023] According to one or more embodiments of the present disclosure, a selection code is configured to cause at least one processor to select a group from a list based on an index signaled in a bitstream, the index indicating which group from the list should be used to obtain the warp model.
[0024] Embodiments of the present disclosure provide a non-temporary computer-readable medium for storing computer code. The computer code enables a decoder that, when executed by at least one processor, causes at least one processor to perform the following steps: receive a bitstream including a coding picture; obtain a plurality of motion vectors pointing to positions in adjacent blocks adjacent to the current block in the coding picture, wherein the plurality of motion vectors include at least one time motion vector; obtain a motion sample for a warp model based on the plurality of motion vectors; and decode the current block using the warp model and the motion sample including at least one time motion vector for prediction. [Brief explanation of the drawing]
[0025] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the following detailed description and accompanying drawings. [Figure 1] This is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 2]1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment. [Figure 3] 3 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment. [Figure 4] 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment. [Figure 5A] 9 is a schematic diagram of a partition tree in VP9 according to an embodiment. [Figure 5B] 12 is a schematic diagram of a partition tree in AV1 according to an embodiment. [Figure 6A] 15 is a schematic diagram of block partitioning of a quad-tree plus binary tree (QTBT) structure according to an embodiment. [Figure 6B] 18 is a schematic diagram of a corresponding tree representation of the block-partitioned QTBT structure shown in FIG. 6A. [Figure 7A] 21 is a schematic diagram of vertical center-side ternary tree partitioning according to an embodiment. [Figure 7B] 24 is a schematic diagram of horizontal center-side ternary tree partitioning according to an embodiment. [Figure 8] 27 is a schematic diagram of search points of a merge mode with motion vector difference (MMVD) using motion vector differences of two reference frames according to an embodiment. [Figure 9] 30 is a schematic diagram of exemplary motion samples used to derive model parameters of a block using local warped motion prediction according to an embodiment. [Figure 10] 33 is a schematic diagram of an example of a warp sample collection scanning order according to an embodiment. [Figure 11] 36 is a schematic diagram of motion vectors within a block using a warp extension mode according to an embodiment. [Figure 12] 39 is a schematic diagram of an MMVD search process for a current frame using two reference frames according to an embodiment. [Figure 13]This is a schematic diagram of the current block and adjacent blocks used to create a warp model according to one embodiment. [Figure 14] This is a schematic diagram of the current block and adjacent blocks used to create a warp model according to one embodiment. [Figure 15] This is a schematic diagram of the current block and adjacent blocks used to create a warp model according to one embodiment. [Figure 16] This is a schematic diagram of the current block and adjacent blocks used to create a warp model according to one embodiment. [Figure 17] This is a block diagram of computer code according to an embodiment. [Figure 18] This is a diagram of a computer system suitable for implementing an embodiment of the present disclosure. [Modes for carrying out the invention]
[0026] A detailed description of exemplary embodiments follows with reference to the accompanying drawings. The same reference numerals in different drawings may identify the same or similar elements.
[0027] The above disclosure provides examples and explanations, but is not intended to be exhaustive or to limit implementations to the exact forms disclosed. Modifications and alterations are possible in light of this disclosure, or may be obtained from the implementation. Furthermore, one or more features or components of one embodiment may be incorporated into or combined with other embodiments (or one or more features of other embodiments). Furthermore, it should be understood that in the flowcharts and descriptions of operations provided below, one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) simultaneously, and the order of one or more operations may be changed.
[0028] It is evident that the systems and / or methods described herein may be implemented in different forms, such as hardware, software, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited to the implementation. It is understood that the software and hardware may be designed to implement the systems and / or methods based on the descriptions herein.
[0029] The features described below may be used separately or in any order. Furthermore, embodiments may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium.
[0030] Any elements, actions, or instructions used herein should not be construed as important or essential unless expressly stated otherwise. Furthermore, where used herein, the singular is intended to include one or more items and may be used interchangeably with “one or more.” When one item is intended, “one” or a similar term is used. Also, where used herein, terms such as “has,” “have,” “having,” “include,” and “including” are intended to be open-ended terms. Furthermore, the phrase “based on” is intended to mean “based at least partially” unless otherwise specified. Additionally, expressions such as “at least one of A and B” or “at least one of A or B” should be understood as including A only, B only, or both A and B.
[0031] Figure 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 includes at least two terminals 110, 120 interconnected via a network 150. For one-way data transmission, the first terminal 110 may encode local location video data for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive encoded video data from the other terminal via the network 150, decode the encoded data, and display the restored video data. One-way data transmission may be common in media delivery applications and the like.
[0032] Figure 1 shows a second pair of terminals 130, 140 provided to support the bidirectional transmission of encoded video that may occur, for example, during a video conference. For bidirectional data transmission, each terminal 130, 140 may encode video data captured at its local location for transmission to other terminals via the network 150. Each terminal 130, 140 may also receive encoded video data transmitted by other terminals, decode the encoded data, and display the restored video data on a local display device.
[0033] In Figure 1, terminals 110-140 may be represented as servers, personal computers, smartphones, and / or any other type of terminal. For example, terminals 110-140 may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks that transmit encoded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched channels and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of network 150 are not important to the operation of this disclosure unless described below herein.
[0034] Figure 2 shows an example of an application of the subject matter of the disclosure, illustrating the arrangement of a video encoder and decoder in a streaming environment. The subject matter of the disclosure is similarly applicable to other video-enabled applications, including, for example, video conferencing, digital TV, and the storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).
[0035] As shown in Figure 2, the streaming system 200 may include a capture subsystem 213 which can include a video source 201 and an encoder 203. For example, the video source 201 may be a digital camera and may be configured to produce an uncompressed video sample stream 202. The uncompressed video sample stream 202 may provide a higher data volume compared to the encoded video bitstream and can be processed by the encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the subject matter of disclosure, as will be described in more detail below. The encoded video bitstream 204 may contain a lower data volume compared to the sample stream and can be stored in the streaming server 205 for future use. One or more streaming clients 206 can access the streaming server 205 to obtain a video bitstream 209 which may become a copy 208 of the encoded video data 204.
[0036] In an embodiment, the streaming server 205 may also function as a Media-Aware Network Element (MANE). For example, the streaming server 205 may be configured to prune an encoded video bitstream 204 to align potentially different bitstreams with one or more of the streaming clients 206. In an embodiment, the MANE may be provided separately from the streaming server 205 in the streaming system 200.
[0037] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 can, for example, decode a video bitstream 209, which is an input copy of an encoded video bitstream 204, and generate an output video sample stream 210 that can be rendered on the display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204,209 can be encoded according to a specific video encoding / compression standard. Examples of these standards include, but are not limited to, ITU-T Recommendation H.265. Video encoding standards under development are informally known as VVC (Versatile Video Coding). Embodiments of this disclosure may be used in the context of VVC.
[0038] Figure 3 shows an exemplary functional block diagram of a video decoder 210 mounted on a display 212 according to one embodiment of the present disclosure.
[0039] The video decoder 210 may include a channel 312, a receiver 310, a buffer memory 315, an entropy decoder / parser 320, a scaler / inverse unit 351, an intra-prediction unit 352, a motion compensation prediction unit 353, an aggregator 355, a loop filter unit 356, a reference picture memory 357, and a current picture memory. In at least one embodiment, the video decoder 210 may include an integrated circuit, a series of integrated circuits, and / or other electronic circuits. The video decoder 210 may also be partially or entirely embodied in software running on one or more CPUs having associated memory.
[0040] In this embodiment and other embodiments, the receiver 310 may receive one or more encoded video sequences to be decoded by the decoder 210, or it may receive one encoded video sequence at a time, and the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequences may be received from channel 312, which may be a hardware / software link to a storage device that stores encoded video data. The receiver 310 may receive the encoded video data together with other data (e.g., encoded audio data and / or auxiliary data streams), which may be transferred to their respective user entities (not shown). The receiver 310 may isolate the encoded video sequences from other data. To prevent network jitter, a buffer memory 315 may be coupled between the receiver 310 and the entropy decoder / parser 320 (hereinafter referred to as the "parser"). If the receiver 310 is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory 315 may not be used or may be reduced in size. For use in best-effort packet networks such as the internet, a buffer memory of 315 may be required, which can be relatively large and adapted to different sizes.
[0041] The video decoder 210 may include a parser 320 for reconstructing symbols 321 from an entropy-encoded video sequence. These symbol categories may include, for example, information used to manage the operation of the decoder 210, and potentially information for controlling a rendering device, such as a display 212, which may be coupled to the decoder as shown in Figure 2. The rendering device control information may be in the form of, for example, Supplementary Enhancement Information (SEI) (SEI messages) or Video Usability Information (VUI) parameter set fragments (not shown). The parser 320 may parse / entropy-decode the received encoded video sequence. The encoding of the encoded video sequence may follow video encoding techniques or standards, and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser 320 may extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to a group. Subgroups may include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transformation units (TU), prediction units (PU), etc. Parser 320 may also extract information from the coded video sequence, such as transformation coefficients, quantization parameter values, and motion vectors.
[0042] The parser 320 may perform an entropy decoding / analysis operation on the video sequence received from the buffer memory 315 in order to generate the symbol 321.
[0043] The reconstruction of symbol 321 may involve multiple different units, depending on the type of encoded video picture or portion thereof (e.g., interpicture and intrapicture, interblock and intrablock) and other factors. Which units are involved and how they are involved can be controlled by subgroup control information analyzed from the encoded video sequence by parser 320. The flow of such subgroup control information between parser 320 and the following multiple units is not illustrated for clarity.
[0044] In addition to the functional blocks described above, the decoder 210 can be conceptually subdivided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be integrated at least partially. However, for the purpose of illustrating the subject of disclosure, it is appropriate to conceptually subdivide it into the following functional units.
[0045] One unit is the scaler / inverse unit 462. The scaler / inverse unit 351 may receive quantized transformation coefficients as symbol 321 from the parser 320, along with control information (including which transformation to use, block size, quantization coefficients, quantization scaling matrix, etc.). The scaler / inverse unit 351 can output a block containing sample values that can be input to the aggregator 355.
[0046] In some cases, the output samples of the scaler / inverse unit 351 may relate to intra-encoded blocks, i.e., blocks that do not use predictive information from previously reconstructed pictures but can use predictive information from previously reconstructed portions of the current picture. Such predictive information can be provided by the intra-picture predictive unit 352. In some cases, the intra-picture predictive unit 352 generates blocks of the same size and shape as the block being reconstructed, using already reconstructed surrounding information retrieved from the (partially reconstructed) current picture from the current picture memory 358. In some cases, the aggregator 355 adds the predictive information generated by the intra-picture predictive unit 352 to the output sample information provided by the scaler / inverse unit 351, sample by sample.
[0047] In other cases, the output samples of the scaler / inverse unit 351 may be associated with intercoded and potentially motion-compensated blocks. In such cases, the motion-compensated prediction unit 353 may access the reference picture memory 357 to retrieve samples to be used for prediction. After motion-compensating the retrieved samples according to the symbols 321 associated with the blocks, these samples can be added by the aggregator 355 to the output of the scaler / inverse unit 351 (in this case, called residual samples or residual signals) to generate output sample information. The address in the reference picture memory 357 from which the motion-compensated prediction unit 353 retrieves the prediction samples can be controlled by a motion vector. The motion vector may be available to the motion-compensated prediction unit 353 in the form of a symbol 321, which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values retrieved from the reference picture memory 357 when the precise motion vectors of the subsamples are used, motion vector prediction mechanisms, etc.
[0048] The output samples of the aggregator 355 can undergo various loop filtering techniques within the loop filter unit 356. The video compression technique may include an in-loop filtering technique, which is controlled by parameters included in the encoded video bitstream and made available to the loop filter unit 356 as symbols 321 from the parser 320, and can respond to metadata obtained during the decoding of earlier parts (in decoding order) of the encoded picture or encoded video sequence, as well as to previously reconstructed and loop-filtered sample values.
[0049] The output of the loop filter unit 356 can be a sample stream, which can be output to a rendering device such as the display 212, and can also be stored in the reference picture memory 357 for use in future interpicture prediction.
[0050] A particular encoded picture, once fully reconstructed, can be used as a reference picture for future predictions. Once an encoded picture is fully reconstructed and identified as a reference picture (e.g., by parser 320), the current reference picture can become part of reference picture memory 357, and new current picture memory can be reallocated before starting the reconstruction of subsequent encoded pictures.
[0051] The video decoder 210 may perform decoding operations according to a predetermined video compression technique that may be documented in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that it conforms to the syntax of the video compression technique or standard, as specified in the video compression technique or standard, and particularly in its profile document. Also, for compliance with certain video compression techniques or standards, the complexity of the encoded video sequence may be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level can be further limited through the Hypothetical Reference Decoder (HRD) specification and metadata about HRD buffer management transmitted in the encoded video sequence.
[0052] In one embodiment, the receiver 310 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder 210 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, time, space, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.
[0053] Figure 4 shows an exemplary functional block diagram of a video encoder 203 associated with a video source 201 according to one embodiment of the present disclosure.
[0054] The video encoder 203 may include, for example, an encoder which is a source coder 430, an encoding engine 432, a (local) decoder 433, a reference picture memory 434, a predictor 435, a transmitter 440, an entropy coder 445, a controller 450, and a channel 460.
[0055] The encoder 203 may receive video samples from a video source 201 (which is not part of the encoder), and the video source 201 may capture video images to be encoded by the encoder 203.
[0056] The video source 201 may provide a source video sequence to be encoded by the encoder 203 in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media delivery system, the video source 201 may also be a storage device that stores pre-prepared video. In a video conferencing system, the video source 203 may also be a camera that captures local image information as a video sequence. The video data may be provided as a series of individual pictures that convey motion when viewed sequentially. The picture itself may be composed of a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.
[0057] According to one embodiment, the encoder 203 may encode and compress the pictures of the source video sequence into an encoded video sequence 443 in real time or under any other time constraints required by the application. Achieving an appropriate encoding speed is one function of the controller 450. The controller 450 may also control other functional units, as described below, and may be functionally coupled to these units. The couplings are not shown for clarity. Parameters set by the controller 450 may include rate control-related parameters (such as picture skipping, quantization, lambda values for rate distortion optimization techniques), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art will readily recognize other functions of the controller 450 that may be relevant to a video encoder 203 optimized for a particular system design.
[0058] Some video encoders operate in what is readily recognizable to those skilled in the art as an "encoding loop." In very simplified terms, the encoding loop can consist of an encoding portion of a source coder 430 (responsible for generating symbols based on the input picture and reference picture to be encoded) and a (local) decoder 433 embedded in the encoder 203. The decoder 433 reconstructs the symbols to generate sample data similar to what a (remote) decoder would generate, provided that the compression between the symbols and the encoded video bitstream is reversible in a given video compression technique. The reconstructed sample stream may be fed into a reference picture memory 434. Since decoding the symbol stream yields bitwise-accurate results independent of the decoder's location (local or remote), the contents of the reference picture memory are also bitwise-accurate between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values as the reference picture samples that the decoder "sees" when using predictions during decoding. This basic principle of the synchronization of reference pictures (including, for example, drift resulting from the inability to maintain synchronization due to channel errors) is known to those skilled in the art.
[0059] The operation of the "local" decoder 433 can be the same as that of the "remote" decoder 210, which has already been described in detail above in relation to Figure 3. However, since symbols are available and the encoding / decoding of symbols to the encoded video sequence by the entropy coder 445 and parser 320 can be reversible, the entropy decoding portion of the decoder 210, including channel 312, receiver 310, buffer memory 315 and parser 320, does not have to be fully implemented in the local decoder 433.
[0060] A consideration that can be made at this point is that any decoder technique present within the decoder, excluding analysis / entropy decoding, must exist in substantially the same functional form within the corresponding encoder. Therefore, the subject of this disclosure will focus on decoder operation. A description of encoder techniques can be omitted, as it may be the inverse of a comprehensive description of decoder techniques. More detailed explanations are necessary only in specific areas and are provided below.
[0061] As part of its operation, the source coder 430 may perform motion-compensated predictive coding, which predictively codes the input frame by referencing one or more previously coded frames from a video sequence designated as “reference frames”. In this way, the coding engine 432 codes the difference between the pixel blocks of the input frame and the pixel blocks of the reference frame that may be selected as a predictive reference for the input frame.
[0062] The local decoder 433 may decode the encoded video data of a frame that may be designated as a reference frame based on the symbols generated by the source coder 430. The operation of the encoding engine 432 may, advantageously, be lossy. If the encoded video data can be decoded by a video decoder (not shown in Figure 4), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local decoder 433 may replicate the decoding process that may be performed by the video decoder on the reference frame and store the reconstructed reference frame in the reference picture memory 434. Thus, the encoder 203 may locally store a copy of the reconstructed reference frame having common content as a reconstructed reference frame (without transmission errors) to be acquired by the far-end video decoder.
[0063] The predictor 435 may perform a predictive search for the coding engine 432. That is, for a new frame to be coded, the predictor 435 may search the reference picture memory 434 for sample data (as candidate reference pixel blocks) or specific metadata (reference picture motion vector, block shape, etc.). These may function as appropriate predictive references for the new picture. The predictor 435 may operate sample block-by-pixel block to find appropriate predictive references. In some cases, the input picture determined by the search results obtained by the predictor 435 may have predictive references drawn from multiple reference pictures stored in the reference picture memory 434.
[0064] The controller 450 may manage the encoding operation of the source coder 430, including, for example, setting parameters and subgroup parameters used to encode video data.
[0065] The outputs of all the above functional units may undergo entropy coding in the entropy coder 445. The entropy coder converts the symbols generated by the various functional units into coded video sequences by lossless compression of the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, arithmetic coding, etc.
[0066] Transmitter 440 may buffer the encoded video sequence generated by entropy coder 445 and prepare it for transmission over communication channel 460, which may be a hardware / software link to a storage device that stores the encoded video data. Transmitter 440 may merge the encoded video data from coder 430 with other data to be transmitted (e.g., encoded audio data and / or auxiliary data streams (not shown)).
[0067] The controller 450 may manage the operation of the encoder 203. During encoding, the controller 450 may assign a specific encoded picture type to each encoded picture. This encoded picture type may influence the encoding technique that can be applied to each picture. For example, a picture may often be assigned as an intra-picture (I-picture), a predictive picture (P-picture), or a bidirectional predictive picture (B-picture).
[0068] An intra-picture (I-picture) may be one that can be encoded and decoded without using other frames in the sequence as a source for prediction. Some video codecs enable different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will recognize these variations of I-pictures, as well as their respective uses and characteristics.
[0069] The prediction picture (P-picture) may be encoded and decoded using intra-prediction or inter-prediction, which uses at most one motion vector and reference index to predict the sample values of each block.
[0070] A bidirectional predictive picture (B-picture) may be encoded and decoded using intra-prediction or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values of each block. Similarly, a multiple predictive picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.
[0071] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block may be coded. Blocks may be coded predictively by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, blocks of picture I may be coded unpredictably, or they may be coded predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded unpredictably by referencing one previously coded reference picture, via spatial prediction or temporal prediction. Blocks of picture B may be coded unpredictably by referencing one or two previously coded reference pictures, via spatial prediction or temporal prediction.
[0072] The video encoder 203 may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In these operations, the video encoder 203 may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to the syntax specified by the video coding technique or standard being used.
[0073] In one embodiment, the transmitter 440 may transmit additional data along with the encoded video. The source coder 430 may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, and the like.
[0074] [Block partitioning in VP9 and AV1]
[0075] As described above, AV1 is an open video coding format designed for video transmission over the internet, developed as a successor to VP9. Referring to Figure 5A, VP9 uses a four-way partition tree starting at 64x64 levels down to 4x4 levels, with some further limitations on blocks below 8x8 (as shown in the upper half of Figure 5A). Partitions designated as R may be called recursive in that the same partition tree may be repeated at lower scales until the partition reaches the lowest 4x4 level.
[0076] As shown in Figure 5B, AV1 not only extends the partition tree to a 10-way structure but also increases the maximum size (called a superblock in VP9 / AV1 terminology) to start at 128x128. This may include 4:1 / 1:4 rectangular partitions that were not present in VP9. None of the rectangular partitions may be further subdivided. Furthermore, AV1 adds more flexibility to the use of partitions below the 8x8 level, in the sense that 2x2 chromainter prediction becomes possible in certain cases.
[0077] [Block partitioning in HEVC]
[0078] In HEVC, a coding tree unit (CTU) may be divided into coding units (CU) by using a quad-tree (QT) structure called a coding tree to adapt to various local characteristics. The decision of whether to code a picture region using interpicture (time) prediction or intrapicture (spatial) prediction may be made at the CU level. Each CU can be further divided into one, two, or four prediction units (PU) according to the PU partitioning type. Within a single PU, the same prediction process may be applied, and relevant information may be transmitted to the decoder for each PU. After obtaining residual blocks by applying the prediction process based on the PU partitioning type, the CU can be partitioned into transform units (TU) according to other QT structures such as coding trees for the CU. One of the key features of the HEVC structure is that it has multiple partition concepts, including CUs, PUs, and TUs. In HEVC, CUs or TUs can only be square in shape, while PUs can be square or rectangular in shape for inter-prediction blocks. In HEVC, a single coding block may be further divided into four square subblocks, and a transformation is performed on each subblock, i.e., the TU. Each TU may be further recursively divided into smaller TUs (using QT partitioning), which is called a Residual Quad-Tree (RQT).
[0079] At picture boundaries, HEVC employs implicit quadtree partitioning so that blocks maintain their quadtree partitioning until their size fits the picture boundary.
[0080] [Block partitioning in VVC]
[0081] 1. Block partitioning structure using quad-tree (QT) + BT (binary tree)
[0082] The QTBT block structure eliminates the concept of multiple partition types. That is, the QTBT block structure eliminates the separation of the concepts of CU, PU, and TU, and supports more flexibility for the CU partition shape. In the QTBT block structure, the CU may have a square or rectangular shape. Referring to Figures 6A and 6B, the coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a binary tree structure. There are two types of BT partitioning: symmetric horizontal partitioning and symmetric vertical partitioning. The BT leaf nodes may also be called CUs, and their segmentation is used for prediction and transformation processing without further partitioning. This means that CU, PU, and TU have the same block size in the QTBT coding block structure. In the joint exploration model (JEM), the CU may consist of coding blocks (CBs) of different color components. For example, in the case of prediction (P) and binary (B) slices of a 4:2:0 chroma format, one CU may contain one luma CB and two chroma CBs. A CU may also consist of a single-component CB or contain a single-component CB. For example, in the case of an I slice, one CU may contain only one luma CB or only two chroma CBs.
[0083] The following parameters may be defined for the QTBT partitioning scheme. (a) CTU size: The size of the root node of a quadtree, which can be considered the same concept as HEVC. (b) MinQTSize: Minimum allowable quadtree leaf node size (c) MaxBTSize: Maximum allowable binary tree root node size (d) MaxBTDepth: Maximum allowable binary tree depth (e) MinBTSize: Minimum allowable binary tree leaf node size
[0084] In one example of a QTBT partitioning structure, the CTU size may be set to a 128x128 chroma sample with two corresponding 64x64 blocks of chroma samples, MinQTSize may be set to 16x16, MaxBTSize may be set to 64x64, MinBTSize (for both width and height) may be set to 4x4, and MaxBTDepth may be set to 4. The quadtree partitioning may be applied to the CTU first to generate QT leaf nodes. QT leaf nodes may have sizes ranging from 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If a leaf QT node is 128x128, it does not need to be further partitioned by a binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, the leaf QT node may be further partitioned by a binary tree. Thus, the QT leaf node is also the root node of the BT and has a BT depth of zero. When the BT depth reaches MaxBTDepth (i.e., 4), further partitioning may not be considered. When a BT node has a width equal to MinBTSize (i.e., 4), further horizontal partitioning may not be considered. Similarly, when a BT node has a height equal to MinBTSize, further vertical partitioning may not be considered. The leaf nodes of the BT may be further processed by prediction and transformation processes without further partitioning. In JEM, for example, the maximum CTU size may be 256 × 256 luma samples.
[0085] Figure 6A shows an example of block partitioning using QTBT, and Figure 6B shows the corresponding tree representation. Solid lines represent QT partitions, and dotted lines represent BT partitions. At each BT partition (i.e., non-leaf) node, a flag may be signaled to indicate which partition type (i.e., horizontal or vertical) is used, where 0 indicates a horizontal partition and 1 indicates a vertical partition. In the case of QT partitions, it may not be necessary to indicate the partition type, as a QT partition can always divide a block both horizontally and vertically to produce four subblocks of equal size.
[0086] The QTBT method may support flexibility for luma and chroma so that they have separate QTBT structures. Currently, for P-slice and B-slice, the luma CTB and chroma CTB within one CTU share the same QTBT structure. However, for I-slice, the luma CTB is partitioned into CUs by the QTBT structure, and the chroma CTB is partitioned into chroma CUs by the other QTBT structure. This means that the CUs in I-slice consist of coding blocks for the luma component or coding blocks for the two chroma components, and the CUs in P-slice or B-slice consist of coding blocks for all three color components.
[0087] In HEVC, inter-prediction for small blocks is limited to reduce motion-compensated memory access, bi-prediction is not supported for 4x8 and 8x4 blocks, and inter-prediction is not supported for 4x4 blocks. These limitations are removed in QTBT, which is implemented in JEM-7.0.
[0088] 2. Block partitioning using a ternary tree (TT)
[0089] In VVC, a multi-type-tree (MTT) structure is included, as shown in Figures 7A and 7B. This adds vertical central ternary trees and horizontal central ternary trees to the QTBT. Figure 7A shows a vertical central ternary tree partition, and Figure 7B shows a horizontal central ternary tree partition.
[0090] The main advantages of ternary tree partitioning include, but are not limited to, (a) providing a complement to quadtree and binary tree partitioning, i.e., while quadtrees and binary trees always partition along the center of a block, ternary tree partitioning can capture objects located at the center of a block, and (b) the width and height of the proposed ternary tree partitions may always be powers of 2, and therefore no further transformation may be required.
[0091] Two-level tree design is primarily motivated by complexity reduction. Theoretically, the complexity of traversing a tree is T D Here, T represents the number of partition types and D is the depth of the tree.
[0092] [Merge mode with motion vector differences (MMVD)]
[0093] In merge mode, implicitly derived motion information is used directly for predicting samples for the current CU. A merge mode with motion vector differences (MMVD) is introduced in VVC. The MMVD flag is transmitted immediately after the skip and merge flags are sent to specify whether MMVD mode is used for the CU.
[0094] In MMVD, after a merge candidate is selected, this may be further refined by signaled motion vector difference (MVD) information. The MVD information may include a merge candidate flag, an index to specify the magnitude of the motion, and an index to indicate the direction of the motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the motion vector (MV) base. A merge candidate flag may be signaled to specify which of the first two candidates is used.
[0095] A distance index may be used, or motion magnitude information may be specified, indicating a predetermined offset from the starting point. Figure 8 shows the MMVD search points of two reference frames according to several embodiments. As shown in Figure 8, the offset may be added to either the horizontal or vertical component of the starting MV. The relationship between the distance index and the predetermined offset is specified in Table 1 below. [Table 1]
[0096] A direction index may be used, which may represent the direction of the MVD relative to the starting point. The direction index may represent one of four directions, as shown in Table 2 below. The meaning of the MVD code may change according to the information of the starting MV. When the starting MV is a single-prediction MV or a bi-prediction MV, and both lists point to the same side of the current picture (i.e., the picture order counts (POCs) of the two references are either greater than or less than the POC of the current picture), the code in Table 2 specifies the code of the MV offset added to the starting MV. If the starting MV is a bipredicted MV, and the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and the difference in POCs in List 0 (L0) is greater than the difference in POCs in List 1 (L1), then the sign in Table 2 specifies the sign of the MV offset added to the L0 MV component of the starting MV, and the sign of the L1 MV has the opposite value. Otherwise, if the difference in POCs in L1 is greater than that of L0, then the sign in Table 2 specifies the sign of the MV offset added to the L1 MV component of the starting MV, and the sign of the L0 MV has the opposite value.
[0097] The MVD may be scaled according to the difference in POCs in each direction. If the difference in POCs in both lists is the same, scaling may not be necessary. Otherwise, if the difference in POCs in L0 is greater than the difference in L1, the MVD of L1 is scaled. If the difference in POCs in L1 is greater than that of L0, the MVD of L0 may be scaled in the same way. If the initial MV is single-predicted, the MVD may be added to the available MVs. [Table 2]
[0098] [Symmetric MVD coding]
[0099] In VVC, in addition to the signal transmission of MVD in the normal unidirectional and bidirectional prediction modes, a symmetric MVD mode may be applied for the signal transmission of bidirectional MVD. In symmetric MVD mode, motion information including the reference picture indices of both L0 and L1 and the MVD of L1 may be derived, and may not be explicitly signaled.
[0100] The decoding process for symmetric MVD mode may also be as follows:
[0101] At the slice level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 may be derived as follows: If mvd_l1_zero_flag is 1, BiDirPredFlag is set to equal 0. Otherwise, if the nearest reference picture in L0 and the nearest reference picture in L1 form a forward-backward or backward-backward reference picture pair, BiDirPredFlag is set to 1, and both the L0 and L1 reference pictures are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0.
[0102] At the CU level, if the CU is bipredictively coded and BiDirPredFlag is equal to 1, a symmetric mode flag indicating whether symmetric mode is used may be explicitly signaled. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 may be explicitly signaled. The reference indices of L0 and L1 may be set to equal the pair of reference pictures, respectively. MVD1 may be set to equal (-MVD0).
[0103] [Intermode coding in CWG-B018]
[0104] In AV1, for each coding block within an interframe, if the current block's mode is intercoding mode and not skip mode, other flags may be signaled to indicate whether single-reference mode or composite-reference mode is used for the current block. In single-reference mode, the prediction block may be generated by a single motion vector. In composite-reference mode, the prediction block may be generated by a weighted average of two prediction blocks derived from two motion vectors.
[0105] For a single reference, the following modes may be transmitted: (a) NEARMV - Uses one of the motion vector predictors (MVPs) in the list indicated by the Dynamic Reference List (DRL) index. (b) NEWMV - Use one of the MVPs in the list signaled by the DRL index as a reference and apply the difference to the MVP. (c)GLOBALMV - Uses motion vectors based on frame-level global motion parameters
[0106] In the case of composite references, the following modes may be transmitted as signals: (a) NEAR_NEARMV - Uses one of the MVPs in the list signaled by the DRL index. (b) NEAR_NEWMV - Uses one of the MVPs in the list signaled by the DRL index as a reference and sends a differential MV for the second MV. (c) NEW_NEARMV - Uses one of the MVPs in the list signaled by the DRL index as a reference and sends a differential MV for the first MV. (d) NEW_NEWMV - Uses one of the MVPs in the list signaled by the DRL index as a reference and sends the differential MV for both MVs. (e)GLOBAL_GLOBALMV - Uses MV from each reference based on frame-level global motion parameters.
[0107] [Motion vector difference coding in AV1]
[0108] AV1 enables motion vector accuracy (or precision) of 1 / 8 pixel, and the following syntax may be used to signal the motion vector difference in reference frame L0 or L1.
[0109] According to the embodiment, the syntax mv_joint specifies which component of the motion vector difference is non-zero. A value of 0 may indicate that no non-zero MVD exists along either the horizontal or vertical direction. A value of 1 may indicate that a non-zero MVD exists along the horizontal direction only. A value of 2 may indicate that a non-zero MVD exists along the vertical direction only. A value of 3 may indicate that a non-zero MVD exists along both the horizontal and vertical directions.
[0110] According to one embodiment, the syntax element mv_sign specifies whether the motion vector difference is positive or negative.
[0111] According to the embodiment, the syntax element mv_class specifies the class of the motion vector difference. As shown in Table 3 below, a higher class means that the motion vector difference has a larger magnitude. [Table 3]
[0112] According to one embodiment, the syntax element mv_bit specifies the integer part of the offset between the motion vector difference and the magnitude of the start of each MV class.
[0113] According to one embodiment, the syntax element mv_fr specifies the first two fractional bits of the motion vector difference.
[0114] According to one embodiment, the syntax mv_hp specifies the third fractional bit of the motion vector difference.
[0115] [Adaptive MVD resolution in CWG-B092]
[0116] For NEW_NEARMV and NEAR_NEWMV modes, the precision of the MVD may depend on the relevant class and the size of the MVD.
[0117] First, fractional MVDs may only be permitted if the size of the MVD is 1 pixel or less.
[0118] Next, when the value of the relevant MV class is MV_CLASS_1 or greater, only one MVD value may be allowed, and the MVD values for each MV class may be derived as 4, 8, 16, 32, and 64 for MV class 1 (MV_CLASS_1), MV class 2 (MV_CLASS_2), MV class 3 (MV_CLASS_3), MV class 4 (MV_CLASS_4), or MV class 5 (MV_CLASS_5).
[0119] The permissible MVD values for each MV class are shown in Table 4 below. [Table 4]
[0120] In some embodiments, if the current block is coded using NEW_NEARMV or NEAR_NEWMV mode, one context may be used to signal mv_joint or mv_class. If the current block is not coded using NEW_NEARMV or NEAR_NEWMV mode, another context may be used to signal mv_joint or mv_class.
[0121] [Joint MVD Coding (JMVD) in CWG-B092]
[0122] A new intercoding mode named JOINT_NEWMV may be applied to indicate whether the MVDs for two reference lists are signaled together. If the inter-prediction mode is equal to the JOINT_NEWMV mode, the MVDs for reference lists L0 and L1 are signaled together. Thus, only one MVD named joint_mvd may be signaled and transmitted to the decoder, and the difference MV for reference lists L0 and L1 may be derived from joint_mvd.
[0123] The JOINT_NEWMV mode may be signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes. No further context is required.
[0124] When the JOINT_NEWMV mode is signaled and the POC distances between two reference frames and the current frame are different, the MVD may be scaled with respect to reference list L0 or reference list L1 based on the POC distance. Specifically, the distance between reference frame list L0 and the current frame is shown as td0, and the distance between reference frame list L1 and the current frame is shown as td1. If td0 is greater than or equal to td1, joint_mvd may be used directly for reference list L0, and the mvd for reference L1 may be derived from joint_mvd based on the following equation (1). derived_mvd=td1 / td0*joint_mvd Formula (1)
[0125] Otherwise, if td1 is greater than or equal to td0, joint_mvd may be used directly for the reference list L1, and mvd for the reference list L0 may be derived from joint_mvd based on the following equation (2). derived_mvd=td0 / td1*joint_mvd Formula (2)
[0126] [Improvements to Adaptive MVD in CWG-C011]
[0127] A new intercoding mode named AMVDMV may be added for single references. When the AMVDMV mode is selected, this may indicate that AMVD is applied to the signal MVD.
[0128] To indicate whether AMVD is applied to a joint MVD coding mode, a flag named amvd_flag may be added under the JOINT_NEWMV mode. When adaptive MVD resolution is applied to a joint MVD coding mode named joint AMVD coding, the MVDs for two reference frames may be signaled together, and the precision of the MVD may be implicitly determined by the size of the MVD. Otherwise, the MVDs for two (or more) reference frames are signaled together, and MVD coding is applied.
[0129] [Adaptive motion vector resolution (AMVR) in CWG-C012 and CWG-C020]
[0130] AMVR was first proposed in CWG-C012 and supports a total of seven MV accuracies (i.e., 8, 4, 2, 1, 1 / 2, 1 / 4, and 1 / 8). For each prediction block, the AVM encoder searches all supported accuracy values and signals the best accuracy to the decoder.
[0131] To reduce encoder execution time, two precision sets may be supported. Each precision set may contain four predetermined precisions. The precision set may be adaptively selected at the frame level based on the maximum precision value of the frame. Similar to AV1, the maximum precision may be signaled in the frame header. Table 5 below summarizes the precision values supported based on the frame-level maximum precision. [Table 5]
[0132] In AVM software (similar to AV1), a frame-level flag may exist to indicate whether the frame's MV includes sub-PEL precision. AMVR may only be enabled if the value of the cur_frame_force_integer_mv flag is 0. In AMVR, if the block precision is lower than the maximum precision, the motion model and interpolation filter may not be transmitted. If the block precision is lower than the maximum precision, the motion mode may be estimated as translational motion, and the interpolation filter may be estimated as a REGULAR interpolation filter. Similarly, if the block precision is either 4 PEL or 8 PEL, the inter-intra mode may not be transmitted and may be estimated as 0.
[0133] [Warp movement mode in AV1]
[0134] Motion compensation typically assumes a translational motion model between a reference block and a target block. However, warped motion utilizes an affine model.
[0135] The affine motion model can be expressed by the following equation (3).
number
[0136] 1. Global warp motion compensation
[0137] In global warp motion compensation, global motion information, including the global motion type and several motion parameters, may be signaled for each interreference frame. The number of global motion types and associated parameters are listed in Table 6 below. [Table 6]
[0138] If a global motion is selected after the reference frame index has been signaled, the global motion type and parameters associated with the given reference frame may now be used for the coding block.
[0139] 2. Local warp motion compensation
[0140] Local warp movement may be permitted for an interconnecting block if the following conditions are met: (1) The current block uses single reference prediction. (2) The width or height of the interconnecting block is 8 or greater. (3) At least one of the adjacent neighboring blocks uses the same reference frame as the current block.
[0141] When local warp motion is used for the current block, the affine model parameters may be estimated by minimizing the mean square of the difference between the reference projection and the modeled projection, based on the MV of the current block and its adjacent neighboring blocks. To estimate the parameters of local warp motion, if the adjacent block uses the same reference frame as the current block, projected sample pairs may be obtained between the central sample in the adjacent block and its corresponding sample in the reference frame. Subsequently, three additional samples may be created by shifting the central position by a quarter sample in one or both dimensions. These additional samples may also be considered projected sample pairs to ensure stability in the model parameter estimation process.
[0142] The motion samples of adjacent blocks used to derive motion parameters may be called motion samples. Motion samples may be selected from adjacent blocks that use the same reference frame as the current block. The warp motion prediction mode may be enabled only for blocks that use a single reference frame. For example, referring to Figure 9, the motion samples of adjacent blocks B0, B1, and B2 are called MV0, MV1, and MV2, respectively. Figure 6 shows exemplary motion samples used to derive model parameters of blocks using local warp motion prediction in several embodiments. Figure 9 shows exemplary motion samples used to derive model parameters of blocks using local warp motion prediction in several embodiments. The current block 600 may be predicted using a single prediction with reference frame Ref0. Adjacent block B0 may be predicted using a composite prediction with reference frames Ref0 and Ref1. Adjacent block B1 may be predicted using a single prediction with reference frame Ref0. Adjacent block B2 may be predicted using a composite prediction with reference frames Ref0 and Ref2. The motion vectors MV0Ref0 of adjacent block B0, MV1Ref0 of adjacent block B1, and MV2Ref0 of adjacent block B2 may now be used as motion samples to derive the affine motion parameters of block 600.
[0143] 3. Sample acquisition process for local warp motion in AV1 and AVM
[0144] For local warp motion, samples and motion vectors from adjacent blocks may be collected in a stack to compute a warp model using regression / least squares error. For AV1 and AVM, the sample positions of one group in a reference picture and their reference positions pointed to by its MV may be collected. Up to eight pairs of positions may be stored in the stack for regression / least squares error warp model computation.
[0145] Figure 10 is a schematic diagram of an example of a warp sample acquisition scan sequence. As shown in Figure 10, only spatially adjacent blocks may be scanned. If an adjacent block has the same reference picture as the current block 610 (for example, if the adjacent block is composite, one of the reference pictures is the same as the reference picture of the current block), the center position of the adjacent block and the center position of the reference block of the adjacent block pointed to by the MV of the adjacent block may be acquired. Referring to Figure 10, the scan sequence may be defined as follows: (1) the adjacent block 611 in the top row is scanned, (2) the adjacent block 612 in the left column is scanned, (3) the upper left adjacent block 613 is scanned, and (4) the upper right adjacent block 614 is scanned.
[0146] [Local warp extension mode and local warp difference mode from CWG-C050]
[0147] In addition to translational motion, AVM also supports warp motion compensation. Two types of warp motion models are supported: a global warp model and a local warp model. The global warp model is associated with each reference frame, with each of the four non-translational parameters having 12-bit precision, and the translational motion vector being coded with 15-bit precision. The coding block may choose to use it directly (a reference frame index is provided). The global warp model captures frame-level scaling and rotation, and therefore primarily focuses on inflexible motion across the entire frame.
[0148] Furthermore, local warp models at the coding block level are also supported. In local warp mode, also known as WARPED_CAUSAL, the warp parameters of the current block may be derived by fitting the model to nearby motion vectors using least squares.
[0149] 1. Warp Expansion Mode
[0150] The new warp movement mode is called WARP_EXTEND. In WARP_EXTEND mode, the movement of adjacent blocks smoothly extends to the current block, but retains some ability to modify warp parameters. This allows for the representation of complex warping movements that spread across multiple blocks while minimizing blocking artifacts.
[0151] To achieve this, the WARP_EXTEND mode applied to the NEWMV block constructs a new warp model based on two constraints: (1) the per-pixel motion vectors generated by the new warp model must be continuum with the per-pixel motion vectors in adjacent blocks, and (2) the pixel at the center of the current block must have a per-pixel motion vector that matches the motion vectors signaled for the block as a whole.
[0152] For example, referring to Figure 11, if the adjacent block 622 to the left of block 620 is currently warped, a warp model that fits the motion vector may be used. Figure 11 shows motion vectors within a block using a warp extension mode according to several embodiments.
[0153] The two constraints above for constructing a new warp model may mean specific equations containing the warp parameters of the adjacent block and the current block. These equations may then be solved to compute the warp model for the current block.
[0154] For example, if (A,...,F) represents adjacent warp models and (A',...,F') represents a new warp model, the first constraint is given by equation (4) below at each point along the common edge.
number
[0155] Points along the edge have different values for y, but they all have the same value for x. This means that the coefficient of y must be the same on both sides (i.e., B'=B and D'=D). On the other hand, the coefficient of x provides two equations with respect to the other coefficients, as shown in equations 5-8 below.
number
[0156] The second constraint specifies that the motion vector of the center of the block must be equal to that transmitted using the NEWMV mechanism. This gives rise to two further equations, resulting in a system of six equations in six variables, each with a unique solution.
[0157] These equations can be efficiently solved in both software and hardware. The solutions can be found using basic addition, subtraction, multiplication, and division by powers of two. Therefore, this mode is not significantly more complex than least-squares-based local warp modes.
[0158] In some embodiments, there may be multiple adjacent blocks that can serve as the source for extension. In such cases, it is useful to have several ways of selecting which block to extend from. This problem is similar to the problems faced in motion vector prediction. In particular, there may be several possible motion vectors from nearby blocks, and one of them must be selected as the base for NEWMV coding.
[0159] A solution to the above problem may include extending the algorithm to handle the need for WARP_EXTEND. This may be done by tracking the source of each motion vector prediction. WARP_EXTEND may then be enabled only if the selected motion vector prediction is taken from a directly adjacent block. That block may then be used as a single "adjacent block" for the rest of the algorithm.
[0160] In some cases, the adjacent warp model will be perfectly fine as is, without requiring any further modifications. To code this more cheaply, WARP_EXTEND may be used in the NEARMV block. The adjacent selection may be the same as in NEWMV, except that the selection in NEWMV requires the adjacent to be warped rather than simply translated. However, if this is true and WARP_EXTEND is selected, the adjacent warp model parameters may now be copied into the block.
[0161] 2. Warp Difference Mode
[0162] In some embodiments, a motion mode called WARP_DELTA may be provided. In this mode, the block's warp model is coded as a difference from the predicted warp model, similar to how the motion vector is coded as a difference from the predicted motion vector. The prediction may be supplied from either the global motion model (if any) or the adjacent block.
[0163] To avoid having multiple ways to encode the same predictive warp model, the following restrictions may apply: If the mode is NEARMV or NEWMV, the same adjacency selection logic described for WARP_EXTEND may be used. If this results in a warped adjacency block, the model of that block (without applying the rest of the WARP_EXTEND logic) may be used as the predictor. Otherwise, the global warp model may be used as the base.
[0164] Next, the difference for each of the non-translational parameters may be coded. Finally, the translational portion of the model may be adjusted so that the motion vector for each pixel at the center of the block matches the overall motion vector of the block.
[0165] This tool (i.e., WARP_DELTA) uses more bits to encode than other warp modes because it involves explicitly coding the difference for each warp parameter. Therefore, WARP_DELTA may be disabled for blocks smaller than 16x16. However, the decoding logic is extremely simple and can represent more complex movements that are not possible with other warp modes.
[0166] [Merge with Motion Vector Difference (MMVD)]
[0167] Merge with Motion Vector Difference (MMVD) may be used for either skip mode or merge mode using a motion vector representation method. Figure 12 shows the MMDV search process for the current frame 630, including the current block 632, using two reference frames 634 and 636 (for example, the two reference frames shown in Figure 8). MMVD reuses merge candidates within the VVC. Candidates are selected from the merge candidates and may be further extended by a motion vector representation method. MMVD provides a new motion vector representation using simple signal transmission. The representation method may include the starting point, the magnitude of the motion, and the direction of the motion. The MMVD technique may use the merge candidate list of the VVC. However, only candidates of the default merge type (MRG_TYPE_DEFAULT_N) may be considered for MMVD extension. A base candidate index (IDX) may define the starting point. The base candidate IDX may indicate the best candidate among the candidates in the list, as shown in Table 7 below. [Table 7]
[0168] If the number of base candidates is equal to 1, base candidate IDX does not need to be signaled.
[0169] A distance index representing the magnitude of the movement may be used. The distance index may indicate a predetermined distance from the starting point information. The predetermined distance may be as shown in Table 8 below. [Table 8]
[0170] A direction index may be used to represent the direction of the MVD relative to the starting point. The direction index may represent four directions, as shown in Table 9 below. [Table 9]
[0171] The MMVD flag may be signaled immediately after the skip and merge flags have been transmitted. If the skip and merge flags are true, the MMVD flag may be parsed. If the MMVD flag is equal to 1, the MMVD syntax may be parsed. However, if it is not 1, the AFFINE flag may be parsed. If the AFFINE flag is equal to 1, the AFFINE mode may be used. However, if it is not 1, the skip / merge index may be parsed for the VTM's skip / merge mode.
[0172] [Challenges in related technologies]
[0173] In related technologies, only a single group of samples is used to compute the warp model, and only spatially adjacent blocks are considered. This is suboptimal, as samples collected from different groups may result in different warp models that are better than those from the group used in related technologies.
[0174] [Example Embodiments]
[0175] The embodiments of this disclosure may be used separately or combined in any order. Furthermore, each of the embodiments of this disclosure (e.g., methods, encoders, and decoders) may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium. Hereinafter, the term “block” may be interpreted as a prediction block, coding block, or coding unit (CU).
[0176] According to one embodiment, time-motion information may be used to derive a better warp model.
[0177] According to one embodiment, the time motion vector (MV) may be used as an addition to or replacement of the spatial MV during warp motion sample acquisition.
[0178] According to one or more embodiments, the time MV may be used in a position among the spatially adjacent blocks of the current block that has not yet been configured. In this case, embodiments of the present disclosure may determine the center position of an adjacent block and its reference position pointed to by the time MV, assuming that the adjacent block size is a time MV storage grid (e.g., 8 × 8). Exemplary time block positions T1 to T5 in which the time MV may be used are shown in Figure 13. Referring to Figure 13, the current block 610, the adjacent block(s) in the row above 611, the adjacent block(s) in the left column 612, the upper left adjacent block 613, the upper right adjacent block 614, the lower left adjacent block 615, and the time block positions T1 to T5 are shown. According to this embodiment, time block position T1 may correspond to the lower right adjacent block, time block position T2 may correspond to the adjacent block immediately above the lower right adjacent block and to the right of the current block 610, time block position T3 may correspond to the adjacent block immediately to the left of the lower right adjacent block and to the right of the current block 610, time block position T4 may correspond to the adjacent block immediately below the upper right adjacent block 614 and to the right of the current block 610, and time block position T5 may correspond to the adjacent block immediately to the right of the lower left adjacent block 615 and to the right of the current block 610. According to this embodiment, the scanning order of time adjacent blocks is not fixed.
[0179] According to one or more embodiments, the time MV may be used at a location where a spatial adjacent block is reconfigured, but this spatial adjacent block does not have a valid MV (for example, this block is coded using intra-mode). Referring to Figure 13, such exemplary locations may correspond to one or more spatial adjacent blocks, such as one or more of the upper row adjacent block(s) 611, the left column adjacent block(s) 612, the upper left adjacent block(s) 613, the upper right adjacent block(s) 614, and the lower left adjacent block(s) 615. In this case, embodiments of the present disclosure may use either the actual block size of the spatial adjacent block or the size of the time MV storage grid (e.g., 8x8) to calculate the center position of the current block(s) in the adjacent block(s) of the current block(s) in the current picture and its reference position pointed to by the time MV. According to embodiments, the scanning order of the time adjacent blocks does not have to be fixed.
[0180] According to one or more embodiments, time MV is used at a position where the current block is at a predefined relative position plus a motion offset (or displacement) that is signaled or derived from the MV used by the adjacent block.
[0181] According to one embodiment, a subset of available adjacent blocks is used in a regression / least squares error method to compute the warp model.
[0182] According to one or more embodiments, referring to Figure 14, currently only the position in the adjacent block 611 above block 610 is used in the warp model calculation. For example, the position (or the motion vector of such a position) may be used in the regression / least squares error method to compute the warp model.
[0183] According to one or more embodiments, referring to Figure 14, only the position in the adjacent block 611 above block 610 and the position in the adjacent block 614 to the upper right are used in the warp model calculation. For example, the position (or the motion vector of such a position) may be used in the regression / least squares error method to calculate the warp model.
[0184] According to one or more embodiments, referring to Figure 14, currently only the positions of the adjacent block 611 above block 610, the adjacent block 614 to the upper right, and the adjacent block 613 to the upper left are used in the warp model calculation. For example, the positions (or motion vectors of such positions) may be used in the regression / least squares error method to calculate the warp model.
[0185] According to one embodiment, a spatial / temporal adjacent block enclosing one or more corners of the current block is used to collect samples and MVs used in warp model calculations.
[0186] According to one or more embodiments, referring to Figure 15, spatially adjacent blocks S and temporally adjacent blocks T that currently enclose corners P1, P2, P3, and P4 of block 610 are used to compute the warp model. For example, a sample / MV of such blocks may be collected and used in the warp model computation.
[0187] According to one or more embodiments, referring to Figure 16, a spatially adjacent block S and a temporally adjacent block T enclosing a plurality of samples (e.g., samples at angles P1, P2, and P4, or samples at angles P1, P3, and P4) are used to compute the warp model. For example, samples / MV of such blocks may be collected and used in the warp model computation.
[0188] According to one embodiment, referring to Figure 16, intermediate spatially adjacent blocks S, or intermediate temporally adjacent blocks T, or subsets of these blocks are used in the warp model calculation. For example, a sample / MV of such blocks may be collected and used in the warp model calculation.
[0189] According to one embodiment, multiple groups of spatial / temporal adjacent blocks are gathered into a group list. This group list may be constructed on both the encoder and decoder sides (e.g., by the encoder and decoder) and may be used to compute a warp model. The list capacity (e.g., the maximum number of groups) may be predefined or signaled by high-level syntax. Each group in the list may be used to compute a different warp model individually. An index indicating which group is used to compute the warp model ultimately used in the configuration (e.g., encoding or decoding the current block) may be signaled by a bitstream. The group may be constructed using temporal adjacent blocks, spatial / temporal adjacent blocks enclosing one or more corners, and spatial / temporal adjacent blocks in intermediate positions.
[0190] In one or more embodiments, the group list may be a fixed group list. Construction rules for the group list may be predefined on both the encoder and decoder sides. For example, the first group of the group list may include or consist of adjacent blocks in the AV1 design as shown in Figure 10, the second group of the group list may include or consist of spatial adjacent blocks S and temporal adjacent blocks T surrounding corners P1, P3, and P4 as shown in Figure 15, the third group of the group list may include or consist of spatial adjacent blocks S and temporal adjacent blocks T surrounding corners P2, P3, and P4, and the fourth group of the group list may include the upper adjacent blocks of the current block (e.g., the upper adjacent blocks 611 and upper right adjacent blocks 614 in Figure 14), and the index of the group that gives the best warp model may be signaled in a bitstream. If not all of the groups contain valid samples used to compute the warp model, the index and warp motion mode indicator may not be signaled. For example, an encoder may not signal the index and warp motion mode indicator in the bitstream based on its determination that none of the groups contain valid samples to be used to compute the warp model, and a decoder may determine that none of the groups contain valid samples to be used to compute the warp model based on its failure to receive the index and warp motion mode indicator in the bitstream.
[0191] According to one or more embodiments, the group list may be a dynamic group list. That is, if, during the construction of the group list, one of the groups does not contain a valid sample for computing the warp model, this group is not inserted into the list. If all given groups have already been examined and the list is still shorter than the maximum number of groups in the group list, the list may be padded with zeros. If the list is full before all given groups have been examined (e.g., the list size is equal to the maximum number of groups), the list construction terminates.
[0192] According to one or more embodiments, during the construction of the group list, a checked group that is identical or similar to a group already in the group list is not inserted. For example, the encoder and / or decoder may compare the checked group to one or more groups already in the group list for similarity, and based on the results of the comparison that satisfy a predetermined similarity criterion, the encoder and / or decoder may not insert the checked group into the group list.
[0193] According to one or more embodiments, during the construction of the group list, no checked groups are inserted that have elements identical or similar to elements already present in the group list. For example, the first group in the group list has two current picture positions (-1, If a group currently being examined has (-1) and (-1, -2), and the group currently being examined consists of picture positions (-1, -2) and (-1, -1), then this group is not inserted. For example, the encoder and / or decoder may compare one or more elements (e.g., picture positions) of the examined group to one or more elements (e.g., picture positions) of groups already in the group list for similarity, and based on the results of a comparison that satisfies a predetermined similarity criterion, the encoder and / or decoder may choose not to insert the examined group into the group list.
[0194] According to one or more embodiments, in addition to the group list construction method described above, extended warp models from adjacent blocks (not limited to the blocks pointed to by the MVP index) can also be used as candidates in the group list. When extended warp models from adjacent blocks are used, the encoder and / or decoder may decide to skip the regression / least squares calculation. Instead, an exemption projection, or simply copying the warp model from the adjacent block, is used. According to embodiments, the adjacent block is not limited to adjacent spatial adjacent blocks, and warp models from temporal adjacent blocks or non-adjacent spatial blocks can also be used.
[0195] According to one embodiment, the collected MV of adjacent blocks (i.e., motion samples) can be improved before calculating the warp model. In one example, an MV smoothing filter may be applied to both the horizontal and vertical components of the collected MV. In another example, motion vector differences (e.g., MMVDs) can be signaled to correct the collected MV. That is, for example, a decoder may correct the collected MV based on motion vector differences that can be signaled in a bitstream by an encoder.
[0196] According to the embodiment, when multiple groups (or sets) of motion samples are available and the selection of a group index is signaled (for example, by an encoder), the groups of motion samples can be sorted by the decoder before the group index is signaled in the encoder, or sorted by the encoder after the group index has been analyzed in the decoder.
[0197] According to one or more embodiments, the groups are sorted based on the regression error when the derived warp model is applied to the moving sample.
[0198] According to one or more embodiments, the groups are sorted based on how many motion samples are derived from a local warp coding block, or based on how many motion samples are derived from an adjacent block coded using a signal-transmitted MVD.
[0199] According to the embodiment, at least one processor and memory for storing computer program instructions (i.e., computer code) may be provided. When executed by at least one processor, the computer program instructions may implement an encoder or decoder and perform any number of functions described in the disclosure, including functions performed by the encoder or decoder.
[0200] For example, referring to Figure 17, at least one processor may implement an encoder 702 or a decoder 704. Computer program instructions may include, for example, an encoding (or decoding) code 710 configured to cause at least one processor to code (or decode) at least one picture received according to embodiments of the present disclosure. The encoding (or decoding) code 710 may include, for example, a motion vector acquisition code 720, a motion sample / warp model acquisition code 730, a warp motion compensation code 740, a motion offset code 750, a list construction code 760, a selection code 770, and an improvement code 780. While exemplary code for performing specific functions is described herein, embodiments of the present disclosure may include other code for performing any other functions described herein.
[0201] The motion vector acquisition code 720 may be configured to cause at least one processor to acquire a plurality of motion vectors pointing to positions in adjacent blocks (e.g., spatial and temporal adjacent blocks) adjacent to the current block, in accordance with embodiments of the present disclosure. The motion vectors may include at least one temporal motion vector.
[0202] The motion sample / warp model acquisition code 730 may be configured to cause at least one processor to acquire motion samples and / or warp models based on motion vectors and / or warp models of adjacent blocks, in accordance with embodiments of the present disclosure.
[0203] The warp motion compensation code 740 may be configured to cause at least one processor to decode the current block by performing warp motion compensation, for example, using a warp model and motion samples, according to embodiments of the present disclosure.
[0204] The motion offset code 750 may be configured to cause at least one processor to apply a motion offset to at least one time-motion vector, in accordance with embodiments of the present disclosure.
[0205] The list construction code 760 may be configured to cause at least one processor to construct a list (e.g., a candidate list) in accordance with embodiments of the present disclosure. The list may include, for example, multiple groups of adjacent blocks, warp models of adjacent blocks, and / or motion vectors (also called motion samples) of multiple groups of adjacent blocks.
[0206] The selection code 770 may be configured to cause at least one processor to select from a list, for example, an entry to be used for predicting the current block, according to embodiments of the present disclosure.
[0207] The improved code 780 may be configured to cause at least one processor to improve the acquired motion vectors before computing a warp model based on the improved motion vectors according to embodiments of the present disclosure.
[0208] When encoder 702 is provided, the computer program instructions may further include transmit code 790 which may be configured to cause at least one processor to transmit, for example, a bitstream containing a coded picture to a decoder (e.g., decoder 704) according to embodiments of the present disclosure. Encoder 702 may provide one or more syntax elements, such as those described herein, in the bitstream according to embodiments of the present disclosure.
[0209] The technology of embodiments of the present invention is implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 18 shows a computer system 900 suitable for implementing embodiments of the subject matter disclosed.
[0210] Computer software can be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms, and can create code that includes instructions that can be executed directly or through interpretation, microcode execution, etc., by a computer's central processing unit (CPU), graphics processing unit (GPU), etc.
[0211] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game devices, and Internet of Things devices.
[0212] The components of computer system 900 shown in Figure 18 are essentially illustrative and are not intended to imply any limitations on the scope or functionality of computer software implementing embodiments of this disclosure. Furthermore, the configuration of the components should not be construed as having any dependence or requirement on any one or combination of components shown in the exemplary embodiments of computer system 900.
[0213] The computer system 900 may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, through tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, applause, etc.), visual input (gestures, etc.), or olfactory input (not shown). The human interface input devices can also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (voices, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), and video (2D video, 3D video including stereoscopic images, etc.).
[0214] The human interface input device may include one or more of the following: keyboard 901, mouse 902, trackpad 903, touch screen 910, data glove, joystick 905, microphone 906, scanner 907, and camera 908 (only one of each is shown).
[0215] The computer system 900 may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touch screen 910, data glove, or joystick 905, although there may also be tactile feedback devices that do not function as input devices). For example, such devices may include audio output devices (speakers 909, headphones (not shown), etc.), visual output devices (screens 910 including CRT screens, LCD screens, plasma screens, OLED screens, etc., with or without touch screen input functionality and with or without tactile feedback functionality, some of which can output two-dimensional or more-than-three-dimensional visual output through means such as stereoscopic image output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).
[0216] The computer system 900 may also include human-accessible storage devices and their associated media, such as optical media or similar media 921 including CD / DVD ROM / RW 920 having CD / DVD, thumb drives 922, removable hard drives or solid-state drives 93, legacy magnetic media such as tapes and floppy disks (not shown), and special ROM / ASIC / PLD-based devices such as security dongles (not shown).
[0217] Those skilled in the art should also understand that, when used in connection with the subject matter currently disclosed, the term “computer-readable medium” does not encompass a transmission medium, carrier wave, or other transient signal.
[0218] The computer system 900 may also include interfaces to one or more communication networks. These networks can be, for example, wireless, wired, or optical. Furthermore, these networks can be local, wide-area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet® and wireless LAN; cellular networks including GSM®, 3G, 4G, 5G, LTE, etc.; wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial TV; and vehicle and industrial networks including CANBus, etc. Certain networks generally require an external network interface adapter (e.g., a USB port on the computer system 900) attached to a specific general-purpose data port or peripheral bus 949, while others are generally integrated into the core of the computer system 900 by attachment to a system bus (e.g., an Ethernet® interface to a PC computer system or a cellular network interface to a smartphone computer system), as described later. Using any of these networks, the computer system 900 can communicate with other entities. Such communications can be unidirectional, receive-only (e.g., broadcast TV), unidirectional, transmit-only (e.g., from a specific CANbus to a specific CANbus device), or bidirectional, for example, using a local or wide-area digital network to other computer systems. Such communications may include communications to a cloud computing environment 955. As described above, specific protocols and protocol stacks can be used in each of these networks and network interfaces.
[0219] The above-mentioned human interface device, human-accessible storage device, and network interface 954 can be mounted on the core 940 of the computer system 900.
[0220] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) 943, hardware accelerators 944 for specific tasks, etc. These devices may be connected via a system bus 948 along with read-only memory (ROM) 945, random access memory (RAM) 946, internal non-user-accessible hard drives, SSDs, or other internal mass storage 947. In some computer systems, the system bus 948 is accessible in the form of one or more physical plugs to allow for expansion with further CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus 948 or via a peripheral bus 949. The architecture of the peripheral bus includes PCI, USB, etc. A graphics adapter 950 may be included in the core 940.
[0221] The CPU 941, GPU 942, FPGA 943, and accelerator 944 can execute specific instructions that, when combined, constitute the computer code described above. This computer code can be stored in ROM 945 or RAM 946. Temporary data can be stored in RAM 946, while permanent data can be stored, for example, in internal mass storage 947. High-speed storage and retrieval of any of the memory devices can be enabled through the use of cache memory that can be closely associated with one or more CPUs 941, GPUs 942, mass storage 947, ROM 945, RAM 946, etc.
[0222] A computer-readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.
[0223] As an example, and not an limitation, a computer system 900 having an architecture and in particular a core 940 can provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of the core 940 of a non-transient nature, such as core internal mass storage 947 or ROM 945. Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core 940. The computer-readable media can include one or more memory devices or chips according to specific needs. The software can cause the core 940 and in particular the processor (including CPU, GPU, FPGA, etc.) within it to execute specific processes or specific parts of specific processes as described herein, including defining data structures to be stored in RAM 946 and modifying such data structures according to processes defined by the software. Furthermore, or alternatively, a computer system may provide functionality as a result of being embodied in logic hardwired or otherwise in circuitry (e.g., in accelerator 944), which may operate in place of or in conjunction with software to execute specific processes or specific parts of specific processes described herein. References to software include logic, and vice versa, as appropriate. References to computer-readable media may include circuitry that stores software for execution (e.g., integrated circuits (ICs)), circuitry that embodies logic for execution, or both, where appropriate. This disclosure encompasses any appropriate combination of hardware and software.
[0224] While this disclosure describes several non-limiting exemplary embodiments, there are various modifications, substitutions, and equivalent alternatives that fall within the scope of this disclosure. Therefore, those skilled in the art will understand that various systems and methods, not expressly shown or described herein, embody the principles of this disclosure and thus fall within the spirit and scope of this disclosure.
Claims
[Claim 1] A method performed by the decoder, The steps include receiving a bitstream containing a coding picture, A step of obtaining a plurality of motion vectors that point to positions in adjacent blocks adjacent to the current block in the coding picture, wherein the plurality of motion vectors include at least one time motion vector. A step of obtaining motion samples for a warp model based on the aforementioned plurality of motion vectors, The steps of decoding the current block by using the warp model and the motion sample, which includes the at least one time motion vector, for prediction; A method that includes this.
