Method for video coding, computing system, and computer program

Abstract

To provide a method and system for coding a current coding block.SOLUTION: A method includes: retrieving one or more motion vector (MV) candidates from a reference MV bank; in accordance with a determination that a first condition is satisfied, inserting the one or more MV candidates from the reference MV bank into an MVP list associated with a current coding block after derived MV candidates are inserted into the MVP list; in accordance with a determination that the first condition is not satisfied, inserting the one or more MV candidates from the reference MV bank into the MVP list associated with the current coding block before the derived MV candidates are inserted into the MVP list; and predicting the current coding block based on the MVP list.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] This application is a continuation of and claims priority to U.S. Patent Application No. 18 / 196,368, filed May 11, 2023, entitled "Systems and Methods for Motion Vector Predictor List Improvements," and also claims priority to U.S. Provisional Patent Application No. 63 / 349,754, filed June 7, 2022, entitled "Motion Vector Predictor List Improvements," which is incorporated herein by reference in its entirety.

[0002] The disclosed embodiments relate generally to video coding and include, but are not limited to, systems and methods for motion vector predictor list refinement. [Background technology]

[0003] Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video teleconferencing equipment, and video streaming devices. These electronic devices transmit, receive, or otherwise communicate digital video data across communication networks and store the digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video coding may be used to compress the video data according to one or more video coding standards before the video data is communicated or stored.

[0004] Several video codec standards have been developed. For example, video coding standards include AOMedia Video 1 (AV1), Versatile Video Coding (VVC), Joint Search Model (JEM), High Efficiency Video Coding (HEVC / H.265), Advanced Video Coding (AVC / H.264), and Moving Picture Experts Group (MPEG) coding. Video coding generally uses prediction methods (e.g., inter-prediction, intra-prediction, or the like) that exploit the redundancy inherent in video data. Video coding aims to compress video data into a form that uses a lower bitrate while avoiding or minimizing degradation to video quality.

[0005] HEVC, also known as H.265, is a video compression standard designed as part of the MPEG-H project. ITU-T and ISO / IEC published the HEVC / H.265 standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). Versatile Video Coding (VVC), also known as H.266, is a video compression standard intended as the successor to HEVC. ITU-T and ISO / IEC published the VVC / H.266 standard in 2020 (version 1) and 2022 (version 2). AV1 is an open video coding format designed as a replacement for HEVC. Certified version 1.0.0 with specification errata was released on January 8, 2019. Summary of the Invention

[0006] This disclosure describes an improved method and system for decoding video data based on a motion vector prediction (MVP) list. In video codec standards, the result of motion vector prediction for a block being coded or decoded is a list of motion vectors. The main idea behind forming this list is that the motion vector for a current block has a high probability of being similar to the motion vectors for previously coded neighboring blocks, and therefore can be used as a prediction. Currently, the process of ordering motion vector (MVP) candidates in the MVP list is suboptimal because some MV candidates that are more important (e.g., that produce more accurate predictions) are listed at later positions. In addition, a predetermined weight-based sorting process is used to sort only neighboring spatial MVPs (SMVPs), temporal MVPs (TMVPs), and non-neighboring SMVPs, which is suboptimal for other MV candidates.

[0007] Therefore, improved methods and systems for constructing MVP lists for encoding and decoding video data are desired. For example, because spatial MV candidates are typically more accurate than combined (e.g., scaled) candidates and padded candidates (e.g., zero MVs), according to some aspects of the present disclosure, when one or more conditions are met, candidates from a reference MV candidate bank are inserted into an MVP list before other MV candidates. Because MV candidates from the reference MV candidate bank are primarily from spatial neighbors, the present disclosure generates improved MVP lists with more accurate motion vector predictions, thereby resulting in increased coding efficiency.

[0008] According to some embodiments, a method for constructing a decoding target for a current coding block is performed in a computing system having one or more processors and a memory. The method includes retrieving one or more motion vector (MV) candidates from a reference MV bank. The method includes, in accordance with a determination that a first condition is satisfied, inserting the one or more MV candidates from the reference MV bank into a motion vector predictor (MVP) list associated with the current coding block after the derived MV candidates are inserted into the MVP list. The method includes, in accordance with a determination that the first condition is not satisfied, inserting the one or more MV candidates from the reference MV bank into an MVP list associated with the current coding block before the derived MV candidates are inserted into the MVP list. The method includes predicting the current coding block based on the MVP list.

[0009] According to some embodiments, a computing system, such as a streaming system, server system, personal computer system, or other electronic device, is provided. The computing system includes control circuitry and memory storing one or more instruction sets. The one or more instruction sets include instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and / or a decoder component.

[0010] According to some embodiments, a non-transitory computer-readable storage medium is provided that stores one or more instruction sets for execution by a computing system, the one or more instruction sets including instructions for performing any of the methods described herein.

[0011] Accordingly, methods, as well as apparatus and systems, for encoding and decoding video are disclosed that can complement or replace conventional methods, apparatus and systems for video encoding / decoding.

[0012] The features and advantages described herein are not necessarily all-inclusive, and some additional features and advantages will become apparent to those skilled in the art, particularly in light of the drawings, specification, and claims provided in this disclosure. It should also be noted that the language used herein has been chosen primarily for ease of reading and instructional purposes, and not necessarily to delineate or bound the subject matter described herein. [Brief explanation of the drawings]

[0013] In order that the present disclosure may be understood in more detail, a more particular description will be made by reference to features of various embodiments, some of which are illustrated in the accompanying drawings. However, the accompanying drawings are merely illustrative of relevant features of the present disclosure and therefore should not be considered as necessarily limiting, and for purposes of explanation may recognize other useful features as those skilled in the art will read and understand this disclosure. [Figure 1] FIG. 1 is a block diagram illustrating an example of a communication system according to some embodiments. [Figure 2A] FIG. 2 is a block diagram illustrating example elements of an encoder component according to some embodiments. [Figure 2B] FIG. 2 is a block diagram illustrating example elements of a decoder component according to some embodiments. [Figure 3] FIG. 1 is a block diagram illustrating an example of a server system according to some embodiments. [Figure 4] 1 illustrates block partitioning according to some embodiments. [Figure 5A] 5A and 5B illustrate a quadtree (QT) plus binary tree (QTBT) structure according to some embodiments. [Figure 5B] 5A and 5B illustrate a quadtree (QT) plus binary tree (QTBT) structure according to some embodiments. [Figure 6A] 6A and 6B illustrate ternary tree (TT) partitioning according to some embodiments. [Figure 6B] 6A and 6B illustrate ternary tree (TT) partitioning according to some embodiments. [Figure 7] 1 illustrates example search points for merge mode with motion vector differential (MMVD), according to some embodiments. [Figure 8] 1 illustrates using spatially neighboring blocks for MV prediction according to some embodiments. [Figure 9] 1 illustrates motion field estimation by linear projection, according to some embodiments; [Figure 10] 10 illustrates the use of block position to derive a temporal MV predictor, according to some embodiments. [Figure 11] 10 illustrates additional motion vector candidate generation for a single inter-predicted block according to some embodiments. [Figure 12] 10 illustrates additional motion vector candidate generation for a composite prediction block according to some embodiments. [Figure 13] 10 illustrates a reference MV candidate bank update process according to some embodiments. [Figure 14] 1 illustrates a process for building an MVP list, according to some embodiments. [Figure 15] 1 is a flow diagram of a method for decoding a current coding block according to some embodiments.

[0014] According to common practice, the various features illustrated in the drawings are not necessarily drawn to scale, and like reference numerals may be used to refer to like features throughout the specification and drawings. DETAILED DESCRIPTION OF THE INVENTION

[0015] This disclosure describes decoding a current coding block based on a motion vector prediction (MVP) list. One or more motion vector (MV) candidates are retrieved from a reference MV bank. When a first condition is met, the MV candidates from the reference MV bank are inserted into an MVP list associated with the current coding block after the derived MV candidates are inserted into the MVP list. When the first condition is not met, the MV candidates from the reference MV bank are inserted into an MVP list associated with the current coding block before the derived MV candidates are inserted into the MVP list. The current coding block is predicted based on the MVP list. The derived MV candidates are MV predictors derived from either one or more motion vectors in a single reference frame based on inter prediction or one or more composite motion vectors from multiple adjacent reference frames in a hybrid inter prediction mode.

[0016] Examples of systems and equipment 1 is a block diagram illustrating a communication system 100 according to some embodiments. Communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic device 120-1 through electronic device 120-m) communicatively coupled to each other via one or more networks. In some embodiments, communication system 100 is a streaming system for use with video-enabled applications, such as, for example, video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0017] Source device 102 includes a video source 104 (e.g., a camera component or media storage) and an encoder component 106. In some embodiments, video source 104 is a digital camera (e.g., configured to create an uncompressed video sample stream). Encoder component 106 generates one or more encoded video bitstreams from the video stream. The video stream from video source 104 may be a larger amount of data than encoded video bitstream 108 generated by encoder component 106. Because encoded video bitstream 108 is a smaller amount of data (less data) than the video stream from the video source, encoded video bitstream 108 requires less bandwidth to transmit and less storage space to store than the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video data to network(s) 110).

[0018] One or more networks 110 represent any number of networks that carry information between source device 102, server system 112, and / or electronic device 120, including, for example, wireline and / or wireless communication networks. One or more networks 110 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet.

[0019] One or more networks 110 include a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, server system 112 is or includes a streaming server (e.g., configured to store and / or distribute video content, such as encoded video streams from source devices 102). Server system 112 includes a coder component 114 (e.g., configured to encode and / or decode video data). In some embodiments, coder component 114 includes an encoder component and / or a decoder component. In various embodiments, coder component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, coder component 114 is configured to decode encoded video bitstream 108 and re-encode the video data using a different encoding standard and / or methodology to generate encoded video data 116. In some embodiments, server system 112 is configured to generate multiple video formats and / or encodings from encoded video bitstream 108.

[0020] In some embodiments, server system 112 functions as a Media-Aware Network Element (MANE). For example, server system 112 may be configured to prune encoded video bitstream 108 to tailor potentially different bitstreams to one or more of electronic devices 120. In some embodiments, a MANE is provided separate from server system 112.

[0021] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, decoder component 122 is configured to decode encoded video data 116 to generate an output video stream that can be rendered on a display or other type of rendering device. In some embodiments, one or more of electronic devices 120 does not include a display component (e.g., is communicatively coupled to an external display device and / or includes media storage). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0022] Source device and / or electronic devices 120 may be referred to as “terminal devices” or “user devices.” In some embodiments, source device 102 and / or one or more of electronic devices 120 are instances of a server system, a personal computer, a portable device (e.g., a smartphone, tablet, or laptop), a wearable device, a videoconferencing device, and / or other types of electronic devices.

[0023] In an example operation of communication system 100, source device 102 transmits encoded video bitstream 108 to server system 112. For example, source device 102 may code a stream of pictures captured by the source device. Server system 112 may receive encoded video bitstream 108 and decode and / or encode encoded video bitstream 108 using coder component 114. For example, server system 112 may apply more optimal coding to the video data for network transmission and / or storage. Server system 112 may transmit encoded video data 116 (e.g., one or more encoded video bitstreams) to one or more of electronic devices 120. Each electronic device 120 may decode encoded video data 116 to recover the video pictures and, optionally, display them.

[0024] In some embodiments, the transmission is a one-way data transmission. One-way data transmission may be used in media serving applications and the like. In some embodiments, the transmission is a two-way data transmission. Two-way data transmission may be used in video conferencing applications and the like. In some embodiments, the coded video bitstream 108 and / or the coded video data 116 are encoded and / or decoded according to any of the video coding / compression standards described herein, such as HEVC, VVC, and / or AV1.

[0025] 2A is a block diagram illustrating example elements of the encoder component 106 according to some embodiments. The encoder component 106 receives a source video sequence from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., a transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives a video sequence from a remote video source (e.g., a video source that is a component of a different device than the encoder component 106). The video source 104 may provide the source video sequence in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 Y CrCB, or RGB), and any suitable sampling structure (e.g., Y CrCb 4:2:0, or Y CrCb 4:4:4). In some embodiments, the video source 104 is a storage device that stores pre-captured / prepared video. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. Video data may be provided as multiple individual pictures that, when viewed in sequence, convey motion. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc. used. Those skilled in the art will readily understand the relationship between pixels and samples. The following discussion focuses on samples.

[0026] The encoder component 106 is configured to encode and / or compress pictures of a source video sequence into an encoded video sequence (643) in real time or under other time constraints required by the application. Enforcing an appropriate encoding rate is one function of the controller 204. In some embodiments, the controller 204 controls and is recursively coupled to other functional units, as described below. Parameters set by the controller 204 may include rate control-related parameters (picture skip, quantizer, and / or lambda values ​​for rate-distortion optimization techniques), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can readily identify other functions of the controller 204, as they may be relevant to optimizing the encoder component 106 for a particular system design.

[0027] In some embodiments, the encoder component 106 is configured to operate in a coding loop. In a simplified example, the coding loop includes a source coder 202 (e.g., responsible for creating symbols, e.g., a symbol stream, based on an input picture to be coded and one or more reference pictures) and a (local) decoder 210. The decoder 210 reconstructs the symbols to create sample data in a manner similar to a (remote) decoder (when the compression between the symbols and the coded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory 208. Because decoding the symbol stream yields bit-accurate results independent of the decoder location (local or remote), the contents in the reference picture memory 208 are also bit-accurate between the local and remote encoders. Thus, the prediction portion of the encoder interprets the same sample values ​​as reference picture samples as the decoder interprets them when using prediction during decoding. This principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is known to those skilled in the art.

[0028] The operation of decoder 210 may be the same as that of a remote decoder, such as decoder component 122, which is described in more detail below in connection with Figure 2B. However, briefly referring to Figure 2B, because symbols are available and the encoding / decoding of symbols into a coded video sequence by entropy coder 214 and parser 254 may be lossless, the entropy decoding portion of decoder component 122, including buffer memory 252 and parser 254, may not be fully implemented in local decoder 210.

[0029] It can be noted at this point that decoder technology, with the exception of parsing / entropy decoding, present in the decoder necessarily must also be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosure focuses on decoder operation. A description of the encoder technology can be omitted, as it is the reverse of the decoder technology, which is described in detail below. Only in certain areas is further detail required, which is provided below.

[0030] As part of its operation, source coder 202 may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from a video sequence designated as reference frames. Thus, coding engine 212 codes differences between pixel blocks of the input frame and pixel blocks of one or more reference frames that may be selected as prediction reference(s) for the input frame. Controller 204 may manage the coding operations of source coder 202, including, for example, setting parameters and subgroup parameters used to encode the video data.

[0031] The decoder 210 decodes the encoded video data of frames that may be designated as reference frames based on symbols created by the source coder 202. The operation of the coding engine 212 may advantageously be a lossy process. When the encoded video data is decoded by a video decoder (not shown in FIG. 2A ), the reconstructed video sequence may be a replica of the source video sequence, with some error. The decoder 210 may replicate the decoding process that may be performed by a remote video decoder on the reference frames, causing the reconstructed reference frames to be stored in the reference picture memory 208. In this way, the encoder component 106 locally stores copies of reconstructed reference frames that have content in common with the reconstructed reference frames that will be obtained by the remote video decoder.

[0032] The predictor 206 may perform a predictive search for the coding engine 212. That is, for a new frame to be coded, the predictor 206 may search the reference picture memory 208 for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors or block shapes, that can serve as suitable predictive references for the new picture. The predictor 206 may operate on a pixel block-by-pixel block basis to find suitable predictive references. In some cases, as determined by the search results obtained by the predictor 206, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory 208.

[0033] The outputs of all the aforementioned functional units may be subjected to entropy coding in entropy coder 214. Entropy coder 214 converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable length coding, and / or arithmetic coding).

[0034] In some embodiments, the output of the entropy coder 214 is coupled to a transmitter. The transmitter may be configured to buffer the coded video sequence(s) generated by the entropy coder 214 and prepare them for transmission over a communication channel 218. The communication channel 218 may be a hardware / software link to a storage device that stores the coded video data. The transmitter may be configured to merge the coded video data from the source coder 202 with other data to be transmitted, such as coded audio data and / or ancillary data streams (sources not shown). In some embodiments, the transmitter may transmit additional data along with the coded video. The source coder 202 may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, and the like.

[0035] The controller 204 may manage the operation of the encoder component 106. During coding, the controller 204 may assign each coded picture a specific coded picture type, which may affect the coding technique applied to the respective picture. For example, a picture may be assigned as an intra picture (I picture), a predicted picture (P picture), or a bidirectionally predicted picture (B picture). An intra picture may be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow several different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will be aware of these variants of I pictures and their respective uses and characteristics, and therefore will not be repeated here. A predicted picture may be coded and decoded using intra prediction or inter prediction, using at most one motion vector and reference index to predict the sample values ​​of each block. Bidirectionally predicted pictures can be coded and decoded using intra- or inter-prediction, using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0036] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to their respective pictures. For example, blocks of an I-picture may be coded non-predictively, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be coded non-predictively, via spatial prediction, or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be coded non-predictively, via spatial prediction, or via temporal prediction with reference to one or two previously coded reference pictures.

[0037] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) uses spatial correlation within a given picture, while inter-picture prediction uses correlation (temporal or other) between pictures. In one example, a particular picture being coded / decoded, called the current picture, is partitioned into multiple blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture may be coded by a vector called a motion vector. A motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0038] Encoder component 106 may perform coding operations according to a given video coding technique or standard, such as any described herein. In doing so, encoder component 106 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to a syntax defined by the video coding technique or standard used.

[0039] 2B is a block diagram illustrating example elements of decoder component 122 according to some embodiments. Decoder component 122 of FIG. 2B is coupled to channel 218 and display 124. In some embodiments, decoder component 122 includes a transmitter coupled to loop filter 256 and configured to transmit data to display 124 (e.g., via a wired or wireless connection).

[0040] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more coded video sequences to be decoded by decoder component 122. In some embodiments, the decoding of each coded video sequence is independent of the other coded video sequences. Each coded video sequence may be received from channel 218, which may be a hardware / software link to a storage device that stores the coded video data. The receiver may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams. These other data may be forwarded to their respective using entities (not shown). The receiver may separate the coded video sequence from the other data. In some embodiments, the receiver receives additional (redundant) data along with the coded video. The additional data may be included as part of the coded video sequence(s). The additional data may be used by decoder component 122 to decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, and the like.

[0041] According to some embodiments, the decoder component 122 includes a buffer memory 252, a parser 254 (sometimes referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra-picture prediction unit 262, a motion compensated prediction unit 260, an aggregator 268, a loop filter unit 256, a reference picture memory 266, and a current picture memory 264. In some embodiments, the decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuitry. In some embodiments, the decoder component 122 is implemented at least partially in software.

[0042] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to address network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, in addition to buffer memory 252 internal to decoder component 122 (e.g., configured to handle playback timing), a separate buffer memory is provided external to decoder component 122 (e.g., to address network jitter). When receiving data from a store / forward device of sufficient bandwidth and controllability or from an isosynchronous network, buffer memory 252 may not be required or can be small. For use over a best-effort packet network, such as the Internet, buffer memory 252 may be required and can be relatively large, advantageously of an adaptable size, and implemented, at least in part, in an operating system or similar element (not shown) external to decoder component 122.

[0043] Parser 254 is configured to reconstruct symbols 270 from the coded video sequence. The symbols may include, for example, information used to manage the operation of decoder component 122 and / or information to control a rendering device, such as display 124. The control information for the rendering device(s) may be in the form of, for example, a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). Parser 254 parses (entropy decodes) the coded video sequence. The coding of the coded video sequence may be according to a video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context dependency, etc. Parser 254 may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. Parser 254 may also extract information from the coded video sequence information, such as transform coefficients, quantization parameter values, motion vectors, etc.

[0044] The reconstruction of symbols 270 may involve several different units, depending on the type of video picture or portion thereof being coded and other factors (e.g., inter-picture and intra-picture, inter-block and intra-block, etc.). Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by parser 254. The flow of such subgroup control information between parser 254 and the following units is not shown for clarity.

[0045] Beyond the functional blocks already described, the decoder component 122 can be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating within commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, we will maintain the conceptual division into the following functional units:

[0046] The scaler / inverse transform unit 258 receives the quantized transform coefficients and control information (e.g., which transform to use, block size, quantization coefficients, and / or quantization scaling matrix) as symbol(s) 270 from the parser 254. The scaler / inverse transform unit 258 may output blocks containing sample values ​​that may be input to the aggregator 268.

[0047] In some cases, the output samples of the scaler / inverse transform unit 258 relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit 262. The intra-picture prediction unit 262 may generate blocks of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from the current (partially reconstructed) picture from the current picture memory 264. The aggregator 268 may add, on a sample-by-sample basis, the prediction information generated by the intra-picture prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258.

[0048] In other cases, the output samples of the scaler / inverse transform unit 258 relate to a possibly inter-coded, motion-compensated block. In such cases, the motion-compensated prediction unit 260 may access the reference picture memory 266 to fetch samples used for prediction. After motion-compensating the fetched samples according to symbols 270 related to the block, these samples may be appended by the aggregator 268 to the output of the scaler / inverse transform unit 258 (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory 266 from which the motion-compensated prediction unit 260 fetches the prediction samples may be controlled by a motion vector. The motion vector may be available to the motion-compensated prediction unit 260 in the form of symbols 270, which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory 266 when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.

[0049] The output samples of aggregator 268 may be subjected to various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filter techniques, which are controlled by parameters included in the coded video bitstream and made available to loop filter unit 256 as symbols 270 from parser 254, but may also be responsive to meta-information obtained during decoding of previous portions of the coded picture or coded video sequence (in decoding order), as well as to previously reconstructed and loop-filtered sample values.

[0050] The output of the loop filter unit 256 may be a sample stream that can be output to a rendering device, such as the display 124, which may also be stored in the reference picture memory 266 for use in future inter-picture prediction.

[0051] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 254), the current reference picture can become part of reference picture memory 266, and a new current picture memory can be reallocated before starting reconstruction of the next coded picture.

[0052] Decoder component 122 may perform decoding operations according to a given video compression technology, which may be documented in a standard, such as any of the standards described herein. The coded video sequence may conform to the syntax specified by the video compression technology or standard used, in the sense of adhering to the syntax of the video compression technology or standard as specified in the video compression technology document or standard, and specifically in the profile document therein. Also, for compliance with some video compression technologies or standards, the complexity of the coded video sequence may be within a range specified by the level of the video compression technology or standard. In some cases, the level constrains the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further constrained through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled within the coded video sequence.

[0053] 3 is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes a control circuit 302, one or more network interfaces 304, a memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuit 302 includes one or more processors (e.g., a CPU, a GPU, and / or a DPU). In some embodiments, the control circuit includes one or more field programmable gate arrays (FPGAs), hardware accelerators, and / or one or more integrated circuits (e.g., application specific integrated circuits).

[0054] The network interface(s) 304 may be configured to interface with one or more communications networks (e.g., wireless, wired, and / or optical networks). Communications networks may be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, etc. Examples of communications networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, and vehicular and industrial networks including CANbus. Such communications may be unidirectional receive only (e.g., broadcast TV), unidirectional transmit only (e.g., CANbus to a specific CANbus device), or bidirectional (e.g., to another computer system using a local or wide-area digital network). Such communications may include communications to one or more cloud computing networks.

[0055] The user interface 306 includes one or more output devices 308 and / or one or more input devices 310. The input device(s) 310 may include one or more of a keyboard, a mouse, a trackpad, a touchscreen, a data glove, a joystick, a microphone, a scanner, a camera, or the like. The output device(s) 308 may include one or more of an audio output device (e.g., a speaker), a visual output device (e.g., a display or monitor), or the like.

[0056] Memory 314 may include high-speed random-access memory (e.g., DRAM, SRAM, DDR RAM, and / or other random-access solid-state memory devices, etc.) and / or non-volatile memory (e.g., one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices, etc.). Memory 314 optionally includes one or more storage devices located remotely from control circuitry 302. Memory 314, or alternatively, a non-volatile solid-state memory device within memory 314, comprises a non-transitory computer-readable storage medium. In some embodiments, memory 314, or the non-transitory computer-readable storage medium of memory 314, stores the following programs, modules, instructions, and data structures, or a subset or superset thereof: · an operating system 316 that includes procedures for handling various basic system services and for performing hardware-dependent tasks; · a network communications module 318 used to connect the server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections); A coding module 320 that performs various functions related to encoding and / or decoding data, such as video data. In some embodiments, the coding module 320 is an instance of the coder component 114. The coding module 320 may include, but is not limited to, one or more of the following: a decoding module 322 that performs various functions related to decoding the encoded data, such as those described above with respect to the decoder component 122; and an encoding module 340 that performs various functions related to encoding data, such as those described above with respect to the encoder component 106; and A picture memory 352 that stores pictures and picture data, e.g., for use with coding module 320. In some embodiments, picture memory 352 includes one or more of reference picture memory 208, buffer memory 252, current picture memory 264, and reference picture memory 266.

[0057] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions described above with respect to the parser 254), a transform module 326 (e.g., configured to perform various functions described above with respect to the scaler / inverse transform unit 258), a prediction module 328 (e.g., configured to perform various functions described above with respect to the motion compensation prediction unit 260 and / or the intra-picture prediction unit 262), and a filter module 330 (e.g., configured to perform various functions described above with respect to the loop filter 256).

[0058] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described with respect to the source coder 202 and / or coding engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 include a subset of the modules shown in Figure 3. For example, a shared prediction module is used by both the decoding module 322 and the encoding module 340.

[0059] Each of the above-identified modules stored in memory 314 corresponds to an instruction set for performing the functions described herein. The above-identified modules (e.g., instruction sets) need not be implemented as separate software programs, procedures, or modules; thus, in various embodiments, various subsets of these modules may be combined or otherwise rearranged. For example, coding module 320 optionally does not include separate decoding and encoding modules, but instead uses the same set of modules to perform both sets of functionality. In some embodiments, memory 314 stores a subset of the above-identified modules and data structures. In some embodiments, memory 314 stores additional modules and data structures not described above, such as an audio processing module.

[0060] In some embodiments, server system 112 includes a web or Hypertext Transfer Protocol (HTTP) server, a File Transfer Protocol (FTP) server, and web pages and applications implemented using Common Gateway Interface (CGI) scripts, the PHP Hypertext Preprocessor (PHP), Active Server Pages (ASP), Hypertext Markup Language (HTML), Extensible Markup Language (XML), Java, JavaScript, Asynchronous JavaScript+XML (AJAX), XHP, Javelin, Wireless Universal Resource Files (WURFL), and the like.

[0061] While FIG. 3 illustrates a server system 112 according to some embodiments, FIG. 3 is intended as a functional illustration of various features that may be present in one or more server systems, rather than a structural schematic of the embodiments described herein. In practice, items shown separately may be combined, and some items may be separated, as will be recognized by those skilled in the art. For example, some items shown separately in FIG. 3 may be implemented on a single server, and single items may be implemented by one or more servers. The actual number of servers used to implement server system 112, and how features are allocated among them, will vary from implementation to implementation and, optionally, depend in part, on the amount of data traffic the server system handles during peak and average usage periods.

[0062] Block Partitioning

[0063] Figure 4 illustrates block partitioning according to some embodiments. The top half of Figure 4 illustrates block partitioning in VP9, ​​which uses a 4-way partition tree starting from the 64x64 level down to the 4x4 level, with some additional constraints for blocks 8x8 and below. Partitions designated as "R" are referred to as recursive in that the same partition tree is repeated at lower scales until the lowest 4x4 level is reached. The bottom half of Figure 4 illustrates block partitioning in AV1, which not only extends the partition tree to a 10-way structure but also increases the maximum size (called a superblock in VP9 / AV1 terminology) to start at 128x128. The 10-way structure includes 4:1 / 1:4 rectangular partitions, none of which can be further subdivided. AV1 adds more flexibility to the use of partitions below the 8x8 level. For example, 2x2 chroma inter prediction is possible in certain cases.

[0064] In some embodiments, block partitioning involves dividing coding tree units (CTUs) into coding units (CUs) by using a quadtree structure called a coding tree to adapt to various local characteristics. The decision of whether a picture area should be coded using inter-picture (temporal) prediction or intra-picture (spatial) prediction is made at the CU level. Each CU can be further divided into one, two, or four prediction units (PUs) according to the PU partition type. The same prediction process is applied within a PU, and related information is transmitted to the decoder on a PU-by-PU basis. After obtaining a residual block by applying a prediction process based on the PU partition type, the CU can be partitioned into transform units (TUs) according to another quadtree structure, such as a coding tree for the CU. In some embodiments, a block is partitioned using multiple partition concepts, including CUs, PUs, and TUs. In some embodiments, a CU or TU is only square in shape, while a PU can be square or rectangular in shape for inter-predicted blocks. In some embodiments, a coding block may be further divided into four square sub-blocks, and a transform is performed on each sub-block, i.e., TU. Each TU may be further recursively divided (using quadtree partitioning) into smaller TUs, which are called residual quadtrees (RQTs).

[0065] In some embodiments, block partitioning involves using a quadtree (QT) + binary tree (BT) structure (also known as a QTBT structure). The QTBT structure eliminates the concept of multiple partition types; that is, QTBT eliminates the separation of the concepts of CU, PU, ​​and TU, and supports more flexibility for CU partition shapes. In the QTBT block structure, CUs can have either square or rectangular shapes. This is shown in FIG. 5A. Coding tree units (CTUs) are first partitioned using a quadtree structure. The quadtree leaf nodes are further partitioned using a binary tree structure. There are two types of binary tree partitioning: symmetric horizontal partitioning and symmetric vertical partitioning. The binary tree leaf nodes are called coding units (CUs), and their segmentation is used for prediction and transform processes without any further partitioning. This means that in the QTBT coding block structure, CUs, PUs, and TUs have the same block size. In some embodiments, a CU contains coding blocks (CBs) of different color components. For example, in the case of P slices and B slices in 4:2:0 chroma format, one CU includes one luma CB and two chroma CBs, and sometimes consists of a single component CB. As another example, in the case of an I slice, one CU includes only one luma CB or two chroma CBs.

[0066] In some embodiments, the following parameters are defined for the QTBT partitioning scheme: (i) CTU size: quad-tree root node size (e.g., the same concept as in HEVC); (ii) MinQTSize: minimum allowed quad-tree leaf node size; (iii) MaxBTSize: maximum allowed binary tree root node size; (iv) MaxBTDepth: maximum allowed binary tree depth; and (v) MinBTSize: minimum allowed binary tree leaf node size.

[0067] In one example of a QTBT partitioning structure, the CTU size is set as 128x128 luma samples with two corresponding 64x64 blocks of chroma samples, MinQTSize is set as 16x16, MaxBTSize is set as 64x64, MinBTSize (for both width and height) is set as 4x4, and MaxBTDepth is set as 4. First, quadtree division is applied to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes can have sizes from 16x16 (MinQTSize) to 128x128 (CTU size). If the leaf quadtree node is 128x128, it will not be further divided by the binary tree because its size exceeds MaxBTSize (64x64). Otherwise, the leaf quadtree node can be further partitioned by the binary tree. Therefore, the quadtree leaf node is also the root node for the binary tree, which has the binary tree depth as 0. Once the binary tree depth reaches MaxBTDepth(4), no further splits are considered. Once a binary tree node has a width equal to MinBTSize(4), no further horizontal splits are considered. Similarly, once a binary tree node has a height equal to MinBTSize, no further vertical splits are considered. The binary tree leaf node is further processed by the prediction and transform process without further partitioning. In JEM, the maximum CTU size is 256x256 luma samples.

[0068] Figure 5A shows an example of partitioning a block 502 using QTBT. Figure 5B shows the corresponding tree representation 504. Solid lines indicate quadtree partitioning, and dotted lines indicate binary tree partitioning. At each partition of a node (e.g., a non-leaf node) in the binary tree, a flag is signaled to indicate which partition type (e.g., horizontal or vertical) is used, with 0 indicating a horizontal partition and 1 indicating a vertical partition. In quadtree partitioning, there is no need to indicate the partition type because quadtree partitioning always divides a block both horizontally and vertically to generate four sub-blocks of equal size.

[0069] In some embodiments, the QTBT scheme supports the flexibility of luma and chroma having separate QTBT structures. Currently, in P and B slices, the luma CTB and chroma CTB of one CTU share the same QTBT structure. However, in an I slice, the luma CTB is partitioned into CUs by a QTBT structure, and the chroma CTB is partitioned into chroma CUs by a different QTBT structure. This means that a CU in an I slice consists of a coding block of a luma component or a coding block of two chroma components, and a CU in a P or B slice consists of coding blocks of all three color components.

[0070] In some embodiments, inter-prediction for small blocks is restricted to reduce motion compensation memory accesses, resulting in bidirectional prediction not being supported for 4x8 and 8x4 blocks, and inter-prediction not being supported for 4x4 blocks. In some embodiments, these restrictions are removed in QTBT.

[0071] In some embodiments, block partitioning is implemented using a ternary tree (triple tree; TT) (also known as a multi-type tree (MTT) structure). FIGS. 6A and 6B show MTT structures 602 and 604 according to some embodiments. The MTT structure adds horizontal center-side triple trees and vertical center-side triple trees in addition to the QTBT. Benefits of ternary tree partitioning include: (i) it is complementary to quad-tree and binary tree partitioning, and can capture objects located at block centers, whereas quad-tree and binary trees always split along block centers; and (ii) the width and height of the proposed ternary tree partitions are always powers of two, and therefore no additional transformations are required.

[0072] The design of two-level trees is primarily motivated by reduced complexity: theoretically, the complexity of traversing the tree is T D where T represents the number of split types and D is the depth of the tree.

[0073] Merge Mode with Motion Vector Difference (MMVD)

[0074] In merge mode, the implicitly derived motion information is directly used to generate the predicted samples for the current CU. In merge mode with motion vector differences (MMVD), the MMVD flag can be signaled immediately after sending the skip and merge flags to specify whether the MMVD mode is used for the CU.

[0075] In MMVD, after a merge candidate is selected, it can be further refined by signaled motion vector differential (MVD) information (one or more). The signaled information may include a merge candidate flag, an index specifying the magnitude of motion, and an index to indicate the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. A merge candidate flag may be signaled to specify which one is used.

[0076] The distance index defines the magnitude of the motion and indicates a predetermined offset from the starting point. As shown in Figure 7, the offset can be added to either the horizontal or vertical component of the starting motion vector (MV). The relationship between the distance index and the predetermined offset can be defined in Table 1. [Table 1]

[0077] The direction index represents the direction of the MVD relative to the starting point. The direction index can represent four directions, as shown in Table 2. [Table 2]

[0078] In some embodiments, the meaning of the sign of MVD can change according to the information of the starting MV. When the starting MV is a unidirectionally predicted MV or a bidirectionally predicted MV where both lists point to the same side of the current picture (e.g., the picture order counts (POC) of the two references are both greater than that of the current picture, or both less than that of the current picture), the sign in Table 2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV and the two MVs point to different sides of the current picture (e.g., the POC of one reference is greater than that of the current picture, and the POC of the other reference is less than that of the current picture), and the difference in POC in list 0 is greater than that in list 1, the sign in Table 1 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign for the list 1 MV has the opposite value. Otherwise, if the POC difference in List 1 is greater than List 0, the sign in Table 2 specifies the sign of the MV offset that is added to the List 1 MV component of the starting MV, and the sign for the List 0 MV has the opposite value.

[0079] In some embodiments, the MVD is scaled according to the POC difference in each direction. If the POC difference in both lists is the same, no scaling is required. Otherwise, if the POC difference in list 0 is larger than that of list 1, the MVD of list 1 is scaled. If the POC difference in L1 is larger than L0, the MVD of list 0 is scaled as well. If the starting MV is unidirectionally predicted, the MVD is added to the available MV.

[0080] Symmetric Motion Vector Differential (MVD) Coding

[0081] In symmetric MVD mode, motion information including reference picture indices for both list 0 and list 1 and the MVD for list 1 is derived without being signaled.

[0082] In some embodiments, the decoding process for symmetric MVD mode at the slice level is as follows: The variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows: (i) if mvd_l1_zero_flag is 1, then BiDirPredFlag is set equal to 0. (ii) Otherwise, if the closest reference picture in list 0 and the closest reference picture in list 1 form a forward and backward pair of reference pictures or a backward and forward pair of reference pictures, then BiDirPredFlag is set to 1, and both the list 0 reference picture and the list 1 reference picture are short-term reference pictures. (iii) Otherwise, BiDirPredFlag is set to 0.

[0083] In some embodiments, the decoding process for symmetric MVD mode at the CU level is as follows: If a CU is bidirectionally predictive coded and BiDirPredFlag is equal to 1, a symmetric mode flag is explicitly signaled indicating whether symmetric mode is used. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 are explicitly signaled. The reference indices of list 0 and list 1 are set equal to the reference picture pair, respectively. MVD1 is set equal to (-MVD0).

[0084] Intermode Coding

[0085] In some embodiments, for blocks coded in interframes, if the mode of the current block is an inter-coding mode rather than a skip mode, a separate flag is signaled to indicate whether a single or mixed reference mode is used for the current block. In a single reference mode, a predictive block may be generated by one motion vector. In a mixed reference mode (e.g., the current block has two reference pictures), a predictive block is generated by a weighted average of two predictive blocks derived from two motion vectors.

[0086] In some embodiments, the single reference mode is signaled as follows: NEARMV - use one of the motion vector predictors (MVP) in the list indicated by the DRL (Dynamic Reference List) index; · NEWMV - uses one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference and applies a delta to the MVP; ·GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0087] In some embodiments, the following modes are signaled in the mixed reference mode: · NEAR_NEARMV - use one of the motion vector predictors (MVP) in the list signaled by the DRL index; · NEAR_NEWMV - use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference and send a delta MV for the second MV; · NEW_NEARMV - use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference and send a delta MV for the first MV; · NEW_NEWMV - use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as reference and send delta MVs for both MVs; ·GLOBAL_GLOBALMV - Use MV from each reference based on frame-level global motion parameters.

[0088] Motion Vector Differential (MVD) Coding

[0089] AV1 allows for 1 / 8 pixel motion vector precision (or accuracy). To signal motion vector differentials in reference frame list 0 or list 1, the following syntax is used: mv_joint specifies which components of the motion vector differential are non-zero - 0 indicates that there are no non-zero MVDs along either the horizontal or vertical direction - 1 indicates that there is a non-zero MVD only along the horizontal direction - 2 indicates that there is a non-zero MVD only along the vertical direction - 3 indicates that there is a non-zero MVD along both the horizontal and vertical directions mv_sign specifies whether the motion vector differential is positive or negative mv_class specifies the class of the motion vector difference. As shown in Table 3, a higher class means a larger magnitude of the motion vector difference. Hhmv_bit specifies the integer part of the offset between the motion vector differential and the starting magnitude of each MV class mv_fr specifies the first two fractional bits of the motion vector differential · mv_hp specifies the third fractional bit of the motion vector differential. [Table 3]

[0090] Adaptive MVD Resolution

[0091] For NEW_NEARMV and NEAR_NEWMV modes, the precision of the MVD depends on the associated class and the size of the MVD. For example, fractional MVD is only allowed if the size of the MVD is 1 pixel or less. As another example, when the value of the associated MV class is MV_CLASS_1 or greater, only one MVD value is allowed, and the MVD value for each MV class is derived as 4, 8, 16, 32, or 64 for MV class 1 (MV_CLASS_1), 2 (MV_CLASS_2), 3 (MV_CLASS_3), 4 (MV_CLASS_4), or 5 (MV_CLASS_5). As another example, when the current block is coded as NEW_NEARMV or NEAR_NEWMV mode, one context is used to signal mv_joint or mv_class. If this is not the case (eg, the current block is not coded as NEW_NearMV), then a separate context is used to signal mv_joint or mv_class.

[0092] Table 4 shows the allowable MVD values ​​for each MV class. [Table 4]

[0093] In some embodiments, an inter-coding mode called JOINT_NEWMV is applied to indicate whether the MVDs for two reference lists are signaled jointly. When the inter-prediction mode is equal to JOINT_NEWMV mode, the MVDs for reference list 0 and reference list 1 are signaled jointly. Then, only one MVD called joint_mvd can be signaled and sent to the decoder, and the delta MVs for reference list 0 and reference list 1 are derived from joint_mvd.

[0094] In some embodiments, the JOINT_NEWMV mode is signaled along with the NEAR_NEARMV mode, NEAR_NEWMV mode, NEW_NEARMV mode, NEW_NEWMV mode, and GLOBAL_GLOBALMV mode. No additional context is added.

[0095] In some embodiments, when JOINT_NEWMV mode is signaled and the POC distances between two reference frames and the current frame are different, the MVD is scaled for reference list 0 or reference list 1 based on the POC distances. The distance between reference frame list 0 and the current frame is denoted as td0, and the distance between reference frame list 1 and the current frame is denoted as td1. If td0 is greater than or equal to td1, joint_mvd is used directly for reference list 0, and mvd for reference list 1 is calculated from joint_mvd using Equation (1):

number

[0096] Otherwise, if td1 is greater than or equal to td0, then joint_mvd is used directly for reference list 1, and mvd for reference list 0 is obtained from joint_mvd by equation (2):

number

[0097] In some embodiments, an inter-coding mode called AMVDMV is added for the single-reference case. When the AMVDMV mode is selected, it indicates that AMVD is applied to signal the MVD. Under the JOINT_NEWMV mode, a flag (amvd_flag) is added to indicate whether AMVD is applied to the joint MVD coding mode. When adaptive MVD resolution is applied to the joint MVD coding mode, it is called joint AMVD coding, in which the MVDs for two reference frames are signaled together, and the precision of the MVD is implicitly determined by the size of the MVD. Otherwise, the MVDs for two (or more) reference frames are signaled together, and conventional MVD coding is applied.

[0098] In some embodiments, Adaptive Motion Vector Resolution (AMVR) supports seven MV precisions: 8, 4, 2, 1, 1 / 2, 1 / 4, and 1 / 8. For each prediction block, the AOM Video Model (AVM) encoder searches all supported precision values ​​and signals the best precision to the decoder. In some embodiments, to reduce encoder execution time, two precision sets are supported, with each precision set containing four predetermined precisions. At the frame level, the precision set is selected adaptively based on the maximum precision value for that frame. In some embodiments, the maximum precision is signaled in the frame header.

[0099] Table 5 summarizes the supported accuracy values ​​based on frame-level maximum accuracy. [Table 5]

[0100] In some embodiments, a frame-level flag indicates whether the MV of a frame includes sub-pel precision. AMVR is enabled only if the value of the cur_frame_force_integer_mv flag is 0. In AMVR, if the precision of a block is less than the maximum precision, the motion model and interpolation filter are not signaled. If the precision of a block is less than the maximum precision, the motion mode is estimated to be translational motion and the interpolation filter is estimated to be a regular interpolation filter. Similarly, if the precision of a block is either 4-pel or 8-pel, the inter-intra mode is not signaled and is estimated to be 0.

[0101] Motion Vector Predictor (MVP) List

[0102] A spatial motion vector predictor (SMVP) (e.g., both adjacent SMVP and non-adjacent SMVP), a temporal MV predictor (TMVP), extra MV candidates in AV1, and additional derived MVPs and reference bank MVPs are further added in the AVM design. To store the MVP lists, stacks with fixed sizes are generated at both the encoder and decoder ends.

[0103] In some embodiments, the SMVP is derived from spatial neighboring blocks. The spatial neighboring blocks include adjacent spatial neighboring blocks that are direct neighbors to the top and left of the current block, and non-adjacent spatial neighboring blocks that are not directly adjacent to the current block. An exemplary set of spatial neighboring blocks for a luma block is shown in Figure 8, where each spatial neighboring block is an 8x8 block. The spatial neighboring blocks are examined to find one or more MVs associated with the same reference frame index as the current block. For the current block, the search order of the spatially adjacent 8x8 luma blocks is as shown by numbers 1-8 in Figure 8: 1. The adjacent rows above are checked from left to right; 2. The left adjacent column is checked from top to bottom; 3. The upper right adjacent block is checked; 4. The blocks adjacent to the top left block are checked; 5. The first non-adjacent row above is checked from left to right; 6. The first non-adjacent column on the left is checked from top to bottom; 7. The second non-adjacent row above is checked from left to right; 8. The second non-adjacent row to the left is checked from top to bottom.

[0104] In some embodiments, neighboring candidates (candidates 1-3 in FIG. 8) are inserted into the MV predictor list before TMVP, and non-neighboring candidates (also known as outer candidates, i.e., candidates 4-8 in FIG. 8) are placed into the MV predictor list after TMVP. All SMVP candidates should have the same reference picture as the current block. If the current block has a single reference picture and the MVP candidate has a single reference picture and this reference picture is the same as the reference picture of the current block, or if the MVP candidate has a mixed reference picture (e.g., two reference pictures) and one of the reference pictures is the same as the reference picture of the current block, then the MVP candidate will be placed into the MV predictor list. If the current block has two reference pictures, then the MVP candidate will be placed into the MV predictor list only if the MVP candidate has two reference pictures and these two reference pictures are the same as the reference picture of the current block.

[0105] In some embodiments, to reduce line memory, the outer candidates represented by lines 5 and 7 in FIG. 8 are further removed from the non-adjacent candidates.

[0106] In some embodiments, TMVP can be derived using collocated blocks in a reference frame. To generate TMVP, first, the MVs of the reference frame are stored together with the reference indexes associated with each reference frame. Then, for each 8x8 block of the current frame, the MVs of the reference frame whose trajectory passes through the 8x8 block are identified and stored in a temporal MV buffer together with the reference frame index. In inter prediction using a single reference frame, regardless of whether the reference frame is a forward reference frame or a backward reference frame, the MVs are stored in 8x8 units for temporal motion vector prediction of future frames. In hybrid inter prediction, only forward MVs are stored in 8x8 units for temporal motion vector prediction of future frames.

[0107] FIG. 9 shows a motion vector MVref 910 corresponding to reference frame 1 (R1) 908 pointing from R1 to its reference frame 902. In doing so, MVref 910 passes through an 8x8 block 914 of the current frame 906. MVref 910 is stored in a temporal MV buffer associated with the 8x8 block 914. In the motion projection process for deriving TMVP, the reference frames are scanned in a predetermined order: LAST_FRAME, BWDREF_FRAME, ALTREF_FRAME, ALTREF2_FRAME, and LAST2_FRAME. MVs from reference frames with higher indices (in the scan order) do not replace previously identified MVs assigned by reference frames with lower indices (in the scan order). Given a given block coordinate, the associated MV stored in the temporal MV buffer is identified and projected onto the current block to derive a temporal MV predictor (e.g., MV0 912) pointing from the current block 914 to its reference frame 904.

[0108] 10 illustrates predetermined block positions for deriving a temporal MV predictor for a 16x16 block, according to some embodiments. Up to seven blocks (e.g., blocks B0-B6) are checked for valid temporal MV predictors. The temporal MV predictors are checked after adjacent spatial MV predictors but before non-adjacent spatial MV predictors.

[0109] To derive the MV predictor, all spatial and temporal MV candidates are pooled, and each predictor is assigned a weight determined during the scanning of spatial and temporal neighboring blocks. Based on their associated weights, the candidates are sorted and ranked, and up to four candidates are identified and added to the MV predictor list. This list of MV predictors, also called the dynamic reference list (DRL), is further used in the dynamic MV prediction mode, as described in the next subsection.

[0110] Further exploration of extra MVP candidates In some embodiments, if the MVP list is still not full, an additional search is performed to use extra MVP candidates to fill the MVP list, including, for example, global MVs, zero MVs, and unscaled combined composite MVs.

[0111] MVP candidate sorting process In some embodiments, adjacent SMVP candidates, TMVP candidates, and non-adjacent SMVP candidates added to the MVP list are reordered. In some embodiments, the reordering process is based on the weight of each candidate. The candidate weight is predetermined depending on the overlapping area between the current block and the candidate block. In some embodiments, the weighting of outer / non-adjacent candidates and TMVP candidates is not considered during the reordering process, which means that the reordering process only affects adjacent candidates.

[0112] Derived MVP candidates In some embodiments, the derived MVP candidates include both single reference picture and mixed mode derived MVPs.

[0113] Single Inter Prediction In some embodiments, if a neighboring block's reference frame is different from the current block's reference frame but in the same direction, a temporal scaling algorithm can be used to scale its MV to its reference frame to form an MVP for the current block's motion vector. Figure 11 shows an example in which MV1 1102 from a neighboring block 1106 can be used to derive an MVP for a motion vector MV0 1104 of a current block 1108 using temporal scaling.

[0114] Combined Inter Prediction In some embodiments, composite MVs from different neighboring blocks are used to derive the MVP for the current block, but the reference frame of the composite MVs must be the same as the current block. As shown in Figure 12, the composite MVs, consisting of MV2 1202 and MV3 1206, have the same reference frame as the current block 1210, but they are from different neighboring blocks. Figure 12 shows a motion vector MV0 1204 for the current block 1210 in the current frame F4, which points to another reference block 1212 in frame F3, and a motion vector MV1 1208 for the current block 1210, which points (e.g., in the other direction) to another reference block 1214 in frame F5.

[0115] 13 illustrates a reference MV candidate bank 1302 that may be part of or communicatively connected to a computing system (e.g., server system 112, source device 102, or electronic device 120) according to some embodiments. In some embodiments, each buffer corresponds to a unique reference frame type, corresponding to a single reference frame or pair of reference frames covering single inter-mode and multiple inter-mode, respectively. In some embodiments, all buffers are the same size. In some embodiments, when a new MV is added to a full buffer, existing MVs are evicted to make room for the new MV.

[0116] A coding block can collect reference MV candidates by referring to the MV candidate bank in addition to those obtained in the reference MV list generation. After coding a superblock, the MV candidate bank 1302 is updated with the MVs used by the coding block of the superblock.

[0117] In some embodiments, each tile has an independent MV reference bank that is used by all super-blocks within that tile. At the start of encoding each tile, the corresponding bank is emptied. Then, during coding each super-block within that tile, MVs from the bank can be used as MV reference candidates. At the end of encoding a super-block, the bank is updated.

[0118] 13 illustrates a superblock-based reference MV candidate bank 1302 update process according to some embodiments. After a superblock is coded, the first, e.g., up to 64, candidate MVs used by the coding blocks in the superblock are added to the reference MV candidate bank 1302. In some embodiments, a pruning process is also included during this update. In some embodiments, after scanning for reference MV candidates, if there are free slots in the candidate list, the codec algorithm can refer to the MV candidate bank 1302 (in the buffer of the matching reference frame type) for further MV candidates. Starting from the end of the buffer to the beginning, MVs in the bank buffer are added to the candidate list if they are not already in the list.

[0119] 14 illustrates a process 1400 for constructing a motion vector predictor (MVP) list according to some embodiments. In some embodiments, the MVP list is constructed using complete pruning in the following order: In some embodiments, the process begins 1402 by inserting adjacent SMVPs 1404. The process includes a reordering step 1406 of existing candidates, followed by TMVPs 1408, non-adjacent SMVPs 1410, derived MVP candidates 1412, extra MVP candidates 1414, and candidates 1416 from a reference MV candidate bank (e.g., reference MV candidate bank 1302).

[0120] 15 is a flow diagram illustrating a method 1500 for decoding a current coding block according to some embodiments. Method 1500 may be performed on a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and memory storing instructions for execution by the control circuitry. In some embodiments, method 1500 is performed by executing instructions stored in a memory (e.g., memory 314) of the computing system.

[0121] The system retrieves one or more motion vector (MV) candidates from a reference MV bank (1502). In accordance with a determination that a first condition is met, the system inserts the one or more MV candidates from the reference MV bank into a motion vector prediction (MVP) list associated with a current coding block after the derived MV candidate is inserted into the MVP list (1504). In accordance with a determination that the first condition is not met, the system inserts the one or more MV candidates from the reference MV bank into an MVP list associated with the current coding block before the derived MV candidate is inserted into the MVP list (1506). The system predicts the current coding block based on the MVP list (1508).

[0122] (A1) In one aspect, some embodiments include a method (e.g., method 1500) for decoding a current coding block. In some embodiments, the method is implemented in a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is implemented in a coding module (e.g., coding module 320). The method includes retrieving one or more motion vector (MV) candidates from a reference MV bank. The method includes: In some embodiments, candidates from a reference candidate bank (e.g., reference MV candidate bank 1302) are inserted into the MVP list before derived candidates, so that the MVP list is constructed in the following order: adjacent SMVPs, sorting process for existing candidates, TMVP candidates, non-adjacent SMVP candidates, candidates from the reference MV candidate bank, derived candidates, and extra candidates.

[0123] In some embodiments, the derived MV candidate is an MV predictor derived from either one or more motion vectors in a single reference frame based on an inter prediction mode, or a composite motion vector from multiple adjacent reference frames in a composite inter prediction mode.

[0124] In some embodiments, candidates from the reference candidate bank are conditionally inserted into the MVP list before derived candidates, for example, if a condition is met, the original MVP list order is used, otherwise, if the condition is not met, a new MVP list with a different order or content is constructed.

[0125] In some embodiments, candidates from the reference candidate bank are conditionally inserted into the MVP list before the derived candidates. For example, if one condition is met, the original MVP list order is used; otherwise (e.g., if that condition is not met), the MVP list is constructed in the following order, namely, adjacent SMVP, the sorting process for existing candidates, TMVP candidates, non - adjacent SMVP candidates, candidates from the reference MV candidate bank, derived candidates, and extra candidates, in that order.

[0126] In some embodiments, when the condition of width ≧ N and / or height ≧ N is met (e.g., N can be equal to, but not limited to, 8, 16, 32, or 64 luminance samples below), the original MVP list order is used (e.g., candidates from the reference MV candidate bank are inserted at the end of the MVP list); otherwise, when width < N or height < N, candidates from the reference MV candidate bank are inserted before the derived candidates.

[0127] In some embodiments, when the condition of width ≧ N and / or height ≧ N is met (e.g., N can be equal to, but not limited to, 8, 16, 32, or 64 luminance samples), the original MVP list order is used (e.g., candidates from the reference MV candidate bank are placed at the end of the MVP list); otherwise, when width < N and height < N, candidates from the reference MV candidate bank are inserted before the derived candidates.

[0128] In some embodiments, when the condition of width × height ≧ N × N is met (e.g., N can be equal to, but not limited to, 8, 16, 32, or 64 luminance samples), the original MVP list order is used (e.g., candidates from the reference MV candidate bank are placed at the end of the MVP list); otherwise, when width × height < N × N, candidates from the reference MV candidate bank will be inserted before the derived candidates.

[0129] In some embodiments, if the current MV coding mode includes "NEAR" (e.g., NEAR M in the single-reference case, or NEAR_NEWMV, NEW_NEARMV, NEAR_NEARMV in the mixed-reference case, and their extensions, such as optical flow), the original MVP list order is used (e.g., candidates from the reference MV candidate bank are placed at the end of the MVP list); otherwise, if the current MV coding mode does not include "NEAR" (e.g., NEW M in the single-reference case, and NEW_NEWMV and JOINT_NEWMV in the mixed-reference case, and their extensions, such as optical flow), candidates from the reference MV candidate bank are inserted before the derived candidates.

[0130] In some embodiments, if the current MV coding mode includes NEAR for each reference frame list (e.g., NEAR MVS in the single-reference case, NEAR_NEARMV in the mixed-reference case, and their extensions such as optical flow), the original MVP list order is used (e.g., candidates from the reference MV candidate bank are placed at the end of the MVP list); otherwise, if the current MV coding mode does not include NEAR for all reference frame lists (e.g., NEW MVS in the single-reference case, and NEW_NEWMV, NEAR_NEWMV, NEW_NEARMV, and JOINT_NEWMV in the mixed-reference case, and their extensions such as optical flow), candidates from the reference MV candidate bank are inserted before the derived candidates.

[0131] In some embodiments, if the current motion mode is translational, the original MVP list order is used; otherwise (e.g., the motion mode is warp or obmc), a new MVP list with a different order or content is constructed.

[0132] 15 depicts some logical stages in a particular order, stages that are not order-dependent may be rearranged, and other stages may be combined or separated. Any rearrangements or other groupings not specifically mentioned will be apparent to those skilled in the art, and the ordering and groupings presented herein are not exhaustive. It should also be recognized that these stages may be implemented in hardware, firmware, software, or any combination thereof.

[0133] Reference will now be made to some example embodiments.

[0134] (A1) In one aspect, some embodiments include a method (e.g., method 1500) for decoding a current coding block. In some embodiments, the method is implemented in a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is implemented in a coding module (e.g., coding module 320). The method includes retrieving one or more motion vector (MV) candidates from a reference MV bank. In accordance with a determination that a first condition is satisfied, the method includes inserting the one or more MV candidates from the reference MV bank into an MVP list associated with the current coding block after a derived MV candidate is inserted into the MVP list; in accordance with a determination that the first condition is not satisfied, inserting the one or more MV candidates from the reference MV bank into an MVP list associated with the current coding block before the derived MV candidate is inserted into the MVP list; and predicting the current coding block based on the MVP list.

[0135] (A2) In some embodiments of A1, the one or more MV candidates include a spatial MV predictor (SMVP) derived from spatial neighboring blocks adjacent to the current coding block, as shown in FIG.

[0136] (A3) In some embodiments of A1 or A2, the first condition includes at least one of: (i) the current coding block has a width of N luma samples or more; and (ii) the current coding block has a height of N luma samples or more, where N is an integer greater than or equal to 8.

[0137] (A4) In some embodiments of any of A1-A3, the first condition includes a first sub-condition that (i) the current coding block has a width of N luma samples or more, and (ii) a second sub-condition that the current coding block has a height of N luma samples or more, where N is an integer greater than or equal to 8.

[0138] (A5) In some embodiments of any of A1-A4, when the first condition is not met, the MVP list has the following order: adjacent SMVP, sorted series of MV candidates, temporal motion vector predictor (TMVP) candidate, non-adjacent SMVP candidate, second candidate from the reference MV candidate bank, and derived candidate.

[0139] (A6) In some embodiments of any of A1-A5, the first condition includes a condition that the product of the width of the current coding block and the height of the current coding block is greater than or equal to N×N, where N is an integer representing the number of luma samples, and N is greater than or equal to 8.

[0140] (A7) In some embodiments of any of A1-A6, the first condition includes that the current MV coding mode includes the term “NEAR” (e.g., NEAR MV in the single-reference case, or NEAR_NEWMV, NEW_NEARMV, NEAR_NEARMV, and their extensions, such as optical flow, in the mixed-reference case).

[0141] (A8) In some embodiments of any of A1-A7, the first condition includes a condition that the current MV coding mode has a NEAR mode for each reference frame list in the video stream acquired by the computing system.

[0142] (A9) In some embodiments of any of A1-A8, the first condition includes a condition that the current motion mode is a translational mode.

[0143] (A10) In some embodiments of any of A1-A9, the derived MV candidate is an MV predictor derived from either a motion vector (or one or more motion vectors) in a single reference frame based on an inter prediction mode, or a composite motion vector (or one or more composite motion vectors) from multiple adjacent reference frames in a composite inter prediction mode.

[0144] In another aspect, some embodiments include a computing system (e.g., server system 112) including control circuitry (e.g., control circuitry 302) and a memory (e.g., memory 314) coupled to the control circuitry, the memory storing one or more instruction sets configured to be executed by the control circuitry, the one or more instruction sets including instructions for performing any of the methods described herein (e.g., A1-A9 above).

[0145] In yet another aspect, some embodiments include a non-transitory computer-readable storage medium having stored thereon one or more instruction sets for execution by control circuitry of a computing system, the one or more instruction sets including instructions for performing any of the methods described herein (e.g., A1-A9 above).

[0146] It is understood that although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms; these terms are used only to distinguish one element from another.

[0147] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the claims. When used in the description of embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It is also understood that the term "and / or," as used herein, refers to and includes any and all possible combinations of one or more of the associated listed items. It is further understood that the terms "comprise" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0148] As used herein, the term "if" may be interpreted to mean "when," or "responding to," or "in response to determining," or "pursuant to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that the stated condition precedent is true," or "if the stated condition precedent is true," or "when the stated condition precedent is true" may be interpreted to mean "in response to determining," or "in response to determining," or "pursuant to determining," or "in response to detecting," or "in response to detecting," depending on the context.

[0149] The foregoing description has been described with reference to specific embodiments for purposes of explanation. However, the exemplary description above is not intended to be exhaustive or to limit the claims to the precise form disclosed. Numerous modifications and variations are possible in light of the above teachings. These embodiments were chosen and described in order to best explain the principles of operation and practical application, thereby enabling others skilled in the art.

Claims

[Claim 1] 1. A method for coding a current coding block executed by a computing system having one or more processors and a memory, the method comprising: Retrieving one or more motion vector (MV) candidates from a reference MV bank; according to determining that a first condition is satisfied, inserting the one or more MV candidates from the reference MV bank into a motion vector prediction (MVP) list associated with the current coding block after a derived MV candidate is inserted into the MVP list; according to determining that the first condition is not satisfied, inserting the one or more MV candidates from the reference MV bank into the MVP list associated with the current coding block before inserting the derived MV candidate into the MVP list; predicting the current coding block based on the MVP list; A method having the following.

Images