Method, computing system, and computer program for video coding - Patents.com

Abstract

Various implementations described herein include methods and systems for decoding a current coding block. In one aspect, the method includes retrieving one or more motion vector (MV) candidates from a reference MV bank. The method includes, in accordance with a determination that a first condition is satisfied, inserting the one or more MV candidates from the reference MV bank into an MVP list associated with the current coding block after a derived MV candidate is inserted into the MVP list. The method includes, in accordance with a determination that a first condition is not satisfied, inserting the one or more MV candidates from the reference MV bank into an MVP list associated with the current coding block before a derived MV candidate is inserted into the MVP list. The method includes predicting the current coding block based on the MVP list.

Description

[Technical field]

[0001] This application is a continuation of, and claims priority to, U.S. patent application Ser. No. 18 / 196,368, filed May 11, 2023, entitled “Systems and Methods for Motion Vector Predictor List Improvements,” and also claims priority to U.S. Provisional Patent Application Ser. No. 63 / 349,754, filed June 7, 2022, entitled “Motion Vector Predictor List Improvements,” which are incorporated herein by reference in their entireties.

[0002] The disclosed embodiments relate generally to video coding and include, but are not limited to, systems and methods for motion vector predictor list refinement. [Background technology]

[0003] Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc. These electronic devices transmit, receive, or otherwise communicate digital video data across communication networks and store the digital video data in storage devices. Due to the limited bandwidth capacity of communication networks and limited memory resources of storage devices, video coding may be used to compress the video data according to one or more video coding standards before the video data is communicated or stored.

[0004] Several video codec standards have been developed. For example, video coding standards include AOMedia Video 1 (AV1), Versatile Video Coding (VVC), Joint Search Model (JEM), High Efficiency Video Coding (HEVC / H.265), Advanced Video Coding (AVC / H.264), and Moving Picture Experts Group (MPEG) coding. Video coding generally employs prediction techniques (e.g., inter-prediction, intra-prediction, or the like) that exploit the redundancy inherent in video data. Video coding aims to compress video data into a form that uses a lower bitrate while avoiding or minimizing degradation to video quality.

[0005] HEVC, also known as H.265, is a video compression standard designed as part of the MPEG-H project. ITU-T and ISO / IEC published the HEVC / H.265 standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). Versatile Video Coding (VVC), also known as H.266, is a video compression standard intended as a successor to HEVC. ITU-T and ISO / IEC published the VVC / H.266 standard in 2020 (version 1) and 2022 (version 2). AV1 is an open video coding format designed as a replacement for HEVC. On January 8, 2019, the certified version 1.0.0 with errata for the specification was released. Summary of the Invention

[0006] This disclosure describes an improved method and system for decoding video data based on a motion vector prediction (MVP) list. In a video codec standard, the result of motion vector prediction of a block being coded or decoded is a list of motion vectors. The main idea of ​​forming this list is that the motion vector for a current block has a high probability of being little different from the motion vectors for neighboring blocks coded before it, and therefore can be used as a prediction. Currently, the process of ordering the motion vector (MV) candidates in the MVP list is suboptimal because some MV candidates that are more important (e.g., produce more accurate predictions) are listed at later positions. Also, a predefined weight-based sorting process is used to sort only neighboring spatial MVPs (SMVPs), temporal MVPs (TMVPs), and non-neighboring SMVPs, which is suboptimal for other MV candidates.

[0007] Therefore, an improved method and system for constructing an MVP list for encoding and decoding video data is desired. For example, because spatial MV candidates are usually more accurate than combined (e.g., scaled) candidates and padded candidates (e.g., zero MVs, etc.), according to some aspects of the present disclosure, when one or more conditions are met, candidates from a reference MV candidate bank are inserted into an MVP list before other MV candidates. Because MV candidates from the reference MV candidate bank are mainly from spatial neighbors, the present disclosure generates an improved MVP list with more accurate motion vector prediction, thereby resulting in increased coding efficiency.

[0008] According to some embodiments, a method for constructing a current coding block for decoding is performed in a computing system having one or more processors and a memory. The method includes retrieving one or more motion vector (MV) candidates from a reference MV bank. The method includes, in accordance with a determination that a first condition is satisfied, inserting the one or more MV candidates from the reference MV bank into a motion vector predictor (MVP) list associated with the current coding block after a derived MV candidate is inserted into the MVP list. The method includes, in accordance with a determination that a first condition is not satisfied, inserting the one or more MV candidates from the reference MV bank into an MVP list associated with the current coding block before a derived MV candidate is inserted into the MVP list. The method includes predicting the current coding block based on the MVP list.

[0009] According to some embodiments, a computing system, such as a streaming system, a server system, a personal computer system, or other electronic device, is provided. The computing system includes control circuitry and a memory having stored thereon one or more instruction sets. The one or more instruction sets include instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and / or a decoder component.

[0010] According to some embodiments, a non-transitory computer readable storage medium is provided that stores one or more sets of instructions for execution by a computing system, the one or more sets of instructions including instructions for performing any of the methods described herein.

[0011] Accordingly, disclosed are methods, as well as apparatus and systems, for encoding and decoding video, which may complement or replace conventional methods, apparatus and systems for video encoding / decoding.

[0012] The features and advantages described herein are not necessarily all-inclusive, and some additional features and advantages will become apparent to those skilled in the art, particularly in light of the drawings, specification, and claims provided in this disclosure. It should also be noted that the language used herein has been selected primarily for ease of reading and instructional purposes, and not necessarily to delineate or bound the subject matter described herein. [Brief description of the drawings]

[0013] So that the present disclosure may be understood in more detail, a more particular description will be made by reference to features of various embodiments, some of which are illustrated in the accompanying drawings, which are, however, merely illustrative of the relevant features of the present disclosure and therefore should not be considered as necessarily limiting, and may, for the purposes of illustration, recognize other useful features as those skilled in the art will read and understand this disclosure. [Figure 1] FIG. 1 is a block diagram illustrating an example of a communication system according to some embodiments. [Figure 2A] FIG. 2 is a block diagram illustrating example elements of an encoder component according to some embodiments. [Figure 2B] FIG. 2 is a block diagram illustrating example elements of a decoder component according to some embodiments. [Diagram 3] FIG. 1 is a block diagram illustrating an example of a server system according to some embodiments. [Figure 4] 1 illustrates block partitioning according to some embodiments. [Figure 5A] 5A and 5B illustrate a quadtree (QT)+binary tree (QTBT) structure according to some embodiments. [Figure 5B] 5A and 5B illustrate a quadtree (QT)+binary tree (QTBT) structure according to some embodiments. [Figure 6A] 6A and 6B illustrate ternary tree (TT) partitioning according to some embodiments. [Figure 6B] 6A and 6B illustrate ternary tree (TT) partitioning according to some embodiments. [Figure 7] 1 illustrates example search points for merge mode with motion vector difference (MMVD), according to some embodiments. [Figure 8] 1 illustrates using spatially neighboring blocks for MV prediction according to some embodiments. [Figure 9] 1 illustrates motion field estimation by linear projection according to some embodiments. [Figure 10] 13 illustrates the use of block position to derive a temporal MV predictor according to some embodiments. [Figure 11] 11 illustrates additional motion vector candidate generation for a single inter-predicted block according to some embodiments. [Figure 12] 11 illustrates additional motion vector candidate generation for a composite prediction block according to some embodiments. [Figure 13] 1 illustrates a reference MV candidate bank update process according to some embodiments. [Figure 14] 1 illustrates a process for building an MVP list according to some embodiments. [Figure 15] 4 is a flow diagram of a method for decoding a current coding block according to some embodiments.

[0014] According to common practice, the various features illustrated in the drawings are not necessarily drawn to scale and like reference numerals may be used to denote like features throughout the specification and drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The present disclosure describes decoding a current coding block based on a motion vector prediction (MVP) list. One or more motion vector (MV) candidates are retrieved from a reference MV bank. When a first condition is met, the MV candidates from the reference MV bank are inserted into the MVP list associated with the current coding block after the derived MV candidates are inserted into the MVP list. When the first condition is not met, the MV candidates from the reference MV bank are inserted into the MVP list associated with the current coding block before the derived MV candidates are inserted into the MVP list. The current coding block is predicted based on the MVP list. The derived MV candidates are MV predictors derived from either (one or more) motion vectors in a single reference frame based on inter prediction, or (one or more) composite motion vectors from multiple adjacent reference frames in a mixed inter prediction mode.

[0016] Examples of systems and equipment 1 is a block diagram illustrating a communication system 100 according to some embodiments. Communication system 100 includes a source device 102 and a number of electronic devices 120 (e.g., electronic devices 120-1 through 120-m) communicatively coupled to each other via one or more networks. In some embodiments, communication system 100 is a streaming system for use with video-enabled applications, such as, for example, video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0017] Source device 102 includes a video source 104 (e.g., a camera component or media storage) and an encoder component 106. In some embodiments, video source 104 is a digital camera (e.g., configured to create an uncompressed video sample stream). Encoder component 106 generates one or more encoded video bitstreams from the video stream. The video stream from video source 104 may be a larger amount of data compared to encoded video bitstream 108 generated by encoder component 106. Because encoded video bitstream 108 is a smaller amount of data (less data) compared to the video stream from the video source, encoded video bitstream 108 requires less bandwidth to transmit and less storage space to store compared to the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video data to network(s) 110).

[0018] One or more networks 110 represent any number of networks that carry information between source devices 102, server system 112, and / or electronic devices 120, including, for example, wireline and / or wireless communication networks. One or more networks 110 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet.

[0019] The one or more networks 110 include a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is or includes a streaming server (e.g., configured to store and / or distribute video content, such as an encoded video stream from a source device 102). The server system 112 includes a coder component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the coder component 114 includes an encoder component and / or a decoder component. In various embodiments, the coder component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the coder component 114 is configured to decode the encoded video bitstream 108 and re-encode the video data using a different encoding standard and / or methodology to generate encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings from the encoded video bitstream 108.

[0020] In some embodiments, server system 112 functions as a Media-Aware Network Element (MANE). For example, server system 112 may be configured to prune encoded video bitstream 108 to tailor potentially different bitstreams to one or more of electronic devices 120. In some embodiments, a MANE is provided separate from server system 112.

[0021] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, decoder component 122 is configured to decode encoded video data 116 to generate an output video stream that can be rendered on a display or other type of rendering device. In some embodiments, one or more of electronic devices 120 does not include a display component (e.g., is communicatively coupled to an external display device and / or includes media storage). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0022] Source device and / or electronic devices 120 may be referred to as “terminal devices” or “user devices.” In some embodiments, source device 102 and / or one or more of electronic devices 120 are instances of a server system, a personal computer, a portable device (e.g., a smartphone, tablet, or laptop), a wearable device, a videoconferencing device, and / or other types of electronic devices.

[0023] In an example operation of communication system 100, source device 102 transmits encoded video bitstream 108 to server system 112. For example, source device 102 may code a stream of pictures captured by the source device. Server system 112 may receive encoded video bitstream 108 and decode and / or encode encoded video bitstream 108 using coder component 114. For example, server system 112 may apply more optimal encoding to the video data for network transmission and / or storage. Server system 112 may transmit encoded video data 116 (e.g., one or more encoded video bitstreams) to one or more of electronic devices 120. Each electronic device 120 may decode encoded video data 116 to recover video pictures and, optionally, display them.

[0024] In some embodiments, the transmission is a one-way data transmission. The one-way data transmission may be used in media serving applications and the like. In some embodiments, the transmission is a two-way data transmission. The two-way data transmission may be used in video conferencing applications and the like. In some embodiments, the encoded video bitstream 108 and / or the encoded video data 116 are encoded and / or decoded according to any of the video coding / compression standards described herein, such as, for example, HEVC, VVC, and / or AV1.

[0025] 2A is a block diagram illustrating example elements of the encoder component 106 according to some embodiments. The encoder component 106 receives a source video sequence from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., a transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives a video sequence from a remote video source (e.g., a video source that is a component of a different device than the encoder component 106). The video source 104 may provide the source video sequence in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 Y CrCB, or RGB), and any suitable sampling structure (e.g., Y CrCb 4:2:0, or Y CrCb 4:4:4). In some embodiments, the video source 104 is a storage device that stores pre-captured / prepared video. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. Video data may be provided as a number of individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, with each pixel containing one or more samples, depending on the sampling structure, color space, etc. used. Those skilled in the art will readily understand the relationship between pixels and samples. The following discussion focuses on samples.

[0026] The encoder component 106 is configured to encode and / or compress pictures of a source video sequence into an encoded video sequence (643) in real-time or under other time constraints required by an application. Enforcing an appropriate encoding rate is one function of the controller 204. In some embodiments, the controller 204 controls and is recursively coupled to other functional units, as described below. Parameters set by the controller 204 may include rate control related parameters (picture skip, quantizer, and / or lambda values ​​for rate-distortion optimization techniques, picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. One skilled in the art can readily identify other functions of the controller 204 as they may be relevant for the encoder component 106 to be optimized for a particular system design.

[0027] In some embodiments, the encoder component 106 is configured to operate in a coding loop. In a simplified example, the coding loop includes a source coder 202 (e.g., responsible for creating symbols, e.g., a symbol stream, based on an input picture to be coded and reference picture(s)) and a (local) decoder 210. The decoder 210 reconstructs the symbols to create sample data in a similar manner to a (remote) decoder (when the compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory 208. Since the decoding of the symbol stream results in bit-accurate results that are independent of the decoder location (local or remote), the content in the reference picture memory 208 is also bit-accurate between the local and remote encoders. Thus, the prediction part of the encoder interprets the same sample values ​​as reference picture samples that the decoder interprets when using prediction during decoding. This principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is known to those skilled in the art.

[0028] The operation of the decoder 210 may be the same as that of a remote decoder, such as the decoder component 122, which is described in more detail below in connection with Figure 2B. However, with brief reference to Figure 2B, the entropy decoding portion of the decoder component 122, including the buffer memory 252 and the parser 254, may not be fully implemented in the local decoder 210 because symbols are available and the encoding / decoding of the symbols into an encoded video sequence by the entropy coder 214 and the parser 254 may be lossless.

[0029] At this point, it can be noted that the decoder techniques, except for parsing / entropy decoding, present in the decoder must necessarily be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosure focuses on the decoder operation. The description of the encoder techniques can be omitted, since they are the inverse of the decoder techniques, which are described in detail. Only in certain areas are further details necessary, which are provided below.

[0030] As part of its operation, the source coder 202 may perform motion compensated predictive coding, which is predictive coding of an input frame with reference to one or more previously coded frames from the video sequence designated as reference frames. Thus, the coding engine 212 codes differences between pixel blocks of the input frame and pixel blocks of the reference frame(s) that may be selected as the prediction reference(s) for the input frame. A controller 204 may manage the coding operations of the source coder 202, including, for example, setting parameters and subgroup parameters used to encode the video data.

[0031] The decoder 210 decodes the encoded video data of frames that may be designated as reference frames based on symbols created by the source coder 202. The operation of the coding engine 212 may advantageously be a lossy process. When the encoded video data is decoded in a video decoder (not shown in FIG. 2A), the reconstructed video sequence may be a replica of the source video sequence with some errors. The decoder 210 may replicate the decoding process that may be performed by a remote video decoder on the reference frames, causing the reconstructed reference frames to be stored in the reference picture memory 208. Thus, the encoder component 106 locally stores copies of reconstructed reference frames that have a common content with the reconstructed reference frames that will be obtained by the remote video decoder.

[0032] The predictor 206 may perform a predictive search for the coding engine 212. That is, for a new frame to be coded, the predictor 206 may search the reference picture memory 208 for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors or block shapes, that may serve as suitable predictive references for the new picture. The predictor 206 may operate on a pixel block by pixel block basis to find suitable predictive references. In some cases, as determined by the search results obtained by the predictor 206, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory 208.

[0033] The outputs of all the aforementioned functional units may be subjected to entropy coding in entropy coder 214. Entropy coder 214 converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable length coding, and / or arithmetic coding).

[0034] In some embodiments, the output of the entropy coder 214 is coupled to a transmitter. The transmitter may be configured to buffer the encoded video sequence(s) generated by the entropy coder 214 and prepare them for transmission over a communication channel 218. The communication channel 218 may be a hardware / software link to a storage device that stores the encoded video data. The transmitter may be configured to merge the encoded video data from the source coder 202 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown). In some embodiments, the transmitter may transmit additional data along with the encoded video. The source coder 202 may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data, such as redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, and the like.

[0035] The controller 204 may manage the operation of the encoder component 106. In coding, the controller 204 may assign to each coded picture a particular coded picture type, which may affect the coding technique applied to the respective picture. For example, a picture may be assigned as an intra picture (I picture), a predictive picture (P picture), or a bidirectionally predictive picture (B picture). An intra picture may be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow several different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of these variants of I pictures, as well as their respective uses and characteristics, and therefore will not be repeated here. A predictive picture may be coded and decoded using intra prediction or inter prediction, using at most one motion vector and reference index to predict the sample values ​​of each block. Bidirectionally predicted pictures can be encoded and decoded using intra- or inter-prediction, using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0036] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to their respective pictures. For example, blocks of I-pictures may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P-pictures may be coded non-predictively, or via spatial prediction, or via temporal prediction with reference to one previously coded reference picture. Blocks of B-pictures may be coded non-predictively, or via spatial prediction, or via temporal prediction with reference to one or two previously coded reference pictures.

[0037] A video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) uses spatial correlation within a given picture, while inter-picture prediction uses correlation (temporal or other) between pictures. In one example, a particular picture being coded / decoded, called a current picture, is partitioned into multiple blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture may be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0038] Encoder component 106 may perform coding operations according to a given video coding technique or standard, such as any described herein. In so doing, encoder component 106 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to a syntax defined by the video coding technique or standard used.

[0039] 2B is a block diagram illustrating example elements of the decoder component 122 according to some embodiments. The decoder component 122 of FIG. 2B is coupled to the channel 218 and the display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to the loop filter 256 and configured to transmit data to the display 124 (e.g., via a wired or wireless connection).

[0040] In some embodiments, the decoder component 122 includes a receiver coupled to the channel 218 and configured to receive data from the channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more encoded video sequences to be decoded by the decoder component 122. In some embodiments, the decoding of each encoded video sequence is independent of the other encoded video sequences. Each encoded video sequence may be received from the channel 218, which may be a hardware / software link to a storage device that stores the encoded video data. The receiver may receive the encoded video data along with other data, such as encoded audio data and / or auxiliary data streams. The other data may be forwarded to their respective usage entities (not shown). The receiver may separate the encoded video sequence from the other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the decoder component 122 to decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, and so on.

[0041] According to some embodiments, the decoder component 122 includes a buffer memory 252, a parser 254 (sometimes referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra-picture prediction unit 262, a motion compensated prediction unit 260, an aggregator 268, a loop filter unit 256, a reference picture memory 266, and a current picture memory 264. In some embodiments, the decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuitry. In some embodiments, the decoder component 122 is implemented at least partially in software.

[0042] The buffer memory 252 is coupled between the channel 218 and the parser 254 (e.g., to handle network jitter). In some embodiments, the buffer memory 252 is separate from the decoder component 122. In some embodiments, a separate buffer memory is provided between the output of the channel 218 and the decoder component 122. In some embodiments, in addition to the buffer memory 252 internal to the decoder component 122 (e.g., configured to handle playback timing), a separate buffer memory is provided external to the decoder component 122 (e.g., to handle network jitter). When receiving data from a store / forward device of sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory 252 may not be required or can be made small. For use over a best-effort packet network, such as the Internet, the buffer memory 252 may be required and can be relatively large, advantageously of an adaptable size, and implemented at least in part in an operating system or similar element (not shown) external to the decoder component 122.

[0043] Parser 254 is configured to reconstruct symbols 270 from the coded video sequence. The symbols may include, for example, information used to manage the operation of decoder component 122 and / or information to control a rendering device, such as display 124. The control information for the rendering device(s) may be in the form of, for example, a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). Parser 254 parses (entropy decodes) the coded video sequence. The coding of the coded video sequence may be according to a video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context dependency, etc. Parser 254 may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. Parser 254 may also extract information from the coded video sequence information, such as transform coefficients, quantization parameter values, motion vectors, etc.

[0044] The reconstruction of symbols 270 may involve a number of different units, depending on the type of video picture or portion thereof being coded and other factors (e.g., inter-picture and intra-picture, inter-block and intra-block, etc.). Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by parser 254. The flow of such subgroup control information between parser 254 and the following units is not shown for clarity.

[0045] Beyond the functional blocks already described, the decoder component 122 can be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating under commercial constraints, many of these units may closely interact with each other and be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, we will maintain the conceptual division into the following functional units:

[0046] The scalar / inverse transform unit 258 receives the quantized transform coefficients and control information (e.g., which transform to use, block size, quantization factor, and / or quantization scaling matrix) as symbol(s) 270 from the parser 254. The scalar / inverse transform unit 258 may output blocks containing sample values ​​that may be input to an aggregator 268.

[0047] In some cases, the output samples of the scaler / inverse transform unit 258 relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed part of the current picture. Such prediction information can be provided by the intra-picture prediction unit 262. The intra-picture prediction unit 262 may generate a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from the current (partially reconstructed) picture from the current picture memory 264. The aggregator 268 may add, on a sample-by-sample basis, the prediction information generated by the intra-picture prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258.

[0048] In other cases, the output samples of the scaler / inverse transform unit 258 relate to a block that may be inter-coded and motion-compensated. In such a case, the motion compensated prediction unit 260 may access the reference picture memory 266 to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols 270 related to the block, these samples may be added by the aggregator 268 to the output of the scaler / inverse transform unit 258 (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory 266 from which the motion compensated prediction unit 260 fetches the prediction samples may be controlled by a motion vector. The motion vector may be available to the motion compensated prediction unit 260 in the form of the symbols 270, which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory 266 when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.

[0049] The output samples of aggregator 268 may be subjected to various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filter techniques, which are controlled by parameters included in the coded video bitstream and made available to loop filter unit 256 as symbols 270 from parser 254, but may also be responsive to meta-information obtained during decoding of previous portions of the coded pictures or coded video sequences (in decoding order), as well as to previously reconstructed loop filtered sample values.

[0050] The output of the loop filter unit 256 may be a sample stream that can be output to a rendering device, such as the display 124, which may also be stored in the reference picture memory 266 for use in future inter-picture prediction.

[0051] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future predictions. Once a coded picture is fully reconstructed and identified as a reference picture (e.g., by parser 254), the current reference picture can become part of reference picture memory 266, and a new current picture memory can be reallocated before starting reconstruction of the next coded picture.

[0052] The decoder component 122 may perform decoding operations according to a given video compression technique, which may be documented in a standard, such as any of the standards described herein. The encoded video sequence may conform to the syntax defined by the video compression technique or standard used, in the sense of adhering to the syntax of the video compression technique or standard defined in the video compression technique document or standard, and specifically in the profile documents therein. Also, for compliance with some video compression techniques or standards, the complexity of the encoded video sequence may be within a range defined by the level of the video compression technique or standard. In some cases, the level constrains the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may in some cases be further constrained through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled within the encoded video sequence.

[0053] 3 is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes a control circuit 302, one or more network interfaces 304, a memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuit 302 includes one or more processors (e.g., a CPU, a GPU, and / or a DPU). In some embodiments, the control circuit includes one or more field programmable gate arrays (FPGAs), hardware accelerators, and / or one or more integrated circuits (e.g., application specific integrated circuits).

[0054] The network interface(s) 304 may be configured to interface with one or more communication networks (e.g., wireless, wired, and / or optical networks). The communication networks may be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of communication networks include local area networks, such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CANbus, etc. Such communications may be unidirectional receive only (e.g., broadcast TV), unidirectional transmit only (e.g., CANbus to a particular CANbus device), or bidirectional (e.g., to another computer system using a local or wide area digital network). Such communications may include communications to one or more cloud computing networks.

[0055] The user interface 306 includes one or more output devices 308 and / or one or more input devices 310. The input device(s) 310 may include one or more of a keyboard, a mouse, a trackpad, a touch screen, a data glove, a joystick, a microphone, a scanner, a camera, or the like. The output device(s) 308 may include one or more of an audio output device (e.g., a speaker), a visual output device (e.g., a display or monitor), or the like.

[0056] Memory 314 may include high-speed random access memory (such as, for example, DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices) and / or non-volatile memory (such as, for example, one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices). Memory 314 optionally includes one or more storage devices located remotely from control circuitry 302. Memory 314, or alternatively a non-volatile solid-state memory device within memory 314, comprises a non-transitory computer-readable storage medium. In some embodiments, memory 314, or the non-transitory computer-readable storage medium of memory 314, stores the following programs, modules, instructions, and data structures, or a subset or superset thereof: · an operating system 316 that contains procedures for handling various basic system services and for performing hardware dependent tasks; · a network communications module 318 used to connect the server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections); A coding module 320 that performs various functions related to encoding and / or decoding data, such as video data. In some embodiments, the coding module 320 is an instance of the coder component 114. The coding module 320 may include, but is not limited to, one or more of the following: A decoding module 322 that performs various functions related to decoding the encoded data, such as those described above with respect to the decoder component 122; and An encoding module 340 that performs various functions related to encoding data, such as those described above with respect to the encoder component 106; and A picture memory 352 that stores pictures and picture data, e.g., for use with coding module 320. In some embodiments, the picture memory 352 includes one or more of the reference picture memory 208, the buffer memory 252, the current picture memory 264, and the reference picture memory 266.

[0057] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions described above with respect to the parser 254), a transform module 326 (e.g., configured to perform various functions described above with respect to the scaler / inverse transform unit 258), a prediction module 328 (e.g., configured to perform various functions described above with respect to the motion compensation prediction unit 260 and / or the intra-picture prediction unit 262), and a filter module 330 (e.g., configured to perform various functions described above with respect to the loop filter 256).

[0058] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described with respect to the source coder 202 and / or the coding engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to the predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 include a subset of the modules shown in Figure 3. For example, a shared prediction module is used by both the decoding module 322 and the encoding module 340.

[0059] Each of the above-identified modules stored in memory 314 corresponds to an instruction set for performing functions described herein. The above-identified modules (e.g., instruction sets) need not be implemented as separate software programs, procedures, or modules, and thus, in various embodiments, various subsets of these modules may be combined or otherwise rearranged. For example, coding module 320 optionally does not include separate decoding and encoding modules, but instead uses the same set of modules to perform both sets of functions. In some embodiments, memory 314 stores a subset of the above-identified modules and data structures. In some embodiments, memory 314 stores additional modules and data structures not described above, such as, for example, an audio processing module.

[0060] In some embodiments, the server system 112 includes a web or HyperText Transfer Protocol (HTTP) server, a File Transfer Protocol (FTP) server, and web pages and applications implemented using Common Gateway Interface (CGI) scripts, PHP Hypertext Preprocessor (PHP), Active Server Pages (ASP), Hypertext Markup Language (HTML), Extensible Markup Language (XML), Java, JavaScript, Asynchronous JavaScript+XML (AJAX), XHP, Javelin, Wireless Universal Resource Files (WURFL), and the like.

[0061] FIG. 3 illustrates a server system 112 according to some embodiments, but is intended as a functional illustration of various features that may be present in one or more server systems, rather than a structural schematic of the embodiments described herein. In practice, items shown separately may be combined and some items may be separated, as will be recognized by those skilled in the art. For example, some items shown separately in FIG. 3 may be implemented on a single server, and a single item may be implemented by one or more servers. The actual number of servers used to implement the server system 112, and how features are allocated among them, will vary from implementation to implementation, and optionally depend in part on the amount of data traffic the server system handles during peak and average usage periods.

[0062] Block Partitioning

[0063] FIG. 4 illustrates block partitioning according to some embodiments. The top half of FIG. 4 illustrates block partitioning in VP9, ​​which uses a 4-way partition tree starting from the 64×64 level down to the 4×4 level, with some additional constraints for blocks 8×8 and below. Partitions designated as “R” are referred to as recursive in that the same partition tree is repeated at lower scales until the lowest 4×4 level is reached. The bottom half of FIG. 4 illustrates block partitioning in AV1, which not only extends the partition tree to a 10-way structure, but also increases the maximum size (called a superblock in VP9 / AV1 terminology) to start at 128×128. The 10-way structure includes 4:1 / 1:4 rectangular partitions, none of which can be further subdivided. AV1 adds more flexibility to the use of partitions below the 8×8 level. For example, 2×2 chroma inter prediction is possible in certain cases.

[0064] In some embodiments, block partitioning includes dividing coding tree units (CTUs) into coding units (CUs) by using a quadtree structure called a coding tree to adapt to different local characteristics. The decision of whether a picture area should be coded using inter-picture (temporal) prediction or intra-picture (spatial) prediction is made at the CU level. Each CU can be further divided into one, two, or four prediction units (PUs) according to the PU partition type. Within one PU, the same prediction process is applied, and related information is transmitted to the decoder on a PU basis. After obtaining the residual block by applying the prediction process based on the PU partition type, the CU can be partitioned into transform units (TUs) according to another quadtree structure such as a coding tree for the CU. In some embodiments, the block is partitioned using multiple partition concepts including CU, PU, ​​and TU. In some embodiments, the CU or TU is only square in shape, while the PU can be square or rectangular in shape for the inter-prediction block. In some embodiments, a coding block may be further divided into four square sub-blocks, and a transform is performed on each sub-block, i.e., TU. Each TU may be further recursively divided (using quadtree partitioning) into smaller TUs, which are called residual quadtrees (RQTs).

[0065] In some embodiments, the block partitioning includes using a quadtree (QT) + binary tree (BT) structure (also known as a QTBT structure). The QTBT structure removes the concept of multi-partition type, i.e., QTBT removes the separation of the concepts of CU, PU, ​​and TU, and supports more flexibility for CU partition shapes. In the QTBT block structure, a CU can have either a square or rectangular shape. This is shown in FIG. 5A. A coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a binary tree structure. There are two partition types in the binary tree partition: symmetric horizontal partition and symmetric vertical partition. The binary tree leaf nodes are called coding units (CUs), and their segmentation is used for prediction and transform processes without any further partitioning. This means that in the QTBT coding block structure, CUs, PUs, and TUs have the same block size. In some embodiments, a CU includes coding blocks (CBs) of different color components. For example, for P and B slices in 4:2:0 chroma format, one CU includes one luma CB and two chroma CBs, and sometimes consists of a single component CB. As another example, for an I slice, one CU includes only one luma CB or two chroma CBs.

[0066] In some embodiments, the following parameters are defined for the QTBT partitioning scheme: (i) CTU size: quad-tree root node size (e.g., the same concept as in HEVC); (ii) MinQTSize: minimum allowed quad-tree leaf node size; (iii) MaxBTSize: maximum allowed binary tree root node size; (iv) MaxBTDepth: maximum allowed binary tree depth; and (v) MinBTSize: minimum allowed binary tree leaf node size.

[0067] In one example of a QTBT partitioning structure, the CTU size is set as 128x128 luma samples with two corresponding 64x64 blocks of chroma samples, MinQTSize is set as 16x16, MaxBTSize is set as 64x64, MinBTSize (for both width and height) is set as 4x4, and MaxBTDepth is set as 4. First, a quadtree partition is applied to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes can have sizes from 16x16 (MinQTSize) to 128x128 (CTU size). If the leaf quadtree node is 128x128, it will not be further partitioned by the binary tree since the size exceeds MaxBTSize (64x64). Otherwise, the leaf quadtree node can be further partitioned by the binary tree. Therefore, the quadtree leaf node is also the root node for the binary tree, which has the binary tree depth as 0. Once the binary tree depth reaches MaxBTDepth (4), no further splits are considered. Once a binary tree node has a width equal to MinBTSize (4), no further horizontal splits are considered. Similarly, once a binary tree node has a height equal to MinBTSize, no further vertical splits are considered. The leaf node of the binary tree is further processed by the prediction and transform process without further partitioning. In JEM, the maximum CTU size is 256x256 luma samples.

[0068] Figure 5A shows an example of partitioning a block 502 using QTBT. Figure 5B shows the corresponding tree representation 504. The solid lines indicate quadtree partitioning and the dotted lines indicate binary tree partitioning. At each partition of a node (e.g., a non-leaf node) of the binary tree, one flag is signaled to indicate which partition type (e.g., horizontal or vertical) is used, with 0 indicating a horizontal partition and 1 indicating a vertical partition. In quadtree partitioning, there is no need to indicate the partition type, because it always partitions the block both horizontally and vertically to generate four sub-blocks of equal size.

[0069] In some embodiments, the QTBT scheme supports the flexibility of luma and chroma having separate QTBT structures. Currently, in P and B slices, the luma CTB and chroma CTB of one CTU share the same QTBT structure. However, in an I slice, the luma CTB is partitioned into CUs by a QTBT structure, and the chroma CTB is partitioned into chroma CUs by another QTBT structure. This means that a CU in an I slice is composed of a coding block of a luma component or a coding block of two chroma components, and a CU in a P or B slice is composed of coding blocks of all three color components.

[0070] In some embodiments, inter prediction for small blocks is restricted to reduce memory accesses for motion compensation, resulting in bi-prediction not being supported for 4x8 and 8x4 blocks, and inter prediction not being supported for 4x4 blocks. In some embodiments, these restrictions are removed in QTBT.

[0071] In some embodiments, block partitioning is implemented using a ternary tree (triple tree; TT) (also known as a multi-type tree (MTT) structure). FIGS. 6A and 6B show MTT structures 602 and 604 according to some embodiments. The MTT structure adds horizontal center-side triple trees and vertical center-side triple trees in addition to the QTBT. Benefits of ternary tree partitioning include: (i) it is complementary to quad-tree and binary tree partitioning, and can capture objects located at block centers, whereas quad-tree and binary trees always split along block centers, and (ii) the width and height of the proposed ternary tree partitions are always powers of two, and therefore no additional transformations are required.

[0072] The design of two-level trees is primarily motivated by reduced complexity. Theoretically, the complexity of traversing a tree is T D where T represents the number of split types and D is the depth of the tree.

[0073] Merge Mode with Motion Vector Difference (MMVD)

[0074] In merge mode, the implicitly derived motion information is directly used to generate the prediction samples for the current CU. In merge mode with motion vector differences (MMVD), the MMVD flag can be signaled immediately after sending the skip and merge flags to specify whether the MMVD mode is used for the CU.

[0075] In MMVD, after a merge candidate is selected, it can be further refined by signaled motion vector differential (MVD) information (one or more). The signaled information may include a merge candidate flag, an index specifying the magnitude of motion, and an index to indicate the motion direction. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. A merge candidate flag may be signaled to specify which one is used.

[0076] The distance index defines the motion magnitude information and indicates a predetermined offset from the starting point. As shown in Figure 7, the offset can be added to either the horizontal or vertical component of the starting motion vector (MV). The relationship between the distance index and the predetermined offset can be defined in Table 1. [Table 1]

[0077] The direction index represents the direction of the MVD relative to the starting point. The direction index can represent four directions, as shown in Table 2. [Table 2]

[0078] In some embodiments, the meaning of the sign of MVD can change according to the information of the start MV. When the start MV is a unidirectionally predicted MV or a bidirectionally predicted MV with both lists pointing to the same side of the current picture (e.g., the picture order counts (POC) of the two references are both greater than the POC of the current picture, or both less than the POC of the current picture), the sign in Table 2 specifies the sign of the MV offset added to the start MV. When the start MV is a bidirectionally predicted MV and the two MVs point to different sides of the current picture (e.g., the POC of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), and the difference in POC in list 0 is greater than the difference in POC in list 1, the sign in Table 1 specifies the sign of the MV offset added to the list 0 MV component of the start MV, and the sign for list 1 MV has the opposite value. Otherwise, if the POC difference in List 1 is greater than List 0, then the sign in Table 2 specifies the sign of the MV offset that is added to the List 1 MV component of the Start MV, and the sign for the List 0 MV has the opposite value.

[0079] In some embodiments, the MVD is scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is required. Otherwise, if the difference in POC in list 0 is greater than that of list 1, the MVD of list 1 is scaled. If the POC difference of L1 is greater than L0, the MVD of list 0 is scaled as well. If the starting MV is unidirectionally predicted, the MVD is added to the available MV.

[0080] Symmetric Motion Vector Differential (MVD) Coding

[0081] In symmetric MVD mode, motion information including reference picture indexes for both list 0 and list 1 and the MVD for list 1 is derived without being signaled.

[0082] In some embodiments, the decoding process for symmetric MVD mode at slice level is as follows: The variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows: (i) if mvd_l1_zero_flag is 1, then BiDirPredFlag is set equal to 0. (ii) otherwise, if the closest reference picture in list 0 and the closest reference picture in list 1 form a forward and backward pair of reference pictures or a backward and forward pair of reference pictures, then BiDirPredFlag is set to 1 and both the list 0 reference picture and the list 1 reference picture are short-term reference pictures. (iii) otherwise, BiDirPredFlag is set to 0.

[0083] In some embodiments, the decoding process for symmetric MVD mode at CU level is as follows: If a CU is bi-predictively coded and BiDirPredFlag is equal to 1, then a symmetric mode flag is explicitly signaled indicating whether symmetric mode is used or not. When the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 are explicitly signaled. The reference indexes of list0 and list1 are set equal to the pair of reference pictures, respectively. MVD1 is set equal to (-MVD0).

[0084] Intermode Coding

[0085] In some embodiments, for blocks coded in interframes, if the mode of the current block is an inter-coding mode rather than a skip mode, another flag is signaled to indicate whether a single or mixed reference mode is used for the current block. In a single reference mode, a predictive block may be generated by one motion vector. In a mixed reference mode (e.g., the current block has two reference pictures), a predictive block is generated by a weighted average of two predictive blocks derived from two motion vectors.

[0086] In some embodiments, the single reference mode is signaled as follows: · NEARMV - use one of the motion vector predictors (MVP) in the list indicated by the DRL (Dynamic Reference List) index; · NEWMV - use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference and apply a delta to the MVP; ·GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0087] In some embodiments, the following modes are signaled in the mixed reference mode: · NEAR_NEARMV - use one of the motion vector predictors (MVP) in the list signaled by the DRL index; · NEAR_NEWMV - use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference and send a delta MV for the second MV; · NEW_NEARMV - use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference and send a delta MV for the first MV; · NEW_NEWMV - use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as reference and send delta MVs for both MVs; · GLOBAL_GLOBALMV - Use MV from each reference based on frame-level global motion parameters.

[0088] Motion Vector Differential (MVD) Coding

[0089] AV1 allows for 1 / 8 pixel motion vector precision (or accuracy). To signal motion vector differentials in reference frame list 0 or list 1, the following syntax is used: mv_joint specifies which components of the motion vector differential are non-zero - 0 indicates that there is no non-zero MVD along either the horizontal or vertical direction - 1 indicates that there is non-zero MVD only along the horizontal direction - 2 indicates that there is a non-zero MVD only along the vertical direction - 3 indicates the presence of non-zero MVD along both the horizontal and vertical directions mv_sign specifies whether the motion vector differential is positive or negative mv_class specifies the class of the motion vector difference. As shown in Table 3, a higher class means that the motion vector difference has a larger magnitude. Hhmv_bit specifies the integer part of the offset between the motion vector differential and the starting magnitude of each MV class. mv_fr specifies the first two fractional bits of the motion vector differential · mv_hp specifies the third fractional bit of the motion vector differential. [Table 3]

[0090] Adaptive MVD Resolution

[0091] For NEW_NEARMV and NEAR_NEWMV modes, the precision of the MVD depends on the associated class and the size of the MVD. For example, fractional MVD is allowed only if the size of the MVD is less than or equal to 1 pixel. As another example, when the value of the associated MV class is MV_CLASS_1 or more, only one MVD value is allowed, and the MVD value in each MV class is derived as 4, 8, 16, 32, 64 for MV class 1 (MV_CLASS_1), 2 (MV_CLASS_2), 3 (MV_CLASS_3), 4 (MV_CLASS_4), or 5 (MV_CLASS_5). As another example, if the current block is coded as NEW_NEARMV or NEAR_NEWMV mode, one context is used to signal mv_joint or mv_class. If this is not the case (eg, the current block is not coded as NEW_NearMV), then a separate context is used to signal mv_joint or mv_class.

[0092] Table 4 shows the allowable MVD values ​​for each MV class. [Table 4]

[0093] In some embodiments, an inter-coding mode called JOINT_NEWMV is applied to indicate whether the MVDs for the two reference lists are signaled together. When the inter-prediction mode is equal to JOINT_NEWMV mode, the MVDs for reference list 0 and reference list 1 are signaled together. Then, only one MVD called joint_mvd can be signaled and sent to the decoder, and the delta MVs for reference list 0 and reference list 1 are derived from joint_mvd.

[0094] In some embodiments, the JOINT_NEWMV mode is signaled along with the NEAR_NEARMV mode, the NEAR_NEWMV mode, the NEW_NEARMV mode, the NEW_NEWMV mode, and the GLOBAL_GLOBALMV mode. No additional context is added.

[0095] In some embodiments, when JOINT_NEWMV mode is signaled and the POC distances between two reference frames and the current frame are different, the MVD is scaled for reference list 0 or reference list 1 based on the POC distance. The distance between reference frame list 0 and the current frame is denoted as td0, and the distance between reference frame list 1 and the current frame is denoted as td1. If td0 is equal to or greater than td1, then joint_mvd is used directly for reference list 0, and the mvd for reference list 1 is calculated from joint_mvd according to Equation (1):

number

[0096] Otherwise, if td1 is greater than or equal to td0, then joint_mvd is used directly for reference list 1, and the mvd for reference list 0 is obtained from joint_mvd by equation (2):

number

[0097] In some embodiments, for the single reference case, an inter-coding mode called AMVDMV is added. When the AMVDMV mode is selected, it indicates that AMVD is applied to signal MVD. Under the JOINT_NEWMV mode, a flag (amvd_flag) is added to indicate whether AMVD is applied to the joint MVD coding mode. When the adaptive MVD resolution is applied to the joint MVD coding mode, it is called joint AMVD coding, and the MVDs for two reference frames are signaled together, and the accuracy of the MVD is implicitly determined by the size of the MVD. Otherwise, the MVDs for two (or more) reference frames are signaled together, and conventional MVD coding is applied.

[0098] In some embodiments, Adaptive Motion Vector Resolution (AMVR) supports seven MV precisions (8, 4, 2, 1, 1 / 2, 1 / 4, 1 / 8). For each prediction block, the AOM Video Model (AVM) encoder searches all supported precision values ​​and signals the best precision to the decoder. In some embodiments, to reduce encoder execution time, two precision sets are supported. Each precision set contains four predefined precisions. At the frame level, the precision set is adaptively selected based on the maximum precision value for that frame. In some embodiments, the maximum precision is signaled in the frame header.

[0099] Table 5 summarizes the supported precision values ​​based on frame-level maximum precision. [Table 5]

[0100] In some embodiments, a frame level flag indicates whether the MV of a frame includes sub-pel precision. AMVR is enabled only if the value of the cur_frame_force_integer_mv flag is 0. In AMVR, if the precision of a block is less than the maximum precision, the motion model and the interpolation filter are not signaled. If the precision of a block is less than the maximum precision, the motion mode is estimated as translational motion and the interpolation filter is estimated as a REGULAR interpolation filter. Similarly, if the precision of a block is either 4-pel or 8-pel, the inter-intra mode is not signaled and is estimated to be 0.

[0101] Motion Vector Predictor (MVP) List

[0102] A spatial motion vector predictor (SMVP) (e.g., both adjacent SMVP and non-adjacent SMVP), a temporal MV predictor (TMVP), extra MV candidates in AV1, and additional derived MVPs and reference bank MVPs are further added in the AVM design. A stack with a fixed size is generated at both the encoder and decoder ends to store the MVP list.

[0103] In some embodiments, the SMVP is derived from spatial neighboring blocks. The spatial neighboring blocks include adjacent spatial neighboring blocks that are direct neighbors to the top and left of the current block, and non-adjacent spatial neighboring blocks that are not directly adjacent to the current block. An exemplary set of spatial neighboring blocks for a luma block is shown in FIG. 8, where each spatial neighboring block is an 8×8 block. The spatial neighboring blocks are examined to find one or more MVs associated with the same reference frame index as the current block. For the current block, the search order of spatially neighboring 8×8 luma blocks is as shown by numbers 1-8 in FIG. 8: 1.The adjacent rows above are checked from left to right; 2. The left adjacent column is checked from top to bottom; 3. The upper right adjacent block is checked; 4. The blocks adjacent to the top left block are checked; 5.The first non-adjacent row above is checked from left to right; 6. The first non-adjacent column on the left is checked from top to bottom; 7.The second non-adjacent row above is checked from left to right; 8. The next non-adjacent row to the left is checked from top to bottom.

[0104] In some embodiments, adjacent candidates (candidates 1-3 in FIG. 8) are inserted into the MV predictor list before TMVP, and non-adjacent candidates (also known as outer candidates, i.e., candidates 4-8 in FIG. 8) are put into the MV predictor list after TMVP. All SMVP candidates should have the same reference picture as the current block. If the current block has a single reference picture and the MVP candidate has a single reference picture and this reference picture is the same as the reference picture of the current block, or if the MVP candidate has a composite reference picture (e.g., two reference pictures) and one of the reference pictures is the same as the reference picture of the current block, then the MVP candidate will be put into the MV predictor list. If the current block has two reference pictures, then the MVP candidate will be put into the MV predictor list only if the MVP candidate has two reference pictures and these two reference pictures are the same as the reference picture of the current block.

[0105] In some embodiments, to reduce line memory, the outer candidates, represented by lines 5 and 7 in FIG. 8, are further removed from the non-adjacent candidates.

[0106] In some embodiments, TMVP can be derived using collocated blocks in a reference frame. To generate TMVP, first, the MV of the reference frame is stored together with the reference index associated with each reference frame. Then, for each 8x8 block of the current frame, the MV of the reference frame whose trajectory passes through the 8x8 block is identified and stored together with the reference frame index in the temporal MV buffer. In inter prediction using a single reference frame, regardless of whether the reference frame is a forward reference frame or a backward reference frame, the MV is stored in 8x8 units to perform temporal motion vector prediction of future frames. In hybrid inter prediction, only forward MV is stored in 8x8 units to perform temporal motion vector prediction of future frames.

[0107] FIG. 9 shows a motion vector MVref 910 corresponding to reference frame 1 (R1) 908, pointing from R1 to reference frame 902 of R1. In doing so, MVref 910 passes through an 8×8 block 914 of the current frame 906. MVref 910 is stored in a temporal MV buffer associated with the 8×8 block 914. In the motion projection process to derive the TMVP, the reference frames are scanned in a predefined order, namely, LAST_FRAME, BWDREF_FRAME, ALTREF_FRAME, ALTREF2_FRAME, and LAST2_FRAME. An MV from a reference frame with a higher index (in the scan order) does not replace a previously identified MV assigned by a reference frame with a lower index (in the scan order). Given a given block coordinate, the associated MV stored in the temporal MV buffer is identified and projected onto the current block to derive a temporal MV predictor (e.g., MV0 912) pointing from the current block 914 to its reference frame 904.

[0108] Figure 10 shows predetermined block positions for deriving a temporal MV predictor for a 16x16 block, according to some embodiments. Up to seven blocks (e.g., blocks B0-B6) are checked for valid temporal MV predictors. The temporal MV predictors are checked after adjacent spatial MV predictors but before non-adjacent spatial MV predictors.

[0109] To derive the MV predictor, all spatial and temporal MV candidates are pooled and each predictor is assigned a weight determined during the scanning of spatial and temporal neighboring blocks. Based on the associated weights, the candidates are sorted and ranked, and up to four candidates are identified and added to the MV predictor list. This list of MV predictors is also called the Dynamic Reference List (DRL) and is further used in the dynamic MV prediction mode, as described in the next subsection.

[0110] Further exploration of Extra MVP candidates In some embodiments, if the MVP list is still not full, an additional search is performed to use extra MVP candidates to fill the MVP list, including, for example, global MVs, zero MVs, and combined composite MVs without scaling.

[0111] MVP candidate sorting process In some embodiments, the adjacent SMVP candidates, TMVP candidates, and non-adjacent SMVP candidates that are added to the MVP list are reordered. In some embodiments, the reordering process is based on the weight of each candidate. The candidate weight is pre-determined depending on the overlapping area of ​​the current block and the candidate block. In some embodiments, the weighting of the outer / non-adjacent candidates and the TMVP candidates are not considered during the reordering process, which means that the reordering process only affects the adjacent candidates.

[0112] Derived MVP candidates In some embodiments, the derived MVP candidates include both single reference picture and mixed mode derived MVPs.

[0113] Single Inter Prediction In some embodiments, if a neighboring block's reference frame is different from the current block's reference frame but in the same direction, a temporal scaling algorithm can be used to scale its MV to its reference frame to form an MVP for the current block's motion vector. Figure 11 shows an example in which MV1 1102 from a neighboring block 1106 can be used to derive an MVP for a motion vector MV0 1104 of a current block 1108 using temporal scaling.

[0114] Combined Inter Prediction In some embodiments, to derive the MVP of the current block, composite MVs from different neighboring blocks are used, but the reference frame of the composite MVs must be the same as the current block. As shown in Fig. 12, the composite MVs consisting of MV2 1202 and MV3 1206 have the same reference frame as the current block 1210, but they are from different neighboring blocks. Fig. 12 shows a motion vector MV0 1204 of the current block 1210 in the current frame F4, which points to another reference block 1212 in frame F3, and a motion vector MV1 1208 of the current block 1210, which points (e.g., in the other direction) to another reference block 1214 in frame F5.

[0115] FIG. 13 illustrates a reference MV candidate bank 1302 that may be part of or communicatively connected to a computing system (e.g., server system 112, source device 102, or electronic device 120) according to some embodiments. In some embodiments, each buffer corresponds to a unique reference frame type that corresponds to a single reference frame or a pair of reference frames covering single and multiple inter modes, respectively. In some embodiments, all buffers are the same size. In some embodiments, when a new MV is added to a full buffer, existing MVs are evicted to make room for the new MV.

[0116] A coding block can collect reference MV candidates by referring to the MV candidate bank in addition to those obtained in the reference MV list generation. After coding a superblock, the MV candidate bank 1302 is updated with the MVs used by the coding block of the superblock.

[0117] In some embodiments, each tile has an independent MV reference bank that is used by all superblocks in the tile. At the beginning of encoding each tile, the corresponding bank is emptied. Then, during coding each superblock in that tile, MVs from the bank can be used as MV reference candidates. At the end of encoding a superblock, the bank is updated.

[0118] FIG. 13 illustrates a superblock-based reference MV candidate bank 1302 update process according to some embodiments. After a superblock is coded, the first, e.g., (up to 64) candidate MVs used by coding blocks in the superblock are added to the reference MV candidate bank 1302. In some embodiments, a pruning process is also included during this update. In some embodiments, after scanning for reference MV candidates, if there are free slots in the candidate list, the codec algorithm can refer to the MV candidate bank 1302 (in the buffer of the matching reference frame type) for further MV candidates. Starting from the end of the buffer to the beginning, the MVs in the bank buffer are added to the candidate list if they are not already in the list.

[0119] 14 illustrates a process 1400 for building a motion vector predictor (MVP) list according to some embodiments. In some embodiments, the MVP list is built in the following order with complete pruning: In some embodiments, the process starts 1402 by inserting adjacent SMVPs 1404. The process includes a reordering step 1406 for existing candidates, followed by TMVPs 1408, non-adjacent SMVPs 1410, derived MVP candidates 1412, extra MVP candidates 1414, and candidates 1416 from a reference MV candidate bank (e.g., reference MV candidate bank 1302).

[0120] 15 is a flow diagram illustrating a method 1500 for decoding a current coding block according to some embodiments. Method 1500 may be performed in a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and memory storing instructions for execution by the control circuitry. In some embodiments, method 1500 is performed by executing instructions stored in a memory (e.g., memory 314) of the computing system.

[0121] The system retrieves one or more motion vector (MV) candidates from a reference MV bank (1502). In accordance with a determination that a first condition is met, the system inserts the one or more MV candidates from the reference MV bank into a motion vector prediction (MVP) list associated with a current coding block after the derived MV candidate is inserted into the MVP list (1504). In accordance with a determination that a first condition is not met, the system inserts the one or more MV candidates from the reference MV bank into an MVP list associated with a current coding block before the derived MV candidate is inserted into the MVP list (1506). The system predicts the current coding block based on the MVP list (1508).

[0122] (A1) In one aspect, some embodiments include a method (e.g., method 1500) for decoding a current coding block. In some embodiments, the method is implemented in a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is implemented in a coding module (e.g., coding module 320). The method includes retrieving one or more motion vector (MV) candidates from a reference MV bank. The method includes: In some embodiments, candidates from a reference candidate bank (e.g., reference MV candidate bank 1302) are inserted into the MVP list before derived candidates, so that the MVP list is constructed in the following order: adjacent SMVPs, sorting process for existing candidates, TMVP candidates, non-adjacent SMVP candidates, candidates from the reference MV candidate bank, derived candidates, and extra candidates.

[0123] In some embodiments, the derived MV candidates are MV predictors derived from either one or more motion vectors in a single reference frame based on an inter prediction mode, or a composite motion vector from multiple adjacent reference frames in a composite inter prediction mode.

[0124] In some embodiments, candidates from the reference candidate bank are conditionally inserted into the MVP list before derived candidates, e.g., if a condition is met, the original MVP list order is used, otherwise, if the condition is not met, a new MVP list with a different order or content is constructed.

[0125] In some embodiments, candidates from the reference candidate bank are conditionally inserted into the MVP list before the derived candidates. For example, if one condition is met, the original MVP list order is used; otherwise (e.g., if the condition is not met), the MVP list is constructed in the following order: adjacent SMVP, the sorting process for existing candidates, TMVP candidates, non - adjacent SMVP candidates, candidates from the reference MV candidate bank, derived candidates, and extra candidates.

[0126] In some embodiments, if the condition of width ≧ N and / or height ≧ N is met (e.g., N can be equal to, but not limited to, 8, 16, 32, or 64 luminance samples), the original MVP list order is used (e.g., candidates from the reference MV candidate bank are inserted at the end of the MVP list); otherwise, if width < N or height < N, candidates from the reference MV candidate bank are inserted before the derived candidates.

[0127] In some embodiments, if the condition of width ≧ N and / or height ≧ N is met (e.g., N can be equal to, but not limited to, 8, 16, 32, or 64 luminance samples), the original MVP list order is used (e.g., candidates from the reference MV candidate bank are placed at the end of the MVP list); otherwise, if width < N and height < N, candidates from the reference MV candidate bank are inserted before the derived candidates.

[0128] In some embodiments, if the condition of width × height ≧ N × N is met (e.g., N can be equal to, but not limited to, 8, 16, 32, or 64 luminance samples), the original MVP list order is used (e.g., candidates from the reference MV candidate bank are placed at the end of the MVP list); otherwise, if width × height < N × N, candidates from the reference MV candidate bank will be inserted before the derived candidates.

[0129] In some embodiments, if the current MV coding mode includes "NEAR" (e.g., NEARMV in the single reference case, or NEAR_NEWMV, NEW_NEARMV, NEAR_NEARMV in the mixed reference case, and their extensions, such as optical flow), the original MVP list order is used (e.g., candidates from the reference MV candidate bank are placed at the end of the MVP list), otherwise, if the current MV coding mode does not include "NEAR" (e.g., NEWMV in single reference, and NEW_NEWMV and JOINT_NEWMV in mixed reference, and their extensions, such as optical flow), candidates from the reference MV candidate bank are inserted before the derived candidates.

[0130] In some embodiments, if the current MV coding mode includes NEAR for each reference frame list (e.g., NEAR MVS in the single reference case, NEAR_NEARMV in the mixed reference case, and their extensions, such as optical flow), the original MVP list order is used (e.g., candidates from the reference MV candidate bank are placed at the end of the MVP list), otherwise, if the current MV coding mode does not include NEAR for all reference frame lists (e.g., NEWMV in the single reference case, and NEW_NEWMV, NEAR_NEWMV, NEW_NEARMV and JOINT_NEWMV in the mixed reference case, and their extensions, such as optical flow), candidates from the reference MV candidate bank are inserted before the derived candidates.

[0131] In some embodiments, if the current motion mode is translation, the original MVP list order is used, otherwise (e.g., the motion mode is warp or obmc), a new MVP list with a different order or content is constructed.

[0132] 15 shows some logical stages in a particular order, stages that are not order dependent may be reordered and other stages may be combined or broken down. Any reordering or other grouping not specifically mentioned will be apparent to one of ordinary skill in the art, and the ordering and groupings presented herein are not exhaustive. It should also be recognized that these stages may be implemented in hardware, firmware, software, or any combination thereof.

[0133] Reference will now be made to some example embodiments.

[0134] (A1) In one aspect, some embodiments include a method (e.g., method 1500) for decoding a current coding block. In some embodiments, the method is performed in a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed in a coding module (e.g., coding module 320). The method includes retrieving one or more motion vector (MV) candidates from a reference MV bank. The method includes, in accordance with a determination that a first condition is satisfied, inserting the one or more MV candidates from the reference MV bank into an MVP list associated with the current coding block after a derived MV candidate is inserted into the MVP list, in accordance with a determination that the first condition is not satisfied, inserting the one or more MV candidates from the reference MV bank into an MVP list associated with the current coding block before the derived MV candidate is inserted into the MVP list, and predicting the current coding block based on the MVP list.

[0135] (A2) In some embodiments of A1, the one or more MV candidates include a spatial MV predictor (SMVP) derived from spatial neighboring blocks that are adjacent to the current coding block, as shown in FIG.

[0136] (A3) In some embodiments of A1 or A2, the first condition includes at least one of: (i) the current coding block has a width of greater than or equal to N luma samples; and (ii) the current coding block has a height of greater than or equal to N luma samples, where N is an integer greater than or equal to 8.

[0137] (A4) In some embodiments of any of A1-A3, the first condition includes: (i) a first sub-condition that the current coding block has a width of greater than or equal to N luma samples; and (ii) a second sub-condition that the current coding block has a height of greater than or equal to N luma samples, where N is an integer greater than or equal to 8.

[0138] (A5) In some embodiments of any of A1-A4, when the first condition is not met, the MVP list has the following order: adjacent SMVP, a sorted set of MV candidates, a temporal motion vector predictor (TMVP) candidate, a non-adjacent SMVP candidate, a second candidate from a reference MV candidate bank, and a derived candidate.

[0139] (A6) In some embodiments of any of A1-A5, the first condition includes a condition that the product of the width of the current coding block and the height of the current coding block is greater than or equal to N×N, where N is an integer representing the number of luma samples, and N is greater than or equal to 8.

[0140] (A7) In some embodiments of any of A1-A6, the first condition includes a condition that the current MV coding mode includes the term “NEAR” (e.g., NEAR MVS in the single reference case, or NEAR_NEWMV, NEW_NEARMV, NEAR_NEARMV, and extensions thereof, such as optical flow, in the mixed reference case).

[0141] (A8) In some embodiments of any of A1-A7, the first condition includes a condition that the current MV coding mode has a NEAR mode for each reference frame list in the video stream acquired by the computing system.

[0142] (A9) In some embodiments of any of A1-A8, the first condition includes a condition that the current motion mode is a translational mode.

[0143] (A10) In some embodiments of any of A1-A9, the derived MV candidate is an MV predictor derived from either a motion vector (or one or more motion vectors) in a single reference frame based on an inter prediction mode, or a composite motion vector (or one or more composite motion vectors) from multiple adjacent reference frames in a composite inter prediction mode.

[0144] In another aspect, some embodiments include a computing system (e.g., server system 112) including a control circuit (e.g., control circuit 302) and a memory (e.g., memory 314) coupled to the control circuit, the memory storing one or more instruction sets configured to be executed by the control circuit, the one or more instruction sets including instructions for performing any of the methods described herein (e.g., A1-A9 above).

[0145] In yet another aspect, some embodiments include a non-transitory computer-readable storage medium having stored thereon one or more instruction sets for execution by control circuitry of a computing system, the one or more instruction sets including instructions for performing any of the methods described herein (e.g., A1-A9 above).

[0146] It is to be understood that although the terms "first," "second," etc. may be used herein to describe various elements, the elements should not be limited by these terms, and these terms are used only to distinguish one element from another.

[0147] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the claims. As used in the description of the embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is also to be understood that the term "and / or," as used herein, refers to and includes any and all possible combinations of one or more of the associated listed items. It is further to be understood that the terms "comprise" and / or "comprising," as used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0148] As used herein, the term "if" may be interpreted to mean "when" or "response thereto" or "response to determining" or "pursuant to determining" or "response to detecting" the stated precondition is true, depending on the context. Similarly, the phrases "if it is determined that [the stated precondition is true]" or "if [the stated precondition is true]" or "when [the stated precondition is true]" may be interpreted to mean "response to determining" or "response to determining" or "pursuant to determining" or "response to detecting" or "response to detecting" the stated precondition is true, depending on the context.

[0149] The foregoing description has been described with reference to specific embodiments for purposes of explanation. However, the exemplary description above is not intended to be exhaustive or to limit the claims to the precise form disclosed. Numerous modifications and variations are possible in light of the above teachings. These embodiments have been selected and described in order to best explain the principles of operation and practical application, thereby enabling others skilled in the art.

Claims

1. 1. A method for coding a current coding block executed by a computing system having one or more processors and a memory, comprising: retrieving one or more motion vector (MV) candidates from a reference MV bank; inserting the one or more MV candidates from the reference MV bank into a motion vector prediction (MVP) list associated with the current coding block after a derived MV candidate is inserted into the MVP list according to a determination that a first condition is satisfied; according to a determination that the first condition is not satisfied, inserting the one or more MV candidates from the reference MV bank into the MVP list associated with the current coding block before the derived MV candidates are inserted into the MVP list; predicting the current coding block based on the MVP list; The method according to claim 1,

2. The method of claim 1 , wherein the one or more MV candidates include spatial MV predictors (SMVPs) derived from spatial neighboring blocks that are adjacent to the current coding block.

3. The first condition is that N is an integer equal to or greater than 8, the current coding block has a width of N luma samples or more; and the current coding block has a height of N luma samples or more; The method of claim 1 , comprising at least one of:

4. The first condition is that N is an integer equal to or greater than 8, a first sub-condition that the current coding block has a width of N luma samples or more; and a second sub-condition that the current coding block has a height of N luma samples or more; The method of claim 1 , comprising:

5. 2. The method of claim 1, wherein when the first condition is not met, the MVP list has the following order: adjacent SMVP, a reordered set of MV candidates, a temporal motion vector predictor (TMVP) candidate, a non-adjacent SMVP candidate, a second candidate from the reference MV candidate bank, and a derived candidate.

6. 2. The method of claim 1 , wherein the first condition comprises a condition that a product of a width of the current coding block and a height of the current coding block is greater than or equal to N×N, where N is an integer representing a number of luma samples, and N is greater than or equal to 8.

7. The method of claim 1 , wherein the first condition comprises that the current MV coding mode includes the term “NEAR”.

8. The method of claim 1 , wherein the first condition comprises a condition that a current MV coding mode has a NEAR mode for each reference frame list in a video stream acquired by the computing system.

9. The method of claim 1 , wherein the first condition comprises a condition that the current motion mode is a translational mode.

10. The method of claim 1 , wherein the derived MV candidates are MV predictors derived from either a motion vector in a single reference frame based on inter prediction, or a composite motion vector from multiple adjacent reference frames in hybrid inter prediction.

11. 1. A computing system for coding a current coding block, comprising: one or more processors; a memory storing one or more programs; having The one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 10. Computing system.

12. A computer program product causing a computer to carry out the method according to any one of claims 1 to 10.

Images