Systems and methods for candidate list construction

Abstract

An exemplary method for video decoding includes receiving video data including a current block from a video bitstream. The method also includes generating a subblock-based motion vector prediction (SbTMVP) for a subblock of the current block and generating an SbTMVP-MMVD candidate by applying a merged motion vector differential (MMVD) to the SbTMVP. In accordance with a determination that a motion vector (MV) of the SbTMVP-MMVD candidate satisfies one or more criteria, the SbTMVP-MMVD candidate is inserted into a candidate list for the current block. In accordance with a determination that the MV of the SbTMVP-MMVD candidate does not satisfy one or more criteria, the SbTMVP-MMVD candidate is not inserted into the candidate list for the current block. The method further includes reconstructing the current block using the candidate list.

Description

[Technical Field]

[0001] Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 459,566, entitled "Improvement of Candidate List Construction of Subblock Based Motion Vector Predictor With MMVD," filed April 14, 2023, and is a continuation of and claims priority to U.S. Patent Application No. 18 / 619,028, entitled "Systems and Methods for Candidate List Construction," filed March 27, 2024.

[0002] FIELD OF THE INVENTION

[0002] The disclosed embodiments relate generally to image and video coding and compression, including, but not limited to, systems and methods for building candidate lists for motion vector prediction. [Background technology]

[0003]

[0003] Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video gaming consoles, smartphones, video teleconferencing devices, and video streaming devices. The electronic devices transmit and receive or sometimes communicate digital video data across communication networks and / or store the digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video coding may be used to compress the video data according to one or more video coding standards before the video data is communicated or stored. The video coding may be performed by hardware and / or software on the electronic / client device or on a server providing a cloud service.

[0004]

[0004] Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit redundancy inherent in video data. Video coding aims to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality. Multiple video codec standards have been developed. For example, High Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. ITU-T and ISO / IEC published the HEVC / H.265 standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). Versatile Video Coding (VVC / H.266) is a video compression standard intended as a successor to HEVC. ITU-T and ISO / IEC published the VVC / H.266 standard in 2020 (version 1) and 2022 (version 2). AOMedia Video 1 (AV1) is an open video coding format designed as a replacement for HEVC. The validated version 1.0.0 was released on January 8, 2019, along with Errata 1 of the specification. Summary of the Invention [Problem to be solved by the invention]

[0005]

[0005] A block of video data (e.g., a coding unit) may be processed using a predictive transform coding scheme, with prediction derived from either intra-frame reference pixels, inter-frame motion compensation, or some combination of the two. For each inter-predictive coding unit, motion parameters and additional information (such as motion vectors, reference picture indices, and reference picture list usage indices) may be used for inter-prediction sample generation. Each motion parameter may be signaled in an explicit or implicit manner. As an example, when a coding unit is coded in skip mode, the coding is associated with one prediction unit and has no significant residual coefficients, coded motion vector deltas, or reference picture indices. A merge mode may be specified, whereby motion parameters for the current coding unit are obtained from neighboring coding units, including spatial and temporal candidates. Note that the merge mode may be applied to any inter-predictive coding unit, not just skip mode. An alternative to merge mode is to explicitly transmit motion parameters, where for each coding unit, a motion vector, a corresponding reference picture index for each reference picture list, a reference picture list usage flag, and / or other required information are explicitly signaled. [Means for solving the problem]

[0006] This disclosure describes applying subblock-based temporal motion vector prediction (SbTMVP) candidates in a subblock merge motion vector differential (MMVD) merge list. A syntax element, MMVD index, may be signaled to indicate an MMVD offset for an SbTMVP merge candidate, and this MMVD offset may be used to derive a displacement vector (DV) offset value for each SbTMVP-MMVD candidate. A final DV for each SbTMVP-MMVD candidate may be calculated from the DV of the SbTMVP merge candidate along with the selected offset. By using different DV offsets, different subblock-based motion fields may be obtained to form an SbTMVP-MMVD candidate list.

[0007] However, in the absence of a redundant MV check mechanism during candidate list construction, SbTMVP-MMVD candidates with duplicated MV fields may be included in the candidate list. Therefore, in some embodiments, a check mechanism is used to detect duplicated MVs or lack of MV diversity in SbTMVP-MMVD during candidate list construction. Removing redundancy improves coding efficiency because excluding a redundant candidate allows another candidate to be added to the list. This improves list diversity, providing better candidates for coding, which improves video coding. Also, the lack of adjustments for handling MMVD candidate positions with boundary conditions of MV field buffers at collocated positions in collocated pictures. Therefore, in some embodiments, adjustments are made to the derivation of displacement vectors (DVs) for SbTMVP merge candidates for SbTMVP-MMVD candidate list construction. Adjusting MMVD candidates to be within the boundary conditions improves coding efficiency and / or hardware efficiency. For example, if the boundary conditions are enforced in software, hardware does not need to be designed to enforce out-of-bounds values ​​(e.g., allowing for smaller buffer sizes).

[0008] According to some embodiments, a method of video encoding is provided, including: (i) receiving video data (e.g., from a video source 104) including a plurality of blocks including a current block; (ii) generating subblock-based motion vector prediction (SbTMVP) for subblocks of the current block; (iii) generating an SbTMVP-MMVD candidate by applying a merged motion vector differential (MMVD) to the SbTMVP; (iv) inserting the SbTMVP-MMVD candidate into a candidate list for the current block according to a determination that a motion vector (MV) of the SbTMVP-MMVD candidate satisfies one or more criteria; (v) refraining from inserting the SbTMVP-MMVD candidate into the candidate list for the current block according to a determination that the MV of the SbTMVP-MMVD candidate does not satisfy one or more criteria; and (vi) encoding the current block using the candidate list.

[0009] According to some embodiments, a method of video decoding is provided, the method including: (i) receiving video data from a video bitstream, the method including a plurality of blocks including a current block; (ii) generating subblock-based motion vector prediction (SbTMVP) for subblocks of the current block; (iii) generating an SbTMVP-MMVD candidate by applying a merged motion vector differential (MMVD) to the SbTMVP; (iv) inserting the SbTMVP-MMVD candidate into a candidate list for the current block according to a determination that a motion vector (MV) of the SbTMVP-MMVD candidate satisfies one or more criteria; (v) refraining from inserting the SbTMVP-MMVD candidate into the candidate list for the current block according to a determination that the MV of the SbTMVP-MMVD candidate does not satisfy one or more criteria; and (vi) reconstructing the current block using the candidate list.

[0010] According to some embodiments, a computing system, such as a streaming system, a server system, a personal computer system, or other electronic device, is provided. The computing system includes control circuitry and a memory that stores one or more sets of instructions. The one or more sets of instructions include instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a transcoder).

[0011]

[0011] According to some embodiments, a non-transitory computer-readable storage medium is provided that stores one or more sets of instructions for execution by a computing system, the one or more sets of instructions including instructions for performing any of the methods described herein.

[0012]

[0012] Accordingly, disclosed are methods, devices, and systems for encoding and decoding video. Such methods, devices, and systems may complement or replace conventional methods, devices, and systems for video encoding / decoding. The features and advantages described herein are not necessarily all-inclusive, and in particular, some additional features and advantages will be apparent to those skilled in the art in light of the drawings, specification, and claims provided in this disclosure. Moreover, it should be noted that the language used herein has been selected primarily for readability and educational purposes, and not necessarily to define or limit the subject matter described herein.

[0013]

[0013] So that the present disclosure may be more fully understood, a more particular description may be made by reference to features of various embodiments, some of which are illustrated in the accompanying drawings, which, however, illustrate only pertinent features of the present disclosure and should not therefore be considered necessarily limiting, since the description may lead to other useful features, as will be appreciated by those skilled in the art upon reading the present disclosure. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram illustrating an exemplary communication system, according to some embodiments. [Figure 2A]

[0015] FIG. 2 is a block diagram illustrating exemplary elements of an encoder component, according to some embodiments. [Figure 2B]

[0016] 3 is a block diagram illustrating exemplary elements of a decoder component, according to some embodiments. [Figure 3]

[0017] FIG. 1 is a block diagram illustrating an exemplary server system, according to some embodiments. [Figure 4]

[0018] FIG. 1 illustrates sub-block-based motion prediction according to some embodiments. [Figure 5A]

[0019] 1 is a flow diagram illustrating an exemplary method for encoding video, according to some embodiments. [Figure 5B]

[0020] 1 is a flow diagram illustrating an example method for decoding video, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0015]

[0021] According to common practice, the various features illustrated in the drawings are not necessarily drawn to scale and like reference numerals may be used to denote like features throughout the specification and figures.

[0016]

[0022] This disclosure describes, among other things, systems and methods for applying merge motion vector differentials (MMVDs) to sub-block-based motion vector prediction (SbTMVP) candidates (in a sub-block MMVD merge list). An exemplary maximum candidate list size is 16. For example, a syntax element, MMVD index, may be signaled to indicate an MMVD offset for an SbTMVP merge candidate, and the MMVD offset may be used to derive a displacement vector (DV) offset value for each SbTMVP-MMVD candidate. A final DV for each SbTMVP-MMVD candidate may be calculated from the DV of the SbTMVP merge candidate along with the selected offset. Different sub-block-based motion fields may be obtained using different DV offsets at different MMVD offset positions.

[0017]

[0023] However, in the absence of an MV check mechanism, SbTMVP-MMVD candidates with MV fields that are equivalent (or nearly equivalent) to other candidates may be placed in the candidate list. In some embodiments, a check mechanism is used to detect duplicated MVs or MV diversity in SbTMVP-MMVD during candidate list construction. Removing redundancy can improve coding efficiency. For example, excluding a redundant candidate allows another candidate to be added to the list, which may improve list diversity and provide better candidates for coding (thereby improving the accuracy / precision of video coding).

[0018]

[0024] Another issue is that the displacement vectors (DVs) of SbTMVP-MMVD candidates may have positions that are outside the boundary conditions of the MV field buffer at the co-located position in the co-located picture. In some embodiments, the derivation of DVs for SbTMVP merge candidates is adjusted during SbTMVP-MMVD candidate list construction. Adjusting the DVs of MMVD candidates to be within the boundary conditions improves coding efficiency and / or hardware efficiency. For example, if the boundary conditions are enforced in software, the coding hardware does not need to handle / enforce out-of-bounds values ​​(e.g., allowing for smaller buffer sizes).

[0019] Exemplary Systems and Devices

[0025] 1 is a block diagram illustrating a communication system 100 according to some embodiments. Communication system 100 includes a source device 102 and multiple electronic devices 120 (e.g., electronic devices 120-1 through 120-m) communicatively coupled to each other via one or more networks. In some embodiments, communication system 100 is a streaming system for use with video-enabled applications, such as, for example, video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0020]

[0026] Source device 102 includes a video source 104 (e.g., a camera component or media storage) and an encoder component 106. In some embodiments, video source 104 is a digital camera (e.g., configured to create an uncompressed video sample stream). Encoder component 106 generates one or more encoded video bitstreams from the video stream. The video stream from video source 104 may be of a high data volume compared to the encoded video bitstream 108 generated by encoder component 106. Because the encoded video bitstream 108 has a lower data volume (less data) compared to the video stream from the video source, the encoded video bitstream 108 requires less bandwidth to transmit and less storage space to store compared to the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video to network(s) 110).

[0021]

[0027] The one or more networks 110 represent any number of networks that convey information between the source device 102, the server system 112, and / or the electronic device 120, including, for example, wireline (wired) and / or wireless communication networks. The one or more networks 110 may exchange data in circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet.

[0022]

[0028] The one or more networks 110 include a server system 112 (e.g., a distributed / cloud computing system). In some embodiments, the server system 112 is or includes a streaming server (e.g., configured to store and / or distribute video content, such as an encoded video stream from the source device 102). The server system 112 includes a coder component 114 (e.g., configured to encode and / or decode video data). In some embodiments, the coder component 114 includes an encoder component and / or a decoder component. In various embodiments, the coder component 114 is instantiated as hardware, software, or a combination thereof. In some embodiments, the coder component 114 is configured to decode the encoded video bitstream 108 and re-encode the video data using different encoding standards and / or methodologies to generate encoded video data 116. In some embodiments, the server system 112 is configured to generate multiple video formats and / or encodings from the encoded video bitstream 108. In some embodiments, the server system 112 functions as a media-aware network element (MANE). For example, server system 112 may be configured to prune encoded video bitstream 108 to accommodate potentially different bitstreams to one or more of electronic devices 120. In some embodiments, the MANE is provided separately from server system 112.

[0023]

[0029] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, decoder component 122 is configured to decode encoded video data 116 to generate an outgoing video stream that can be rendered on a display or other type of rendering device. In some embodiments, one or more of electronic devices 120 do not include a display component (e.g., are communicatively coupled to an external display device and / or include media storage). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0024]

[0030] The source device and / or the plurality of electronic devices 120 may be referred to as “terminal devices” or “user devices.” In some embodiments, the source device 102 and / or one or more of the electronic devices 120 are instances of a server system, a personal computer, a portable device (e.g., a smartphone, tablet, or laptop), a wearable device, a videoconferencing device, and / or other types of electronic devices.

[0025]

[0031] In an exemplary operation of communication system 100, source device 102 transmits encoded video bitstream 108 to server system 112. For example, source device 102 may code a stream of pictures captured by the source device. Server system 112 may receive encoded video bitstream 108 and decode and / or encode encoded video bitstream 108 using coder component 114. For example, server system 112 may apply coding to the video data, which is more optimal for network transmission and / or storage. Server system 112 may transmit encoded video data 116 (e.g., one or more coded video bitstreams) to one or more of electronic devices 120. Each electronic device 120 may decode the encoded video data 116 and optionally display the video pictures.

[0026]

[0032] 2A is a block diagram illustrating exemplary elements of the encoder component 106, according to some embodiments. The encoder component 106 receives a source video sequence from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., a transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives a video sequence from a remote video source (e.g., a video source that is a component of a different device than the encoder component 106). The video source 104 may provide the source video sequence in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 Y CrCB, or RGB), and any suitable sampling structure (e.g., Y CrCb 4:2:0 or Y CrCb 4:4:4). In some embodiments, the video source 104 is a storage device that stores previously captured / prepared video. In some embodiments, the video source 104 is a camera that captures local image information as a video sequence. Video data may be provided as multiple individual pictures that, when viewed in sequence, impart motion. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples depending on the sampling structure, color space, etc., in use. Those skilled in the art can readily understand the relationship between pixels and samples. The following discussion focuses on samples.

[0027]

[0033] The encoder component 106 is configured to code and / or compress pictures of a source video sequence into a coded video sequence 216 in real time or under other time constraints required by an application. Enforcing an appropriate coding rate is one function of the controller 204. In some embodiments, the controller 204 controls and is operatively coupled to other functional units as described below. Parameters set by the controller 204 may include rate control-related parameters (e.g., picture skip, quantizer, and / or λ value for rate-distortion optimization techniques), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can readily identify other functions of the controller 204, as they may pertain to the encoder component 106 being optimized for a certain system design.

[0028]

[0034] In some embodiments, the encoder component 106 is configured to operate in a coding loop. In a simplified example, the coding loop includes a source coder 202 (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and one or more reference pictures) and a (local) decoder 210. The decoder 210 reconstructs the symbols to create sample data in a manner similar to a (remote) decoder (when the compression between the symbols and the coded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory 208. Because decoding of the symbol stream yields bit-exact results independent of the decoder location (local or remote), the contents in the reference picture memory 208 are also bit-exact between the local and remote encoders. In this way, the predictive portion of the encoder interprets the same sample values ​​as reference picture samples that the decoder would interpret when using prediction during decoding. This principle of reference picture synchrony (and the resulting drift if synchrony cannot be maintained, for example due to channel errors) is known to those skilled in the art.

[0029]

[0035] The operation of decoder 210 may be the same as that of a remote decoder, such as decoder component 122, which is described in detail below in connection with Figure 2B. However, with brief reference to Figure 2B, because symbols are available and the encoding / decoding of symbols into a coded video sequence by entropy coder 214 and parser 254 may be lossless, the entropy decoding portion of decoder component 122, including buffer memory 252 and parser 254, may not be fully implemented in local decoder 210.

[0030]

[0036] The decoder techniques described herein may exist in substantially equivalent functional form in the corresponding encoder, with the exception of parsing / entropy decoding. For this reason, the disclosed subject matter focuses on decoder operation. A description of the encoder techniques may be omitted, as the encoder techniques may be the inverse of the decoder techniques.

[0031]

[0037] As part of its operation, source coder 202 may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from a video sequence designated as reference frames. In this manner, coding engine 212 codes differences between pixel blocks of the input frame and pixel blocks of reference frame(s) that may be selected as prediction reference(s) for the input frame. Controller 204 may manage the coding operations of source coder 202, including, for example, setting parameters and subgroup parameters used to encode the video data.

[0032]

[0038] The decoder 210 decodes coded video data of frames that may be designated as reference frames based on symbols created by the source coder 202. The operation of the coding engine 212 may advantageously be a lossy process. When the coded video data is decoded in a video decoder (not shown in FIG. 2A ), the reconstructed video sequence may be a replica of the source video sequence with some errors. The decoder 210 replicates the decoding process that may be performed on the reference frames by a remote video decoder, which may cause the reconstructed reference frames to be stored in the reference picture memory 208. In this way, the encoder component 106 locally stores copies of the reconstructed reference frames that have common content as the reconstructed reference frames that will be obtained by the remote video decoder (without transmission errors).

[0033]

[0039] The predictor 206 may perform a predictive search for the coding engine 212. That is, for a new frame to be coded, the predictor 206 may search the reference picture memory 208 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that can serve as suitable predictive references for the new picture. The predictor 206 may operate on a sample block-by-pixel block basis to find suitable predictive references. As determined by the search results obtained by the predictor 206, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory 208.

[0034]

[0040] The output of all the above-mentioned functional units may undergo entropy coding in entropy coder 214. Entropy coder 214 turns the symbols produced by the various functional units into a coded video sequence by losslessly compressing them according to techniques known to those skilled in the art (e.g., Huffman coding, variable length coding, and / or arithmetic coding).

[0035]

[0041] In some embodiments, the output of the entropy coder 214 is coupled to a transmitter. The transmitter may be configured to buffer the coded video sequence(s) created by the entropy coder 214 to prepare them for transmission over a communication channel 218, which may be a hardware / software link to a storage device that will store the coded video data. The transmitter may be configured to merge the coded video data from the source coder 202 with other data to be transmitted, such as coded audio data and / or an auxiliary data stream (source not shown). In some embodiments, the transmitter may transmit additional data along with the coded video. The source coder 202 may include such data as part of the coded video sequence. The additional data may include other forms of redundant data, such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, etc.

[0036]

[0042] The controller 204 may manage the operation of the encoder component 106. During coding, the controller 204 may assign a coded picture type to each coded picture, which may affect the coding technique applied to the respective picture. For example, a picture may be assigned as an intra picture (I picture), a predicted picture (P picture), or a bidirectionally predicted picture (B picture). An intra picture may be coded and decoded without using other frames in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh (IDR) pictures. Those skilled in the art are aware of these variations of I pictures and their respective applications and characteristics, and therefore, they will not be repeated here. A predicted picture may be coded and decoded using intra prediction or inter prediction, using at most one motion vector and reference index to predict sample values ​​for each block. A bidirectionally predicted picture may be coded and decoded using intra or inter prediction, using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-predicted picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0037]

[0043] A source picture is typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be nonpredictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P-picture may be nonpredictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be nonpredictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0038]

[0044] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture to be coded / decoded, called the current picture, is partitioned into blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture may be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are in use.

[0039]

[0045] Encoder component 106 may perform coding operations in accordance with a given video coding technique or standard, such as any of those described herein. In doing so, encoder component 106 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0040]

[0046] 2B is a block diagram illustrating exemplary elements of the decoder component 122, according to some embodiments. The decoder component 122 in FIG. 2B is coupled to the channel 218 and the display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to the loop filter 256 and configured to transmit data to the display 124 (e.g., via a wired or wireless connection).

[0041]

[0047] In some embodiments, the decoder component 122 includes a receiver coupled to the channel 218 and configured to receive data from the channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more coded video sequences to be decoded by the decoder component 122. In some embodiments, the decoding of each coded video sequence is independent of the other coded video sequences. Each coded video sequence may be received from the channel 218, which may be a hardware / software link to a storage device that stores the coded video data. The receiver may receive the coded video data along with other data, e.g., coded audio data and / or auxiliary data streams, which may be forwarded to their respective using entities (not shown). The receiver may separate the coded video sequence from the other data. In some embodiments, the receiver receives additional (redundant) data along with the coded video. The additional data may be included as part of the coded video sequence(s). The additional data may be used by the decoder component 122 to decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0042]

[0048] According to some embodiments, the decoder component 122 includes a buffer memory 252, a parser 254 (sometimes referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra-picture prediction unit 262, a motion compensation prediction unit 260, an aggregator 268, a loop filter unit 256, a reference picture memory 266, and a current picture memory 264. In some embodiments, the decoder component 122 is implemented as one or more integrated circuits and / or other electronic circuits. In some embodiments, the decoder component 122 is implemented at least partially in software.

[0043]

[0049] The buffer memory 252 is coupled intermediate the channel 218 and the parser 254 (e.g., to eliminate network jitter). In some embodiments, the buffer memory 252 is separate from the decoder component 122. In some embodiments, a separate buffer memory is provided between the output of the channel 218 and the decoder component 122. In some embodiments, in addition to the buffer memory 252 internal to the decoder component 122 (e.g., configured to handle playout timing), a separate buffer memory is provided external to the decoder component 122 (e.g., to eliminate network jitter). When receiving data from a storage / forwarding device of sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory 252 may not be required or may be small. For use with best-effort packet networks such as the Internet, the buffer memory 252 may be required and may be relatively large, advantageously of adaptive size, and may be implemented at least in part in an operating system or similar element (not shown) external to the decoder component 122.

[0044]

[0050] Parser 254 is configured to reconstruct symbols 270 from the coded video sequence. These symbols may include, for example, information used to manage the operation of decoder component 122 and / or information for controlling a rendering device such as display 124. The control information for the rendering device(s) may be in the form of, for example, a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). Parser 254 parses (entropy decodes) the coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 254 may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to that group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. Parser 254 may also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0045]

[0051] The reconstruction of symbols 270 may involve several different units, depending on the type of coded video picture or portion thereof (e.g., inter-picture and intra-picture, inter-block and intra-block, etc.), and other factors. Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by parser 254. The flow of such subgroup control information between parser 254 and the following units is not shown for clarity.

[0046]

[0052] The decoder component 122 may be conceptually subdivided into several functional units, which in some implementations may interact closely with each other and may be at least partially integrated with each other. However, for clarity, the conceptual subdivision of functional units is maintained herein.

[0047]

[0053] The scaler / inverse transform unit 258 receives the quantized transform coefficients as well as control information (such as which transform to use, block size, quantization factor, and / or quantization scaling metric) from the parser 254 as symbol(s) 270. The scaler / inverse transform unit 258 may output blocks containing sample values ​​that may be input to the aggregator 268.

[0048]

[0054] In some cases, the output samples of the scaler / inverse transform unit 258 relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit 262. The intra-picture prediction unit 262 may generate blocks of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from the current (partially reconstructed) picture from the current picture memory 264. The aggregator 268 may add, on a sample-by-sample basis, the prediction information generated by the intra-picture prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258.

[0049]

[0055] In other cases, the output samples of the scalar / inverse transform unit 258 relate to inter-coded and potentially motion-compensated blocks. In such cases, the motion-compensated prediction unit 260 can access the reference picture memory 266 to fetch samples used for prediction. After motion-compensating the fetched samples according to symbols 270 related to the block, these samples can be added by the aggregator 268 to the output of the scalar / inverse transform unit 258 (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory 266 from which the motion-compensated prediction unit 260 fetches prediction samples can be controlled by motion vectors. The motion vectors can be available to the motion-compensated prediction unit 260, for example, in the form of symbols 270 that can have X, Y, and reference picture components. Motion compensation can also include interpolation of sample values ​​fetched from the reference picture memory 266 when sub-sample exact motion vectors are in use, motion vector prediction mechanisms, and the like.

[0050]

[0056] The output samples of aggregator 268 may undergo various loop filtering techniques in loop filter unit 256. Video compression techniques can include in-loop filter techniques controlled by parameters included in the coded video bitstream and made available to loop filter unit 256 as symbols 270 from parser 254, but can also be responsive to meta-information obtained during decoding of a coded picture or previous portion (in decoding order) of a coded video sequence, as well as to previously reconstructed and loop-filtered sample values. The output of loop filter unit 256 can be a sample stream that can be output to a render device, such as display 124, as well as stored in reference picture memory 266 for use in future inter-picture prediction.

[0051]

[0057] Some coded pictures, when reconstructed, may be used as reference pictures for future prediction. Once a coded picture is reconstructed and the coded picture is identified as a reference picture (e.g., by parser 254), the current reference picture may become part of reference picture memory 266, and a fresh current picture memory may be reallocated before beginning reconstruction of a subsequent coded picture.

[0052]

[0058] Decoder component 122 may perform decoding operations according to a predetermined video compression technology, which may be documented in a standard, such as any of the standards described herein. A coded video sequence may conform to the syntax specified by the video compression technology or standard being used, in that it follows the syntax of the video compression technology or standard as specified in the video compression technology document or standard, and in particular, in a profile document therein. Also, for compliance with some video compression technologies or standards, the complexity of a coded video sequence may be within limits defined by the level of the video compression technology or standard. In some cases, the level restricts a maximum picture size, a maximum frame rate, a maximum reconstruction sample rate (e.g., measured in megasamples per second), a maximum reference picture size, etc. The limits set by the level may, in some cases, be further restricted through a hypothetical reference decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0053]

[0059] 3 is a block diagram illustrating a server system 112, according to some embodiments. The server system 112 includes a control circuit 302, one or more network interfaces 304, a memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuit 302 includes one or more processors (e.g., a CPU, a GPU, and / or a DPU). In some embodiments, the control circuit includes one or more field programmable gate arrays (FPGAs), hardware accelerators, and / or integrated circuits (e.g., application specific integrated circuits).

[0054]

[0060] The network interface(s) 304 may be configured to interface with one or more communication networks (e.g., wireless networks, wireline networks, and / or optical networks). The communication networks may be local, wide-area, metropolitan, vehicular and industrial, real-time, delay-tolerant, etc. Examples of communication networks include local area networks such as Ethernet, wireless LANs, cellular networks, to include GSM, 3G, 4G, 5G, LTE, etc.; TV wireline or wireless wide-area digital networks, to include cable TV, satellite TV, and terrestrial broadcast TV; vehicular and industrial, to include CANbus; etc. Such communications may be unidirectional receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., CANbus to some CANbus devices), or bidirectional (e.g., with other computer systems using local or wide-area digital networks). Such communications may include communications to one or more cloud computing networks.

[0055]

[0061] The user interface 306 includes one or more output devices 308 and / or one or more input devices 310. The input device(s) 310 may include one or more of a keyboard, a mouse, a trackpad, a touchscreen, a data glove, a joystick, a microphone, a scanner, a camera, etc. The output device(s) 308 may include one or more of an audio output device (e.g., a speaker), a visual output device (e.g., a display or monitor), etc.

[0056]

[0062] Memory 314 may include high-speed random-access memory (such as DRAM, SRAM, DDR RAM, and / or other random-access solid-state memory devices) and / or non-volatile memory (such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices). Memory 314 optionally includes one or more storage devices located remotely from control circuitry 302. Memory 314, or alternatively, the non-volatile solid-state memory device(s) within memory 314, comprises a non-transitory computer-readable storage medium. In some embodiments, memory 314, or the non-transitory computer-readable storage medium of memory 314, stores the following programs, modules, instructions, and data structures, or a subset or superset thereof: an operating system 316, which includes procedures for handling various basic system services and for performing hardware-dependent tasks; a network communications module 318 used to connect the server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections); A coding module 320 for performing various functions related to encoding and / or decoding data, such as video data. In some embodiments, the coding module 320 is an instance of the coder component 114. The coding module 320 may include, but is not limited to: a decoding module 322 for performing various functions related to decoding the encoded data, such as those previously described with respect to the decoder component 122; an encoding module 340 for performing various functions related to encoding data, such as those previously described with respect to the encoder component 106; and A picture memory 352 for storing pictures and picture data, e.g., for use with the coding module 320. In some embodiments, the picture memory 352 includes one or more of the reference picture memory 208, the buffer memory 252, the current picture memory 264, and the reference picture memory 266.

[0057]

[0063] In some embodiments, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions previously described with respect to the parser 254), a transform module 326 (e.g., configured to perform various functions previously described with respect to the scalar / inverse transform unit 258), a prediction module 328 (e.g., configured to perform various functions previously described with respect to the motion compensation prediction unit 260 and / or the intra-picture prediction unit 262), and a filter module 330 (e.g., configured to perform various functions previously described with respect to the loop filter 256).

[0058]

[0064] In some embodiments, encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described with respect to source coder 202 and / or coding engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to predictor 206). In some embodiments, decoding module 322 and / or encoding module 340 include a subset of the modules shown in Figure 3. For example, a shared prediction module is used by both decoding module 322 and encoding module 340.

[0059]

[0065] Each of the above-identified modules stored in memory 314 corresponds to a set of instructions for performing functions described herein. The above-identified modules (e.g., sets of instructions) need not be implemented as separate software programs, procedures, or modules; thus, various subsets of these modules may be combined or possibly rearranged in various embodiments. For example, coding module 320 optionally does not include separate decoding and encoding modules, but rather uses the same set of modules to perform both sets of functions. In some embodiments, memory 314 stores a subset of the above-identified modules and data structures. In some embodiments, memory 314 stores additional modules and data structures not described above, such as an audio processing module.

[0060]

[0066] 3 illustrates a server system 112 according to some embodiments, although FIG. 3 is intended as a functional illustration of various features that may be present in one or more server systems, rather than a structural schematic of the embodiments described herein. In practice, and as will be recognized by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some items shown separately in FIG. 3 may be implemented on a single server, and a single item may be implemented by one or more servers. The actual number of servers used to implement server system 112 and how features are allocated among them will vary from implementation to implementation and, optionally, depend in part on the amount of data traffic the server system handles during peak usage periods as well as during average usage periods.

[0061] Exemplary Coding Processes and Techniques

[0067] To improve coding efficiency and reduce motion vector transmission overhead, subblock-level motion vector refinement may be applied to extend CU-level temporal motion vector prediction (TMVP). Subblock-based TMVP (SbTMVP) enables inheriting motion information at the subblock level from a collocated reference picture. For example, each subblock of a large-sized CU may have its own motion information without explicitly transmitting a block partition structure or motion information. SbTMVP may obtain motion information for each subblock in three steps, where the first step is to derive a displacement vector (DV) for the current CU. The second step may be to check the availability of SbTMVP candidates and derive a central motion vector. The third step may be to derive subblock motion information from the corresponding subblock via the DV. Unlike TMVP candidate derivation, which always derives temporal motion vectors from co-located blocks in a reference frame, SbTMVP may apply DVs derived from the MVs of the current CU's left-neighboring CU to find corresponding sub-blocks in the co-located picture for each sub-block of the current CU. If the corresponding sub-block is not inter-coded, the motion information of the current sub-block may be set to be central motion. The same co-located picture used for TMVP may be used for SbTMVP. SbTMVP may differ from TMVP in that (1) TMVP predicts motion at the CU level, while SbTMVP predicts motion at the sub-CU level, and (2) TMVP fetches temporal motion vectors from co-located blocks in the co-located picture (e.g., the co-located block may be the bottom-right or central block relative to the current CU), while SbTMVP may apply a motion shift before fetching temporal motion information from the co-located picture. For example, the motion shift may be obtained from a motion vector from one of the spatially neighboring blocks of the current CU.

[0062]

[0068] FIG. 4 is a diagram illustrating sub-block-based motion prediction according to some embodiments. FIG. 4 shows a current picture 402 with a current block A1. FIG. 4 further shows a co-located picture 404 with a corresponding block A1′. As shown by FIG. 4, a coding unit may be divided into sub-blocks, and the sub-block size may be fixed at 8×8 pixels, for example. For example, as in the affine merge mode, SbTMVP is applicable for coding units larger than 8×8 pixels. A sub-block motion vector may be derived in SbTMVP by applying a shift from a neighbor and scaling the sub-block motion vector. As shown by line 405, a motion shift may be derived on the co-located picture. For example, if the A1 neighbor has a motion vector that uses the co-located picture as its reference picture, that motion vector is used. Otherwise, the motion shift may be set to 0 (e.g., a 0,0 shift). The derived motion shift may be applied in the co-located picture, for example (the shift is added to the coordinates of the current block). Then, from the center of the CU, an initial vector 406 (e.g., to be used as a default value for sub-blocks with unavailable motion information) may be obtained. In some embodiments, when the motion of the center sub-block of the CU is also unavailable, a motion vector predictor (MVP) using the signaled motion vector differential (MVD) is used instead. Then, sub-block motion information 408 (e.g., from the center of the sub-block) is obtained. By applying scaling for the corresponding sub-block of the co-located picture 404, a sub-block motion vector 412 (and a motion vector 414 corresponding to the initial vector 406) and / or a reference index may be derived for the current picture 402. For example, when SbTMVP mode is enabled, the list of merge candidates may include SbTMVP candidates as well as affine merge candidates.

[0063]

[0069] Affine MMVD mode may be used to improve coding gain by using sub-block-based motion information together with affine parameters. Although SbTMVP is one of the candidates in the sub-block-based merge list, MMVD is not supported for SbTMVP in the current ECM design. The SbTMVP derived from neighboring blocks may not be the optimal sub-block-based motion field. To improve SbTMVP, SbTMVP with MMVD and / or advanced motion vector prediction (AMVP) mode may be used. For example, for a CU coded in SbTMVP mode with AMVP, the CU may be predicted similarly to the CU of SbTMVP in merge mode, except that the motion shift is signaled in the bitstream instead of being derived from neighboring blocks.

[0064]

[0070] As an example, when a CU is coded in modified SbTMVP mode, the CU may be split into n×n sub-blocks (e.g., 4×4 or 8×8), and motion for each sub-block may be derived from a corresponding sub-block in a co-located picture (e.g., co-located picture 404). The co-located picture (or other reference picture for the sub-block) may be selected in the same manner as for SbTMVP (e.g., the same co-located picture may be used for both). Corresponding sub-blocks may be identified using a motion vector predictor (MVP) with a signaled motion vector differential (MVD). In some embodiments, a first flag is signaled to indicate whether the current block (e.g., CU) is coded using SbTMVP. If the current block is coded using SbTMVP, a second flag is signaled to indicate whether MVD is available. If MVD is available, the MVD index may be signaled (so that the motion direction and / or magnitude may be obtained). As an example, the number of MVDs may be equal to 16.

[0065]

[0071] In some embodiments, MMVD is performed on SbTMVP candidates in a sub-block MMVD merge list. For example, a syntax element, MMVD index, is signaled to indicate an MMVD offset for the SbTMVP merge candidate, and this MMVD offset is used to derive a displacement vector (DV) offset value for each SbTMVP-MMVD candidate. A final DV' for each SbTMVP-MMVD candidate is calculated from the DV of the SbTMVP merge candidate along with the selected offset. The sub-block-based motion field for the offset DV' point is used as the SbTMVP-MMVD candidate. By using different DV offsets, different sub-block-based motion fields can be obtained to form the SbTMVP-MMVD candidate list. The step size in can be {4, 8, 12, 16, 20}, where the unit of the step size is integer pixel units. The number of SbTMVP-MMVD directions can be 8, which is also used in affine MMVD. As with affine MMVD, the total number of available SbTMVP-MMVD candidates may be less than or equal to 16. As an example, a spiral scan order may be applied, and up to 16 available candidates may be placed in a candidate list. As another example, all MMVD candidates may be scanned in spiral order, and each may be placed in a candidate list if it is an available candidate. Subblock-based template matching (TM) may be applied to all available SbTMVP-MMVD candidates to sort the SbTMVP-MMVD candidate list by using TM cost in ascending order (e.g., only the 16 SbTMVP-MMVD candidates with the smallest TM cost are signaled).

[0066]

[0072] In some embodiments, a picture-level flag for SbTMVP-MMVD mode is used (e.g., in a random access (RA) configuration). This flag may be used to indicate whether SbTMVP-MMVD is enabled (e.g., when both the SPS flag in SbTMVP-MMVD and the SPS control flag in the picture header for SbTMVP-MMVD are true). For example, picture-level enablement and disablement is determined based on a temporal ID, and SbTMVP-MMVD is enabled only in the top two temporal layers for RA (e.g., to improve coding gain and runtime). In some embodiments, a flag is signaled (e.g., at the CU level) to indicate which co-located picture should be used to derive sub-block motion (e.g., the flag identifies one of two reference pictures). In some embodiments, the reference picture for a sub-block is fixed to the reference picture corresponding to an index of 0.

[0067]

[0073] There are several potential problems with using modified SbTMVP candidates. First, in the absence of an MV check mechanism during candidate list construction, modified SbTMVP candidates with duplicated MV fields may be placed in the candidate list. Second, the modified candidate position may exceed the boundary conditions of the MV field buffer at the collocated position in the collocated picture. The techniques and approaches described below address these problems and others (e.g., improve candidate list construction for SbTMVP-MMVD). The techniques / approaches may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0068]

[0074] In some embodiments, a check mechanism is used for duplicated MVs or MV diversity of SbTMVP-MMVD during candidate list construction. For example, if the result indicates that the MVs of SbTMVP-MMVD are duplicated MVs or MV diversity compared to existing candidates in the candidate list, this SbTMVP-MMVD candidate is discarded. When MV diversity is used, the difference between the MV components and a predefined threshold is determined. When the difference is less than (or equal to) the threshold, the MV components are categorized as similar MV components. For example, if both MV components, MVx and MVy, are similar MV components, the uni-predictive merge candidate is not placed in the candidate list. For example, a subblock-based motion vector prediction (SbTMVP) for a subblock of the current block may be generated. Then, an SbTMVP-MMVD candidate may be generated (e.g., by applying the merged motion vector differential (MMVD) to SbTMVP). If the motion vector (MV) of the SbTMVP-MMVD candidate satisfies one or more criteria, the SbTMVP-MMVD candidate may be included in the candidate list for the current block. If the MV of the SbTMVP-MMVD candidate does not satisfy one or more criteria, the SbTMVP-MMVD candidate is not included in the candidate list for the current block.

[0069]

[0075] 5A is a flow diagram illustrating a method 500 for encoding video according to some embodiments. Method 500 may be implemented in a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and memory storing instructions for execution by the control circuitry. In some embodiments, method 500 is implemented by executing instructions stored in memory (e.g., memory 314) of the computing system.

[0070]

[0076] The system receives video data (e.g., from video source 104) including a plurality of blocks, including a current block (502). The system generates subblock-based motion vector prediction (SbTMVP) candidates for subblocks of the current block (504). The system generates modified SbTMVP candidates (508). For example, the system generates SbTMVP-MMVD candidates by applying merged motion vector differential (MMVD) to SbTMVP.

[0071]

[0077] In accordance with a determination that the motion vector (MV) of the modified SbTMVP candidate satisfies one or more criteria, the system includes the modified SbTMVP candidate in a candidate list for the current block (510). In some embodiments, the one or more criteria include a duplicated MV criterion. For example, duplicated MV means that the MV field information in the candidate is equivalent to the MV field information of an existing candidate in the candidate list. In some embodiments, the one or more criteria include an MV diversity criterion. For example, a predefined threshold is used to determine whether the MV difference in each n×n subblock between the candidate and an existing candidate in the list is greater than (or equal to) this value, where typical values ​​of n are 4 or 8.

[0072]

[0078] Following a determination that the MV of the modified SbTMVP candidate does not satisfy one or more criteria, the system refrains from including the modified SbTMVP candidate in a candidate list for the current block (512). In some embodiments, the system encodes the current block using the candidate list.

[0073]

[0079] In some embodiments, when the predefined threshold is a scalar, if at least one component of the MV difference of a subblock is greater than (or equal to) the threshold, the SbTMVP-MMVD candidate will be placed in the candidate list. Otherwise, the SbTMVP-MMVD candidate will not be placed in the candidate list.

[0074]

[0080] In some embodiments, when the predefined threshold is a scalar, if both components of the MV difference of a subblock are greater than (or equal to) the threshold, the SbTMVP-MMVD candidate will be placed in the candidate list. Otherwise, the SbTMVP-MMVD candidate will not be placed in the candidate list.

[0075]

[0081] In some embodiments, when the predefined threshold is a vector of two components (separately set horizontal and vertical components), if the corresponding component of the MV difference of a sub-block is greater than (or equal to) the threshold component, this SbTMVP-MMVD candidate will be put into the candidate list. Otherwise, this SbTMVP-MMVD candidate will not be put into the candidate list.

[0076]

[0082] 5B is a flow diagram illustrating a method 550 of decoding video according to some embodiments. Method 550 may be implemented in a computing system (e.g., server system 112, source device 102, or electronic device 120) having control circuitry and memory storing instructions for execution by the control circuitry. In some embodiments, method 550 is implemented by executing instructions stored in memory (e.g., memory 314) of the computing system.

[0077]

[0083] The system receives video data from a video bitstream, the video data including multiple blocks including a current block (552). The system generates sub-block-based motion vector prediction (SbTMVP) for sub-blocks of the current block (554). The system generates modified SbTMVP candidates (556). For example, the system generates SbTMVP-MMVD candidates by applying merged motion vector differentials (MMVD) to SbTMVP. In some embodiments, an SbTMVP MMVD candidate list is generated with a predefined MVD offset. In some embodiments, the MMVD candidate list is derived from SbTMVP in sub-block merging mode.

[0078]

[0084] Following a determination that the motion vector (MV) of the modified SbTMVP candidate satisfies one or more criteria, the system includes (558) the modified SbTMVP candidate in the candidate list for the current block.

[0079]

[0085] Following a determination that the MV of the modified SbTMVP candidate does not meet one or more criteria, the system refrains from including the modified SbTMVP candidate in the candidate list for the current block (560).

[0080]

[0086] In some embodiments, the system uses the candidate list to reconstruct the current block. As previously described, the encoding process may mirror the decoding process described herein. For brevity, those details will not be repeated here.

[0081]

[0087] 5A and 5B show some logical stages in a particular order, but order-independent stages may be reordered and other stages may be combined or separated. Any reordering or other grouping not specifically described will be apparent to those skilled in the art, and therefore the ordering and grouping presented herein is not exhaustive. Moreover, it should be recognized that the stages may be implemented in hardware, firmware, software, or any combination thereof.

[0082]

[0088] In some embodiments, the derivation of the displacement vectors (DVs) of SbTMVP merge candidates is adjusted for SbTMVP-MMVD candidate list construction (e.g., whether or not the check mechanism described above is applied). A DV offset (sometimes referred to as an MVD offset) may be added to the displacement vectors of the SbTMVP merge candidates (e.g., to obtain a final displacement vector that will point to the corresponding sub-block motion field in the co-located picture).

[0083]

[0089] In some embodiments, the DVs of SbTMVP merge candidates are adjusted to ensure that the MV field data of the MMVD candidate with the furthest MMVD offset does not exceed the boundary of the MV field buffer at the co-located position in the co-located picture. For example, for a coded block of w × h, if the buffer size of the MV field at the co-located position in the co-located picture is M × N and the maximum MMVD offset value is offset max If DV=(x c ,y c ) is calculated by the following equation 1: (x c’ ,y c’ ) should be adjusted.

[0084]

number

[0085]

[0090] In some embodiments, the adjustment is applied to one component of the DV. For example, the adjustment is c or y c In some embodiments, w×h coded blocks are replaced with W×H CTU sizes so that all coded block sizes have the same adjustment formula as shown in Equation 2 below.

[0086]

number

[0087]

[0091] In some embodiments, the size constraint of the collocated position is extended from one CTU row to three CTU rows. For example, this extension may apply only to SbTMVP-MMVD, and there is no size extension for TMVP and SbTMVP.

[0088]

[0092] (A1) In one aspect, some embodiments include a method of video encoding (e.g., method 500). In some embodiments, the method is implemented in a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is implemented in a coding module (e.g., coding module 320). In some embodiments, the method is implemented in a source coding component (e.g., source coder 202), a coding engine (e.g., coding engine 212), and / or an entropy coder (e.g., entropy coder 214). The method includes: (i) receiving video data (e.g., from a video source 104) including a plurality of blocks including a current block; (ii) generating subblock-based motion vector prediction (SbTMVP) candidates for subblocks of the current block; (iii) generating SbTMVP-MMVD candidates by applying merged motion vector differentials (MMVDs) to the SbTMVP; (iv) inserting the SbTMVP-MMVD candidates into a candidate list for the current block according to a determination that the motion vectors (MVs) of the SbTMVP-MMVD candidates satisfy one or more criteria; (v) refraining from inserting the SbTMVP-MMVD candidates into the candidate list for the current block according to a determination that the MVs of the SbTMVP-MMVD candidates do not satisfy one or more criteria; and (vi) encoding the current block using the candidate list.

[0089]

[0093] (A2) In some embodiments of A1, the method further includes signaling the encoded current block via a video bitstream.

[0090]

[0094] (A3) In some embodiments of A1 or A2, the method further includes signaling information about the candidate list in the video bitstream. For example, an index for the candidate list is signaled via the video bitstream.

[0091]

[0095] (A4) In some embodiments of any of A1-A3, the one or more criteria include that the MV of the SbTMVP-MMVD candidate does not match the MV of another candidate in the candidate list.

[0092]

[0096] (A5) In some embodiments of any of A1-A4, the one or more criteria include a criterion that the MV of the SbTMVP-MMVD candidate is at least a threshold difference from the corresponding MV of another candidate in the candidate list. In some embodiments, the threshold difference is a predefined threshold scalar value, and the criterion is that at least one component of the MV differs from the corresponding MV of another candidate in the candidate list by more than the predefined threshold scalar value. In some embodiments, the criterion is that each component of the MV differs from the corresponding MV of another candidate in the candidate list by more than the predefined threshold scalar value. In some embodiments, the threshold difference is a vector of components, and the criterion is that each component of the MV differs from the corresponding MV of another candidate in the candidate list by more than the corresponding component of the vector of components.

[0093]

[0097] (A6) In some embodiments of any of A1-A5, the method further includes (i) determining that a displacement vector (DV) of the SbTMVP-MMVD candidate exceeds a boundary of the MV field buffer, and (ii) adjusting the DV of the SbTMVP-MMVD candidate according to the determination that the DV of the SbTMVP-MMVD candidate exceeds the boundary of the MV field buffer so that the farthest MMVD offset does not exceed the boundary of the MV field buffer. In some embodiments, adjusting the DV of the SbTMVP-MMVD candidate includes adjusting a horizontal component and a vertical component of the DV. In some embodiments, adjusting the DV of the SbTMVP-MMVD candidate includes adjusting only one component of the DV.

[0094]

[0098] (B1) In another aspect, some embodiments include a method of video decoding (e.g., method 550). In some embodiments, the method is implemented in a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is implemented in a coding module (e.g., coding module 320). In some embodiments, the method is implemented in a parser (e.g., parser 254), a motion prediction component (e.g., motion compensated prediction unit 260), and / or an intra prediction component (e.g., intra-picture prediction unit 262). The method includes (i) receiving video data (e.g., a coded video sequence) including a plurality of blocks including a current block from a video bitstream (e.g., a video bitstream of A1), (ii) generating subblock-based motion vector prediction (SbTMVP) for subblocks of the current block, (iii) generating an SbTMVP-MMVD candidate by applying merged motion vector differential (MMVD) to the SbTMVP, (iv) inserting the SbTMVP-MMVD candidate into a candidate list for the current block according to a determination that a motion vector (MV) of the SbTMVP-MMVD candidate satisfies one or more criteria, (v) refraining from inserting the SbTMVP-MMVD candidate into the candidate list for the current block according to a determination that the MV of the SbTMVP-MMVD candidate does not satisfy one or more criteria, and (vi) reconstructing the current block using the candidate list. For example, a check mechanism is used for duplicated MVs or MV diversity of SbTMVP-MMVD during candidate list construction. If the results indicate that the MVs of the SbTMVP-MMVD are duplicated or have MV diversity compared to existing candidates in the candidate list, the SbTMVP-MMVD candidate can be discarded. As an example, MV diversity is used to determine the relationship between the difference of MV components and a predefined threshold. When the difference is less than (or equal to) the threshold, the MV components are categorized as similar MV components.If both MV components, MVx and MVy, are similar MV components, then this uni-predictive merge candidate will not be put into the candidate list.

[0095]

[0099] (B2) In some embodiments of B1, the one or more criteria include that the MV of the SbTMVP-MMVD candidate does not match the MV of another candidate in the candidate list. For example, a duplicated MV means that the MV field information in the candidate is equivalent to the MV field information of an existing candidate in the candidate list.

[0096]

[0100] (B3) In some embodiments of B1 or B2, the one or more criteria include that the MV of the SbTMVP-MMVD candidate is at least a threshold difference from the corresponding MV of other candidates in the candidate list. For example, a predefined threshold is used to determine whether the MV difference in each n×n subblock between the candidate and existing candidates in the list is greater than (or equal to) this value. Exemplary values ​​of n include 4 and 8.

[0097]

[0101] (B4) In some embodiments of B3, (i) the threshold difference is a predefined threshold scalar value, and (ii) the criterion is that at least one component of the MV differs from the corresponding MV of another candidate in the candidate list by more than the predefined threshold scalar value. For example, when the predefined threshold is a scalar, if at least one component of the MV difference of a subblock is greater than (or equal to) the threshold, the SbTMVP-MMVD candidate will be included in the candidate list. Otherwise, the SbTMVP-MMVD candidate will not be included in the candidate list.

[0098]

[0102] (B5) In some embodiments of B3, (i) the threshold difference is a predefined threshold scalar value, and (ii) the criterion is that each component of the MV differs from the corresponding MV of other candidates in the candidate list by more than the predefined threshold scalar value. For example, when the predefined threshold is a scalar, if both components of the MV difference of a subblock are greater than (or equal to) the threshold, the SbTMVP-MMVD candidate will be included in the candidate list. Otherwise, the SbTMVP-MMVD candidate will not be included in the candidate list.

[0099]

[0103] (B6) In some embodiments of B3, (i) the threshold difference is a vector of components, and (ii) the criterion is that each component of the MV differs from the corresponding MV of other candidates in the candidate list by more than the corresponding component of the vector of components. For example, when the predefined threshold is a vector of two components (separately set horizontal and vertical components), if the corresponding component of the MV difference of a subblock is greater than (or equal to) the threshold component, the SbTMVP-MMVD candidate will be included in the candidate list. Otherwise, the SbTMVP-MMVD candidate will not be included in the candidate list.

[0100]

[0104] (B7) In some embodiments of any of B1 to B6, the method further includes (i) determining that a displacement vector (DV) of the SbTMVP-MMVD candidate exceeds a boundary of the MV field buffer; and (ii) adjusting the DV of the SbTMVP-MMVD candidate in accordance with the determination that the DV of the SbTMVP-MMVD candidate exceeds the boundary of the MV field buffer, such that the farthest MMVD offset does not exceed the boundary of the MV field buffer. In some embodiments, refraining from adjusting the DV of the SbTMVP-MMVD candidate in accordance with the determination that the DV of the SbTMVP-MMVD candidate does not exceed the boundary of the MV field buffer. As an example, derivation of displacement vectors (DVs) of SbTMVP merge candidates is adjusted for SbTMVP-MMVD candidate list construction. In some embodiments, the displacement vectors (DVs) of the SbTMVP merge candidates are adjusted such that the MV field data of the MMVD candidate with the farthest MMVD offset does not exceed the boundary of the MV field buffer at the co-located position in the co-located picture.

[0101]

[0105] In some embodiments of B7, adjusting the DV of the SbTMVP-MMVD candidate includes adjusting the horizontal and vertical components of the DV. For example, for a coded block of w × h, if the buffer size of the MV field at the collocated position in the collocated picture is M × N and the maximum MMVD offset value is offsetmax, then DV = (xc, yc) is adjusted to (xc', yc') using Equation 1 above.

[0102]

[0106] (B9) In some embodiments of B7, adjusting the DV of the SbTMVP-MMVD candidate includes adjusting only one component of the DV. For example, the adjustment is applied to one component of the DV (e.g., applied to xc or yc). In some embodiments, a w×h coded block is replaced with a W×H CTU size so that all different coded block sizes have the same adjustment formula as shown in Equation 2.

[0103]

[0107] In another aspect, some embodiments include a computing system (e.g., server system 112) including control circuitry (e.g., control circuitry 302) and a memory (e.g., memory 314) coupled to the control circuitry, the memory storing one or more sets of instructions configured to be executed by the control circuitry, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1-A6 and B1-B9 above). In yet another aspect, some embodiments include a non-transitory computer-readable storage medium storing one or more sets of instructions for execution by control circuitry of a computing system, the one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1-A6 and B1-B9 above).

[0104]

[0108] Although terms such as “first,” “second,” etc. may be used herein to describe various elements, it will be understood that these elements are not limited by these terms. These terms are merely used to distinguish one element from another. The terminology used herein is merely for the purpose of describing particular embodiments and does not limit the scope of the claims. As used in the description of these embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. The term “and / or” as used herein will also be understood to refer to and encompass any and all possible combinations of one or more of the associated listed items. Furthermore, it will be understood that the terms “comprises” and / or “comprising” as used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0105]

[0109] The term "if," as used herein, may be interpreted to mean "when," or "upon," or "in response to determining," or "in accordance with a determination," or "in response to detecting" that a stated condition precedent is true, depending on the context. Similarly, the phrase "if determined [the stated condition precedent is true]," or "if [the stated condition precedent is true]," or "when [the stated condition precedent is true]," may be interpreted to mean "upon determining," or "in response to determining," or "in accordance with a determination," or "upon detecting," or "in response to detecting" that a stated condition precedent is true, depending on the context.

[0106]

[0110] The foregoing description has been set forth with reference to specific embodiments for purposes of explanation. However, the above exemplary description is not intended to be exhaustive or to limit the scope of the claims to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of operation and practical application, thereby enabling others skilled in the art.

Claims

1. 1. A method of video decoding implemented in a computing system having a memory and one or more processors, the method comprising: receiving video data from a video bitstream, the video data including a plurality of blocks including a current block; generating a sub-block-based motion vector prediction (SbTMVP) for a sub-block of the current block; generating a SbTMVP-MMVD candidate by applying a merged motion vector differential (MMVD) to the SbTMVP; including the SbTMVP-MMVD candidate in a candidate list for the current block if the motion vector (MV) of the SbTMVP-MMVD candidate satisfies one or more criteria, wherein the one or more criteria check for duplicated motion vectors in the candidate list; If the MV of the SbTMVP-MMVD candidate does not satisfy the one or more criteria, not including the SbTMVP-MMVD candidate in the candidate list for the current block; reconstructing the current block using the candidate list; and A method comprising:

2. The method of claim 1 , wherein the one or more criteria include a criterion that the MV of the SbTMVP-MMVD candidate does not match the MV of another candidate in the candidate list.

3. 2. The method of claim 1, wherein the one or more criteria include a criterion that the MV of the SbTMVP-MMVD candidate is at least a threshold difference from corresponding MVs of other candidates in the candidate list.

4. the threshold difference is a predefined threshold scalar value; the criterion being that at least one component of the MV differs from the corresponding MV of the other candidates in the candidate list by more than the predefined threshold scalar value; The method of claim 3.

5. the threshold difference is a predefined threshold scalar value; The criterion is that each component of the MV differs from the corresponding MV of the other candidates in the candidate list by more than the predefined threshold scalar value. The method of claim 3.

6. the threshold difference is a vector of components, The criterion is that each component of the MV differs from the corresponding MV of the other candidates in the candidate list by more than the corresponding component of the vector of components. The method of claim 3.

7. determining that a displacement vector (DV) of the SbTMVP-MMVD candidate exceeds a boundary of an MV field buffer; adjusting the DV of the SbTMVP-MMVD candidate according to the determination that the DV of the SbTMVP-MMVD candidate exceeds the boundary of the MV field buffer so that the farthest MMVD offset does not exceed the boundary of the MV field buffer; The method of claim 1 further comprising:

8. The method of claim 7 , wherein adjusting the DV of the SbTMVP-MMVD candidate comprises adjusting a horizontal component and a vertical component of the DV.

9. 8. The method of claim 7, wherein adjusting the DV of the SbTMVP-MMVD candidate comprises adjusting only one component of the DV.

10. a control circuit; Memory and one or more sets of instructions stored in the memory and configured for execution by the control circuitry; wherein the one or more sets of instructions receiving video data including a plurality of blocks including a current block; generating a sub-block-based motion vector prediction (SbTMVP) for a sub-block of the current block; generating a modified SbTMVP candidate by applying a merged motion vector differential (MMVD) to the SbTMVP; including the modified SbTMVP candidate in a candidate list for the current block if the motion vector (MV) of the modified SbTMVP candidate satisfies one or more criteria, wherein the one or more criteria check for duplicated motion vectors in the candidate list; refraining from including the modified SbTMVP candidate in the candidate list for the current block if the MV of the modified SbTMVP candidate does not satisfy the one or more criteria; encoding the current block using the candidate list; 1. A computing system comprising instructions for performing

11. The computing system of claim 10 , wherein the one or more criteria include a criterion that the MV of the modified SbTMVP candidate does not match the MV of another candidate in the candidate list.

12. 11. The computing system of claim 10, wherein the one or more criteria include a criterion that the MV of the modified SbTMVP candidate is at least a threshold difference from corresponding MVs of other candidates in the candidate list.

13. the threshold difference is a predefined threshold scalar value; the criterion being that at least one component of the MV differs from the corresponding MV of the other candidates in the candidate list by more than the predefined threshold scalar value; The computing system of claim 12.

14. the threshold difference is a predefined threshold scalar value; The criterion is that each component of the MV differs from the corresponding MV of the other candidates in the candidate list by more than the predefined threshold scalar value. The computing system of claim 12.

15. the threshold difference is a vector of components, The criterion is that each component of the MV differs from the corresponding MV of the other candidates in the candidate list by more than the corresponding component of the vector of components. The computing system of claim 12.

16. The one or more sets of instructions determining that a displacement vector (DV) of the modified SbTMVP candidate exceeds a boundary of an MV field buffer; adjusting the DV of the modified SbTMVP candidate such that the furthest MMVD offset does not exceed the boundary of the MV field buffer according to the determination that the DV of the modified SbTMVP candidate exceeds the boundary of the MV field buffer; The computing system of claim 10 further comprising instructions for:

17. 17. The computing system of claim 16, wherein adjusting the DV of the modified SbTMVP candidate comprises adjusting a horizontal component and a vertical component of the DV.

18. 17. The computing system of claim 16, wherein adjusting the DV of the modified SbTMVP candidate comprises adjusting only one component of the DV.

19. 1. A non-transitory computer-readable storage medium storing one or more sets of instructions configured for execution by a computing device having control circuitry and a memory, the one or more sets of instructions comprising: obtaining a source video sequence corresponding to a plurality of blocks including a first block; performing a conversion between the source video sequence and a video bitstream of visual media data; and the video bitstream comprises instructions to: a plurality of coded blocks corresponding to the plurality of blocks, the plurality of coded blocks including a coded first block, the plurality of coded blocks being coded using a candidate list that does not include a duplicated motion vector; a first indicator indicating a merge motion vector differential (MMVD) for a sub-block-based motion vector prediction (SbTMVP) candidate for the first block; 1. A non-transitory computer-readable storage medium comprising:

20. 20. The non-transitory computer-readable storage medium of claim 19, wherein the candidate list is constructed according to a criterion that the motion vector (MV) of each candidate is at least a threshold difference from the corresponding MV of other candidates in the candidate list.

Images