Improvements to signaling inter-prediction modes

Abstract

Various implementations described herein include methods and systems for coding video. In one embodiment, the method includes receiving a video bitstream comprising a plurality of blocks including a current block. The method includes determining that the current block is encoded using motion information from a first reference block and a second reference block. The method includes (i) selecting a composite inter prediction mode for the current block from a first set of composite inter prediction modes when the first and second reference blocks are in different reference frames, and (ii) selecting a composite inter prediction mode for the current block from a second set of composite inter prediction modes when the first and second reference blocks are in the same reference frame. The method includes reconstructing the current block using the composite inter prediction mode and motion information from the first and second reference blocks.

Description

Technology Field

[0001] Related applications

[0002] This application is a continuation of U.S. Patent Application No. 19 / 207,254 filed on May 13, 2025, claiming priority to U.S. Provisional Patent Application No. 63 / 701,647, filed on October 1, 2024, the entirety of which is incorporated by reference in this specification, titled "Improvements on Signaling Inter Prediction Mode".

[0003] Technology field

[0004] The disclosed embodiments relate to video coding, which generally includes, but is not limited to, systems and methods for inter-predicted mode coding and signaling. Background Technology

[0005] Digital video is supported by various electronic devices such as digital televisions, laptops or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video teleconferencing devices, and video streaming devices. Electronic devices transmit and receive digital video data over communication networks or communicate in other ways, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video coding may be used to compress video data according to one or more video coding standards before it is transmitted or stored. Video coding can be performed by hardware and / or software on electronic / client devices or servers that provide cloud services.

[0006] Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, etc.) that take advantage of the inherent redundancy in video data. Video coding aims to compress video data into a form that uses lower bit rates while avoiding or minimizing degradation in video quality. Numerous video codec standards have been developed. For example, High Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. The ITU-T and ISO / IEC released the HEVC / H.265 standard in 2013 (Version 1), 2014 (Version 2), 2015 (Version 3), and 2016 (Version 4). VVC / H.266 (Versatile Video Coding) is a video compression standard intended as a successor to HEVC. ITU-T and ISO / IEC published the VVC / H.266 standard in 2020 (Version 1) and 2022 (Version 2). AOMedia Video1 (AV1) is an open video coding format designed as an alternative to HEVC. On January 8, 2019, verified version 1.0.0 with Errata 1 of the specification was released.

[0007] The present disclosure describes, above all, a set of methods for video (image) compression, and more specifically relates to inter-prediction modes. In inter-prediction, a current block may be encoded using motion vector information from one or more reference frames. Some embodiments described herein include signaling syntax in a bitstream to indicate whether the current block is encoded using information from the same reference frame or from two reference blocks from two different reference frames. For example, when the current block is encoded using information from two reference blocks belonging to the same reference frame, a first subset of composite inter-prediction modes is used. In this example, when the current block is encoded using information from two reference blocks belonging to two different reference frames, a second subset of composite inter-prediction modes is used. By using and signaling a subset of composite inter-prediction modes depending on whether the reference blocks are from the same or different reference frames, coding efficiency is increased.

[0008] According to some embodiments, a video decoding method is provided. The method comprises: (i) receiving a video bitstream comprising a plurality of blocks including a current block in a current frame; (ii) determining that the current block is encoded using motion information from a first reference block and a second reference block; (iii) selecting a composite inter prediction mode for the current block from a first set of composite inter prediction modes when the first reference block and the second reference block are in different reference frames; (iv) selecting a composite inter prediction mode for the current block from a second set of composite inter prediction modes when the first reference block and the second reference block are in the same reference frame—the second set of composite inter prediction modes is a subset of the first set of composite inter prediction modes—; and (v) reconstructing the current block using the composite inter prediction mode and motion information from the first reference block and the second reference block.

[0009] According to some embodiments, a video encoding method is provided. The method comprises: (i) receiving video data including a plurality of blocks including a current block; (ii) determining that the current block should be encoded using motion information from a first reference block and a second reference block; (iii) selecting a composite inter prediction mode for the current block from a first set of composite inter prediction modes when the first reference block and the second reference block are in different reference frames; (iv) selecting a composite inter prediction mode for the current block from a second set of composite inter prediction modes when the first reference block and the second reference block are in the same reference frame—the second set of composite inter prediction modes is a subset of the first set of composite inter prediction modes—; and (v) encoding the current block using the composite inter prediction mode and motion information from the first reference block and the second reference block.

[0010] According to some embodiments, a method for processing visual media data is provided. The method comprises: (i) acquiring a source video sequence comprising a plurality of frames; and (ii) performing a conversion between a video bitstream of visual media data and a source video sequence according to a format rule. The video bitstream comprises a plurality of encoded blocks including a current block. The format rule specifies: (a) that the current block must be decoded using motion information from a first reference block and a second reference block; (b) that when the first reference block and the second reference block are in different reference frames, a composite inter prediction mode must be selected for the current block from a first set of composite inter prediction modes; (c) that when the first reference block and the second reference block are in the same reference frame, a composite inter prediction mode must be selected for the current block from a second set of composite inter prediction modes, wherein the second set of composite inter prediction modes is a subset of the first set of composite inter prediction modes; and (d) that the current block must be reconstructed using a composite inter prediction mode and motion information from the first reference block and the second reference block.

[0011] According to some embodiments, a video decoding method is provided. The method comprises: (i) receiving a video bitstream comprising a plurality of blocks including a current block in a current frame; (ii) parsing from the video bitstream a first indicator indicating whether a composite inter prediction mode for the current block should be selected from a first set of composite inter prediction modes or from a second set of composite inter prediction modes; (iii) parsing from the video bitstream a second indicator indicating a mode index for a composite inter prediction mode; (iv) selecting a composite inter prediction mode from the first set of composite inter prediction modes using the mode index when the first indicator indicates that a composite inter prediction mode for the current block should be selected from the first set of composite inter prediction modes; (v) selecting a composite inter prediction mode from the second set of composite inter prediction modes using the mode index when the first indicator indicates that a composite inter prediction mode for the current block should be selected from the second set of composite inter prediction modes; and (vi) reconstructing the current block using the composite inter prediction mode.

[0012] According to some embodiments, a video encoding method is provided. The method comprises: (i) receiving video data including a plurality of blocks including a current block in a current frame; (ii) determining whether to use a first set of complex inter prediction modes or a second set of complex inter prediction modes for the current block; (iii) selecting a complex inter prediction mode from the first set of complex inter prediction modes using a mode index when using the first set of complex inter prediction modes; (iv) selecting a complex inter prediction mode from the second set of complex inter prediction modes when using the second set of complex inter prediction modes; (v) signaling a first indicator through a video bitstream indicating whether the complex inter prediction mode for the current block is selected from the first set of complex inter prediction modes or from the second set of complex inter prediction modes; (vi) signaling a second indicator through a video bitstream indicating a mode index for the complex inter prediction mode; and (vii) encoding the current block using the complex inter prediction mode.

[0013] According to some embodiments, a computing system such as a streaming system, a server system, a personal computer system, or other electronic device is provided. The computing system includes a control circuit and a memory storing one or more instruction sets. One or more instruction sets include instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a transcoder).

[0014] According to some embodiments, a non-transient computer-readable storage medium is provided. The non-transient computer-readable storage medium stores one or more instruction sets for execution by a computing system. One or more instruction sets include instructions for performing any of the methods described herein.

[0015] Accordingly, devices and systems having methods for encoding and decoding video are disclosed. These methods, devices, and systems may complement or replace conventional methods, devices, and systems for video encoding / decoding.

[0016] The features and benefits described herein are not necessarily all of the same, and in particular, some additional features and benefits will be apparent to a person skilled in the art by considering the drawings, specification, and claims provided in this disclosure. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and directional purposes and has not been chosen to describe or limit the subject matter described herein. Brief explanation of the drawing

[0017] To enable a more detailed understanding of the present disclosure, a more specific description may be made by referring to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the accompanying drawings merely illustrate the relevant features of the present disclosure and, therefore, should not be construed as limiting, as a person skilled in the art will understand this when reading the present disclosure and the description may allow for other effective features. FIG. 1 is a block diagram illustrating an exemplary communication system according to some embodiments. FIG. 2a is a block diagram illustrating exemplary elements of an encoder component according to some embodiments. FIG. 2b is a block diagram illustrating exemplary elements of a decoder component according to some embodiments. FIG. 3 is a block diagram illustrating an exemplary server system according to some embodiments. FIG. 4a illustrates the generation of exemplary motion vector candidates for a single inter prediction block according to some embodiments. FIG. 4b illustrates the generation of exemplary motion vector candidates for a composite prediction block according to some embodiments. FIG. 5a illustrates an exemplary video decoding process according to some embodiments. FIG. 5b illustrates an exemplary video encoding process according to some embodiments. According to general practice, the various features illustrated in the drawings are not necessarily drawn in a fixed proportion, and similar reference numbers may be used to indicate similar features throughout the specification and drawings. Specific details for implementing the invention

[0018] The present disclosure describes video / image compression techniques related to inter-prediction modes. Some embodiments select a composite inter-prediction mode for the current block from a first set of composite inter-prediction modes based on the determination that the current block is encoded using motion information from a first reference block and a second reference block located in different reference frames. Some embodiments select a composite inter-prediction mode for the current block from a second set of composite inter-prediction modes based on the determination that the current block is encoded using motion information from a first reference block and a second reference block located in the same reference frame. Some embodiments reconstruct the current block using the selected composite inter-prediction mode and motion information from the first reference block and the second reference block. Selecting a composite inter-prediction mode from each set of composite inter-prediction modes based on whether the reference blocks are from the same or different reference frames reduces redundancy and improves coding efficiency. For example, when both reference blocks are from the same reference frame (e.g., because the results of some modes will be very similar to each other), fewer modes need to be considered, which improves coding efficiency.

[0019] Exemplary systems and devices

[0020] FIG. 1 is a block diagram illustrating a communication system (100) according to some embodiments. The communication system (100) includes a source device (102) and a plurality of electronic devices (120) (e.g., electronic device (120-1) to electronic device (120-m)) that are coupled to communicate with each other through one or more networks. In some embodiments, the communication system (100) is a streaming system for use with video-enabled applications such as, for example, video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0021] The source device (102) includes a video source (104) (e.g., a camera component or media storage) and an encoder component (106). In some embodiments, the video source (104) is a digital camera (e.g., configured to generate an uncompressed video sample stream). The encoder component (106) generates one or more encoded video bitstreams from the video stream. The video stream from the video source (104) may have a high data volume compared to the encoded video bitstream (108) generated by the encoder component (106). Because the encoded video bitstream (108) has a lower data volume (less data) compared to the video stream from the video source, the encoded video bitstream (108) requires less bandwidth to transmit and less storage space to store compared to the video stream from the video source (104). In some embodiments, the source device (102) does not include an encoder component (106) (e.g., configured to transmit uncompressed video to network(s) (110)).

[0022] One or more networks (110) represent any number of networks that transmit information between a source device (102), a server system (112), and / or electronic devices (120), including, for example, wired and / or wireless communication networks. One or more networks (110) may exchange data on circuit-switched and / or packet-switched channels. Representative networks include telecommunication networks, local area networks, wide area networks, and / or the Internet.

[0023] One or more networks (110) include a server system (112) (e.g., a distributed / cloud computing system). In some embodiments, the server system (112) is or includes a streaming server (e.g., configured to store and / or distribute video content, such as an encoded video stream from a source device (102). The server system (112) includes a coder component (114) (e.g., configured to encode and / or decode video data). In some embodiments, the coder component (114) includes an encoder component and / or a decoder component. In various embodiments, the coder component (114) is instantiated as hardware, software, or a combination thereof. In some embodiments, the coder component (114) is configured to decode an encoded video bitstream (108) and re-encode the video data using a different encoding standard and / or methodology to produce encoded video data (116). In some embodiments, the server system (112) is configured to generate multiple video formats and / or encodings from an encoded video bitstream (108). In some embodiments, the server system (112) functions as a Media-Aware Network Element (MANE). For example, the server system (112) may be configured to prune the encoded video bitstream (108) to potentially customize different bitstreams for one or more of the electronic devices (120). In some embodiments, the MANE is provided separately from the server system (112).

[0024] The electronic device (120-1) includes a decoder component (122) and a display (124). In some embodiments, the decoder component (122) is configured to decode encoded video data (116) to generate an outgoing video stream that can be rendered on a display or other type of rendering device. In some embodiments, one or more of the electronic devices (120) do not include a display component (e.g., coupled to communicate with an external display device and / or including media storage). In some embodiments, the electronic devices (120) are streaming clients. In some embodiments, the electronic devices (120) are configured to access a server system (112) to obtain the encoded video data (116).

[0025] The source device and / or a plurality of electronic devices (120) are sometimes referred to as "terminal devices" or "user devices". In some embodiments, one or more of the source device (102) and / or electronic devices (120) are instances of a server system, a personal computer, a portable device (e.g., a smartphone, tablet, or laptop), a wearable device, a video conferencing device, and / or other types of electronic devices.

[0026] In an exemplary operation of the communication system (100), the source device (102) transmits an encoded video bitstream (108) to the server system (112). For example, the source device (102) may encode a stream of pictures captured by the source device. The server system (112) receives the encoded video bitstream (108) and may decode and / or encode the encoded video bitstream (108) using a coder component (114). For example, the server system (112) may apply encoding to video data that is more optimal for network transmission and / or storage. The server system (112) may transmit the encoded video data (116) (e.g., one or more encoded video bitstreams) to one or more of the electronic devices (120). Each electronic device (120) may decode the encoded video data (116) and optionally display the video pictures.

[0027] FIG. 2a is a block diagram illustrating exemplary elements of an encoder component (106) according to some embodiments. The encoder component (106) receives video data (e.g., source video sequence) from a video source (104). In some embodiments, the encoder component includes a receiver (e.g., transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component (106) receives a video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder component (106)). The video source (104) may provide the source video sequence in the form of a digital video sample stream which may be any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601Y CrCB, or RGB), and any suitable sampling structure (e.g., Y CrCb4:2:0 or Y CrCb4:4:4). In some embodiments, the video source (104) is a storage device that stores previously captured / prepared video. In some embodiments, the video source (104) is a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that impart motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, wherein each pixel may include one or more samples depending on the sampling structure, color space, etc., being used. A person skilled in the art can easily understand the relationship between the pixels and the samples.

[0028] The encoder component (106) is configured to code and / or compress pictures of a source video sequence into a coded video sequence (216) in real time or under other time constraints as required by the application. In some embodiments, the encoder component (106) is configured to perform a conversion between the source video sequence and a bitstream of visual media data (e.g., a video bitstream). Enforcing an appropriate coding rate is one function of the controller (204). In some embodiments, the controller (204) controls other function units and is functionally coupled to other function units as described below. Parameters set by the controller (204) may include rate-control-related parameters (e.g., picture skips, quantizers, and / or lambda values ​​of rate-distortion optimization techniques), picture size, picture group (GOP) layout, maximum motion vector search range, etc. A person skilled in the art can easily identify other functions of the controller (204) because they may be related to encoder components (106) that are optimized for a specific system design.

[0029] In some embodiments, the encoder component (106) is configured to operate in a coding loop. In a simplified example, the coding loop includes a source coder (202) (responsible for generating symbols, such as a symbol stream, based on the input picture to be coded and reference picture(s), e.g.,) and a (local) decoder (210). The decoder (210) reconstructs the symbols to generate sample data in a manner similar to the (remote) decoder (when compression between the symbols and the coded video bitstream is lossless). The reconstructed sample stream (sample data) is input into the reference picture memory (208). Because the decoding of the symbol stream leads to bit-exact results independently of the decoder location (local or remote), the content within the reference picture memory (208) is also bit-exact between the local encoder and the remote encoder. In this way, the prediction part of the encoder interprets the same sample values ​​that the decoder will interpret as reference picture samples when using the prediction during decoding.

[0030] The operation of the decoder (210) may be the same as the operation of a remote decoder such as the decoder component (122) described in detail below in relation to FIG. 2b. However, with brief reference to FIG. 2b, since symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (214) and parser (254) may be lossless, the entropy decoding portions of the decoder component (122), including the buffer memory (252) and parser (254), may not be fully implemented in the local decoder (210).

[0031] Except for parsing / entropy decoding, the decoder techniques described herein may exist in substantially the same functional form in the corresponding encoder. For this reason, the subject matter disclosed focuses on decoder operations. Additionally, descriptions of encoder techniques may be abbreviated because they may be the inverses of decoder techniques.

[0032] As part of the operation, the source coder (202) may perform motion-compensated predictive coding to predictively code an input frame by referencing one or more previously coded frames from a video sequence designated as reference frames. In this way, the coding engine (212) codes the differences between pixel blocks of the input frame and pixel blocks of the reference frame(s) that can be selected as predictive reference(s) for the input frame. The controller (204) may manage the coding operations of the source coder (202), including, for example, the settings of parameters and subgroup parameters used to encode video data.

[0033] The decoder (210) decodes the coded video data of frames that can be designated as reference frames based on symbols generated by the source coder (202). The operations of the coding engine (212) may advantageously be lossy processes. When the coded video data is decoded in a video decoder (not shown in FIG. 2a), the reconstructed video sequence may be a copy of the source video sequence having some errors. The decoder (210) may duplicate the decoding processes that can be performed by a remote video decoder on the reference frames and allow the reconstructed reference frames to be stored in the reference picture memory (208). In this way, the encoder component (106) locally stores copies of the reconstructed reference frames having common content as reconstructed reference frames to be acquired by a remote video decoder (without transmission errors).

[0034] The predictor (206) can perform predictive searches for the coding engine (212). That is, for a new frame to be coded, the predictor (206) can search the reference picture memory (208) for specific metadata or sample data (as candidate reference pixel blocks), such as reference picture motion vectors, block shapes, etc., which can serve as appropriate predictive references for the new pictures. The predictor (206) can operate on a sample block-by-pixel block basis to find appropriate predictive references. As determined by the search results obtained by the predictor (206), the input picture may have predictive references derived from a number of reference pictures stored in the reference picture memory (208).

[0035] The outputs of all the aforementioned functional units may undergo entropy coding in an entropy coder (214). The entropy coder (214) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to a person skilled in the art (e.g., Huffman coding, variable length coding, and / or arithmetic coding).

[0036] In some embodiments, the output of the entropy coder (214) is coupled to a transmitter. The transmitter may be configured to buffer the coded video sequence(s) generated by the entropy coder (214) to prepare them for transmission over a communication channel (218), which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter may be configured to merge the coded video data from the source coder (202) with other data to be transmitted, e.g., coded audio data and / or auxiliary data streams (sources not shown). In some embodiments, the transmitter may transmit additional data along with the encoded video. The source coder (202) may include this data as part of the coded video sequence. The additional data may include other forms of redundant data such as time / space / SNR enhancement layers, redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0037] The controller (204) can manage the operation of the encoder component (106). During coding, the controller (204) can assign a specific coded picture type to each coded picture, which may affect the coding techniques applied to each picture. For example, pictures may be assigned as intra pictures (I pictures), prediction pictures (P pictures), or bidirectional prediction pictures (B pictures). Intra pictures may be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. A person skilled in the art is aware of such variations of I pictures and their respective applications and features, so they are not repeated here. Prediction pictures may be coded and decoded using intra prediction or inter prediction, which use at most one motion vector and reference index to predict the sample values ​​of each block. Bidirectional prediction pictures can be coded and decoded using intra-prediction or inter-prediction, which use up to two motion vectors and reference indices to predict sample values ​​of each block. Similarly, multi-prediction pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0038] Source pictures are generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and can be coded on a block-by-block basis. Blocks can be predictively coded by referencing other (already coded) blocks determined by the coding assignment applied to each of the blocks' pictures. For example, blocks of I pictures can be non-predictively coded, or they can be predictively coded by referencing already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of P pictures can be non-predictively coded via spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks of B pictures can be non-predictively coded via spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0039] Video can be captured as multiple source pictures (video pictures) within a time sequence. Intra-picture prediction (often abbreviated as intra prediction) uses spatial correlations within a given picture, while inter-picture prediction uses (temporal or other) correlations between pictures. In one example, a specific picture being encoded / decoded, referred to as the current picture, is partitioned into blocks. When a block in the current picture is similar to a reference block in a previously encoded and still buffered reference picture within the video, the block in the current picture can be encoded by a vector referred to as a motion vector. The motion vector points to the reference block in the reference picture and, if multiple reference pictures are in use, may have three dimensions that identify the reference picture.

[0040] The encoder component (106) may perform coding operations according to a predetermined video coding technique or standard, such as any one described herein. In such operations, the encoder component (106) may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancies in the input video sequence. Accordingly, the coded video data may comply with the syntax specified by the video coding technique or standard being used.

[0041] FIG. 2b is a block diagram illustrating exemplary elements of a decoder component (122) according to some embodiments. The decoder component (122) of FIG. 2b is coupled to a channel (218) and a display (124). In some embodiments, the decoder component (122) includes a transmitter coupled to a loop filter (256) and configured to transmit data to the display (124) (e.g., via a wired or wireless connection).

[0042] In some embodiments, the decoder component (122) includes a receiver coupled to the channel (218) and configured to receive data from the channel (218) (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more coded video sequences to be decoded by the decoder component (122). In some embodiments, the decoding of each coded video sequence is independent of other coded video sequences. Each coded video sequence may be received from the channel (218), which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive the encoded video data along with other data, e.g., encoded audio data and / or auxiliary data streams, which may be forwarded to their respective use entities (not described). The receiver may separate the coded video sequence from the other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. This additional data may be included as part of the coded video sequence(s). Additional data may be used by a decoder component (122) to decode the data and / or reconstruct the original video data more accurately. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0043] According to some embodiments, the decoder component (122) includes a buffer memory (252), a parser (254) (sometimes referred to as an entropy decoder), a scaler / inverse conversion unit (258), an intra-picture prediction unit (262), a motion compensation prediction unit (260), an aggregator (268), a loop filter unit (256), a reference picture memory (266), and a current picture memory (264). In some embodiments, the decoder component (122) is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuits. The decoder component (122) may be implemented at least partially in software.

[0044] A buffer memory (252) is coupled between the channel (218) and the parser (254) (e.g., to prevent network jitter). In some embodiments, the buffer memory (252) is separate from the decoder component (122). In some embodiments, a separate buffer memory is provided between the output of the channel (218) and the decoder component (122). In some embodiments, in addition to the buffer memory (252) inside the decoder component (122) (e.g. configured to handle playback timing), a separate buffer memory is provided outside the decoder component (122) (e.g., to prevent network jitter). When receiving data from a storage / forward device with sufficient bandwidth and controllability, or from an isochronous synchronous network, the buffer memory (252) may not be required or may be small. For use on best effort packet networks such as the Internet, a buffer memory (252) may be required, may be relatively large and / or adaptive in size, and may be implemented at least partially in an operating system or similar elements outside the decoder component (122).

[0045] The parser (254) is configured to reconstruct symbols (270) from a coded video sequence. The symbols may include, for example, information used to manage the operation of a decoder component (122) and / or information for controlling a rendering device such as a display (124). The control information for the rendering device(s) may be in the form of, for example, Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not described). The parser (254) parses (entropy-decodes) the coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow principles widely known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (254) can extract a set of subgroup parameters for at least one of the subgroups of pixels in a video decoder from a coded video sequence based on at least one parameter corresponding to a group. The subgroups may include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, Coding Units (CU), blocks, Transform Units (TU), Prediction Units (PU), etc. The parser (254) can also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the coded video sequence.

[0046] The reconstruction of the symbols (270) may involve a number of different units depending on the type of the coded video picture or its parts (such as inter- and intra-pictures, inter- and intra-blocks) and other factors. Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by the parser (254). The flow of such subgroup control information between the parser (254) and the number of units below is not illustrated for clarity.

[0047] The decoder component (122) can be conceptually subdivided into a number of functional units, and in some implementations, these units may interact closely with each other and be integrated at least partially. However, for clarity, the conceptual subdivision of the functional units is retained in this specification.

[0048] The scaler / inverse transform unit (258) receives quantized transform coefficients as well as control information (such as which transform to use, block size, quantization factor, and / or quantization scaling matrices) as symbol(s) (270) from the parser (254). The scaler / inverse transform unit (258) may output blocks containing sample values ​​that can be input to the aggregator (268). In some cases, the output samples of the scaler / inverse transform unit (258) are of intra-coded blocks; that is: blocks that do not use prediction information from previously reconstructed pictures, but can use prediction information from previously reconstructed parts of the current picture. Such prediction information may be provided by the intra-picture prediction unit (262). The intra-picture prediction unit (262) may generate a block of the same size and shape as the block being reconstructed by using surrounding already reconstructed information fetched from the current (partially reconstructed) picture from the current picture memory (264). The aggregator (268) can add, on a sample-by-sample basis, the prediction information generated by the intra-picture prediction unit (262) to the output sample information provided by the scaler / inverse transformation unit (258).

[0049] In other cases, the output samples of the scaler / inverse unit (258) are intercoded and potentially related to a motion-compensated block. In these cases, the motion compensation prediction unit (260) may access the reference picture memory (266) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (270) regarding the block, these samples may be added by the aggregator (268) to the output of the scaler / inverse unit (258) (in this case, called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (266) where the motion compensation prediction unit (260) fetches the prediction samples may be controlled by motion vectors. Motion vectors may be available to the motion compensation prediction unit (260) in the form of symbols (270) that may have, for example, X, Y, and reference picture components. Motion compensation may also include motion vector prediction mechanisms, such as interpolation of sample values ​​fetched from reference picture memory (266) when, for example, sub-sample exact motion vectors are in use.

[0050] The output samples of the aggregator (268) may undergo various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filter techniques that are made available to the loop filter unit (256) as symbols (270) from the parser (254) and are controlled by parameters included in the coded video bitstream, but which respond not only to meta-information acquired during the decoding of previous (in decoding order) parts of the coded picture or coded video sequence, but also to previously reconstructed and loop-filtered sample values. The output of the loop filter unit (256) may be a sample stream that can be output to a render device such as a display (124), as well as stored in a reference picture memory (266) for use in future inter-picture predictions.

[0051] Certain coded pictures, once reconstructed, can be used as reference pictures for future predictions. When a coded picture is reconstructed and the coded picture is identified as a reference picture (e.g., by a parser (254)), the current reference picture may become part of the reference picture memory (266), and a new current picture memory may be reallocated before starting the reconstruction of the next coded picture.

[0052] The decoder component (122) may perform decoding operations according to a predetermined video compression technology that may be documented in a standard such as any of the standards described in this specification. The coded video sequence may comply with the syntax specified by the video compression technology or standard being used in that it adheres to the syntax of the video compression technology or standard as specified in the video compression technology document or standard and specifically in the profile document therein. Additionally, to comply with some video compression technologies or standards, the complexity of the coded video sequence may be within boundaries defined by the levels of the video compression technology or standard. In some cases, the levels limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the levels may, in some cases, be further limited by HRD (Hypothetical Reference Decoder) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0053] FIG. 3 is a block diagram illustrating a server system (112) according to some embodiments. The server system (112) includes a control circuit (302), one or more network interfaces (304), memory (314), a user interface (306), and one or more communication buses (312) for interconnecting these components. In some embodiments, the control circuit (302) includes one or more processors (e.g., CPU, GPU, and / or DPU). In some embodiments, the control circuit includes field programmable gate array(s), hardware accelerators, and / or integrated circuit(s) (e.g., application integrated circuit).

[0054] Network interface(s) (304) may be configured to interface with one or more communication networks (e.g., wireless, wired, and / or optical networks). The communication networks may be local, wide-area, metropolitan, automotive and industrial, real-time, latency-tolerant, etc. Examples of communication networks include local area networks such as Ethernet and wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; and automotive and industrial networks including CANBus. Such communication may be unidirectional, receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., CANbus for specific CANbus devices), or bidirectional (e.g., to other computer systems using local or wide-area digital networks). Such communication may include communication to one or more cloud computing networks.

[0055] The user interface (306) includes one or more output devices (308) and / or one or more input devices (310). The input device(s) (310) may include one or more of a keyboard, mouse, trackpad, touch screen, data glove, joystick, microphone, scanner, camera, etc. The output device(s) (308) may include one or more of an audio output device (e.g., speaker), a visual output device (e.g., display or monitor), etc.

[0056] The memory (314) may include high-speed random access memory (e.g., DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices) and / or non-volatile memory (e.g., one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices). The memory (314) optionally includes one or more storage devices located remotely from the control circuit (302). The memory (314), or alternatively, the non-volatile solid-state memory device(s) within the memory (314), includes a non-transient computer-readable storage medium. In some embodiments, the memory (314), or the non-transient computer-readable storage medium of the memory (314), contains the following programs, modules, instructions, and data structures, or these:

[0057] An operating system (316) including procedures for handling various basic system services and performing hardware-dependent tasks;

[0058] A network communication module (318) used to connect the server system (112) to other computing devices through one or more network interfaces (304) (e.g., through wired and / or wireless connections);

[0059] A subset or superset of coding modules (320) for performing various functions related to encoding and / or decoding data such as video data is stored. In some embodiments, the coding module (320) is an instance of the coder component (114). The coding module (320) includes, but is not limited to, one or more of the following:

[0060] A decoding module (322) for performing various functions related to decoding encoded data, such as those previously described in relation to the decoder component (122); and

[0061] An encoding module (340) for performing various functions related to encoding data, such as those previously described in relation to the encoder component (106); and

[0062] A picture memory (352) for use with, for example, a coding module (320), storing pictures and picture data. In some embodiments, the picture memory (352) includes one or more of a reference picture memory (208), a buffer memory (252), a current picture memory (264), and a reference picture memory (266).

[0063] In some embodiments, the decoding module (322) includes a parsing module (324) configured to perform various functions previously described in relation to, for example, a parser (254), a conversion module (326) configured to perform various functions previously described in relation to, for example, a scalar / inverse conversion unit (258), a prediction module (328) configured to perform various functions previously described in relation to, for example, a motion compensation prediction unit (260) and / or an intra-picture prediction unit (262), and a filter module (330) configured to perform various functions previously described in relation to, for example, a loop filter (256).

[0064] In some embodiments, the encoding module (340) includes a code module (342) (e.g., configured to perform various functions previously described in relation to the source coder (202) and / or coding engine (212)) and a prediction module (344) (e.g., configured to perform various functions previously described in relation to the predictor (206). In some embodiments, the decoding module (322) and / or the encoding module (340) includes a subset of the modules illustrated in FIG. 3. For example, a shared prediction module is used by both the decoding module (322) and the encoding module (340).

[0065] Each of the modules identified above stored in memory (314) corresponds to a set of instructions for performing the functions described herein. The modules identified above (e.g., sets of instructions) do not need to be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or rearranged in other ways in various embodiments. For example, the coding module (320) may not optionally include separate decoding and encoding modules, but rather use the same set of modules to perform the functions of both. In some embodiments, memory (314) stores a subset of the modules and data structures identified above. In some embodiments, memory (314) stores additional modules and data structures not described above.

[0066] FIG. 3 illustrates a server system (112) according to some embodiments, but FIG. 3 is intended more as a functional description of various features that may exist in one or more server systems rather than as a structural schematic diagram of the embodiments described herein. In practice, items illustrated individually may be combined and some items may be separated. For example, some items illustrated individually in FIG. 3 may be implemented on single servers, and single items may be implemented by one or more servers. The actual number of servers used to implement the server system (112), and how features are allocated among them, will vary from implementation to implementation and, optionally, depend in part on the amount of data traffic processed by the server system during average usage periods as well as during peak usage periods.

[0067] Exemplary coding techniques

[0068] The coding processes and techniques described below may be performed on the devices and systems described above (e.g., source device (102), server system (112), and / or electronic device (120)). According to some embodiments, methods for using signaling of a composite inter-prediction mode are described below.

[0069] FIG. 4a illustrates the generation of exemplary motion vector candidates for a single inter-prediction block according to some embodiments. As shown in FIG. 4a, mv1 from neighbor block A is used to derive an MVP for the motion vector mv0 of the current block having time scaling.

[0070] For complex inter-prediction, constructed MVs from different neighboring blocks are used to derive the MVP of the current block, but the reference frames of the constructed MVs must be identical to those of the current block. FIG. 4b illustrates the generation of exemplary motion vector candidates for a complex prediction block according to some embodiments. As shown in FIG. 4b, the constructed MVs (mv2, mv3) have reference frames identical to those of the current block but are from different neighboring blocks.

[0071] As mentioned above, an inter-coded block may utilize one or more associated motion vectors. In some embodiments, syntax is signaled in the bitstream to indicate whether the block is in a single inter-prediction mode (e.g., whether the block is encoded using motion information from a single reference block) or a complex inter-prediction mode (e.g., whether the block is encoded using motion information from two or more reference blocks). A single prediction mode has one associated motion vector, whereas a complex inter-prediction mode has multiple associated motion vectors.

[0072] In the case of a composite inter prediction mode, multiple reference blocks may be from the same reference frame. In some embodiments, for a composite inter prediction mode, two reference blocks may be from different reference frames. For the composite reference case, the composite inter prediction modes described in detail in Table 1 below may be used.

[0073]

[0074] Table 1 - Composite Reference Modes

[0075] In some embodiments, for complex inter prediction modes, a syntax is signaled to indicate whether optical flow-based motion vector refinement is applied. In some embodiments, the seven complex inter prediction modes shown in Table 1 are signaled with a single flag. In some embodiments, the seven complex inter prediction modes shown in Table 1 are signaled with six contexts, and the context derivation is shown in Table 2 below.

[0076]

[0077] Table 2 - Context Derivation.

[0078] In some embodiments, the composite inter-predicted modes shown in Table 1 are divided into multiple groups based on whether these modes are joint MVD coding modes. Then, the modes within each group are signaled individually. Some embodiments signal the inter-composite mode by first signaling whether it is a joint MVD coding mode. This flag is signaled in two contexts. A first context is used when two reference frames are from different sides and the distance between the reference frames and the current frame is the same. Otherwise, a second context is used. In some embodiments, if the current mode is a joint MVD coding mode, a flag is signaled to indicate whether the JOINT_NEWMV or JOINT_AMVDNEWMV mode is used. This flag is signaled in only one context. If the current mode is not a joint MVD coding mode, a flag is signaled to indicate which of the following five modes {NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, GLOBAL_GLOBALMV, NEW_NEWMV} is used. These five modes are signaled from the AVM anchor to the same context derivation process. The proposed signaling of the composite modes is shown in Table 3 below.

[0079]

[0080] Table 3 - Signaling of Intercomplex Mode.

[0081] Splitting complex inter-prediction modes into two groups and signaling the modes in each group individually can improve coding efficiency, for example, by reducing context memory from 570 bits to 450 bits. Table 4 below illustrates improvements to encoding and decoding based on simulations performed using current designs (e.g., AVM Study-v8.0.0 anchor) with various video data (e.g., representing AOM Common Test Conditions v7.0).

[0082]

[0083] Table 4 - Simulation Results

[0084] In some embodiments, the NEW_NEARMV mode is removed when two reference frames are identical. In some embodiments, because JOINT_NEWMV and JOINT_AMVDNEWMV are designed for motion vectors from two reference frames having different orientations, these two modes are also removed when two reference frames are from the same side. In some embodiments, removing these modes improves coding efficiency by reducing the number of contexts from 30 to 20, reducing the RAM size from 400 bits to 250 bits, and reducing the ROM size from 445 to 295 bits.

[0085] Table 5 compares context memory when NEW_NEARMV mode is removed, and when JOINT_NEWMV and JOINT_AMVDNEWMV are also removed.

[0086]

[0087] Table 5 - Comparison of Context Memory

[0088] FIG. 5a is a flowchart illustrating a method (500) for decoding video according to some embodiments. The method (500) may be performed in a computing system (e.g., a server system (112), a source device (102), or an electronic device (120)) having a control circuit and a memory storing instructions for execution by the control circuit. In some embodiments, the method (500) is performed by executing instructions stored in the memory of the computing system (e.g., memory (314)).

[0089] The system receives a video bitstream (e.g., a coded video sequence) containing a plurality of blocks (e.g., corresponding to a set of pictures) that include the current block within the current frame (502). The system determines that the current block is encoded using motion information from a first reference block and a second reference block (504). When the first reference block and the second reference block are in different reference frames, the system selects a composite inter prediction mode for the current block from a first set of composite inter prediction modes (506). When the first reference block and the second reference block are in the same reference frame, the system selects a composite inter prediction mode for the current block from a second set of composite inter prediction modes (508), wherein the second set of composite inter prediction modes is a subset of the first set of composite inter prediction modes. The system reconstructs the current block using the composite inter prediction mode and motion information from the first reference block and the second reference block (510). In this way, for a composite inter prediction mode, only a subset of composite inter prediction modes is used when two motion vectors point to the same reference frame.

[0090] In some embodiments, for the NEAR_NEWMV and NEW_NEARMV modes, only one of them is allowed and signaling occurs when both motion vectors point to the same reference frame. For example, only NEAR_NEWMV may be allowed.

[0091] In some embodiments, JOINT_NEWMV and JOINT_AMVDNEWMV modes are not allowed and are not signaled when the motion vectors of both point to the same reference frame.

[0092] In some embodiments, when two motion vectors point to the same reference frame, the following four composite inter-prediction modes, namely NEAR_NEARMV, NEAR_NEWMV, NEW_NEWMV, and GLOBAL_GLOBALMV, are allowed and signaled.

[0093] In some embodiments, GLOBAL_GLOBALMV is not allowed when two motion vectors point to the same reference frame.

[0094] In some embodiments, the allowed composite inter prediction modes are NEAR_NEARMV, NEAR_NEWMV, and NEW_NEWMV.

[0095] In some embodiments, complex inter-predicted modes are classified into two or more groups / sets, and one flag is signaled to indicate the set index of the complex inter-predicted mode, and another flag is signaled to indicate the index of the mode in the corresponding set.

[0096] In some embodiments, composite inter prediction modes are classified into two sets based on whether they have a joint motion vector difference (MVD) coding mode.

[0097] In some embodiments, the composite modes in the first set are JOINT_NEWMV and JOINT_AMVDNEWMV, while the composite modes in the second set are NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV.

[0098] In some embodiments, the context for signaling the set index of composite inter prediction modes may rely on information from two reference frames.

[0099] In some embodiments, the context for signaling the set index of composite inter prediction modes may depend on whether the two reference frames are from the same side.

[0100] In some embodiments, the context for signaling the set index of composite inter-prediction modes may depend on the distance between two reference frames and the current frame.

[0101] In some embodiments, the context for signaling the set index of composite inter-prediction modes may depend on whether the distance between two reference frames and the current frame is the same.

[0102] In some embodiments, the context for signaling the set index of composite inter prediction modes may rely on a time layer index or qindex.

[0103] In some embodiments, the context depends on whether they have the same time layer index or the same qindex.

[0104] In some embodiments, composite inter prediction modes are classified into two sets based on whether they have MVDs for both reference frames.

[0105] In some embodiments, the composite inter prediction modes in the first set are NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, and GLOBAL_GLOBALMV, while the composite inter prediction modes in the second set are NEW_NEWMV, JOINT_NEWMV, and JOINT_AMVDNEWMV.

[0106] In some embodiments, composite inter prediction modes are classified into two sets based on whether the adaptive MVD precision method is applied to one mode.

[0107] In some embodiments, the composite inter prediction modes in the first set are NEAR_NEARMV, NEW_NEWMV, and JOINT_NEWMV. The composite inter prediction modes in the second set are NEW_NEARMV, NEAR_NEWMV, and JOINT_AVMDNEWMV.

[0108] FIG. 5b is a flowchart illustrating a method (550) for encoding video according to some embodiments. The method (550) may be performed in a computing system (e.g., a server system (112), a source device (102), or an electronic device (120)) having a control circuit and a memory storing instructions for execution by the control circuit. In some embodiments, the method (550) is performed by executing instructions stored in the memory of the computing system (e.g., memory (314)). In some embodiments, the method (550) is performed by the same system as the method (500) described above.

[0109] The system receives video data (e.g., source video sequence) containing a plurality of blocks (e.g., corresponding to a set of pictures) including the current block (552). The system determines that the current block should be encoded using motion information from a first reference block and a second reference block (554). When the first reference block and the second reference block are in different reference frames, the system selects a composite inter prediction mode for the current block from a first set of composite inter prediction modes (556). When the first reference block and the second reference block are in the same reference frame, the system selects a composite inter prediction mode for the current block from a second set of composite inter prediction modes (558). The second set of composite inter prediction modes is a subset of the first set of composite inter prediction modes. The system encodes the current block using the composite inter prediction mode and motion information from the first reference block and the second reference block (560). As previously described, the encoding process may mirror the decoding processes (e.g., composite inter prediction) described herein. For the sake of brevity, these details are not repeated here.

[0110] FIGS. 5a and 5b illustrate a number of logical stages in a specific order, but stages that are not order-dependent may be rearranged, and other stages may be combined or separated. Since some rearrangements or other groupings not specifically mentioned will be obvious to those skilled in the art, the orderings and groupings presented herein are not comprehensive. Furthermore, it should be recognized that the stages may be implemented in hardware, firmware, software, or any combination thereof.

[0111] Now, refer to some exemplary embodiments.

[0112] (A1) In one embodiment, some embodiments include a method of video decoding (e.g., method (500)). In some embodiments, the method is performed in a computing system having memory and control circuits (e.g., server system (112)). In some embodiments, the method is performed in a coding module (e.g., coding module (320)). In some embodiments, the method is performed in a source coding component (e.g., source coder (202)), a coding engine (e.g., coding engine (212)), and / or an entropy coder (e.g., entropy coder (214)). The method comprises: (i) receiving a video bitstream (e.g., coded video sequence) comprising a plurality of blocks including a current block within a current frame; (ii) determining that the current block is encoded using motion information from a first reference block and a second reference block; (iii) selecting a composite inter prediction mode for the current block from a first set of composite inter prediction modes when the first reference block and the second reference block are in different reference frames; (iv) selecting a composite inter prediction mode for the current block from a second set of composite inter prediction modes when the first reference block and the second reference block are in the same reference frame—the second set of composite inter prediction modes is a subset of the first set of composite inter prediction modes—; and (v) reconstructing the current block using the composite inter prediction mode and motion information from the first reference block and the second reference block. For example, for a composite inter prediction mode, only a subset of composite inter prediction modes is used when two motion vectors point to the same reference frame. In some embodiments, a composite inter prediction mode for the current block is selected from a subset of available composite inter prediction modes based on the determination that the first and second reference blocks are in the same reference frame.

[0113] (A2) In some embodiments of A1, the second set of composite inter prediction modes does not include a joint motion vector difference (MVD) composite inter prediction mode. For example, JOINT_NEWMV and JOINT_AMVDNEWMV modes may not be allowed (and may not be signaled) when both motion vectors point to the same reference frame.

[0114] (A3) In some embodiments of A2, the second set of composite inter prediction modes does not include JOINT_NEWMV mode and JOINT_AMVDNEWMV mode.

[0115] (A4) In some embodiments of any of A1-A3, the second set of composite inter-prediction modes includes only one of the NEAR_NEWMV mode and the NEW_NEARMV mode. For example, for the NEAR_NEWMV and NEW_NEARMV modes, only one of them is allowed and is signaled when both motion vectors point to the same reference frame.

[0116] (A5) In some embodiments of any of A4, the second set of composite inter-prediction modes includes the NEAR_NEWMV mode and does not include the NEW_NEARMV mode. For example, only NEAR_NEWMV is allowed for the second set.

[0117] (A6) In some embodiments of any of A1-A5, the second set of composite inter prediction modes consists of four composite inter prediction modes. For example, when two motion vectors point to the same reference frame, the following four composite inter prediction modes, namely NEAR_NEARMV, NEAR_NEWMV, NEW_NEWMV, and GLOBAL_GLOBALMV, may be allowed and signaled. In some embodiments, the second set of composite inter prediction modes has a size of four or fewer.

[0118] (A7) In some embodiments of any of A1-A6, the second set of composite inter prediction modes consists of three composite inter prediction modes. For example, the allowed composite inter prediction modes are NEAR_NEARMV, NEAR_NEWMV, and NEW_NEWMV. In some embodiments, the second set of composite inter prediction modes does not include any global MV modes or co-MV modes.

[0119] (A8) In some embodiments of any of A1-A7, the second set of composite inter-prediction modes does not include a global motion vector mode. For example, GLOBAL_GLOBALMV is not allowed when two motion vectors point to the same reference frame.

[0120] (A9) In some embodiments of any of A1-A8, the method further comprises the step of parsing a first indicator in a video bitstream to determine whether to select a complex inter prediction mode from a first set of complex inter prediction modes or to select a complex inter prediction mode from a second set of complex inter prediction modes. For example, complex inter prediction modes may be classified into two or more sets, and a flag may be signaled to indicate a set index for two or more sets.

[0121] (A10) In some embodiments of any of A1-A8, the method further comprises the step of parsing a second indicator in a video bitstream, wherein the second indicator indicates an index for a composite inter-prediction mode. For example, a second flag may be signaled to indicate an index of a mode in a corresponding set.

[0122] (B1) In other embodiments, some embodiments include a method of video encoding (e.g., method (550)). In some embodiments, the method is performed in a computing system having memory and control circuits (e.g., a server system (112)). In some embodiments, the method is performed in a coding module (e.g., a coding module (320)). The method comprises: (i) receiving video data (e.g., a source video sequence) comprising a plurality of blocks including a current block; (ii) determining that the current block should be encoded using motion information from a first reference block and a second reference block; (iii) selecting a composite inter prediction mode for the current block from a first set of composite inter prediction modes when the first reference block and the second reference block are in different reference frames; (iv) selecting a composite inter prediction mode for the current block from a second set of composite inter prediction modes when the first reference block and the second reference block are in the same reference frame—the second set of composite inter prediction modes is a subset of the first set of composite inter prediction modes—; and (v) includes the step of encoding the current block using composite inter-prediction mode and motion information from the first reference block and the second reference block. In some embodiments, the encoded current block is signaled in a video bitstream.

[0123] (B2) In some embodiments of B1, the second set of composite inter prediction modes does not include a joint motion vector difference (MVD) composite inter prediction mode.

[0124] (B3) In some embodiments of B1 or B2, the method further includes the step of signaling a first indicator in the video bitstream that indicates whether to select a complex inter prediction mode from a first set of complex inter prediction modes or to select a complex inter prediction mode from a second set of complex inter prediction modes.

[0125] (B4) In some embodiments of any of B3, the method further comprises the step of signaling a second indicator in a video bitstream, and the second indicator indicates an index for a composite inter prediction mode.

[0126] (B5) In some embodiments of any of B1-B4, the method further comprises the step of signaling a second set of composite inter-prediction modes when the first reference block and the second reference block are in the same reference frame. For example, the second set is signaled and the first set is not signaled.

[0127] (B6) In some embodiments of any of B1-B5, the method further comprises the step of signaling a first set of composite inter prediction modes when the first reference block and the second reference block are in different reference frames. In some embodiments, the first set of composite inter prediction modes includes all available composite inter prediction modes.

[0128] (C1) In other embodiments, some embodiments include a method for processing visual media data. In some embodiments, the method is performed in a computing system having memory and control circuits (e.g., a server system (112)). In some embodiments, the method is performed in a coding module (e.g., a coding module (320)). The method comprises: (i) acquiring a source video sequence comprising a plurality of frames; and (ii) performing a conversion between a video bitstream of visual media data and a source video sequence according to a format rule. The video bitstream comprises a plurality of encoded blocks including a current block. The format rule is such that (a) the current block must be decoded using motion information from a first reference block and a second reference block; and (b) when the first reference block and the second reference block are in different reference frames, a composite inter prediction mode must be selected for the current block from a first set of composite inter prediction modes; (c) when the first reference block and the second reference block are in the same reference frame, a composite inter prediction mode is selected for the current block from a second set of composite inter prediction modes, and the second set of composite inter prediction modes is a subset of the first set of composite inter prediction modes; and (d) specifies that the current block is to be reconstructed using the composite inter prediction mode and motion information from the first reference block and the second reference block.

[0129] (D1) In other embodiments, some embodiments include a method (of video decoding). In some embodiments, the method is performed in a computing system having memory and control circuits (e.g., a server system (112)). In some embodiments, the method is performed in a coding module (e.g., a coding module (320)). In some embodiments, the method is performed in a source coding component (e.g., a source coder (202)), a coding engine (e.g., a coding engine (212)), and / or an entropy coder (e.g., an entropy coder (214)). The method comprises: (i) receiving a video bitstream comprising a plurality of blocks including a current block in a current frame; (ii) parsing from the video bitstream a first indicator indicating whether a complex inter-prediction mode for the current block should be selected from a first set of complex inter-prediction modes or from a second set of complex inter-prediction modes; (iii) parsing from the video bitstream a second indicator indicating a mode index for a complex inter-prediction mode; (iv) a step of selecting a composite inter prediction mode from a first set of composite inter prediction modes using a mode index when a first indicator indicates that a composite inter prediction mode for the current block should be selected from a first set of composite inter prediction modes; (v) a step of selecting a composite inter prediction mode from a second set of composite inter prediction modes using a mode index when a first indicator indicates that a composite inter prediction mode for the current block should be selected from a second set of composite inter prediction modes; and (vi) a step of reconstructing the current block using a composite inter prediction mode. For example, for composite inter prediction modes, they are classified into two or more groups / sets, one flag is signaled to indicate the set index of the composite inter prediction mode, and another flag is signaled to indicate the index of the mode in the corresponding set.

[0130] (D2) In some embodiments of D1, only one of the first set of composite inter prediction modes and the second set of composite inter prediction modes includes a joint MVD mode. For example, composite inter prediction modes may be classified into two sets based on whether they are joint MVD coding modes. For example, the composite modes in the first set may include JOINT_NEWMV and JOINT_AMVDNEWMV, while the composite modes in the second set may include NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV.

[0131] (D3) In some embodiments of D1 or D2, only one of the first set of composite inter prediction modes and the second set of composite inter prediction modes includes composite modes having MVDs for both reference frames. For example, composite inter prediction modes may be classified into two sets based on whether they have MVDs for both reference frames. For example, the composite inter prediction modes in the first set may include NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, and GLOBAL_GLOBALMV, while the composite inter prediction modes in the second set may include NEW_NEWMV, JOINT_NEWMV, and JOINT_AMVDNEWMV.

[0132] (D4) In some embodiments of any of D1-D3, only one of the first set of composite inter prediction modes and the second set of composite inter prediction modes includes composite modes having adaptive MVD precision. For example, composite inter prediction modes may be classified into two sets based on whether the adaptive MVD precision method is applied to one mode. For example, the composite inter prediction modes in the first set may include NEAR_NEARMV, NEW_NEWMV, and JOINT_NEWMV. The composite inter prediction modes in the second set may include NEW_NEARMV, NEAR_NEWMV, and JOINT_AVMDNEWMV.

[0133] (D5) In some embodiments of any of D1-D4, the current block is reconstructed using composite inter-prediction mode and motion information from the first reference frame and the second reference frame.

[0134] (D6) In some embodiments of D5, the first indicator is entropy decoded using the first context, and the first context is selected based on information regarding the first reference frame and the second reference frame. For example, the context for signaling the set index of complex inter-prediction modes may rely on information from the two reference frames.

[0135] (D7) In some embodiments of D6, information regarding the first reference frame and the second reference frame includes the first reference frame and the second reference frame being on the same side of the current frame. For example, the context for signaling the set index of composite inter-predict modes may depend on whether the two reference frames are from the same side. In some embodiments, the first context is selected when the first and second reference frames are on the same side, and the second context is selected when the first and second reference frames are on different sides.

[0136] (D8) In some embodiments of D6 or D7, information regarding the first reference frame and the second reference frame includes information regarding the distances between the current frame and the first and second reference frames. For example, the context for signaling the set index of composite inter-prediction modes may depend on the distance between the two reference frames and the current frame.

[0137] (D9) In some embodiments of any of D6-D8, a first context is selected when the distances between the current frame and the first and second reference frames are the same, and a second context is selected when the distances between the current frame and the first and second reference frames are different. For example, a context for signaling a set index of composite inter-prediction modes may depend on whether the distance between two reference frames and the current frame is the same.

[0138] (D10) In some embodiments of any of D6-D9, information regarding the first reference frame and the second reference frame includes a time layer index and / or a quantization index. For example, the context for signaling the set index of complex inter-prediction modes may depend on the time layer index or qindex. In some embodiments, the first context is selected when the first and second reference frames have the same time layer index (or qindex), and the second context is selected when the first and second reference frames have different time layer indices (or qindexes). For example, the context may depend on whether the reference frames have the same time layer index or the same qindex.

[0139] (E1) In other embodiments, some embodiments include a video encoding method. In some embodiments, the method is performed in a computing system having memory and control circuits (e.g., a server system (112)). In some embodiments, the method is performed in a coding module (e.g., a coding module (320)). The method comprises: (i) receiving video data including a plurality of blocks including a current block in a current frame; (ii) determining whether to use a first set of complex inter prediction modes or a second set of complex inter prediction modes for the current block; (iii) when using the first set of complex inter prediction modes, selecting a complex inter prediction mode from the first set of complex inter prediction modes using a mode index; (iv) when using the second set of complex inter prediction modes, selecting a complex inter prediction mode from the second set of complex inter prediction modes; (v) signaling a first indicator through a video bitstream that indicates whether the complex inter prediction mode for the current block is selected from the first set of complex inter prediction modes or from the second set of complex inter prediction modes; (vi) signaling a second indicator indicating a mode index for a composite inter prediction mode through a video bitstream; and (vii) encoding the current block using the composite inter prediction mode.

[0140] In another aspect, some embodiments include a computing system (e.g., a server system (112)) comprising a control circuit (e.g., a control circuit (302)) and a memory coupled to the control circuit (e.g., a memory (314)), wherein the memory stores one or more instruction sets configured to be executed by the control circuit, and the one or more instruction sets include instructions for performing any of the methods described herein (e.g., A1-A10, B1-B6, C1, D1-10, and E1 above).

[0141] In another aspect, some embodiments comprise a non-transient computer-readable storage medium storing one or more instruction sets for execution by a control circuit of a computing system, wherein the one or more instruction sets comprise instructions for performing any of the methods described herein (e.g., A1-A10, B1-B6, C1, D1-10, and E1 above).

[0142] Unless otherwise specified, any of the syntax elements (e.g., indicators) described herein may be High-Level Syntax (HLS). As used herein, HLS is signaled at a level higher than the block level. For example, HLS may correspond to the sequence level, frame level, slice level, or tile level. As another example, HLS elements may be signaled in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), adaptation parameter set (APS), slice header, picture header, tile header, and / or CTU header.

[0143] It will be understood that while terms such as “first,” “second,” etc., may be used in this specification to describe various elements, these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. The terms used in this specification are intended only to describe specific embodiments and are not intended to limit the claims. As used in the description of the embodiments and the appended claims, singular forms (“a,” “an,” and “the”) are intended to include plural forms unless the context otherwise clearly indicates. It will also be understood that the term “and / or,” as used in this specification, refers to and encompasses any and all possible combinations of one or more of the associated enumerated items. It will be further understood that the terms “comprising” and / or “comprising,” when used in this specification, specify the presence of the mentioned features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0144] As used herein, the term “if” may be interpreted, depending on the context, to mean “when” or “at” or “in response to a decision” or “according to a decision” or “in response to a detection” that the mentioned antecedent is true. Similarly, the phrases "[when the mentioned antecedent is determined to be true]” or "[when the mentioned antecedent is true]” or "[when the mentioned antecedent is true]” may be interpreted, depending on the context, to mean “when the mentioned antecedent is determined to be true” or “in response to a decision” or “according to a decision” or “when it is detected” or “in response to a detection”.

[0145] The foregoing description has been made with reference to specific embodiments for the purposes of explanation. However, the exemplary discussions above are not intended to be comprehensive or to limit the claims to the exact forms disclosed. Many modifications and variations are possible in light of the teachings above. The embodiments have been selected and described to enable other people skilled in the art to best explain the principles of operation and actual application.

Claims

Claim 1 A method of video decoding performed in a computing system having memory and one or more processors, comprising: receiving a video bitstream comprising a plurality of blocks including a current block in a current frame; determining that the current block is encoded using motion information from a first reference block and a second reference block; selecting a composite inter prediction mode for the current block from a first set of composite inter prediction modes when the first reference block and the second reference block are in different reference frames; selecting a composite inter prediction mode for the current block from a second set of composite inter prediction modes when the first reference block and the second reference block are in the same reference frame, wherein the second set of composite inter prediction modes is a subset of the first set of composite inter prediction modes; and reconstructing the current block using the composite inter prediction mode and motion information from the first reference block and the second reference block. Claim 2 A method according to claim 1, wherein the second set of composite inter prediction modes does not include a joint motion vector difference (MVD) composite inter prediction mode. Claim 3 A method according to paragraph 2, wherein the second set of composite inter-prediction modes does not include JOINT_NEWMV mode and JOINT_AMVDNEWMV mode. Claim 4 A method according to claim 1, wherein the second set of the composite inter-prediction modes comprises only one of the NEAR_NEWMV mode and the NEW_NEARMV mode. Claim 5 In paragraph 4, the second set of the composite inter-prediction modes includes the NEAR_NEWMV mode and does not include the NEW_NEARMV mode. Claim 6 A method according to claim 1, wherein the second set of the composite inter prediction modes consists of four composite inter prediction modes. Claim 7 A method according to claim 1, wherein the second set of the composite inter prediction modes consists of three composite inter prediction modes. Claim 8 A method according to claim 1, wherein the second set of the composite inter-prediction modes does not include a global motion vector mode. Claim 9 A method according to claim 1, further comprising the step of parsing a first indicator in the video bitstream to determine whether to select the complex inter prediction mode from a first set of complex inter prediction modes or to select the complex inter prediction mode from a second set of complex inter prediction modes. Claim 10 A method according to claim 9, further comprising the step of parsing a second indicator in the video bitstream, wherein the second indicator indicates an index for the composite inter prediction mode. Claim 11 A method of video encoding performed in a computing system having memory and one or more processors, comprising: receiving video data including a plurality of blocks including a current block; determining that the current block should be encoded using motion information from a first reference block and a second reference block; selecting a composite inter prediction mode for the current block from a first set of composite inter prediction modes when the first reference block and the second reference block are in different reference frames; selecting a composite inter prediction mode for the current block from a second set of composite inter prediction modes when the first reference block and the second reference block are in the same reference frame, wherein the second set of composite inter prediction modes is a subset of the first set of composite inter prediction modes; and encoding the current block using the composite inter prediction mode and motion information from the first reference block and the second reference block. Claim 12 In claim 11, the method wherein the second set of the composite inter-prediction modes does not include a joint motion vector difference (MVD) composite inter-prediction mode. Claim 13 A method according to claim 11, further comprising the step of signaling a first indicator in a video bitstream indicating whether to select the complex inter prediction mode from a first set of complex inter prediction modes or the complex inter prediction mode from a second set of complex inter prediction modes. Claim 14 A method according to claim 13, further comprising the step of signaling a second indicator in the video bitstream, wherein the second indicator indicates an index for the composite inter prediction mode. Claim 15 A method according to claim 11, further comprising the step of signaling a second set of composite inter-prediction modes when the first reference block and the second reference block are in the same reference frame. Claim 16 A method according to claim 11, further comprising the step of signaling a first set of composite inter-prediction modes when the first reference block and the second reference block are in different reference frames. Claim 17 A non-transient computer-readable storage medium storing a video bitstream generated by a video encoding method, wherein the video encoding method comprises: receiving video data including a plurality of blocks including a current block; determining that the current block should be encoded using motion information from a first reference block and a second reference block; selecting a complex inter prediction mode for the current block from a first set of complex inter prediction modes when the first reference block and the second reference block are in different reference frames; selecting a complex inter prediction mode for the current block from a second set of complex inter prediction modes when the first reference block and the second reference block are in the same reference frame, wherein the second set of complex inter prediction modes is a subset of the first set of complex inter prediction modes; and encoding the current block using the complex inter prediction mode and motion information from the first reference block and the second reference block; wherein the video bitstream includes the encoded current block. Claim 18 In paragraph 17, the video bitstream comprises a representation of a first set of composite inter-prediction modes when the first reference block and the second reference block are in different reference frames, in a non-transient computer-readable storage medium. Claim 19 In paragraph 17, the video bitstream comprises a representation of a second set of composite inter-prediction modes when the first reference block and the second reference block are in the same reference frame, in a non-transient computer-readable storage medium. Claim 20 In paragraph 17, the second set of the above-mentioned complex inter-prediction modes is a non-transient computer-readable storage medium that does not include a joint motion vector difference (MVD) complex inter-prediction mode.

Images