Improvements to inter-frame prediction mode signaling

By introducing a composite inter-frame prediction mode set into video coding and selecting an appropriate prediction mode set based on different reference frames, the problem of high redundancy in inter-frame prediction modes is solved, thereby improving coding efficiency and performance.

CN123125852APending Publication Date: 2026-11-17TENCENT AMERICA LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202580027264.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2025-05-13
Filing Date
2025-05-15
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing video coding techniques suffer from high redundancy and low coding efficiency in inter-frame prediction modes, especially when using motion vector information from different reference frames, making it difficult to effectively utilize mode differences.

Method used

By introducing a composite inter-frame prediction mode set, different prediction mode sets are selected based on whether the reference block comes from the same or different reference frames, reducing redundancy and improving coding efficiency. Specifically, the method involves signaling in the video bitstream whether reference block information from the same or different reference frames is used, and selecting an appropriate inter-frame prediction mode from the corresponding prediction mode set.

Benefits of technology

It improves the efficiency of video coding, reduces redundancy, and enhances coding performance, especially when using motion vector information from different reference frames, thereby reducing data volume and storage requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123125852A_ABST
    Figure CN123125852A_ABST
Patent Text Reader

Abstract

The various implementations described herein include methods and systems for encoding and decoding video. In one aspect, the method includes: receiving a video bitstream comprising a plurality of blocks, said plurality of blocks including a current block. The method includes: determining that the current block is encoded using motion information from a first reference block and a second reference block. The method includes: (i) selecting a composite inter-frame prediction mode for the current block from a first set of composite inter-frame prediction modes when the first reference block and the second reference block are located in different reference frames, and (ii) selecting a composite inter-frame prediction mode for the current block from a second set of composite inter-frame prediction modes when the first reference block and the second reference block are located in the same reference frame. The method includes: reconstructing the current block using the composite inter-frame prediction mode and the motion information from the first and second reference blocks.
Need to check novelty before this filing date? Find Prior Art

Description

Related applications

[0001] This application is a continuation of U.S. Patent Application No. 19 / 207,254, filed May 13, 2025, which claims priority to U.S. Provisional Patent Application No. 63 / 701,647, filed October 1, 2024, entitled “Improvements on Signaling Inter Prediction Mode,” both of which are incorporated herein by reference in their entirety. Technical Field

[0002] The disclosed implementations generally relate to video coding and decoding, including but not limited to systems and methods for inter-frame predictive mode coding and signaling. Background Technology

[0003] Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital camera devices, digital recording devices, digital media players, video game consoles, smartphones, video conferencing equipment, video streaming devices, etc. Electronic devices send and receive or otherwise transmit digital video data across communication networks, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video encoding can be used to compress video data according to one or more video encoding standards before transmission or storage. Video encoding can be performed by hardware and / or software on servers or electronic / client devices providing cloud services.

[0004] Video coding typically uses prediction methods that leverage the inherent redundancy in video data (e.g., inter-frame prediction, intra-frame prediction, etc.). Video coding aims to compress video data into a form that uses a lower bitrate while avoiding or minimizing video quality degradation. Several video codec standards have been developed. For example, High-Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H (Moving Picture Experts Group-H) project. The ITU-T (International Telecommunication Union-Telecommunication Standardization Sector) and ISO / IEC (International Organization for Standardization / International Electrotechnical Commission) released the HEVC / H.265 standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Versatile Video Coding (VVC / H.266) is a video compression standard designed as a successor to HEVC. The ITU-T and ISO / IEC released the VVC / H.266 standard in 2020 (Revision 1) and 2022 (Revision 2). AOMedia Video 1 (Alliance for Open Media Video 1) AV1 is an open video coding format designed as an alternative to HEVC. On January 8, 2019, a validated version 1.0.0 of the specification with errata table 1 was released. Summary of the Invention

[0005] Among other things, this disclosure describes a set of methods for video (image) compression, more specifically relating to inter-frame prediction modes. In inter-frame prediction, motion vector information from one or more reference frames can be used to encode the current block. Some implementations described herein include signaling a syntax in the bitstream to indicate whether the current block is encoded using information from two reference blocks belonging to the same reference frame or information from two reference blocks belonging to two different reference frames. For example, when encoding the current block using information from two reference blocks belonging to the same reference frame, a first composite inter-frame prediction mode subset is employed. In this example, when encoding the current block using information from two reference blocks belonging to two different reference frames, a second composite inter-frame prediction mode subset is employed. By employing and signaling composite inter-frame prediction mode subsets based on whether the reference block comes from the same or different reference frames, coding efficiency is improved.

[0006] According to some embodiments, a method for video decoding is provided. The method includes: (i) receiving a video bitstream comprising a plurality of blocks, the plurality of blocks including a current block in a current frame; (ii) determining that the current block is encoded using motion information from a first reference block and a second reference block; (iii) when the first reference block and the second reference block are located in different reference frames, selecting a composite inter-frame prediction mode for the current block from a first set of composite inter-frame prediction modes; (iv) when the first reference block and the second reference block are located in the same reference frame, selecting a composite inter-frame prediction mode for the current block from a second set of composite inter-frame prediction modes, wherein the second set of composite inter-frame prediction modes is a subset of the first set of composite inter-frame prediction modes; and (v) reconstructing the current block using the composite inter-frame prediction mode and the motion information from the first reference block and the second reference block.

[0007] According to some embodiments, a video encoding method is provided. The method includes: (i) receiving video data comprising a plurality of blocks, the plurality of blocks including a current block; (ii) determining to encode the current block using motion information from a first reference block and a second reference block; (iii) when the first reference block and the second reference block are located in different reference frames, selecting a composite inter-frame prediction mode for the current block from a first set of composite inter-frame prediction modes; (iv) when the first reference block and the second reference block are located in the same reference frame, selecting a composite inter-frame prediction mode for the current block from a second set of composite inter-frame prediction modes, wherein the second set of composite inter-frame prediction modes is a subset of the first set of composite inter-frame prediction modes; and (v) encoding the current block using the composite inter-frame prediction mode and the motion information from the first reference block and the second reference block.

[0008] According to some embodiments, a method for processing visual media data is provided. The method includes: (i) obtaining a source video sequence comprising multiple frames; and (ii) performing a conversion between the source video sequence and a video bitstream of the visual media data according to format rules. The video bitstream comprises multiple encoded blocks, the multiple encoded blocks including a current block. The format rules specify that: (a) the current block is to be decoded using motion information from a first reference block and a second reference block; (b) when the first reference block and the second reference block are located in different reference frames, a composite inter-frame prediction mode is selected for the current block from a first set of composite inter-frame prediction modes; (c) when the first reference block and the second reference block are located in the same reference frame, a composite inter-frame prediction mode is selected for the current block from a second set of composite inter-frame prediction modes, wherein the second set of composite inter-frame prediction modes is a subset of the first set of composite inter-frame prediction modes; and (d) the current block is reconstructed using the composite inter-frame prediction mode and motion information from the first and second reference blocks.

[0009] According to some embodiments, a method for video decoding is provided. The method includes: (i) receiving a video bitstream comprising a plurality of blocks, the plurality of blocks including a current block in a current frame; (ii) parsing a first indicator from the video bitstream, the first indicator indicating whether a composite inter-frame prediction mode will be selected for the current block from a first set of composite inter-frame prediction modes or from a second set of composite inter-frame prediction modes; (iii) parsing a second indicator from the video bitstream, the second indicator indicating a mode index of the composite inter-frame prediction mode; (iv) when the first indicator indicates that a composite inter-frame prediction mode will be selected for the current block from the first set of composite inter-frame prediction modes, selecting a composite inter-frame prediction mode from the first set of composite inter-frame prediction modes using the mode index; (v) when the first indicator indicates that a composite inter-frame prediction mode will be selected for the current block from the second set of composite inter-frame prediction modes, selecting a composite inter-frame prediction mode from the second set of composite inter-frame prediction modes using the mode index; and (vi) reconstructing the current block using the composite inter-frame prediction mode.

[0010] According to some embodiments, a video encoding method is provided. The method includes: (i) receiving video data comprising a plurality of blocks, the plurality of blocks including a current block in a current frame; (ii) determining whether to use a first set of composite inter-frame prediction modes or a second set of composite inter-frame prediction modes for the current block; (iii) when using the first set of composite inter-frame prediction modes, selecting a composite inter-frame prediction mode from the first set of composite inter-frame prediction modes using a mode index; (iv) when using the second set of composite inter-frame prediction modes, selecting a composite inter-frame prediction mode from the second set of composite inter-frame prediction modes; (v) signaling a first indicator via a video bitstream, the first indicator indicating whether the composite inter-frame prediction mode for the current block is selected from the first set of composite inter-frame prediction modes or from the second set of composite inter-frame prediction modes; (vi) signaling a second indicator via a video bitstream, the second indicator indicating the mode index of the composite inter-frame prediction mode; and (vii) encoding the current block using the composite inter-frame prediction mode.

[0011] According to some embodiments, a computing system, such as a streaming system, server system, personal computer system, or other electronic device, is provided. The computing system includes a control circuitry system and a memory storing one or more sets of instructions. The one or more sets of instructions include instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a transcoder).

[0012] According to some embodiments, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more sets of instructions executable by a computing system. The one or more sets of instructions include instructions for performing any of the methods described herein.

[0013] Therefore, methods, apparatus, and systems for encoding and decoding video are disclosed. Such methods, apparatus, and systems can supplement or replace conventional methods, apparatus, and systems for video encoding / decoding.

[0014] The features and advantages described in this specification are not necessarily all included, and in particular, some additional features and advantages will be apparent to those skilled in the art in light of the accompanying drawings, specification, and claims provided in this disclosure. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes and is not necessarily intended to depict or limit the subject matter described herein. Attached Figure Description

[0015] To provide a more detailed understanding of this disclosure, a more specific description can be made by referring to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only show relevant features of this disclosure and are therefore not necessarily intended to be limiting, as those skilled in the art will understand upon reading this disclosure that other valid features may be permissible.

[0016] Figure 1 This is a block diagram illustrating an example communication system according to some implementations.

[0017] Figure 2A This is a block diagram illustrating example elements of an encoder component according to some embodiments.

[0018] Figure 2B This is a block diagram illustrating example elements of a decoder component according to some embodiments.

[0019] Figure 3 This is a block diagram illustrating an example server system according to some implementation methods.

[0020] Figure 4A An example motion vector candidate generation for a single inter-frame prediction block is shown according to some implementations.

[0021] Figure 4B An example motion vector candidate generation for a composite prediction block is shown according to some implementations.

[0022] Figure 5A An example video decoding process according to some implementation methods is shown.

[0023] Figure 5B An example video encoding process according to some implementation methods is shown.

[0024] By convention, the various features shown in the accompanying drawings are not necessarily drawn to scale, and similar reference numerals may be used to indicate similar features throughout the specification and the drawings. Detailed Implementation

[0025] This disclosure describes video / image compression techniques related to inter-frame prediction modes. Some implementations select a composite inter-frame prediction mode (CIMM) for the current block from a first set of CIMMs based on a determination that the current block is encoded using motion information from first and second reference blocks located in different reference frames. Some implementations select a CIMM from a second set of CIMMs based on a determination that the current block is encoded using motion information from first and second reference blocks located in the same reference frame. Some implementations use the selected CIMM and motion information from the first and second reference blocks to reconstruct the current block. Selecting a CIMM from the appropriate set of CIMMs based on whether the reference blocks come from the same or different reference frames reduces redundancy and improves coding efficiency. For example, when two reference blocks come from the same reference frame, fewer modes need to be considered (e.g., because the results of some modes will be very similar to each other), which improves coding efficiency.

[0026] Example systems and devices

[0027] Figure 1 This is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 to 120-m) communicatively coupled to each other via one or more networks. In some embodiments, the communication system 100 is a streaming system, for example, for use with video-enabled applications such as video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0028] Source device 102 includes a video source 104 (e.g., a camera device component or media storage device) and an encoder component 106. In some embodiments, the video source 104 is a digital camera device (e.g., configured to create an uncompressed video sample stream). The encoder component 106 generates one or more encoded video bitstreams from the video stream. The video stream from video source 104 can have a higher data volume compared to the encoded video bitstream 108 generated by encoder component 106. Because the encoded video bitstream 108 has a lower data volume (less data) compared to the video stream from video source 104, it requires less bandwidth to transmit and less storage space to store compared to the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video to network 110).

[0029] One or more networks 110 represent any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (connected) and / or wireless communication networks. One or more networks 110 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet.

[0030] One or more networks 110 include server systems 112 (e.g., distributed / cloud computing systems). In some embodiments, server system 112 is a streaming server or includes streaming servers (e.g., configured to store and / or distribute video content, such as encoded video streams from source device 102). Server system 112 includes codec components 114 (e.g., configured to encode and / or decode video data). In some embodiments, codec components 114 include encoder components and / or decoder components. In various embodiments, codec components 114 are instantiated as hardware, software, or a combination of hardware and software. In some embodiments, codec components 114 are configured to decode encoded video bitstream 108 and re-encode video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, server system 112 is configured to generate multiple video formats and / or encodings based on encoded video bitstream 108. In some embodiments, server system 112 is used as a Media-Aware Network Element (MANE). For example, server system 112 can be configured to trim the encoded video bitstream 108 to customize potentially different bitstreams for one or more electronic devices 120. In some implementations, MANE is provided separately from server system 112.

[0031] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode encoded video data 116 to generate an outgoing video stream that can be rendered on a display or other type of rendering device. In some embodiments, one or more of the electronic devices 120 do not include a display component (e.g., communicatively coupled to an external display device and / or includes a media storage device). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0032] The source device and / or multiple electronic devices 120 are sometimes referred to as “terminal devices” or “user devices”. In some embodiments, one or more of the electronic devices 120 and / or the source device 102 are instances of server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing equipment, and / or other types of electronic devices.

[0033] In an example operation of communication system 100, source device 102 transmits an encoded video bitstream 108 to server system 112. For example, source device 102 may encode a stream of images captured by the source device. Server system 112 receives the encoded video bitstream 108 and may decode and / or encode the encoded video bitstream 108 using codec component 114. For example, server system 112 may apply encoding to the video data that is better suited for network transmission and / or storage. Server system 112 may transmit encoded video data 116 (e.g., one or more encoded video bitstreams) to one or more electronic devices 120. Each electronic device 120 may decode the encoded video data 116 and optionally display video images.

[0034] Figure 2A This is a block diagram illustrating example elements of an encoder component 106 according to some embodiments. The encoder component 106 receives video data (e.g., a source video sequence) from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives the video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder component 106). The video source 104 may provide the source video sequence in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 YCrCb or RGB), and any suitable sampling structure (e.g., YCrCb 4:2:0 or YCrCb 4:4:4). In some embodiments, the video source 104 is a storage device storing previously captured / prepared video. In some embodiments, the video source 104 is a camera device that captures local image information as a video sequence. Video data can be provided as multiple individual images that are given motion when viewed sequentially. Each image can be organized as a spatial array of pixels, where, depending on the sampling structure, color space, etc., each pixel may include one or more samples. The relationship between pixels and samples will be readily understood by those skilled in the art.

[0035] Encoder component 106 is configured to encode and / or compress images of a source video sequence into an encoded video sequence 216 in real time or under other time constraints required by the application. In some embodiments, encoder component 106 is configured to perform a conversion between the source video sequence and a visual media data bitstream (e.g., a video bitstream). Implementing an appropriate encoding rate is a function of controller 204. In some embodiments, controller 204 controls and is functionally coupled to other functional units as described below. Parameters set by controller 204 may include rate control-related parameters (e.g., image skipping, quantizer and / or rate-distortion optimization techniques with λ values), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Other functions of controller 204 can be readily identified by those skilled in the art, as these functions may belong to encoder component 106 optimized for a particular system design.

[0036] In some implementations, encoder component 106 is configured to operate within an encoding / decoding loop. In a simplified example, the encoding / decoding loop includes a source encoder 202 (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder 210. Decoder 210 reconstructs the symbols in a manner similar to that of the (remote) decoder to create sample data (assuming lossless compression between the symbols and the encoded video bitstream). The reconstructed sample stream (sample data) is input to a reference image memory 208. Since decoding the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference image memory 208 are also bit-accurate between the local and remote encoders. In this way, the encoder's prediction portion interprets the same sample values ​​as the sample values ​​that the decoder interprets during prediction as reference image samples.

[0037] The operation of decoder 210 can be combined with a remote decoder, for example, as shown below. Figure 2B The operation of the decoder component 122 is the same as described in the detailed description. However, a brief reference is provided. Figure 2B Since the symbols are available and the encoding of the symbols into an encoded video sequence by the entropy encoder 214 and the decoding of the symbols by the parser 254 can be lossless, the entropy decoding portion of the decoder component 122, including the buffer memory 252 and the parser 254, does not need to be fully implemented in the local decoder 210.

[0038] Besides parsing / entropy decoding, the decoder techniques described in this paper can exist in the corresponding encoders in essentially the same form. For this reason, the subject matter focuses on decoder operations. Additionally, the description of the encoder techniques can be simplified, as the encoder techniques are inverses of the decoder techniques.

[0039] As part of the operation of source encoder 202, source encoder 202 can perform motion-compensated predictive coding, which predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence designated as reference frames. In this way, encoding engine 212 encodes the difference between pixel blocks of the input frame and pixel blocks of the reference frame, which can be selected as the predictive reference for the input frame. Controller 204 can manage the encoding operations of source encoder 202, including, for example, setting parameters and subgroup parameters for encoding video data.

[0040] Decoder 210 decodes encoded video data based on symbols created by source encoder 202 that can be designated as reference frames. The operation of encoding engine 212 can advantageously handle lossy processing. When encoded video data is processed by video decoder (… Figure 2A When decoded at (not shown), the reconstructed video sequence can be a copy of the source video sequence with some errors. Decoder 210 replicates the decoding process performed on the reference frame by a remote video decoder, and the reconstructed reference frame can be stored in reference image memory 208. In this way, encoder component 106 locally stores a copy of the reconstructed reference frame that shares common content (no transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.

[0041] Predictor 206 can perform a prediction search against encoding engine 212. That is, for a new frame to be encoded, predictor 206 can search the reference image memory 208 for sample data (as candidate reference pixel blocks) or specific metadata such as reference image motion vectors, block shapes, etc., that can be used as appropriate prediction references for the new image. Predictor 206 can operate pixel-by-pixel based on the sample blocks to find appropriate prediction references. As determined by the search results obtained by predictor 206, the input image can have prediction references obtained from multiple reference images stored in reference image memory 208.

[0042] The outputs of all the functional units mentioned above can undergo entropy encoding in the entropy encoder 214. The entropy encoder 214 converts the symbols, such as those generated by the various functional units, into an encoded video sequence by lossless compression according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, and / or arithmetic coding).

[0043] In some implementations, the output of entropy encoder 214 is coupled to a transmitter. The transmitter can be configured to buffer encoded video sequences, such as those created by entropy encoder 214, in preparation for transmission via communication channel 218, which can be a hardware / software link to a storage device storing the encoded video data. The transmitter can be configured to combine encoded video data from source encoder 202 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown). In some implementations, the transmitter can transmit additional data along with the encoded video. Source encoder 202 can include such data as part of the encoded video sequence. Additional data can include temporal / spatial / SNR (Signal-to-Noise Ratio, SNR) enhancement layers, other forms of redundant data such as redundant images and slices, Supplementary Enhancement Information (SEI) messages, fragments of Visual Usability Information (VUI) parameter sets, etc.

[0044] Controller 204 can manage the operation of encoder component 106. During encoding, controller 204 can assign a specific type of encoded picture to each encoded picture, which may affect the encoding technique applied to the corresponding picture. For example, pictures can be assigned as intra-pictures (I-pictures), predictive pictures (P-pictures), or bidirectional predictive pictures (B-pictures). Intra-pictures can be encoded and decoded without using any other frames in the sequence as prediction sources. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are familiar with those variations of I-pictures and their corresponding applications and characteristics, and therefore they will not be repeated here. Predictive pictures can be encoded and decoded using inter-frame prediction or intra-frame prediction that uses at most one motion vector and reference index to predict sample values ​​for each block. Bidirectional predictive pictures can be encoded and decoded using inter-frame prediction or intra-frame prediction that uses at most two motion vectors and reference indexes to predict sample values ​​for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0045] The source image can typically be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and encoded on a block-by-block basis. These blocks can be predictively encoded with reference to other (already encoded) blocks, determined by the coding assignments of the corresponding images applied to the blocks. For example, blocks of image I can be unpredictably encoded, or blocks of image I can be predictively encoded (spatial prediction or intra-frame prediction) with reference to already encoded blocks of the same image. Pixel blocks of image P can be unpredictably encoded with reference to a previously encoded reference image via spatial prediction or temporal prediction. Blocks of image B can be unpredictably encoded with reference to one or two previously encoded reference images via spatial prediction or temporal prediction.

[0046] Video can be captured as multiple source images (video images) in a time-series manner. Intra-frame image prediction (often simply called intra-prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In the example, a specific image during encoding / decoding—referred to as the current image—is segmented into blocks. Where a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. The motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector can have a third dimension that identifies the reference images.

[0047] Encoder component 106 can perform encoding operations according to any predetermined video coding technique or standard described herein. In operation, encoder component 106 can perform various compression operations, including predictive coding operations utilizing temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0048] Figure 2B This is a block diagram illustrating example elements of a decoder component 122 according to some embodiments. Figure 2B The decoder component 122 is coupled to channel 218 and display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to loop filter 256 and configured to transmit data to display 124 (e.g., via a wired or wireless connection).

[0049] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more encoded video sequences to be decoded by decoder component 122. In some embodiments, decoding of each encoded video sequence is independent of other encoded video sequences. Each encoded video sequence can be received from channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive encoded video data along with other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective user entities (not depicted). The receiver may separate the encoded video sequences from other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. This additional data may be included as part of the encoded video sequence. Decoder component 122 may use the additional data to decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0050] According to some embodiments, the decoder component 122 includes a buffer memory 252, a parser 254 (sometimes also referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra-frame image prediction unit 262, a motion compensation prediction unit 260, an aggregator 268, a loop filter unit 256, a reference image memory 266, and a current image memory 264. In some embodiments, the decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuit systems. The decoder component 122 can be implemented at least partially in software.

[0051] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to combat network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, a separate buffer memory is provided outside decoder component 122 (e.g., to combat network jitter) in addition to buffer memory 252 inside decoder component 122 (e.g., configured to handle broadcast timing). Buffer memory 252 may not be necessary or may be small when receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network. Buffer memory 252 may be required to make the best use of packet networks such as the Internet. Buffer memory 252 may be relatively large and / or have an adaptive size and may be implemented at least partially in an operating system or similar component outside decoder component 122.

[0052] Parser 254 is configured to reconstruct symbols 270 from the encoded video sequence. Symbols may include, for example, information for managing the operation of decoder component 122, and / or information for controlling rendering devices such as display 124. Control information for the rendering device may be in the form of, for example, Supplemental Enhancement Information (SEI) messages or Video Availability Information (VUI) parameter set fragments (not depicted). Parser 254 parses (entropy decodes) the encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 254 may extract a subgroup parameter set from the encoded video sequence for pixels in a subgroup of at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to a group. Subgroups may include picture groups (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser 254 can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0053] Depending on the type of encoded video picture or a portion thereof (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 270 may involve multiple different units. Which units are involved and how they are involved can be controlled by parser 254 through subgroup control information parsed from the encoded video sequence. For clarity, the flow of such subgroup control information between parser 254 and the following multiple units is not depicted.

[0054] The decoder component 122 can be conceptually subdivided into multiple functional units, and in some implementations, these units interact closely with each other and can be at least partially integrated with each other. However, for clarity, this document retains the conceptual subdivision of functional units.

[0055] The scaler / inverse transform unit 258 receives quantized transform coefficients as symbols 270 and control information (e.g., which transform to use, block size, quantization factor, and / or quantization scaling matrix) from the parser 254. The scaler / inverse transform unit 258 can output blocks including sample values, which can be input to the aggregator 268. In some cases, the output samples of the scaler / inverse transform unit 258 belong to intra-frame encoded blocks; that is, blocks that do not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. Such predictive information can be provided by the intra-frame image prediction unit 262. The intra-frame image prediction unit 262 can use surrounding reconstructed information obtained from the current (partially reconstructed) image from the current image memory 264 to generate blocks with the same size and shape as the blocks in the reconstruction. The aggregator 268 can add the prediction information already generated by the intra-frame image prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258 based on each sample.

[0056] In other cases, the output samples of the scaler / inverse transform unit 258 belong to inter-frame coded blocks and potentially to motion-compensated blocks. In such cases, the motion compensation prediction unit 260 can access the reference image memory 266 to obtain samples for prediction. After motion compensation of the obtained samples according to the symbol 270 belonging to the block, these samples can be added by the aggregator 268 to the output of the scaler / inverse transform unit 258 (referred to in this case as residual samples or residual signals) to generate output sample information. The address in the reference image memory 266 from which the motion compensation prediction unit 260 obtains the predicted samples can be controlled by the motion vector. The motion vector can be used by the motion compensation prediction unit 260 in the form of symbol 270, which can have, for example, an X component, a Y component, and a reference image component. Motion compensation can also include, for example, interpolation of sample values ​​obtained from the reference image memory 266 when using subsampled precise motion vectors, and motion vector prediction mechanisms.

[0057] The output samples of aggregator 268 can undergo various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and available to loop filter unit 256 as symbols 270 from parser 254. However, video compression techniques may also respond to metadata obtained during decoding of previous (in decoding order) portions of the encoded picture or encoded video sequence, and to sample values ​​obtained from previous reconstruction and loop filtering. The output of loop filter unit 256 may be a sample stream, which can be output to a rendering device such as display 124 and stored in reference picture memory 266 for use in future inter-frame picture prediction.

[0058] Once reconstructed, certain encoded images can be used as reference images for future predictions. Once an encoded image has been reconstructed and has been identified as a reference image (e.g., by parser 254), the current reference image can become part of the reference image memory 266, and a new current image memory can be reallocated before reconstructing subsequent encoded images begins.

[0059] Decoder component 122 can perform decoding operations according to a predetermined video compression technique that may be recorded in any standard such as those described herein. As specified in a video compression technique document or standard, and particularly in a configuration file therein, an encoded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence follows the syntax of the video compression technique or standard. Furthermore, the complexity of the encoded video sequence may be within a range defined by the hierarchy of the video compression technique or standard in order to conform to some video compression techniques or standards. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured, for example, megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further restricted by the Hypothetical Reference Decoder (HRD) specification and metadata managed by the HRD buffer used to represent signals in the encoded video sequence.

[0060] Figure 3This is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes a control circuitry system 302, one or more network interfaces 304, a memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuitry system 302 includes one or more processors (e.g., a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and / or a DPU (Data Processing Unit)). In some embodiments, the control circuitry system includes a field-programmable gate array, a hardware accelerator, and / or an integrated circuit (e.g., an application-specific integrated circuit).

[0061] Network interface 304 can be configured to interface with one or more communication networks (e.g., wireless networks, wired networks, and / or optical networks). Communication networks can be local, wide area, metropolitan area, vehicular and industrial, real-time, latency-tolerant, etc. Examples of communication networks include: local area networks such as Ethernet, wireless LANs; cellular networks including GSM (Global System for Mobile Communications), 3G (the Third Generation), 4G (the Fourth Generation), 5G (the Fifth Generation), LTE (Long Term Evolution), etc.; wired or wireless wide area digital networks for TV including cable TV, satellite TV, and terrestrial broadcast TV; vehicular and industrial networks including CANBus (Controller Area Network-BUS), etc. Such communication can be one-way receiving (e.g., broadcast TV), one-way transmitting (e.g., to a CANBus device), or bidirectional (e.g., to other computer systems using local area digital networks or wide area digital networks). Such communication can include communication across one or more cloud computing networks.

[0062] User interface 306 includes one or more output devices 308 and / or one or more input devices 310. Input devices 310 may include one or more of the following: keyboard, mouse, touchpad, touch screen, data glove, joystick, microphone, scanner, camera, etc. Output devices 308 may include one or more of the following: audio output devices (e.g., speakers), visual output devices (e.g., displays or monitors), etc.

[0063] Memory 314 may include high-speed random access memory (e.g., DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), DDR RAM (Double Data Rate Random Access Memory), and / or other random access solid-state memory devices) and / or non-volatile memory (e.g., one or more disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state memory devices). Memory 314 may optionally include one or more storage devices located remotely from the control circuitry system 302. Memory 314, or alternatively, the non-volatile solid-state memory devices within memory 314 include non-transitory computer-readable storage media. In some embodiments, memory 314 or the non-transitory computer-readable storage media of memory 314 stores programs, modules, instructions, and data structures, or subsets or supersets thereof:

[0064] • Operating system 316, which includes procedures for handling various basic system services and for performing hardware-related tasks;

[0065] • Network communication module 318, which is used to connect server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections);

[0066] • Encoding / decoding module 320, which performs various functions related to encoding and / or decoding data, such as video data. In some embodiments, encoding / decoding module 320 is an example of codec component 114. Encoding / decoding module 320 includes, but is not limited to, one or more of the following:

[0067] ◦ Decoding module 322, which performs various functions related to decoding encoded data, such as those previously described with respect to decoder component 122; and

[0068] ◦ Encoding module 340, which performs various functions related to encoding data, such as those previously described with respect to encoder component 106; and

[0069] • For example, an image memory 352 for storing images and image data, used in conjunction with the encoding / decoding module 320. In some embodiments, the image memory 352 includes one or more of the following: a reference image memory 208, a buffer memory 252, a current image memory 264, and a reference image memory 266.

[0070] In some implementations, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions previously described with respect to the parser 254), a transformation module 326 (e.g., configured to perform various functions previously described with respect to the scaler / inverse transformation unit 258), a prediction module 328 (e.g., configured to perform various functions previously described with respect to the motion compensation prediction unit 260 and / or the intra-frame image prediction unit 262), and a filter module 330 (e.g., configured to perform various functions previously described with respect to the loop filter 256).

[0071] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described with respect to source encoder 202 and / or encoding engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 include... Figure 3 A subset of the modules shown. For example, a prediction module shared by both decoding module 322 and encoding module 340.

[0072] Each of the modules identified above and stored in memory 314 corresponds to a set of instructions for performing the functions described herein. The modules identified above (e.g., instruction sets) do not need to be implemented as separate software programs, processes, or modules, and therefore various subsets of these modules can be combined or otherwise rearranged in various implementations. For example, the codec module 320 may optionally not include separate decoding and encoding modules, but instead use the same set of modules to perform both sets of functions. In some implementations, memory 314 stores a subset of the modules and data structures identified above. In some implementations, memory 314 stores additional modules and data structures not described above.

[0073] Although Figure 3 A server system 112 according to some embodiments is shown, but Figure 3 This is intended more as a functional description of various features that can exist in one or more server systems than as a structural diagram of the implementation described herein. In practice, items shown individually may be combined together, and some items may be separate. For example, Figure 3 Some items shown individually can be implemented on a single server, and a single item can be implemented by one or more servers. The actual number of servers used to implement server system 112 and how features are distributed among said servers will vary depending on the implementation method, and optionally, it depends in part on the amount of data traffic processed by the server system during peak usage periods and during average usage periods.

[0074] Example encoding and decoding techniques

[0075] The encoding processes and techniques described below can be performed at the devices and systems described above (e.g., source device 102, server system 112, and / or electronic device 120). According to some implementations, a method for signaling using composite inter-frame prediction modes is described below.

[0076] Figure 4A Example motion vector candidate generation for a single inter-frame prediction block is shown according to some implementation methods. Figure 4A As shown, the MVP (Motion Vector Prediction) of the current block's motion vector mv0 is derived using time scaling with mv1 from neighboring block A.

[0077] For composite inter-frame prediction, the MVP of the current block is derived using the MV composed of different neighboring blocks, but the reference frame of the composed MV needs to be the same as that of the current block. Figure 4B Example motion vector candidate generation for composite prediction blocks is shown according to some implementation methods. Figure 4B As shown, the MV (mv2, mv3) has the same reference frame as the current block, but comes from a different neighboring block.

[0078] As mentioned above, inter-frame coded blocks can employ one or more associated motion vectors. In some implementations, a signaling syntax is used in the bitstream to indicate whether a block is encoded in a single inter-frame prediction mode (e.g., using motion information from one reference block) or a composite inter-frame prediction mode (e.g., using motion information from two or more reference blocks). A single prediction mode has one associated motion vector, while a composite inter-frame prediction mode has multiple associated motion vectors.

[0079] In composite inter-frame prediction mode, multiple reference blocks can come from the same reference frame. In some implementations, for composite inter-frame prediction mode, two reference blocks can come from different reference frames. For composite reference cases, the composite inter-frame prediction modes described in detail in Table 1 below can be used.

[0080]

[0081] Table 1 - Composite Reference Modes

[0082] In some implementations, for composite inter-frame prediction modes, a signaling syntax is used to indicate whether optical flow-based motion vector refinement is applied. In some implementations, the seven composite inter-frame prediction modes shown in Table 1 are signaled along with a flag. In some implementations, six contexts are used to signal the seven composite inter-frame prediction modes shown in Table 1, and the context derivation is shown in Table 2 below.

[0083]

[0084] Table 2 - Contextual Deduction.

[0085] In some implementations, the composite inter-frame prediction modes depicted in Table 1 are grouped into multiple groups based on whether these modes are joint MVD (Motion Vector Difference) coding modes. The modes in each group are then signaled separately. Some implementations signal the inter-frame composite mode by first signaling whether the inter-frame composite mode is a joint MVD coding mode. This flag is signaled using two contexts. A first context is used when two reference frames come from different sides and the distance between the reference frames and the current frame is the same. Otherwise, a second context is used. In some implementations, if the current mode is a joint MVD coding mode, a flag is signaled to indicate whether the JOINT_NEWMV mode or the JOINT_AMVDNEWMV mode is used. This flag is signaled using only one context. If the current mode is not a joint MVD coding mode, a flag is signaled to indicate which of the following five modes {NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, GLOBAL_GLOBALMV, NEW_NEWMV} is used. The same context derivation process as in the AVM (AOMedia Video Model, AVM) benchmark is used to signal these five modes. The signaling for the proposed composite mode is shown in Table 3 below.

[0086]

[0087] Table 3 - Signaling for inter-frame composite mode.

[0088] Dividing the composite inter-frame prediction modes into two groups and signaling the modes in each group separately can improve coding efficiency, for example, by reducing the context memory from 570 bits to 450 bits. Table 4 below shows the improvements in encoding and decoding based on simulations using the current design (e.g., AVM Study-v8.0.0 benchmark) and various video data (e.g., representing AOM General Test Conditions v7.0).

[0089]

[0090] Table 4 - Simulation Results

[0091] In some implementations, the NEW_NEARMV mode is removed when the two reference frames are identical. In some implementations, since JOINT_NEWMV and JOINT_AMVDNEWMV are designed for motion vectors from two reference frames with different directions, these two modes are also removed when the two reference frames come from the same side. In some implementations, removing these modes reduces the number of contexts from 30 to 20, the RAM size from 400 bits to 250 bits, and the ROM size from 445 bits to 295 bits, thus improving coding efficiency.

[0092] Table 5 compares the context memory when the NEW_NEARMV mode is removed and when JOINT_NEWMV and JOINT_AMVDNEWMV are also removed.

[0093]

[0094] Table 5 - Comparison of Context Memory

[0095] Figure 5A This is a flowchart illustrating a method 500 for decoding video according to some embodiments. Method 500 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuitry and a memory storing instructions for execution by the control circuitry. In some embodiments, method 500 is executed by executing instructions stored in the computing system's memory (e.g., memory 314).

[0096] The system receives (502) a video bitstream (e.g., an encoded video sequence) comprising multiple blocks (e.g., corresponding to a set of pictures), the multiple blocks including the current block in the current frame. The system determines (504) that the current block is encoded using motion information from a first reference block and a second reference block. When the first reference block and the second reference block are in different reference frames, the system selects (506) a composite inter-frame prediction mode for the current block from a first set of composite inter-frame prediction modes. When the first reference block and the second reference block are in the same reference frame, the system selects (508) a composite inter-frame prediction mode for the current block from a second set of composite inter-frame prediction modes, wherein the second set of composite inter-frame prediction modes is a subset of the first set of composite inter-frame prediction modes. The system uses the composite inter-frame prediction mode and the motion information from the first and second reference blocks to reconstruct (510) the current block. In this way, for composite inter-frame prediction modes, if two motion vectors point to the same reference frame, only a subset of the composite inter-frame prediction modes is used.

[0097] In some implementations, for both NEAR_NEWMV and NEW_NEARMV modes, when two motion vectors point to the same reference frame, only one of these modes is allowed and signaled. For example, only NEAR_NEWMV may be allowed.

[0098] In some implementations, when two motion vectors point to the same reference frame, the JOINT_NEWMV and JOINT_AMVDNEWMV modes are not allowed to be signaled.

[0099] In some implementations, when two motion vectors point to the same reference frame, the following four composite inter-frame prediction modes are allowed and signaled: NEAR_NEARMV, NEAR_NEWMV, NEW_NEWMV, and GLOBAL_GLOBALMV.

[0100] In some implementations, GLOBAL_GLOBALMV is not allowed when two motion vectors point to the same reference frame.

[0101] In some implementations, the permitted composite inter-frame prediction modes are NEAR_NEARMV, NEAR_NEWMV, and NEW_NEWMV.

[0102] In some implementations, composite inter-frame prediction modes are classified into two or more groups / sets, and a flag is signaled to indicate the set index of the composite inter-frame prediction mode, and another flag is signaled to indicate the mode index in the corresponding set.

[0103] In some implementations, composite inter-frame prediction modes are classified into two sets based on whether they have a joint motion vector difference (MVD) coding mode.

[0104] In some implementations, the composite patterns in the first set are JOINT_NEWMV and JOINT_AMVDNEWMV, while the composite patterns in the second set are NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV.

[0105] In some implementations, the context used to signal the set index of the composite inter-frame prediction mode can depend on information from two reference frames.

[0106] In some implementations, the context for signaling the set index of the composite inter-frame prediction mode can depend on whether the two reference frames come from the same side.

[0107] In some implementations, the context for signaling the set index of the composite inter-frame prediction mode can depend on the distance between the two reference frames and the current frame.

[0108] In some implementations, the context for signaling the set index of the composite inter-frame prediction mode may depend on whether the distances between the two reference frames and the current frame are equal.

[0109] In some implementations, the context of the set index used to signal the composite inter-frame prediction mode can depend on the temporal layer index or the q index.

[0110] In some implementations, the context depends on whether they have the same time layer index or the same q index.

[0111] In some implementations, composite inter-frame prediction modes are classified into two sets based on whether they have MVD of two reference frames.

[0112] In some implementations, the composite inter-frame prediction modes in the first set are NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, and GLOBALMV, while the composite inter-frame prediction modes in the second set are NEW_NEWMV, JOINT_NEWMV, and JOINT_AMVDNEWMV.

[0113] In some implementations, composite inter-frame prediction modes are classified into two sets based on whether an adaptive MVD accuracy method has been applied to a mode.

[0114] In some implementations, the composite inter-frame prediction modes in the first set are NEAR_NEARMV, NEW_NEWMV, and JOINT_NEWMV. The composite inter-frame prediction modes in the second set are NEW_NEARMV, NEAR_NEWMV, and JOINT_AVMDNEWMV.

[0115] Figure 5B This is a flowchart illustrating a method 550 for encoding video according to some embodiments. Method 550 can be performed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuitry and a memory storing instructions for execution by the control circuitry. In some embodiments, method 550 is performed by executing instructions stored in the computing system's memory (e.g., memory 314). In some embodiments, method 550 is performed by the same system as method 500 described above.

[0116] The system receives (552) video data (e.g., a source video sequence) comprising multiple blocks (e.g., corresponding to a set of images), including the current block. The system determines (554) that motion information from a first reference block and a second reference block should be used to encode the current block. When the first and second reference blocks are in different reference frames, the system selects (556) a composite inter-frame prediction mode for the current block from a first set of composite inter-frame prediction modes. When the first and second reference blocks are in the same reference frame, the system selects (558) a composite inter-frame prediction mode for the current block from a second set of composite inter-frame prediction modes. The second set of composite inter-frame prediction modes is a subset of the first set of composite inter-frame prediction modes. The system encodes the current block using the composite inter-frame prediction mode and motion information from the first and second reference blocks (560). As previously described, the encoding process can mirror the decoding process described herein (e.g., composite inter-frame prediction). For brevity, these details are not repeated here.

[0117] Although Figure 5A and Figure 5B Multiple logical stages are shown in a specific order, but stages that are not in order can be reordered, and other stages can be combined or split. Some reorderings or other groupings not specifically mentioned will be obvious to those skilled in the art, and therefore the orderings and groupings presented herein are not exhaustive. Furthermore, it should be understood that the stages can be implemented in hardware, firmware, software, or any combination thereof.

[0118] Now let's turn to some example implementations.

[0119] (A1) In one aspect, some embodiments include a method for video decoding (e.g., method 500). In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding / decoding module (e.g., encoding / decoding module 320). In some embodiments, the method is performed at a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving a video bitstream comprising multiple blocks (e.g., an encoded video sequence), the multiple blocks including a current block in a current frame; (ii) determining that the current block is encoded using motion information from a first reference block and a second reference block; (iii) selecting a composite inter-frame prediction mode for the current block from a first set of composite inter-frame prediction modes when the first and second reference blocks are in different reference frames; (iv) selecting a composite inter-frame prediction mode for the current block from a second set of composite inter-frame prediction modes when the first and second reference blocks are in the same reference frame, wherein the second set of composite inter-frame prediction modes is a subset of the first set of composite inter-frame prediction modes; and (v) reconstructing the current block using the composite inter-frame prediction mode and the motion information from the first and second reference blocks. For example, for composite inter-frame prediction modes, if two motion vectors point to the same reference frame, only a subset of composite inter-frame prediction modes is used. In some embodiments, the composite inter-frame prediction mode for the current block is selected from a subset of available composite inter-frame prediction modes based on the determination that the first and second reference blocks are in the same reference frame.

[0120] (A2) In some implementations of A1, the second composite inter-prediction mode set does not include the joint motion vector difference (MVD) composite inter-prediction mode. For example, when two motion vectors point to the same reference frame, the JOINT_NEWMV mode and the JOINT_AMVDNEWMV mode may be disallowed (and not signaled).

[0121] (A3) In some implementations of A2, the second composite inter-frame prediction mode set does not include the JOINT_NEWMV mode and the JOINT_AMVDNEWMV mode.

[0122] (A4) In some implementations of any of A1 to A3, the second composite inter-frame prediction mode set includes only one of the NEAR_NEWMV mode and the NEW_NEARMV mode. For example, for both the NEAR_NEWMV and NEW_NEARMV modes, when two motion vectors point to the same reference frame, only one of the two modes is permitted and signaled.

[0123] (A5) In some implementations of any of A4, the second composite inter-frame prediction mode set includes the NEAR_NEWMV mode but excludes the NEW_NEARMV mode. For example, for the second composite inter-frame prediction mode set, only NEAR_NEWMV is allowed.

[0124] (A6) In some embodiments of any of A1 to A5, the second composite inter-prediction mode set includes four composite prediction modes. For example, when two motion vectors point to the same reference frame, the following four composite inter-frame prediction modes may be permitted and signaled: NEAR_NEARMV, NEAR_NEWMV, NEW_NEWMV, and GLOBAL_GLOBALMV. In some embodiments, the second composite inter-frame prediction mode set has a size of 4 or less.

[0125] (A7) In some embodiments of any of A1 to A6, the second composite inter-prediction mode set includes three composite inter-prediction modes. For example, the allowed composite inter-frame prediction modes are NEAR_NEARMV, NEAR_NEWMV, and NEW_NEWMV. In some embodiments, the second composite inter-frame prediction mode set does not include any global MV mode or joint MV mode.

[0126] (A8) In some implementations of any of A1 to A7, the second composite inter-frame prediction mode set does not include global motion vector modes. For example, GLOBAL_GLOBALMV is not allowed when two motion vectors point to the same reference frame.

[0127] (A9) In some embodiments of any of A1 to A8, the method further includes: parsing a first indicator in the video bitstream to determine whether a composite inter-frame prediction mode is selected from a first set of composite inter-frame prediction modes or from a second set of composite inter-frame prediction modes. For example, the composite inter-frame prediction modes may be classified into two or more sets, and a signaling flag may be used to indicate the set index of the two or more sets.

[0128] (A10) In some embodiments of any of A1 to A8, the method further includes: parsing a second indicator in the video bitstream that indicates an index of a composite inter-frame prediction mode. For example, a second flag may be signaled to indicate an index of a mode in a corresponding set.

[0129] (B1) In another aspect, some implementations include a video encoding method (e.g., method 550). In some implementations, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some implementations, the method is performed at an encoding / decoding module (e.g., encoding / decoding module 320). The method includes: (i) receiving video data (e.g., a source video sequence) comprising a plurality of blocks, said plurality of blocks including a current block; (ii) determining to encode the current block using motion information from a first reference block and a second reference block; (iii) when the first reference block and the second reference block are located in different reference frames, selecting a composite inter-frame prediction mode for the current block from a first set of composite inter-frame prediction modes; (iv) when the first reference block and the second reference block are located in the same reference frame, selecting a composite inter-frame prediction mode for the current block from a second set of composite inter-frame prediction modes, wherein the second set of composite inter-frame prediction modes is a subset of the first set of composite inter-frame prediction modes; and (v) encoding the current block using the composite inter-frame prediction mode and motion information from the first and second reference blocks. In some implementations, the current encoded block is signaled in the video bitstream.

[0130] (B2) In some implementations of B1, the second composite inter-frame prediction mode set does not include the joint motion vector difference (MVD) composite inter-frame prediction mode.

[0131] (B3) In some embodiments of B1 or B2, the method further includes: signaling a first indicator in the video bitstream, the first indicator indicating whether a composite inter-frame prediction mode is selected from a first set of composite inter-frame prediction modes or from a second set of composite inter-frame prediction modes.

[0132] (B4) In some embodiments of any of B3, the method further includes: signaling a second indicator in the video bitstream, the second indicator indicating an index of the composite inter-frame prediction mode.

[0133] (B5) In some embodiments of any of B1 to B4, the method further includes: signaling a second composite inter-frame prediction mode set when the first reference block and the second reference block are located in the same reference frame. For example, the second composite inter-frame prediction mode set is signaled instead of the first composite inter-frame prediction mode set.

[0134] (B6) In some embodiments of any of B1 to B5, the method further includes: signaling a first composite inter-frame prediction mode set when the first reference block and the second reference block are located in different reference frames. In some embodiments, the first composite inter-frame prediction mode set includes all available composite inter-frame prediction modes.

[0135] (C1) In another aspect, some embodiments include a method for processing visual media data. In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at a codec module (e.g., codec module 320). The method includes: (i) obtaining a source video sequence comprising multiple frames; and (ii) performing a conversion between the source video sequence and a video bitstream of visual media data according to format rules. The video bitstream comprises multiple encoded blocks, the multiple encoded blocks including the current block. The format rules specify that: (a) the current block should be decoded using motion information from the first reference block and the second reference block; (b) when the first reference block and the second reference block are in different reference frames, a composite inter-frame prediction mode should be selected for the current block from the first set of composite inter-frame prediction modes; (c) when the first reference block and the second reference block are in the same reference frame, a composite inter-frame prediction mode should be selected for the current block from the second set of composite inter-frame prediction modes, wherein the second set of composite inter-frame prediction modes is a subset of the first set of composite inter-frame prediction modes; and (d) the current block should be reconstructed using the composite inter-frame prediction mode and motion information from the first reference block and the second reference block.

[0136] (D1) In another aspect, some embodiments include a method for video decoding. In some embodiments, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some embodiments, the method is performed at an encoding / decoding module (e.g., encoding / decoding module 320). In some embodiments, the method is performed at a source encoding component (e.g., source encoder 202), an encoding engine (e.g., encoding engine 212), and / or an entropy encoder (e.g., entropy encoder 214). The method includes: (i) receiving a video bitstream comprising a plurality of blocks, the plurality of blocks including a current block in a current frame; (ii) parsing a first indicator from the video bitstream, the first indicator indicating whether a composite inter-frame prediction mode will be selected for the current block from a first set of composite inter-frame prediction modes or from a second set of composite inter-frame prediction modes; (iii) parsing a second indicator from the video bitstream, the second indicator indicating a mode index of the composite inter-frame prediction mode; (iv) when the first indicator indicates that the composite inter-frame prediction mode for the current block will be selected from the first set of composite inter-frame prediction modes, selecting a composite inter-frame prediction mode from the first set of composite inter-frame prediction modes using the mode index; (v) when the first indicator indicates that the composite inter-frame prediction mode for the current block will be selected from the second set of composite inter-frame prediction modes, selecting a composite inter-frame prediction mode from the second set of composite inter-frame prediction modes using the mode index; and (vi) reconstructing the current block using the composite inter-frame prediction mode. For example, for composite inter-frame prediction modes, these modes are classified into two or more groups / sets, and a flag is signaled to indicate the set index of the composite inter-frame prediction mode, and another flag is signaled to indicate the index of the mode in the corresponding set.

[0137] (D2) In some implementations of D1, only one of the first and second composite inter-frame prediction mode sets includes a joint MVD mode. For example, composite inter-frame prediction modes can be classified into two sets based on whether they are joint MVD coding modes. As an example, composite modes in the first set may include JOINT_NEWMV and JOINT_AMVDNEWMV, while composite modes in the second set may include NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV.

[0138] (D3) In some implementations of D1 or D2, only one of the first and second composite inter-frame prediction mode sets includes a composite mode with MVD having two reference frames. For example, composite inter-frame prediction modes can be classified into two sets based on whether the composite inter-frame prediction mode has MVD with two reference frames. As an example, the composite inter-frame prediction modes in the first set may include NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, and GLOBAL_GLOBALMV, while the composite inter-frame prediction modes in the second set may include NEW_NEWMV, JOINT_NEWMV, and JOINT_AMVDNEWMV.

[0139] (D4) In some implementations of any of D1 to D3, only one of the first set of composite inter-frame prediction modes and the second set of composite inter-frame prediction modes includes a composite mode with adaptive MVD accuracy. For example, composite inter-frame prediction modes can be classified into two sets based on whether an adaptive MVD accuracy method is applied to a mode. As an example, the composite inter-frame prediction modes in the first set may include NEAR_NEARMV, NEW_NEWMV, and JOINT_NEWMV. The composite inter-frame prediction modes in the second set may include NEW_NEARMV, NEAR_NEWMV, and JOINT_AVMDNEWMV.

[0140] (D5) In some implementations of any of D1 to D4, the current block is reconstructed using a composite inter-frame prediction mode and motion information from a first reference frame and a second reference frame.

[0141] (D6) In some implementations of D5, the first indicator is entropy-decoded using a first context, wherein the first context is selected based on information about a first reference frame and a second reference frame. For example, the context used to signal the set index of a composite inter-frame prediction mode may depend on information from both reference frames.

[0142] (D7) In some implementations of D6, information about the first and second reference frames includes whether the first and second reference frames are on the same side of the current frame. For example, the context used to signal the set index of the composite inter-frame prediction mode may depend on whether the two reference frames are from the same side. In some implementations, a first context is selected when the first and second reference frames are on the same side, and a second context is selected when the first and second reference frames are on different sides.

[0143] (D8) In some implementations of D6 or D7, the information about the first and second reference frames includes information about the distance between the current frame and the first and second reference frames. For example, the context used to signal the set index of the composite inter-frame prediction mode may depend on the distance between the two reference frames and the current frame.

[0144] (D9) In some embodiments of any of D6 to D8, a first context is selected when the distance between the current frame and the first reference frame and the second reference frame is the same, and a second context is selected when the distance between the current frame and the first reference frame and the second reference frame is different. For example, the context used to signal the set index of the composite inter-frame prediction mode may depend on whether the distance between the two reference frames and the current frame is equal.

[0145] (D10) In some embodiments of any of D6 through D9, the information regarding the first and second reference frames includes a temporal layer index and / or a quantization index. For example, the context used for signaling the set index of the composite inter-frame prediction mode may depend on the temporal layer index or the q index. In some embodiments, a first context is selected when the first and second reference frames have the same temporal layer index (or q index), and a second context is selected when the first and second reference frames have different temporal layer indices (or q indices). For example, the context may depend on whether the reference frames have the same temporal layer index or the same q index.

[0146] (E1) In another aspect, some implementations include a method for video encoding. In some implementations, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some implementations, the method is performed at a codec module (e.g., codec module 320). The method includes: (i) receiving video data comprising a plurality of blocks, the plurality of blocks including a current block in a current frame; (ii) determining whether to use a first set of composite inter-frame prediction modes or a second set of composite inter-frame prediction modes for the current block; (iii) when using the first set of composite inter-frame prediction modes, selecting a composite inter-frame prediction mode from the first set of composite inter-frame prediction modes using a mode index; (iv) when using the second set of composite inter-frame prediction modes, selecting a composite inter-frame prediction mode from the second set of composite inter-frame prediction modes; (v) signaling a first indicator via a video bitstream, the first indicator indicating whether the composite inter-frame prediction mode for the current block is selected from the first set of composite inter-frame prediction modes or from the second set of composite inter-frame prediction modes; (vi) signaling a second indicator via a video bitstream, the second indicator indicating the mode index of the composite inter-frame prediction mode; and (vii) encoding the current block using the composite inter-frame prediction mode.

[0147] On the other hand, some implementations include a computing system (e.g., server system 112) that includes a control circuitry system (e.g., control circuitry system 302) and a memory (e.g., memory 314) coupled to the control circuitry system, the memory storing one or more sets of instructions configured to be executed by the control circuitry system, the set of one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1 to A10, B1 to B6, C1, D1 to D10 and E1 above).

[0148] In another aspect, some implementations include a non-transitory computer-readable storage medium that stores one or more sets of instructions for execution by a control circuitry of a computing system, the set of instructions including instructions for performing any of the methods described herein (e.g., A1 to A10, B1 to B6, C1, D1 to D10 and E1 above).

[0149] Unless otherwise stated, any syntax element described herein (e.g., indicator) can be a High-Level Syntax (HLS). As used herein, the HLS is signaled at a level higher than the block level. For example, the HLS can correspond to a sequence level, frame level, slice level, or tile level. As another example, HLS elements can be signaled in a Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Adaptation Parameter Set (APS), slice header, picture header, tile header, and / or CTU (Coding Tree Unit) header.

[0150] It should be understood that while the terms “first,” “second,” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of embodiments and appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or,” as used herein, refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0151] As used herein, the term "if" may be interpreted, depending on the context, as meaning "when the prerequisite is true," "after the prerequisite is true," "in response to determining that the prerequisite is true," "based on the determination that the prerequisite is true," or "in response to detecting that the prerequisite is true." Similarly, the phrases "if it is determined [the prerequisite is true]," "if [the prerequisite is true]," or "when [the prerequisite is true]" may be interpreted, depending on the context, as meaning "after determining that the prerequisite is true," "in response to determining that the prerequisite is true," "based on the determination that the prerequisite is true," "after detecting that the prerequisite is true," or "in response to detecting that the prerequisite is true."

[0152] For illustrative purposes, the foregoing description has been described with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described to best illustrate the principles of operation and practical application, thereby enabling others skilled in the art to implement them.

Claims

1. A method for video decoding performed at a computing system having a memory and one or more processors, the method comprising: Receive a video bitstream comprising multiple blocks, the multiple blocks including the current block in the current frame; It is determined that the current block is encoded using motion information from the first reference block and the second reference block; When the first reference block and the second reference block are located in different reference frames, a composite inter-frame prediction mode is selected for the current block from the first composite inter-frame prediction mode set. When the first reference block and the second reference block are located in the same reference frame, a composite inter-frame prediction mode is selected for the current block from the second composite inter-frame prediction mode set, wherein the second composite inter-frame prediction mode set is a subset of the first composite inter-frame prediction mode set. as well as The current block is reconstructed using the composite inter-frame prediction mode and motion information from the first reference block and the second reference block.

2. The method according to claim 1, wherein, The second set of composite inter-frame prediction modes does not include the joint motion vector difference (MVD) composite inter-frame prediction mode.

3. The method according to claim 2, wherein, The second composite inter-frame prediction mode set does not include the JOINT_NEWMV mode and the JOINT_AMVDNEWMV mode.

4. The method according to claim 1, wherein, The second composite inter-frame prediction mode set includes only one of the NEAR_NEWMV mode and the NEW_NEARMV mode.

5. The method according to claim 4, wherein, The second composite inter-frame prediction mode set includes the NEAR_NEWMV mode, but does not include the NEW_NEARMV mode.

6. The method according to claim 1, wherein, The second set of composite inter-frame prediction modes includes four composite inter-frame prediction modes.

7. The method according to claim 1, wherein, The second composite inter-frame prediction mode set includes three composite inter-frame prediction modes.

8. The method according to claim 1, wherein, The second composite inter-frame prediction mode set does not include global motion vector modes.

9. The method according to claim 1, further comprising: The first indicator in the video bitstream is parsed to determine whether the composite inter-frame prediction mode is selected from the first set of composite inter-frame prediction modes or from the second set of composite inter-frame prediction modes.

10. The method of claim 9, further comprising: The second indicator in the video bitstream is parsed, and the second indicator indicates the index of the composite inter-frame prediction mode.

11. A method for video encoding performed at a computing system having a memory and one or more processors, the method comprising: Receive video data comprising multiple blocks, including the current block; Determine that motion information from the first reference block and the second reference block should be used to encode the current block; When the first reference block and the second reference block are located in different reference frames, a composite inter-frame prediction mode is selected for the current block from the first composite inter-frame prediction mode set. When the first reference block and the second reference block are located in the same reference frame, a composite inter-frame prediction mode is selected for the current block from the second composite inter-frame prediction mode set, wherein the second composite inter-frame prediction mode set is a subset of the first composite inter-frame prediction mode set. as well as The current block is encoded using the composite inter-frame prediction mode and motion information from the first reference block and the second reference block.

12. The method according to claim 11, wherein, The second set of composite inter-frame prediction modes does not include the joint motion vector difference (MVD) composite inter-frame prediction mode.

13. The method of claim 11, further comprising: In the video bitstream, a signal is used to notify a first indicator, which indicates whether the composite inter-frame prediction mode is selected from the first set of composite inter-frame prediction modes or from the second set of composite inter-frame prediction modes.

14. The method of claim 13, further comprising: A second indicator is signaled in the video bitstream, the second indicator indicating the index of the composite inter-frame prediction mode.

15. The method of claim 11, further comprising: When the first reference block and the second reference block are located in the same reference frame, the second composite inter-frame prediction mode set is notified by a signal.

16. The method of claim 11, further comprising: When the first reference block and the second reference block are located in different reference frames, the first composite inter-frame prediction mode set is notified by a signal.

17. A non-transitory computer-readable storage medium storing a video bitstream generated by a video coding method, the video coding method comprising: Receive video data comprising multiple blocks, including the current block; Determine that motion information from the first reference block and the second reference block should be used to encode the current block; When the first reference block and the second reference block are located in different reference frames, a composite inter-frame prediction mode is selected for the current block from the first composite inter-frame prediction mode set. When the first reference block and the second reference block are located in the same reference frame, a composite inter-frame prediction mode is selected for the current block from a second composite inter-frame prediction mode set, wherein the second composite inter-frame prediction mode set is a subset of the first composite inter-frame prediction mode set; and The current block is encoded using the composite inter-frame prediction mode and motion information from the first and second reference blocks; and The video bitstream includes the currently encoded block.

18. The non-transitory computer-readable storage medium according to claim 17, wherein, The video bitstream includes an indication of the first composite inter-frame prediction mode set when the first reference block and the second reference block are located in different reference frames.

19. The non-transitory computer-readable storage medium according to claim 17, wherein, The video bitstream includes: an indication of the second composite inter-frame prediction mode set when the first reference block and the second reference block are located in the same reference frame.

20. The non-transitory computer-readable storage medium according to claim 17, wherein, The second set of composite inter-frame prediction modes does not include the joint motion vector difference (MVD) composite inter-frame prediction mode.