MMVD signaling improvement

Abstract

To provide a method and an apparatus for video decoding.SOLUTION: The apparatus includes processing circuitry receiving prediction information of multiple blocks in a current picture from a coded video bitstream. The processing circuitry determines, based on the prediction information, whether at least one of a decoder-side motion vector refinement (DMVR) mode and a bi-directional optical flow (BDOF) mode is allowed for the multiple blocks. If the DMVR mode or the BDOF mode is allowed for the multiple blocks, the processing circuitry determines that multiple merge-with-motion-vector-difference (MMVD) merge flags indicating whether the MMVD mode is applied to the multiple blocks, respectively, are inferred as false for the multiple blocks. If the multiple MMVD merge flags are not inferred as false, the processing circuitry reconstructs each of the multiple blocks without applying the MMVD mode.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] Incorporation by Reference This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 233,490, entitled "Merge with Motion Vector Difference Signaling Improvement," filed August 16, 2021, which claims the benefit of priority to U.S. Provisional Patent Application No. 17 / 888,193, entitled "MMVD SIGNALING IMPROVEMENT," filed August 15, 2022. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] This disclosure describes embodiments related generally to video coding. [Background technology]

[0003] The discussion of the background art provided herein is intended to generally present the context of the present disclosure. The work of the inventors named herein is not admitted, explicitly or implicitly, as prior art to the present disclosure, to the extent that such work is described in this Background section, along with aspects of the description that may not otherwise be prior art at the time of filing.

[0004] Uncompressed digital images and / or videos may include a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luminance samples and associated chrominance samples. The sequence of pictures may have a fixed or variable picture rate (also informally known as frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed images and / or videos have specific bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luminance sample resolution at a frame rate of 60 Hz) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0005] One objective of image and / or video coding and decoding may be the reduction of redundancy in the input image and / or video signal through compression. Compression may help reduce the aforementioned bandwidth and / or storage space requirements, possibly by more than one order of magnitude. The description herein uses video encoding / decoding as an illustrative example, but the same techniques may be applied to image encoding / decoding in a similar manner without departing from the spirit of this disclosure. Both lossless and lossy compression, and combinations thereof, may be used. Lossless compression refers to techniques that allow an exact copy of the original signal to be reconstructed from a compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal useful for its intended application. For video, lossy compression is widely adopted. The amount of tolerable distortion depends on the application, for example, a user of a particular consumer streaming application may tolerate higher distortion than a user of a television distribution application. The achievable compression ratio may reflect that the higher the acceptable / tolerable distortion, the higher the compression ratio that can be obtained.

[0006] Video encoders and decoders can utilize techniques from a number of broad categories, including, for example, motion compensation, transform processing, and entropy coding.

[0007] Video codec techniques can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in intra mode, the picture may be an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and therefore can be used as the first picture in a coded video bitstream and video session or as a still image. Samples of an intra block can be transformed and the transform coefficients can be quantized before entropy coding. Intra prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are needed for a given quantization step size to represent the block after entropy coding.

[0008] For example, conventional intra-coding, used in MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to perform prediction based on surrounding sample data and / or metadata obtained during encoding and / or decoding of a data block, for example. Such techniques are hereinafter referred to as "intra-prediction" techniques. It should be noted that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from a reference picture.

[0009] There may be many different forms of intra prediction. If more than one of such techniques can be used in a given video coding technique, the particular technique in use may be coded as a particular intra prediction mode that uses the particular technique. In certain cases, the intra prediction mode may have sub-modes and / or parameters that may be coded separately or included in the mode codeword that defines the prediction mode used. Which codeword is used for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency gains via intra prediction, and so may the entropy coding technique used to convert the codeword into a bitstream.

[0010] A specific mode of intra prediction was introduced in H.264, improved in H.265, and further improved in newer coding techniques such as Joint Search Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). A predictor block can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of the neighboring samples are copied to the predictor block according to the direction. The reference to the direction in use can be coded in the bitstream or can be predicted itself.

[0011] Referring to FIG. 1A, depicted at the bottom right is a subset of 9 predictor directions known from the 33 possible predictor directions defined in H.265 (corresponding to the 33 angle modes of the 35 intra modes). The point where the arrows converge (101) represents the sample being predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right, at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101), at an angle of 22.5 degrees from the horizontal.

[0012] 1A, at the top left is depicted a square block (104) of 4×4 samples (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S", its Y-dimensional position (e.g., row index), and its X-dimensional position (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Since the block is 4×4 samples in size, S44 is at the bottom right. Further shown are reference samples that follow a similar numbering scheme. The reference samples are labeled with R, their Y position (e.g., row index), and X position (column index) relative to the block (104). In both H.264 and H.265, the predicted samples are adjacent to the block being reconstructed, and therefore negative values ​​do not need to be used.

[0013] Intra-picture prediction can work by copying reference sample values ​​from adjacent samples indicated by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling for this block indicating a prediction direction that coincides with the arrow (102), i.e., the sample is predicted from the sample to the upper right, at an angle of 45 degrees from the horizontal. In that case, samples S41, S32, S23, S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example by interpolation, to calculate the reference sample.

[0015] The number of possible directions has increased as video coding techniques have developed. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments have been carried out to identify the most likely directions, and certain techniques of entropy coding are used to represent those likely directions with a small number of bits, accepting a certain penalty for the less likely directions. Furthermore, the direction itself may be predicted from neighboring directions used in neighboring, already decoded blocks.

[0016] FIG. 1B shows a schematic diagram (110) depicting 65 intra prediction directions with JEM to illustrate the increasing number of prediction directions over time.

[0017] The mapping of intra-prediction direction bits to represent directions in the coded video bitstream may vary between video coding techniques. Such mappings may range from simple direct mappings to complex adaptation schemes including codewords, most probable modes, and similar techniques. In most cases, however, there may be certain directions that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, these less likely directions are represented with more bits than more likely directions in well-performing video coding techniques.

[0018] Image and / or video coding and decoding can be performed using inter-picture prediction with motion compensation. Motion compensation can be a lossy compression technique and can refer to a technique in which blocks of sample data from a previously reconstructed picture or part thereof (reference picture) are used to predict a newly reconstructed picture or part of a picture after being spatially shifted in a direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV can have two dimensions X and Y, or three dimensions, the third dimension being an indication of the reference picture in use (the latter can be indirectly a temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular area of ​​sample data can be predicted from other MVs, e.g., from MVs associated with other areas of sample data that are spatially adjacent to the area being reconstructed and that precede that MV in decoding order. Doing so can significantly reduce the amount of data required to code the MV, thereby eliminating redundancy and increasing the compression ratio. MV prediction can work effectively because, for example, when coding an input video signal derived from a camera (known as natural video), there is a statistical likelihood that areas larger than the area to which a single MV is applicable move in similar directions and thus can be predicted using similar motion vectors, possibly derived from MVs of neighboring areas. As a result, the detected MV for a given area is similar or the same as the MV predicted from the surrounding MVs, which can be represented, after entropy coding, with fewer bits than would be used if the MV were coded directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from an original signal (i.e., a sample stream). In other cases, the MV prediction itself may be lossy, for example due to rounding errors when computing a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms offered by H.265, the one described with reference to Fig. 2 is a technique hereafter referred to as "spatial merging".

[0021] Referring to Figure 2, the current block (201) contains samples that have been discovered by the encoder during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of coding its MV directly, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order), using MVs associated with any one of five surrounding samples, denoted A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, MV prediction can use predictors from the same reference picture that neighboring blocks are using. Summary of the Invention [Means for solving the problem]

[0022] Aspects of the present disclosure provide methods and apparatus for video encoding and decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit is configured to receive prediction information for a plurality of blocks in a current picture from a coded video bitstream. The processing circuit determines whether at least one of a decoder-side motion vector refinement (DMVR) mode or a bi-directional optical flow (BDOF) mode is allowed for the plurality of blocks based on the prediction information. In response to determining that at least one of the DMVR mode or the BDOF mode is allowed for the plurality of blocks, the processing circuit determines that a plurality of merge with motion vector difference (MMVD) merge flags are suspected to be false for the plurality of blocks. The plurality of MMVD merge flags respectively indicate whether an MMVD mode is applied for the plurality of blocks. In response to determining that the plurality of MMVD merge flags are suspected to be false for the plurality of blocks, the processing circuit reconstructs each block in the plurality of blocks without applying the MMVD mode.

[0023] In one embodiment, the prediction information includes at least one flag indicating whether at least one of DMVR mode or BDOF mode is allowed for multiple blocks.

[0024] In one example, the processing circuit determines that (i) the current picture references a forward reference picture and a backward reference picture, and (ii) the current picture is intermediate between the forward reference picture and the backward reference picture, determines that the MMVD merge flags are not signaled, and infers the MMVD merge flags based on determining that the MMVD mode is not allowed for the current picture.

[0025] In one example, the processing circuit determines that an absolute value of a first picture order count (POC) difference between the forward reference picture and the current picture is equal to an absolute value of a second POC difference between the backward reference picture and the current picture. For example, the absolute value of the first POC difference and the absolute value of the second POC difference are less than a threshold value.

[0026] In one embodiment, the prediction information includes MMVD merge enable flags for a plurality of blocks, the MMVD merge enable flags indicating that MMVD mode is not enabled for the plurality of blocks, and the processing circuitry determines that a plurality of MMVD merge flags for the plurality of blocks are suspected to be false.

[0027] In one example, the blocks are slices in the current picture.

[0028] In one embodiment, in response to determining that the plurality of MMVD merge flags are not inferred to be false for the plurality of blocks, a respective MMVD merge flag for each block in the plurality of blocks is signaled in the coded video bitstream. The processing circuitry may select a context from among the first context and the second context based on whether one or both of a DMVR mode or a BDOF mode are allowed for the plurality of blocks. The processing circuitry may decode the respective MMVD merge flag for each block in the plurality of blocks using context-adaptive binary arithmetic coding (CABAC) with the selected context.

[0029] In one embodiment, the prediction information indicates whether a condition for selecting the first context is satisfied. The condition includes that one or both of the DMVR mode or the BDOF mode is allowed for the plurality of blocks, the reference pictures of the plurality of blocks include a forward reference picture and a backward reference picture, and an absolute value of a first Picture Order Count (POC) difference between the forward reference picture and the current picture is equal to an absolute value of a second POC difference between the backward reference picture and the current picture. The processing circuit can select the first context as the context based on the condition being satisfied. The processing circuit can select the second context as the context based on one of the conditions not being satisfied.

[0030] In one embodiment, the first context is initialized with a probability indicating that the MMVD merge flag of each block in the plurality of blocks has a higher probability of being false than the probability of the MMVD merge flag being true.

[0031] Aspects of the present disclosure also provide a non-transitory computer-readable storage medium storing a program executable by at least one processor to perform a method for video decoding.

[0032] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0033] [Figure 1A] FIG. 2 is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 2 is a diagram of an example intra-prediction direction. [Diagram 2] FIG. 2 is a diagram showing an example of a current block (201) and surrounding samples. [Diagram 3] FIG. 3 is a schematic diagram of an example block diagram of a communication system (300). [Figure 4]FIG. 4 is a schematic diagram of an example block diagram of a communication system (400). [Diagram 5] FIG. 2 is a schematic diagram of an example block diagram of a decoder. [Figure 6] FIG. 2 is a schematic diagram of an example block diagram of an encoder; [Figure 7] 1 is a block diagram illustrating an exemplary encoder. [Figure 8] FIG. 2 is a block diagram illustrating an exemplary decoder. [Figure 9] FIG. 2 illustrates an example of a bidirectional optical flow (BDOF) mode. [Figure 10] FIG. 2 illustrates an example of a decoder-side motion vector refinement (DMVR) mode. [Figure 11] FIG. 2 illustrates an example of a search process in merge mode by motion vector difference (MMVD). [Figure 12] FIG. 13 is a diagram showing an example of search points in the MMVD mode. [Figure 13] 1 is a flow chart outlining an encoding process according to one embodiment of the present disclosure. [Figure 14] 1 is a flow chart outlining a decoding process according to one embodiment of the present disclosure. [Figure 15] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] FIG. 3 illustrates an example block diagram of a communication system (300). The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) perform unidirectional transmission of data. For example, the terminal device (310) may code video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) may receive the coded video data from the network (350), decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. Unidirectional data transmission may be common, such as in media serving applications.

[0035] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) performing bidirectional transmission of coded video data, for example during a video conference. In the case of bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) may code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) may also receive coded video data transmitted by the other of the terminal devices (330) and (340), decode the coded video data to recover the video pictures, and display the video pictures on an accessible display device according to the recovered video data.

[0036] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) are illustrated as a server, a personal computer, and a smartphone, respectively, although the principles of the present disclosure may not be so limited. The embodiments of the present disclosure apply with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (350) represents any number of networks that convey coded video data between the terminal devices (310), (320), (330), and (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of the network (350) may not be important to the operation of the present disclosure unless otherwise described herein below.

[0037] 4 shows a video encoder and video decoder in a streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0038] The streaming system may include a video source (401), which may include, for example, a capture subsystem (413), which may include, for example, a digital camera, creating a stream of uncompressed video pictures (402). In one example, the stream of video pictures (402) includes samples taken by a digital camera. The stream of video pictures (402), depicted as a thick line to emphasize the large amount of data compared to the encoded video data (404) (or coded video bitstream), may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof that enables or implements aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream), depicted as a thin line to emphasize the small amount of data compared to the stream of video pictures (402), may be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example, in an electronic device (430). The video decoder (410) decodes an input copy (407) of the encoded video data and creates an output stream (411) of video pictures that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) can be encoded according to a particular video coding / compression standard. Examples of such standards include ITU-T Recommendation H.265.In one example, the developing video coding standard is informally known as Versatile Video Coding (VVC), and the disclosed subject matter may be used in the context of VVC.

[0039] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0040] 5 shows an example block diagram of a video decoder (510). The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (410) in the example of FIG. 4.

[0041] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In one embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (531) may receive the encoded video data along with other data, such as coded audio data and / or auxiliary data streams, that may be forwarded to their respective using entities (not shown). The receiver (531) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In other applications, the buffer memory (515) may be external to the video decoder (510) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), e.g., to combat network jitter, plus another buffer memory (515) internal to the video decoder (510), e.g., to handle playout timing. When the receiver (531) is receiving data from a store-and-forward device of sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may be unnecessary or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (515) may be necessary and may be relatively large, may be advantageously of adaptive size, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (510).

[0042] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (510) and potentially information for controlling a rendering device such as a rendering device (512) (e.g., a display screen) that is not an integral part of the electronic device (530) but may be coupled to the electronic device (530) as shown in FIG. 5. The control information for the rendering device(s) may be in the form of a supplemental enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding with or without context dependency, Huffman coding, arithmetic coding, etc. The parser (520) may extract from the coded video sequence a set of subgroup parameters for at least one of a subgroup of pixels in a video decoder based on at least one parameter corresponding to the group. The subgroup may include a collection of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) may also extract information such as transform coefficients, quantization parameter values, motion vectors, etc. from the coded video sequence.

[0043] The parser (520) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (515) to produce symbols (521).

[0044] The reconstruction of the symbols (521) can involve several different units, depending on the type of coded video picture or portion thereof (inter-picture and intra-picture, inter-block and intra-block, etc.), as well as other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not depicted for clarity.

[0045] Beyond the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In an actual implementation operating under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate:

[0046] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients as well as control information from the parser (520) including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. as symbol(s) (521). The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0047] In some cases, the output samples of the scaler / inverse transform unit (551) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed and / or a fully reconstructed current picture. The aggregator (555) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information as provided by the scaler / inverse transform unit (551).

[0048] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (521) related to the block, these samples may be added to the output of the scalar / inverse transform unit (551) by the aggregator (555) to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (553), for example in the form of symbols (521) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.

[0049] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called a coded video bitstream) and provided to the loop filter unit (556) as symbols (521) from the parser (520). Video compression may also be performed in response to meta-information obtained during decoding of a previous portion (in decoding order) of the coded picture or coded video sequence, or in response to previously reconstructed and loop filtered sample values.

[0050] The output of the loop filter unit (556) may be a sample stream that can be output to a rendering device (512) as well as stored in a reference picture memory (557) for use in future inter-picture prediction.

[0051] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0052] The video decoder (510) may perform decoding operations according to a given video compression technique or standard, such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used in the sense that the coded video sequence adheres to both the syntax of the video compression technique or standard and to a profile documented in the video compression technique or standard. In particular, a profile may select certain tools from among all tools available in the video compression technique or standard as tools that are only available to them under that profile. Compliance may also require that the complexity of the coded video sequence be within a range specified by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled in the coded video sequence.

[0053] In one embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0054] 6 shows an example block diagram of a video encoder (603). The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG.

[0055] The video encoder (603) may receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6) that may capture a video image(s) to be coded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0056] The video source (601) may provide a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0057] According to one embodiment, the video encoder (603) may code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraint required. Enforcing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units described below. This coupling is not depicted for clarity. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured with other appropriate functions related to the video encoder (603) optimized for a particular system design.

[0058] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (630) (responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture(s)), as well as a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a similar manner that a (remote) decoder would also create. The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since decoding of the symbol stream leads to a bit-exact result regardless of the location of the decoder (local or remote), the content in the reference picture memory (634) is also bit-exact between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the decoder would "see" when using prediction during decoding as reference picture samples. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, eg, due to channel errors) is also used in several related techniques.

[0059] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510), already described in detail above in conjunction with Figure 5. Referring also briefly to Figure 5, however, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515), and the parser (520), may not be fully implemented in the local decoder (633).

[0060] In one embodiment, the decoder techniques, except for parsing / entropy decoding, present in the decoder are present in the corresponding encoder in the same or substantially the same functional form. Therefore, the subject matter disclosed focuses on the operation of the decoder. The description of the encoder techniques can be omitted since the encoder techniques are the opposite of the decoder techniques, which are described generically. Only in certain areas will more detailed descriptions be provided below.

[0061] In some examples, during operation, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of the reference picture(s) that may be selected as the predictive reference(s) to the input picture.

[0062] The local video decoder (633) may decode the coded video data of pictures that may be designated as reference pictures based on the symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. If the coded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may usually be a copy of the source video sequence with some errors. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in the reference picture memory (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures that will be obtained by the far-end video decoder (without transmission errors).

[0063] The predictor (635) may perform the prediction search of the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (635) may operate on sample blocks, pixel blocks by pixel blocks, to find suitable prediction references. In some cases, as determined by search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0064] The controller (650) may manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0065] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0066] The transmitter (640) may buffer the coded video sequence(s) created by the entropy coder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) may merge the coded video data from the video encoder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0067] The controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0068] An intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0069] A predictive picture (P picture) may be one that can be coded and decoded using intra- or inter-prediction, which predicts the sample values ​​of each block using at most one motion vector and reference index.

[0070] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indexes to predict sample values ​​for each block. Similarly, multi-predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0071] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. A block may be predictively coded with reference to other (already coded) blocks determined by a coding assignment applied to the block's respective picture. For example, a block of an I picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). A pixel block of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. A block of a B picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0072] The video encoder (603) may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0073] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include other types of redundant data, such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0074] A video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits the (temporal or other) correlation between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a previously coded, still buffered reference picture in the video, the block in the current picture may be coded by a vector, called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0075] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both of which precede the current picture in decoding order (but may be in the past and future, respectively, in display order) in the video. A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0076] Furthermore, to improve coding efficiency, merge mode techniques can be used in inter-picture prediction.

[0077] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed on a block-by-block basis. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, and 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, or into four CUs of 32×32 pixels, or into 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) according to temporal predictability and / or spatial predictability. In general, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0078] 7 shows an example diagram of a video encoder (703). The video encoder (703) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture in a sequence of video pictures and encode the processed block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) of the example of FIG. 4.

[0079] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8×8 samples. The video encoder (703) determines whether the processing block is optimally coded using intra-mode, inter-mode, or bi-predictive mode, for example, using rate-distortion optimization. If the processing block is to be coded in intra-mode, the video encoder (703) may encode the processing block into a coded picture using intra-prediction techniques, and if the processing block is to be coded in inter-mode or bi-predictive mode, the video encoder (703) may encode the processing block into a coded picture using inter-prediction techniques or bi-prediction techniques, respectively. In certain video coding techniques, the merge mode may be an inter-picture prediction sub-mode in which a motion vector is derived from one or more motion vector predictors without the aid of coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown), to determine the mode of the processing block.

[0080] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), coupled to each other as shown in FIG. 7.

[0081] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-prediction information (e.g., a description of redundancy information, motion vectors, merge mode information according to an inter-encoding technique), and calculate an inter-prediction result (e.g., a prediction block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that is decoded based on the encoded video information.

[0082] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), possibly compare the block with already coded blocks in the same picture, generate transformed quantized coefficients, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). In one example, the intra encoder (722) also calculates intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same picture.

[0083] The generic controller (721) is configured to determine generic control data and control other components of the video encoder (703) based on the generic control data. In one example, the generic controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, if the mode is an intra mode, the generic controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream, and if the mode is an inter mode, the generic controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0084] The residual calculator (723) is configured to calculate a difference (residual data) between the received block and a prediction result selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra-encoder (722) and the inter-encoder (730) as appropriate. For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and the intra-prediction information. The decoded blocks are appropriately processed to generate decoded pictures, which can be buffered in a memory circuit (not shown) and used as a reference picture in some examples.

[0085] The entropy encoder (725) is configured to format a bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information in the bitstream according to an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include in the bitstream general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information. It should be noted that, in accordance with the disclosed subject matter, when coding a block in a merged sub-mode of either an inter mode or a bi-prediction mode, no residual information is present.

[0086] 8 shows an example diagram of a video decoder (810). The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) is used in place of the video decoder (410) of the example of FIG. 4.

[0087] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter-decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-decoder (872), coupled to each other as shown in FIG.

[0088] The entropy decoder (871) may be configured to reconstruct from the coded picture certain symbols that represent syntax elements of which the coded picture is composed. Such symbols may include, for example, prediction information (e.g., intra-mode, inter-mode, bi-predictive mode, inter-mode and bi-predictive mode of merged or other submodes, etc.) that may identify the mode in which the block is coded, as well as certain samples or metadata used for prediction by the intra-decoder (872) or the inter-decoder (880), respectively (e.g., intra-prediction and inter-prediction information, etc.). The symbols may also include, for example, residual information in the form of quantized transform coefficients, etc. In one example, if the prediction mode is an inter-mode or bi-predictive mode, the inter-prediction information is provided to the inter-decoder (880), and if the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may undergo inverse quantization and be provided to the residual decoder (873).

[0089] The inter decoder (880) is configured to receive the inter prediction information and to generate inter prediction results based on the inter prediction information.

[0090] The intra decoder (872) is configured to receive the intra prediction information and to generate a prediction result based on the intra prediction information.

[0091] The residual decoder (873) is configured to perform inverse quantization to extract inverse quantized transform coefficients, and to process the inverse quantized transform coefficients to transform the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantization parameters (QPs)), which may be provided by the entropy decoder (871) (this may be only a small amount of control information, so a data path is not depicted).

[0092] The reconstruction module (874) is configured to combine, in the spatial domain, the residual information output by the residual decoder (873) and the prediction results (possibly output by the inter-prediction module or the intra-prediction module) to form reconstructed blocks that may become part of a reconstructed picture, which may become part of a reconstructed video. It should be noted that other suitable operations, such as a deblocking operation, may be performed to improve visual quality.

[0093] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0094] Various inter prediction modes can be used in VVC. For inter predicted CUs, the motion parameters can include MV(s), one or more reference picture indexes, reference picture list usage index, and additional information about the specific coding features to be used for inter predicted sample generation. The motion parameters can be signaled explicitly or implicitly. If a CU is coded in skip mode, the CU can be associated with a PU and may not have significant residual coefficients, coded motion vector deltas, MV differences (e.g. MVD), or reference picture indexes. A merge mode can be specified if the motion parameters of the current CU are obtained from neighboring CU(s), including spatial and / or temporal candidates. The merge mode can be applied to inter predicted CUs, not only for skip mode. In one example, an alternative to the merge mode is explicit transmission of motion parameters, in which the MV(s), the corresponding reference picture index of each reference picture list, and a reference picture list usage flag and other information are explicitly signaled for each CU.

[0095] Certain inter-predictive coding tools (e.g., refined inter-predictive coding tools), such as bidirectional optical flow (BDOF) mode, decoder-side motion vector refinement (DMVR) mode, merge by motion vector difference (MMVD) mode, etc., can be used in VVC.

[0096] According to one embodiment of the present disclosure, the BDOF mode is a motion refinement technique used in inter prediction. The BDOF mode can be used to refine the bi-prediction signal of a CU, for example, at a 4x4 sub-block level. The BDOF mode can be applied to a CU when certain conditions are met. In one example, these conditions include: (i) the CU is coded using a "true" bi-prediction mode, e.g., one of the two reference pictures is before the current picture in display order and the other of the two reference pictures is after the current picture in display order; (ii) the distances (e.g., picture order count (POC) difference) from the two reference pictures to the current picture are identical; (iii) the two reference pictures are short-term reference pictures; (iv) the CU is not coded using an affine mode or a sub-block-based temporal motion vector prediction (SbTMVP) merge mode; (v) the CU has more than 64 luma samples; (vi) the CU height and CU width are equal to or greater than 8 luma samples; (vii) the bi-prediction by CU weights (BCW) weight index indicates equal weights; (viii) weighted prediction (WP) is not enabled for the current CU; and (ix) the combined inter-intra prediction (CIIP) mode is not used for the current CU.

[0097] In one embodiment, the BDOF mode is applied only to the luma component. The motion refinement of the BDOF mode is based on the concept of optical flow, which assumes, for example, that the motion of an object is smooth. With reference to FIG. 9, for each subblock, such as a 4×4 subblock (e.g., subblock (920) in a CU (910)), motion refinement (e.g., MV refinement or MVR) (Δmv x ,Δmv y) can be calculated. In one example, the L0 predicted sample and the L1 predicted sample are predicted samples based on the first reference picture (L0) and the second reference picture (L1), respectively. Motion refinement can be used to adjust the bi-predictive sample values ​​in a 4×4 sub-block (e.g., sub-block (920)). The following steps can be applied in the BDOF process.

[0098] Horizontal gradient of the two predicted signals

number

number

number

[0099] In one example, the gradient S 1 , S 2 , S 3 , S 5 , and S 6 The autocorrelation and cross-correlation of are calculated as follows:

number

number

[0100] MVR(Δmv x ,Δmv y ) can be derived using autocorrelation and cross-correlation as follows:

number

number

number

[0101] Based on the motion refinement and gradients, the following adjustment (or prediction refinement) b(x,y) can be calculated for each sample in a sub-block (e.g., a 4×4 sub-block):

number

[0102] The BDOF samples of a CU can be calculated by adjusting the bi-predictive samples as follows:

number

[0103] In one embodiment, these values ​​are selected such that the multipliers in the BDOF process do not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process is kept within 32 bits.

[0104] In one example, to derive the gradient value, we select the predicted sample I in list k (k=0,1) that is outside the current CU boundary. (k) (i,j) will be generated. Referring to FIG. 9, BDOF (e.g., in VVC) can use extended rows and / or extended columns around the boundary of the CU (910). To control the computational complexity of generating out-of-bounds predicted samples (e.g., sample prediction (930)), predicted samples (e.g., (930)) in the extended area (white positions in FIG. 9) can be generated by directly taking reference samples at nearby integer positions (e.g., using a floor() operation on the coordinates) without interpolation. In one example, an 8-tap motion compensation interpolation filter (or 8-tap interpolation filter) is used to generate predicted samples in the CU (910) (gray positions in FIG. 9). In the example shown in FIG. 9, the CU (910) includes 8×8 samples. In one example, these extended sample values ​​are used only in the gradient calculation. In one example, the gradient calculation is performed using Equation 1, as described above.

[0105] 9, if it is necessary to use sample(s) and gradient value(s) outside the CU boundary for the remaining steps of the BDOF process, the sample(s) and gradient value(s) outside the CU boundary can be generated by padding (e.g., iteratively) from corresponding nearest neighbors, as indicated by the arrows in FIG 9. For example, the predicted sample and gradient value at (930) can be padded from the corresponding sample and gradient value at (921).

[0106] If the width and / or height of a CU is greater than a threshold, such as 16 luma samples, the CU may be divided into sub-blocks with width and / or height equal to 16 luma samples. The boundaries of the sub-blocks may be treated as the boundaries of the CU in the BDOF process. The maximum unit size of the BDOF process may be limited to an area, such as 16×16 luma samples. For each sub-block, the BDOF process may be skipped. In one example, if the sum of absolute differences (SAD) between the initial L0 predicted sample and the initial L1 predicted sample is less than a threshold, the BDOF process is not applied to the sub-block. The threshold may be set to 8×W×(H>>1), where W indicates the sub-block width and H indicates the sub-block height. To avoid further complexity of the SAD calculation, the SAD between the initial L0 predicted sample and the initial L1 predicted sample calculated in the DMVR mode or DMVR process may be reused.

[0107] If BCW mode is enabled for the current block, e.g., if the BCW weight index indicates unequal weights, BDOF mode may be disabled. If WP mode is enabled for the current block, e.g., if a flag (e.g., luma_weight_lx_flag) is 1 for either of the two reference pictures, BDOF may be disabled. If the CU is coded in symmetric MVD mode or CIIP mode, BDOF may be disabled.

[0108] According to one aspect of the present disclosure, the accuracy of MV in merge mode can be improved and / or refined using, for example, the DMVR mode used in VVC.

[0109] 10 shows an example of a DMVR mode based on bilateral matching (BM) or bilateral template matching according to one embodiment of the present disclosure. A current block (1011) is in a current picture (1010). Initial MVs (e.g., MV0 and MV1) refer to a reference block (1022) in a reference picture (Ref0) (1020) in a reference picture list L0 and a reference block (1032) in a reference picture (Ref1) (1030) in a reference picture list L1, respectively.

[0110] In some examples, such as VVC, BM-based DMVR can be applied to improve the accuracy of the motion vector of the merge mode. In bi-prediction operation, refined MVs can be searched around the initial MVs (e.g., MV0 and MV1) in the reference picture list L0 and the reference picture list L1, respectively. Refined reference blocks (or candidate blocks) (1021) and (1031) can be determined. The bilateral matching method can calculate distortion based on two candidate blocks (1021), (1031) in the reference picture list L0 and the reference picture list L1.

[0111] In one example, the SAD between candidate blocks (1021), (1031) referenced by respective MV candidates (e.g., MV0′ and MV1′) around an initial MV (e.g., MV0 and MV1) is calculated. The MV candidate (e.g., MV0′ and MV1′) with the lowest SAD can be the refined MV and can be used to generate a bi-predictive signal.

[0112] The relationship between the initial MV and the MV candidates can be explained as follows.

number

[0113] In some examples, such as some applications in VVC, the application of DMVR is restricted. For example, DMVR mode is applied to CUs coded with certain modes and features. In one example, these modes and features include: (i) the CU is coded in CU level merge mode with bi-predictive MV; (ii) one reference picture (e.g., backward reference picture) is in the past with respect to the current picture, and another reference picture (e.g., forward reference picture) is in the future with respect to the current picture; (iii) the distance (e.g., POC difference) from the two reference pictures to the current picture is the same; (iv) the two reference pictures are short-term reference pictures; (v) the CU has more than 64 luma samples; (vi) the CU height and CU width are equal to or greater than 8 luma samples; (vii) the BCW weight index indicates equal weights; (viii) the WP mode is not enabled for the current CU; and (ix) the CIIP mode is not used for the current CU.

[0114] The refined MV(s) derived by the DMVR process can be used to generate inter-prediction samples and can be used in temporal motion vector prediction for future picture coding. The original MV(s) (e.g., initial MV(s)) can be used in the deblocking process and can be used in spatial motion vector prediction for future CU coding.

[0115] In one embodiment, such as in VVC, an MMVD mode may be used to predict samples of a CU (e.g., the current CU) using implicitly derived motion information. For example, after signaling a skip flag or a merge flag, an MMVD merge flag may be signaled to specify whether the MMVD mode is used for the CU.

[0116] In MMVD mode, after a merge candidate (also called an MV basis or MV starting point) is selected, the merge candidate can be refined by additional information such as signaled MVD information. The additional information can include an index used to specify the magnitude of the motion (distance index, e.g., mmvd_distance_idx[x0][y0], etc.) and an index used to indicate the direction of the motion (direction index, e.g., mmvd_direction_idx[x0][y0], etc.). In MMVD mode, one of the first two candidates in the merge list can be selected as the MV basis. For example, a merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) indicates one of the first two candidates in the merge list. The merge candidate flag can be signaled to indicate (e.g., specify) which of the first two candidates is selected. The additional information can indicate the MVD (or motion offset) relative to the MV basis. For example, the motion magnitude indicates the magnitude of the MVD, and the motion direction indicates the direction of the MVD.

[0117] In one example, a merging candidate selected from the merging candidate list is used to provide a starting point or MV starting point in the reference picture. The motion vector of the current block can be represented by a starting point and a motion offset (or MVD) including the magnitude and direction of motion relative to the starting point. On the encoder side, the selection of the merging candidate and the determination of the motion offset can be based on a search process (evaluation process) as shown in Figure 11. On the decoder side, the selected merging candidate and the motion offset can be determined based on signaling from the encoder side.

[0118] FIG. 11 illustrates an example of a search process (1100) in MMVD mode. FIG. 12 illustrates an example of search points in MMVD mode. In some examples, a subset or the entire set of search points in FIG. 12 are used in the search process (1100) of FIG. 11. For example, by performing the search process (1100) on the encoder side, additional information including a merge candidate flag (e.g., mmvd_cand_flag[x0][y0]), a distance index (e.g., mmvd_distance_idx[x0][y0]), and a direction index (e.g., mmvd_direction_idx[x0][y0]) can be determined for a current block (1101) in a current picture (or current frame).

[0119] A first motion vector (1111) and a second motion vector (1121) belonging to a first merging candidate are shown. The first motion vector (1111) and the second motion vector (1121) are MV starting points used in the search process (1100). The first merging candidate may be a merging candidate on the merging candidate list configured for the current block (1101). The first motion vector (1111) and the second motion vector (1121) may be associated with two reference pictures (1102), (1103) in the reference picture list L0 and the reference picture list L1, respectively. Referring to Figures 11-12, the first motion vector (1111) and the second motion vector (1121) may point to two starting points (1211), (1221) in the reference picture (1102) and the reference picture (1103), respectively, as shown in Figure 12.

[0120] 12, two starting points (1211), (1221) in Fig. 12 can be determined in the reference picture (1102) and the reference picture (1103). In one example, based on the starting point (1211) and the starting point (1221), multiple predetermined points extending in the vertical direction (represented by +Y or -Y) or the horizontal direction (represented by +X and -X) from the starting point (1211) and the starting point (1221) in the reference picture (1102) and the reference picture (1103) can be evaluated. In one example, pairs of points that mirror each other with respect to the respective starting point (1211) or starting point (1221), such as pair of points (1214) and (1224) (indicated by a shift of 1S in FIG. 11) or pair of points (1215) and (1225) (indicated by a shift of 2S in FIG. 11), can be used to determine pairs of motion vectors (e.g., MV (1113) and MV (1123) in FIG. 11) that may form motion vector predictor candidates for the current block (1101). Motion vector predictor candidates (e.g., MV (1113) and MV (1123) in FIG. 11) determined based on predetermined points surrounding the starting point (1211) or starting point (1221) can be evaluated.

[0121] The distance index (eg, mmvd_distance_idx[x0][y0]) specifies motion magnitude information and may indicate a given offset (eg, 1S or 2S in FIG. 11) from the starting point indicated by the merge candidate flag.

[0122] Referring to FIG. 11 , an offset (e.g., MVD(1112) or MVD(1122)) may be applied (e.g., added) to the horizontal or vertical component of the starting MV (e.g., MV(1111) or MV(1121)). An example relationship between the distance index (IDX) and the pre-defined offset is specified in Table 1. When full-pel MMVD is off, e.g., a full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is 0, the pre-defined offset of the MMVD may range from ¼ luma sample to 32 luma samples. When full-pel MMVD is off, the pre-defined offset may have a non-integer value, such as a fraction of a luma sample (e.g., ¼ pixel or ½ pixel). The range of the predefined offset of the MMVD may be from 1 luma sample to 128 luma samples when full-pel MMVD is on, e.g., when a full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is 1. In one example, when full-pel MMVD is on, the predefined offset has only integer values, such as one or more luma samples.

[0123] [Table 1]

[0124] The direction index can represent the direction (or motion direction) of the MVD relative to the starting point. In one example, the direction index represents one of the four directions shown in Table 2. The meaning of the MVD code(s) in Table 2 may change depending on the information of the starting MV(s). In one example, if the starting MV is a uni-predictive MV or the starting MV is a bi-predictive MV and both reference lists point to the same side of the current picture (e.g., the POCs of the two reference pictures are both greater than the POC of the current picture, or the POCs of the two reference pictures are both less than the POC of the current picture), the MVD code(s) in Table 2 specifies the code of the MV offset (or MVD) that is added to the starting MV.

[0125] If the start MV is a bi-predictive MV and the two MVs point to different sides of the current picture (e.g., the POC of one reference picture is larger than that of the current picture and the POC of the other reference picture is smaller than that of the current picture), the MVD code in Table 2 specifies the code of the MV offset (or MVD) added to the MV component of list0 of the start MV, and the MVD code of the MV of list1 has the opposite value. Referring to Figure 11, the start MV (1111) and the start MV (1121) are bi-predictive MVs, and the two MVs (1111), (1121) point to different sides of the current picture. The POC of the L1 reference picture (1103) is larger than that of the current picture, and the POC of the L0 reference picture (1102) is smaller than that of the current picture. The MVD sign (e.g., the sign "+" on the x-axis) indicated by the direction index (e.g., 00) in Table 2 specifies the sign (e.g., the sign "+" on the x-axis) of the MVD (e.g., MVD(1112)) added to the MV component of list0 of the starting MV (e.g., (1111)), and the MVD sign of MVD(1122) of the MV component of list1 of the starting MV (e.g., (1121)) has the opposite value, such as the sign "-" opposite to the sign "+" of MVD(1112).

[0126] Referring to Table 2, directional index 00 indicates the positive direction of the x-axis, directional index 01 indicates the negative direction of the x-axis, directional index 10 indicates the positive direction of the y-axis, and directional index 11 indicates the negative direction of the y-axis.

[0127] [Table 2]

[0128] The syntax element mmvd_merge_flag[x0][y0] may be used to represent the MMVD merge flag of the current CU. In one example, the MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) being 1 specifies that the MMVD mode is used to generate the inter prediction parameters of the current CU. The MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) being 0 specifies that the MMVD mode is not used to generate the inter prediction parameters. The array indexes x0 and y0 may specify the position (x0, y0) of the top-left luma sample of the considered coding block (e.g., the current CB) relative to the top-left luma sample of the picture (e.g., the current picture).

[0129] If the MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) is not present for the current CU, the MMVD merge flag (e.g., mmvd_merge_flag[x0][y0]) may be inferred to be 0 for the current CU.

[0130] In some examples, such as the VVC specification, a single context is used to signal the MMVD merge flag (e.g., mmvd_merge_flag). For example, a single context is used to code (e.g., encode and / or decode) the MMVD merge flag in context-adaptive binary arithmetic coding (CABAC).

[0131] The syntax element mmvd_cand_flag[x0][y0] may represent a merge candidate flag. In one example, the merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) specifies whether the first candidate (0) or the second candidate (1) in the merge candidate list is used with the MVD derived from the distance index (e.g., mmvd_distance_idx[x0][y0]) and the direction index (e.g., mmvd_direction_idx[x0][y0]). The array indexes x0 and y0 may specify the location (x0, y0) of the top-left luma sample of the considered coding block (e.g., current CB) relative to the top-left luma sample of the picture (e.g., current picture).

[0132] If a merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) is not present, then the merge candidate flag (e.g., mmvd_cand_flag[x0][y0]) can be inferred to be 0.

[0133] The syntax element mmvd_distance_idx[x0][y0] may represent a distance index. In one example, the distance index (e.g., mmvd_distance_idx[x0][y0]) specifies an index used to derive MmvdDistance[x0][y0] as specified in Table 3. The array indexes x0 and y0 may specify a position (x0, y0) of the top-left luma sample of a considered coding block (e.g., current CB) relative to the top-left luma sample of a picture (e.g., current picture).

[0134] [Table 3]

[0135] The first column of Table 3 indicates the distance index (e.g., mmvd_distance_idx[x0][y0]). The second column of Table 3 indicates the magnitude of movement (e.g., MmvdDistance[x0][y0]) when full-pel MMVD is off, e.g., the full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is 0. The third column of Table 3 indicates the magnitude of movement (e.g., MmvdDistance[x0][y0]) when full-pel MMVD is on, e.g., the full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is 1.

[0136] In one example, the units in the second and third columns of Table 3 are ¼ luma samples. Referring to the first row of Table 3, if the distance index (e.g., mmvd_distance_idx[x0][y0]) is 0, the motion magnitude (e.g., MmvdDistance[x0][y0]) is 1 when full-pel MMVD is off (e.g., slice_fpel_mmvd_enabled_flag is 0). The motion magnitude (e.g., MmvdDistance[x0][y0]) is 1×¼ luma samples or ¼ luma samples. If the distance index (e.g., mmvd_distance_idx[x0][y0]) is 0, the motion magnitude (e.g., MmvdDistance[x0][y0]) is 4 when full-pel MMVD is off (e.g., slice_fpel_mmvd_enabled_flag is 1). The motion magnitude (e.g., MmvdDistance[x0][y0]) is 4×1 / 4 luma samples or 1 luma sample.

[0137] In one example, the second column of Table 3 (in 1 / 4 luma sample units) corresponds to the second row of Table 1 (in luma sample units), and the third column of Table 3 (in 1 / 4 luma sample units) corresponds to the third row of Table 1 (in luma sample units).

[0138] The syntax element mmvd_direction_idx[x0][y0] can represent a direction index. In one example, the direction index (e.g., mmvd_direction_idx[x0][y0]) specifies an index used to derive a motion direction (e.g., MmvdSign[x0][y0]) as specified in Table 4. The array indexes x0 and y0 specify the position (x0, y0) of the top-left luma sample of the considered coding block (e.g., current CB) relative to the top-left luma sample of the picture (e.g., current picture). The first column of Table 4 indicates the direction index (e.g., mmvd_distance_idx[x0][y0]). The second column of Table 4 indicates the first component of the MVD (e.g., MVD x The third column of Table 4 indicates the first sign (e.g., MmvdSign[x0][y0][0]) of the MVD (e.g., MmvdOffset[x0][y0][0]). y Or MmvdOffset[x0][y0][1]), indicates the second sign (e.g., MmvdSign[x0][y0][1]).

[0139] [Table 4]

[0140] The first component (e.g., MmvdOffset[x0][y0][0]) and the second component (e.g., MmvdOffset[x0][y0][1]) of the MVD or offset MmvdOffset[x0][y0] can be derived as follows:

number

[0141] In one example, the distance index (e.g., mmvd_distance_idx[x0][y0]) is 3 and the direction index (e.g., mmvd_distance_idx[x0][y0]) is 2. Based on Table 4 and the direction index (e.g., mmvd_direction_idx[x0][y0]) being 2, the first component of the MVD (e.g., MVD x Or the first sign of MmvdOffset[x0][y0][0] (e.g., MmvdSign[x0][y0][0]) is 0, and the second component of MVD (e.g., MVD y Or the second sign of MmvdOffset[x0][y0][1] (e.g., MmvdSign[x0][y0][1]) is "+1". In this example, the MVD is along the positive vertical direction (+y) and has no horizontal component.

[0142] When the full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is 0 and full-pel MMVD is off, the motion magnitude indicated by MmvdDistance[x0][y0] is 8 based on Table 3 and the distance index (e.g., mmvd_distance_idx[x0][y0]) is 3. Based on Equations 10-11, the first component of MVD (e.g., MmvdOffset[x0][y0][0]) is (8<<2)×0=0 and the second component of MVD (e.g., MmvdOffset[x0][y0][1]) is (8<<2)×(+1)=2 (luma samples).

[0143] When the full-pel MMVD flag (e.g., slice_fpel_mmvd_enabled_flag) is 1 and full-pel MMVD is on, based on Table 3 and the distance index (e.g., mmvd_distance_idx[x0][y0]) is 3, the motion magnitude indicated by MmvdDistance[x0][y0] is 32. Based on Equations 10-11, the first component of MVD (e.g., MmvdOffset[x0][y0][0]) is (32<<2)×0=0, and the second component of MVD (e.g., MmvdOffset[x0][y0][1]) is (32<<2)×(+1)=8 (luma samples).

[0144] The present disclosure includes embodiments relating to improved signaling in MMVD mode.

[0145] In one embodiment, when the BDOF mode and / or the DMVR mode are applied to a coding block in a bi-predictive picture (e.g., a B picture) having both forward and backward reference pictures, the improvement of coding efficiency for the coding block in the bi-predictive picture by the MMVD mode may not be significant. Thus, in some examples, when the BDOF mode and / or the DMVR mode are applied to the coding block, the MMVD mode is not applied to the coding block. On the other hand, in various examples, the MMVD merge flag of the coding block is signaled regardless of whether the MMVD mode is applied to the coding block. For example, the signaling cost of signaling the MMVD merge flag of the coding block may be relatively large because the MMVD merge flag is signaled for the coding block when the MMVD mode is unlikely to be applied to the coding block.

[0146] According to an embodiment of the present disclosure, the MMVD merge flag (e.g., indicated by syntax element mmvd_merge_flag) may not be signaled for each block in the current picture. The MMVD merge flag may be conditionally signaled for a current block in the current picture based on whether one or more requirements are met. The requirements may include a first requirement and a second requirement. The first requirement may include that other(s) inter-prediction mode (e.g., BDOF mode and / or DMVR mode) are allowed for a plurality of blocks including the current block. The second requirement may include that a high-level flag (e.g., MMVD merge enable flag) of the plurality of blocks indicates that the MMVD mode is not enabled for the plurality of blocks. In some embodiments, the first requirement further includes that the reference pictures of the plurality of blocks include a forward reference picture and a backward reference picture. In some embodiments, the first requirement further includes that the reference pictures of the plurality of blocks are symmetric with respect to the current picture.

[0147] In one example, the MMVD merge flag (eg, mmvd_merge_flag) may be conditionally signaled based on the use of DMVR mode and / or BDOF mode.

[0148] At the decoder side, prediction information of the plurality of blocks in the current picture may be decoded from the coded video bitstream. Whether at least one of the requirements is satisfied may be determined based on the prediction information. In some embodiments, the requirements include (i) a first requirement (e.g., at least one of DMVR mode or BDOF mode is allowed for the plurality of blocks) and (ii) a second requirement (e.g., the MMVD merge enable flag indicates that the MMVD mode is not enabled for the plurality of blocks). Whether the plurality of MMVD merge flags are inferred to be false for the plurality of blocks may be determined based on whether at least one of the requirements is satisfied. Each of the plurality of MMVD merge flags may indicate, for a corresponding block of the plurality of blocks, whether the MMVD mode is applied to the corresponding block. In response to determining that the plurality of MMVD merge flags are inferred to be false for the plurality of blocks, each block in the plurality of blocks may be reconstructed without applying the MMVD mode. Coding efficiency is improved because MMVD merge flags for blocks are not signaled (eg, MMVD merge flags are presumed to be false) in the coded video bitstream.

[0149] In one example, the multiple blocks are slices in the current picture.In one example, the multiple blocks are the current picture.

[0150] In one embodiment, the prediction information includes at least one flag indicating whether at least one of the DMVR mode or the BDOF mode is allowed for multiple blocks. For example, the prediction information includes a first slice-level flag indicating whether the DMVR mode is allowed for multiple blocks in a slice (e.g., the current slice) of the current picture. For example, the prediction information includes a second slice-level flag indicating whether the BDOF mode is allowed for multiple blocks in the slice.

[0151] In one embodiment, the merging first requirement requires that at least one of DMVR mode or BDOF mode is allowed for the plurality of blocks, the reference pictures of the plurality of blocks include a forward reference picture and a backward reference picture, and an absolute value of a first Picture Order Count (POC) difference between the forward reference picture and the current picture is equal to an absolute value of a second POC difference between the backward reference picture and the current picture. The plurality of MMVD merging flags of the plurality of blocks may be determined to be inferred to be false, for example, based on the first requirement being satisfied. In some examples, the first requirement further includes that the forward reference picture and the backward reference picture are short-term reference pictures. For example, the first requirement further includes that the absolute value of the first POC difference and the absolute value of the second POC difference are less than a threshold value (e.g., 2). In one example, the threshold value is 2.

[0152] In one embodiment, if DMVR mode and / or BDOF mode are allowed for the current slice (e.g., the plurality of blocks), the current slice has a forward reference picture and a backward reference picture, and the current picture is temporally in between the two reference pictures (e.g., the forward reference picture and the backward reference picture), then MMVD mode is not allowed for the plurality of blocks, and the MMVD merge flag (e.g., mmvd_merge_flag) of each block in the plurality of blocks is not signaled. The MMVD merge flag (e.g., mmvd_merge_flag) of each block in the plurality of blocks may be inferred to be false.

[0153] The probability of using the MMVD mode may depend on the absolute POC difference between the two reference pictures and the current picture (e.g., the absolute value of the first POC difference and the absolute value of the second POC difference). In one embodiment, as the absolute POC difference decreases, the probability of using the MMVD mode decreases. In one example, a further constraint on the temporal distance (e.g., the absolute POC difference) between the current picture and the two reference pictures may be further applied. For example, the absolute POC difference is less than 1 or less than 2.

[0154] According to an embodiment of the present disclosure, the prediction information includes a single high-level MMVD merge enable flag corresponding to all blocks in the plurality of blocks. The single high-level MMVD merge enable flag may indicate whether the MMVD mode is enabled or not enabled for the plurality of blocks. Based on the MMVD merge enable flag, the plurality of MMVD merge flags corresponding to each of the plurality of blocks may be inferred. The MMVD merge enable flag may be a high-level flag such as a slice level flag in a slice header (SH) (e.g., sh_mmvd_merge_enable_flag). The MMVD merge enable flag may be a high-level flag such as a picture level flag in a picture header (PH) (e.g., ph_mmvd_merge_enable_flag).

[0155] In one embodiment, the MMVD merge enable flag may indicate that MMVD mode is not enabled for the plurality of blocks, which satisfies the second requirement, in which case all of the plurality of MMVD merge flags corresponding to each of the plurality of blocks may be inferred to be false.

[0156] In one embodiment, the MMVD merge enable flag may indicate that the MMVD mode is enabled for multiple blocks, which does not meet the second requirement. In this case, the multiple MMVD merge flags are not inferred as false. Instead, multiple MMVD merge flags corresponding to each of the multiple blocks may be signaled in the coded video bitstream and received at the decoder side.

[0157] In one example, the prediction information may include: (i) at least one of DMVR mode or BDOF mode is allowed for the plurality of blocks, (ii) reference pictures of the plurality of blocks include a forward reference picture and a backward reference picture, and (iii) an absolute value of a first POC difference between the forward reference picture and the current picture is equal to an absolute value of a second POC difference between the backward reference picture and the current picture. If the MMVD merge enable flag indicates that the MMVD mode is not enabled for the plurality of blocks, the plurality of MMVD merge flags are not inferred as false.

[0158] The MMVD merge enable flag in a slice header or picture header may indicate whether the MMVD merge mode can be used for the coded slice corresponding to the slice header (e.g., the current slice) or the coded picture corresponding to the picture header (e.g., the current picture).

[0159] In one embodiment, if the MMVD merge enable flag (eg, sh_mmvd_merge_enable_flag) in the slice header corresponding to a slice is FALSE, then the MMVD merge flag (eg, mmvd_merge_flag) is inferred to be FALSE for all blocks in the slice.

[0160] In one embodiment, if the MMVD merge enable flag (eg, ph_mmvd_merge_enable_flag) in the picture header corresponding to a picture is FALSE, then the MMVD merge flag (eg, mmvd_merge_flag) is inferred to be FALSE for all blocks in the picture.

[0161] In one embodiment, in response to determining that the multiple MMVD merge flags are not inferred to false for multiple blocks (e.g., blocks within a slice or blocks within a picture), a separate respective MMVD merge flag for each block within the multiple blocks may be signaled in the coded video bitstream. The multiple MMVD merge flags may be coded using CABAC.

[0162] In the related video coding techniques, a single context (e.g., referred to as an old context) is used during CABAC to code at least one MMVD merge flag in the plurality of MMVD merge flags. According to an embodiment of the present disclosure, multiple contexts may be used to CABAC code (e.g., encode and / or decode) at least one MMVD merge flag in the plurality of MMVD merge flags. In one example, multiple contexts are used in CABAC to code (e.g., encode and / or decode) each MMVD merge flag in the plurality of MMVD merge flags. In one example, the multiple contexts include a first context and a second context. In one example, the second context is an old context used in the related video coding techniques, and the first context is a new context not used in the related video coding techniques. A context may be selected from among the multiple contexts (e.g., the first context and the second context) based on whether one or both of the DMVR mode or the BDOF mode are allowed for the plurality of blocks. CABAC along with the selected context may be used to code (eg, encode and / or decode) respective MMVD merge flags for each block in the plurality of blocks.

[0163] In one embodiment, the prediction information indicates a condition for selecting a first context (e.g., a new context). The condition may be the same as the first requirement or may be different. In one example, the condition is the same as the first requirement. For example, the condition includes: at least one of DMVR mode or BDOF mode is allowed for the plurality of blocks, the reference pictures of the plurality of blocks include a forward reference picture and a backward reference picture, and an absolute value of a first POC difference between the forward reference picture and the current picture is equal to an absolute value of a second POC difference between the backward reference picture and the current picture. If the condition is met, the first context may be selected as the context. The second context may be selected as the context based on one of the conditions not being met.

[0164] In one example, when a first context (e.g., a new context) is selected as the CABAC context, the first context is initialized with a probability indicating that each MMVD merge flag has a higher probability of being false than the probability of the MMVD merge flag being true.

[0165] In one embodiment, the first context (e.g., new context) and the second context (e.g., old context) are contexts used in CABAC coding of an MMVD merge flag (e.g., mmvd_merge_flag). The selection between the first context (e.g., new context) and the second context (e.g., old context) may be based on the use of DMVR mode and / or BDOF mode.

[0166] In one embodiment, the condition for selecting between the first context and the second context is the same as the first requirement: if the condition is met (e.g., is true), the first context (e.g., the new context) is selected; if not, the second context is used.

[0167] In one embodiment, the first context (e.g., a new context) is initialized with a probability indicating that the likelihood that the MMVD merge flag (e.g., mmvd_merge_flag) is false is greater than the likelihood that it is not (e.g., the probability is greater than the probability that the MMVD merge flag is true) when the first context (e.g., a new context) is selected.

[0168] FIG. 13 shows a flow chart outlining an encoding process (1300) according to one embodiment of the present disclosure. In various embodiments, the process (1300) is performed by processing circuitry, such as processing circuitry within terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video encoder (e.g., (403), (603), (703)), etc. In some embodiments, the process (1300) is implemented with software instructions, such that the processing circuitry performs the process (1300) when it executes the software instructions. The process starts at (S1301) and proceeds to (S1310).

[0169] At (S1310), it may be determined whether at least one requirement is satisfied. The requirements may include a first requirement and a second requirement. The first requirement may include that at least one of DMVR mode or BDOF mode is allowed for the plurality of blocks in the current picture. The first requirement may further include (one or more) additional requirements. For example, the first requirement may include that (i) the reference pictures of the plurality of blocks include a forward reference picture and a backward reference picture, and (ii) an absolute value of a first Picture Order Count (POC) difference between the forward reference picture and the current picture is equal to an absolute value of a second POC difference between the backward reference picture and the current picture. The second requirement may include that the MMVD mode is not enabled for the plurality of blocks.

[0170] If at least one of the requirements is met, the process (1300) proceeds to (S1320). If none of the requirements are met, the process (1300) proceeds to (S1340).

[0171] In (S1320), if at least one of the requirements is met, each block in the plurality of blocks may be encoded without applying the MMVD mode.

[0172] In one example, the first requirement further includes that the reference pictures of the multiple blocks include a forward reference picture and a backward reference picture, and the absolute value of the first POC difference is equal to the absolute value of the second POC difference. The first requirement is met. Each block in the multiple blocks can be encoded without applying the MMVD mode.

[0173] In one example, the second requirement (e.g., MMVD mode is not enabled for the multiple blocks) is met, and each block in the multiple blocks can be encoded without applying the MMVD mode.

[0174] As part of the branch that includes (S1320), the process (1300) proceeds to (S1330).

[0175] At (S1330), the MMVD merge flags for the blocks are not encoded or signaled in the video bitstream. In one example, MMVD mode is not enabled for the blocks, and thus a flag, such as a high-level flag (e.g., an MMVD merge enable flag), is encoded and signaled in the bitstream to indicate that MMVD mode is not enabled for the blocks.

[0176] In one example, the first requirement is met and no high-level flag (e.g., MMVD merge enable flag) is signaled, and the decoder can infer that the MMVD merge flags of the blocks are false based on the first requirement being met.

[0177] The process (1300) then proceeds to (S1399) and ends.

[0178] At (S1340), for each block in the plurality of blocks, a separate decision is made as to whether to encode each block in the MMVD mode and encode each block according to the decision.

[0179] In one example, it is determined that the MMVD mode is applied to each block, and each block is encoded by applying the MMVD mode.

[0180] In another example, it is determined that the MMVD mode does not apply to each block, and each block is encoded without applying the MMVD mode.

[0181] As part of the branch that includes (S1340), the process (1300) proceeds to (S1350).

[0182] At (S1350), multiple MMVD merge flags for multiple blocks are encoded and signaled in the video bitstream, and each of the multiple encoded and signaled MMVD merge flags indicates that a respective block in the multiple blocks has been determined to be encoded in MMVD mode.

[0183] In one example, the MMVD mode is enabled for multiple blocks, and thus an MMVD merge enable flag is encoded and signaled in the video bitstream to indicate that the MMVD mode is enabled for multiple blocks.

[0184] After step S1350, the process 1300 proceeds to S1399 and ends.

[0185] The process (1300) can be adapted appropriately for various scenarios, and the steps in the process (1300) can be adjusted accordingly. One or more of the steps in the process (1300) can be adapted, omitted, repeated, and / or combined. Any suitable order can be used to perform the process (1300). Additional step(s) can be added.

[0186] In one embodiment, the first requirement is met and the second requirement is not met. In the example shown in Figure 13, the process (1300) proceeds to a branch including (S1320) and (S1330), where each block in the plurality of blocks can be encoded without applying an MMVD mode (S1320), and the plurality of MMVD merge flags are not encoded or signaled (S1320).

[0187] Alternatively (not shown in FIG. 13), if the first requirement is met, the second requirement is not met, and the MMVD mode is enabled for the plurality of blocks, the process (1300) may proceed to a branch including (S1340) and (S1350). For example, a decision may be made for each block in the plurality of blocks as to whether to encode the respective block in the MMVD mode. The plurality of MMVD merge flags may be encoded and signaled in the bitstream. Additionally, an MMVD merge enable flag may be encoded and signaled.

[0188] FIG. 14 shows a flow chart outlining a decoding process (1400) according to one embodiment of the present disclosure. In various embodiments, the process (1400) is performed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video encoder (403), processing circuitry performing the functions of a video decoder (410), processing circuitry performing the functions of a video decoder (510), processing circuitry performing the functions of a video encoder (603), etc. In some embodiments, the process (1400) is implemented with software instructions, such that the processing circuitry performs the process (1400) when it executes the software instructions. The process starts at (S1401) and proceeds to (S1410).

[0189] At (S1410), prediction information for a plurality of blocks in a current picture may be decoded from a coded video bitstream. The plurality of blocks may be slices in the current picture. The plurality of blocks may be the current picture.

[0190] At (S1420), it may be determined whether at least one requirement is satisfied based on the prediction information. The requirements may include (i) a first requirement (e.g., at least one of a decoder-side motion vector refinement (DMVR) mode or a bidirectional optical flow (BDOF) mode is allowed for the plurality of blocks) and (ii) a second requirement (e.g., an MMVD merge enable flag indicates that a merge by motion vector difference (MMVD) mode is not enabled for the plurality of blocks).

[0191] In one embodiment, the prediction information includes one or more flags indicating whether at least one of DMVR mode or BDOF mode is allowed for the blocks. The one or more flags may include a high-level flag(s). In one example, the one or more flags include (i) a slice-level flag indicating whether DMVR mode is allowed for the slice, and / or (ii) a slice-level flag indicating whether BDOF mode is allowed for the slice.

[0192] At (S1430), it may be determined whether the plurality of MMVD merge flags are inferred to be false for the plurality of blocks based on whether at least one of the requirements is satisfied. Each of the plurality of MMVD merge flags may indicate whether an MMVD mode is applied to a corresponding block in the plurality of blocks, respectively.

[0193] In one embodiment, the first requirement includes: (i) at least one of DMVR mode or BDOF mode is allowed for the plurality of blocks; (ii) the reference pictures of the plurality of blocks include a forward reference picture and a backward reference picture; and (iii) an absolute value of a first Picture Order Count (POC) difference between the forward reference picture and the current picture is equal to an absolute value of a second POC difference between the backward reference picture and the current picture. The plurality of MMVD merge flags of the plurality of blocks may be inferred to be false, for example, when the first requirement is met. In one example, when the first requirement is met, the plurality of MMVD merge flags of the plurality of blocks are not signaled.

[0194] In one example, the first requirement further includes that the absolute value of the first POC difference and the absolute value of the second POC difference are less than a threshold value. The threshold value may be 2.

[0195] In one embodiment, the prediction information includes MMVD merge enable flags for multiple blocks, which may be high-level flags (e.g., higher than blocks), such as slice-level flags in a slice header, picture-level flags in a picture header, etc.

[0196] In one example, the MMVD merge enable flag indicates that MMVD mode is not enabled for the blocks, which satisfies the second requirement. The MMVD merge flags of the blocks may be inferred to be false, for example, when the second requirement is met (e.g., the MMVD merge enable flag indicates that MMVD mode is not enabled for the blocks).

[0197] At (S1440), in response to determining that the plurality of MMVD merge flags are inferred to be false for the plurality of blocks, each block in the plurality of blocks may be reconstructed without applying an MMVD mode. In one example, a DMVR mode and / or a BDOF mode is applied to one or more blocks in the plurality of blocks, and one or more blocks in the plurality of blocks are reconstructed using the DMVR mode and / or the BDOF mode. Because the plurality of MMVD merge flags are inferred to be false, no MMVD merge flags are signaled in the coded video bitstream.

[0198] In one embodiment, in response to determining that the plurality of MMVD merge flags are not inferred to be false, an alternative step (not shown) of (S1440) is performed, in which the plurality of MMVD merge flags are signaled in the coded video bitstream. If a first MMVD merge flag in the plurality of MMVD merge flags signaled in the coded video bitstream indicates that an MMVD mode is applied to a first block in the plurality of blocks, a first block in the plurality of blocks may be reconstructed by applying an MMVD mode. If a second MMVD merge flag in the plurality of MMVD merge flags signaled in the coded video bitstream indicates that an MMVD mode is not applied to a second block in the plurality of blocks, a second block in the plurality of blocks may be reconstructed without applying an MMVD mode.

[0199] After (S1440) or an alternative step, the process (1400) proceeds to (S1499) and ends.

[0200] Process (1400) can be adapted appropriately for various scenarios, and steps within process (1400) can be adjusted accordingly. One or more of the steps within process (1400) can be adapted, omitted, repeated, and / or combined. Any suitable order can be used to perform process (1400). Additional step(s) can be added.

[0201] In one embodiment, in response to determining that the plurality of MMVD merge flags are not inferred to be false for the plurality of blocks, the respective MMVD merge flags for each block in the plurality of blocks may be signaled in the coded video bitstream. A context may be selected from among the first context and the second context based on whether one or both of the DMVR mode or the BDOF mode are allowed for the plurality of blocks. The respective MMVD merge flags for each block in the plurality of blocks may be decoded using context-adaptive binary arithmetic coding (CABAC) with the selected context.

[0202] In one example, the prediction information indicates whether a condition for selecting a first context is satisfied. The conditions include: (i) one or both of DMVR mode or BDOF mode are allowed for the plurality of blocks; (ii) the reference pictures of the plurality of blocks include a forward reference picture and a backward reference picture; and (iii) the absolute value of a first POC difference between the forward reference picture and the current picture is equal to the absolute value of a second POC difference between the backward reference picture and the current picture. The first context may be selected as a context based on the condition being satisfied. The second context may be selected as a context based on one of the conditions not being satisfied.

[0203] In one example, the first context is initialized with a probability indicating that the MMVD merge flag of each block in the plurality of blocks has a higher probability of the MMVD merge flag being false than the probability of the MMVD merge flag being true.

[0204] In one example, the prediction information includes an MMVD merge enable flag for the plurality of blocks, and the MMVD merge enable flag indicates that an MMVD mode is enabled for the plurality of blocks, which does not satisfy the second requirement. If the second requirement is not satisfied (e.g., the MMVD merge enable flag indicates that an MMVD mode is enabled for the plurality of blocks), the plurality of MMVD merge flags are not inferred as false, and it is determined that the plurality of MMVD merge flags for the plurality of blocks are signaled in the coded video bitstream.

[0205] In an example not shown in FIG. 14, regardless of whether the first requirement is met (e.g., the prediction information indicates that (i) at least one of DMVR mode or BDOF mode is allowed for the multiple blocks, (ii) the reference pictures for the multiple blocks include a forward reference picture and a backward reference picture, and (iii) the absolute value of the first POC difference between the forward reference picture and the current picture is equal to the absolute value of the second POC difference between the backward reference picture and the current picture), for example, if the second requirement is not met (e.g., the MMVD merge enable flag indicates that the MMVD mode is enabled for the multiple blocks), the multiple MMVD merge flags are not inferred to be false and are signaled in the coded video bitstream.

[0206] In some embodiments, prediction information for a plurality of blocks in the current picture is received from a coded video bitstream. Whether at least one of a DMVR mode or a BDOF mode is allowed for the plurality of blocks can be determined based on the prediction information. In response to determining that at least one of a DMVR mode or a BDOF mode is allowed for the plurality of blocks, a plurality of MMVD merge flags can be inferred as false for the plurality of blocks, each of the plurality of MMVD merge flags indicating whether an MMVD mode applies to the plurality of blocks. In response to determining that the plurality of MMVD merge flags are inferred as false for the plurality of blocks, each block in the plurality of blocks can be reconstructed without applying an MMVD mode.

[0207] In one embodiment, (i) the current picture references a forward reference picture and a backward reference picture, and (ii) the current picture is intermediate between the forward reference picture and the backward reference picture. Based on condition (i) and condition (ii), it is determined that the multiple MMVD merge flags are not signaled. Thus, the multiple MMVD merge flags can be inferred based on the determination that the MMVD mode is not allowed for the current picture.

[0208] In one embodiment, the prediction information includes an MMVD merge enable flag for the blocks, the MMVD merge enable flag indicating that MMVD mode is not enabled for the blocks, and the MMVD merge flags for the blocks are inferred to be false.

[0209] The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), the encoder, and the decoder may be implemented by processing circuitry (e.g., one or more processors and / or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0210] The techniques described above can be implemented as computer software physically stored on one or more computer readable media using computer readable instructions. For example, Figure 15 illustrates a computer system (1500) suitable for implementing certain embodiments of the disclosed subject matter.

[0211] The computer software can be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similar mechanisms to produce code containing instructions that can be executed directly, or via interpretation, microcode execution, etc., by one or more computer central processing units (CPUs) and graphics processing units (GPUs), etc.

[0212] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smart phones, gaming consoles, Internet of Things devices, and the like.

[0213] 15 for computer system (1500) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1500).

[0214] The computer system (1500) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0215] The input human interface devices may include one or more (only one of each is shown) of a keyboard (1501), a mouse (1502), a trackpad (1503), a touch screen (1510), a data glove (not shown), a joystick (1505), a microphone (1506), a scanner (1507), and a camera (1508).

[0216] The computer system (1500) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1510), data gloves (not shown), or joystick (1505), although there may also be haptic feedback devices that do not function as input devices), audio output devices (such as speakers (1509), headphones (not shown)), visual output devices (such as screens (1510), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, or output in more than three dimensions by means of stereoscopic output, etc.), and printers (not shown).

[0217] The computer system (1500) may also include human accessible storage devices and associated media for such storage devices, such as optical media including CD / DVD ROM / RW (1520) with CD / DVD or similar media (1521), thumb drives (1522), removable hard drives or solid state drives (1523), legacy magnetic media such as tapes or floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0218] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0219] The computer system (1500) may also include an interface (1554) to one or more communication networks (1555). The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, and delay tolerant. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, and vehicular and industrial including CANBus. Certain networks generally require an external network interface adapter attached to a particular general-purpose data port (e.g., a USB port of the computer system (1500)) or peripheral bus (1549), while other networks are generally integrated into the core of the computer system (1500) by connection to a system bus described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1500) can communicate with other entities. Such communication can be unidirectional, receive only (e.g., television broadcast), unidirectional transmit only (e.g., CANbus to a specific CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks can be used with each of the networks and network interfaces as described above.

[0220] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be connected to the core (1540) of the computer system (1500).

[0221] The cores (1540) may include one or more central processing units (CPUs) (1541), graphics processing units (GPUs) (1542), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1543), hardware accelerators for specific tasks (1544), graphics adapters (1550), etc. These devices may be connected via a system bus (1548) along with read only memory (ROM) (1545), random access memory (1546), internal mass storage (1547) such as an internal hard drive that is not accessible to the user, SSDs, etc. In some computer systems, the system bus (1548) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (1548) or via a peripheral bus (1549). In one example, a screen (1510) may be connected to the graphics adapter (1550). Peripheral bus architectures include PCI, USB, and the like.

[0222] The CPU (1541), GPU (1542), FPGA (1543), and accelerator (1544) can execute certain instructions that, in combination, may constitute the aforementioned computer code. The computer code may be stored in a ROM (1545) or a RAM (1546). Temporary data may be stored in the RAM (1546), while persistent data may be stored, for example, in an internal mass storage (1547). The use of cache memory, which may be closely associated with one or more of the CPU (1541), GPU (1542), mass storage (1547), ROM (1545), RAM (1546), etc., may enable fast storage and retrieval from any of the memory devices.

[0223] The computer readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the available kind well known to those skilled in the computer software arts.

[0224] By way of example and not limitation, a computer system (1500) having an architecture, and in particular a core (1540), may provide functionality as a result of a processor (or processors) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as described above, as well as media associated with specific storage of the core (1540) that is non-transitory in nature, such as the core internal mass storage (1547) or ROM (1545). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (1540). The computer-readable media may include one or more memory devices or chips according to specific needs. The software may cause the core (1540), and in particular the processors (including CPUs, GPUs, FPGAs, etc.) therein, to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (1546) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1544)) that may operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0225] Appendix A: Acronyms JEM: joint exploration model VVC: versatile video coding BMS:benchmark set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI: Video Usability Information GOP: Group of Pictures, a collection of pictures TU: Transform Unit PU: Prediction Unit CTU: Coding Tree Unit CTB: Coding Tree Block PB: Prediction Block HRD: Hypothetical Reference Decoder SNR: Signal Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT:Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD:Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Areas SSD: solid-state drive IC: Integrated Circuit CU: Coding Unit RD: Rate-Distortion

[0226] While this disclosure describes several exemplary embodiments, there exist modifications, substitutions, and various alternative equivalents that are within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are within the spirit and scope of the present disclosure. [Explanation of symbols]

[0227] 300 Communication Systems 310,320,330,340 Terminal Devices 350 Network 400 Communication Systems 401 Video Source 402 Video Picture Stream 403 Video Encoder 404 Encoded video data 405 Streaming Server 406 Client Subsystem 407 Copy of encoded video data 408 Client Subsystem 409 Copy of encoded video data 410 Video Decoder 411 Video Picture Output Stream 412 Display 413 Capture Subsystem 420,430 Electronic Devices 501 Channel 510 Video Decoder 512 Rendering Device 515 Buffer Memory 520 Parser 521 Symbols 530 Electronic Devices 531 Receiver 551 Scaler / Descaler Unit 552 Intra-picture prediction unit 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557 Reference Picture Memory 558 Current Picture Buffer 601 Video Sources 603 Video Encoder 620 Electronic Devices 630 Source Coder 632 Coding Engine 633 Decoder 634 Reference Picture Memory 635 Predictor 640 Transmitter 643 coded video sequence 645 Entropy Coder 650 Controller 660 Communication Channels 703 Video Encoder 721 General-purpose controller 722 Intra Encoder 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 Switch 728 Residual Decoder 730 InterEncoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Reconstruction Module 880 Interdecoder 1500 Computer System 1501 Keyboard 1502 Mouse 1503 Trackpad 1505 Joystick 1506 Microphone 1507 Scanner 1508 Camera 1509 Speaker 1510 Touch Screen 1520 CD / DVD ROM / RW 1521 CD / DVD or similar media 1522 Samud Drive 1523 Removable Hard Drive or Solid State Drive 1540 Core 1541 Central Processing Unit (CPU) 1542 Graphics Processing Unit (GPU) 1543 Field Programmable Gate Area (FPGA) 1544 Hardware Accelerator 1545 Read-Only Memory (ROM) 1546 Random Access Memory 1547 Internal Mass Storage 1548 System Bus 1549 Peripheral Bus 1550 Graphics Adapter 1554 Interface 1555 Communication Network

Claims

1. A method for video encoding in a video encoder, comprising: determining whether one of a first requirement and a second requirement is met, the first requirement including at least one of a decoder-side motion vector refinement (DMVR) mode or a bidirectional optical flow (BDOF) mode being allowed for a plurality of blocks in the current picture, and the second requirement including a merge by motion vector difference (MMVD) mode being not enabled for the plurality of blocks; in response to determining that one of the first requirement and the second requirement is satisfied, encoding each block in the plurality of blocks without applying the MMVD mode; Including, A method according to claim 1, wherein MMVD merge flags for the blocks are not encoded and are not signaled in a video bitstream.

2. The first requirement is met, 2. The method of claim 1, wherein the encoding step comprises, in response to determining that the at least one of the DMVR mode or the BDOF mode is allowed for the plurality of blocks in the current picture, encoding each block in the plurality of blocks without applying the MMVD mode.

3. The method described in claim 2, further comprising a step of encoding at least one flag in prediction information of the multiple blocks, the at least one flag indicating that at least one of the DMVR mode or the BDOF mode is allowed for the multiple blocks.

4. The method described in claim 2, wherein the first requirement is met, further comprising the current picture referencing a forward reference picture and a backward reference picture, the current picture being intermediate between the forward reference picture and the backward reference picture.

5. The method described in claim 4, wherein the first requirement further includes that the absolute value of a first picture order count (POC) difference between the forward reference picture and the current picture is equal to the absolute value of a second POC difference between the backward reference picture and the current picture.

6. The method of claim 5, wherein the absolute value of the first POC difference and the absolute value of the second POC difference are less than a threshold value.

7. The second requirement is met, 2. The method of claim 1, wherein the encoding step comprises, in response to determining that the MMVD mode is not enabled for the plurality of blocks, encoding each block in the plurality of blocks without applying the MMVD mode.

8. The method described in claim 7, further comprising a step of encoding an MMVD merge enable flag into prediction information of the multiple blocks, the MMVD merge enable flag indicating that the MMVD mode is not enabled for the multiple blocks.

9. The method of claim 1, wherein the multiple blocks are slices within the current picture.

10. The method of claim 1, further comprising: in response to determining that the first requirement and the second requirement are not satisfied, determining, for each block in the plurality of blocks, whether to encode each block in the MMVD mode; encoding and signaling each MMVD merge flag for each block in the plurality of blocks into a video bitstream; 2. The method of claim 1, comprising:

11. An encoder configured to perform a method according to any one of claims 1 to 10.

12. A computer program for causing a computer to carry out the method according to any one of claims 1 to 10.

Images