METHOD AND APPARATUS FOR VIDEO CODING - Patent application

Abstract

To provide a method and apparatus for video encoding / decoding.SOLUTION: A method includes determining a parameter based on prediction information decoded from a coded video bitstream. The parameter is in a range that depends on a flag indicative of an enabled / disabled status of subblock based temporal motion vector prediction. The method also includes calculating the maximum number of candidates in a subblock based merge candidate list based on the parameter, and reconstructing, in response to a current block in a subblock based prediction mode, samples of the current block based on a candidate selection from a constructed subblock based merge candidate list of the current block. The constructed subblock based merge candidate list of the current block is constrained by the maximum number.SELECTED DRAWING: Figure 18

Description

[Technical field]

[0001] [Incorporated by reference] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 005,511, entitled "METHOD OF SETTING NUMBER OF SUBBLOCK MERGE CANDIDATES," filed on April 6, 2020, which claims the benefit of priority to U.S. Provisional Patent Application No. 17 / 217,595, entitled "METHOD AND APPARATUS FOR VIDEO CODING," filed on March 30, 2021. The entire disclosure of the prior application is hereby incorporated by reference in its entirety.

[0002] This disclosure describes embodiments generally related to video coding. [Background technology]

[0003] The discussion of the background art provided herein is intended to generally present the context of the present disclosure. The inventors' work is not admitted expressly or impliedly as prior art to the present disclosure to the extent that their work is described in this background section, or in aspects of the description that may not otherwise qualify as prior art at the time of filing.

[0004] Video coding and decoding can be performed using intra-image prediction with motion compensation. Uncompressed digital video can include a sequence of images, each having spatial dimensions of, for example, 1920x1080 luminance samples and associated chrominance samples. The sequence of images can have a fixed or variable image rate (also informally called frame rate) of, for example, 60 images per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luminance sample resolution at a frame rate of 60 Hz) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage.

[0005] One goal of video coding and decoding can be the reduction of redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage requirements, in some cases by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from a compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application. For video, lossy compression is widely used. The amount of distortion that is tolerated depends on the application; for example, a user of a particular consumer streaming application may tolerate higher distortion than a user of a television distribution application. The achievable compression ratio can reflect the following: a higher tolerable / acceptable distortion can result in a higher compression ratio.

[0006] Motion compensation can be a lossy compression technique and can be related to a technique in which blocks of sample data from a previously reconstructed image or part thereof (reference image) are spatially shifted in a direction indicated by a motion vector (hereinafter MV) and then used for the prediction of a newly reconstructed image or part thereof. In some cases, the reference image can be the same as the image currently being reconstructed. The MV can have two dimensions, X and Y, or three dimensions, the third dimension being a representation of the reference image being used (the latter can indirectly be the temporal dimension).

[0007] In some video compression techniques, the MV applicable to a region of sample data can be predicted from other MVs, for example from an MV associated with another region of sample data that is spatially adjacent to the region under reconstruction and precedes that MV in decoding order. This can significantly reduce the amount of data required to code the MV, thereby removing redundancy and increasing compression. MV prediction can work effectively, for example, when coding an input video signal derived from a camera (known as natural video), since there is a statistical probability that regions larger than the region to which a single MV is applied will move in a similar direction and therefore, in some cases, can be predicted using similar motion vectors derived from MVs of neighboring regions. As a result, for a given region, the MV predicted from the surrounding MVs is found to be similar or identical, and, after entropy coding, can be represented with a smaller number of bits than would be used if coding the MV directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, the MV prediction itself can be lossy, for example, due to rounding errors in computing the predictor from several surrounding MVs.

[0008] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms provided by H.265, the one described herein is a technique hereinafter referred to as “spatial merging”.

[0009] Referring to Figure 1, the current block (101) contains samples that the encoder found during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of coding the MV directly, the MV can be derived from metadata associated with one or more reference pictures, for example, using the MV associated with any of the five surrounding samples denoted A0, A1, and B0, B1, B2 (102 to 106, respectively) from the most recent (in decoding order) reference picture. In H.265, the MV prediction can use predictors from the same reference picture as the neighboring blocks use. Summary of the Invention

[0010] Aspects of the present disclosure provide a method and apparatus for video encoding / decoding. In some examples, the apparatus for video decoding includes a receiving circuit and a processing circuit. For example, the processing circuit determines a parameter based on prediction information decoded from a coded video bitstream. The parameter is within a range that depends on a flag indicating an enabled / disabled state of sub-block-based temporal motion vector prediction. Then, the processing circuit calculates a maximum number of candidates in a sub-block-based merge candidate list based on the parameter, and reconstructs samples of the current block based on a candidate selection from the configured sub-block-based merge candidate list of the current block in response to the current block in a sub-block-based prediction mode. The configured sub-block-based merge candidate list of the current block is constrained by the maximum number of candidates in the sub-block-based merge candidate list.

[0011] In some examples, the processing circuit determines the maximum number of candidates in the sub-block based merging candidate list by subtracting the parameter from a default number. In one example, the default number is 5.

[0012] In some embodiments, the upper limit of the range depends on a flag indicating the enable / disable status of sub-block-based temporal motion vector prediction.

[0013] In one example, the processing circuit receives the parameters signaled in the coded video bitstream, and in another example, in response to the parameters not being signaled in the coded video bitstream, the processing circuit estimates the parameters based on a default number and a flag indicating an enabled / disabled status of sub-block-based temporal motion vector prediction.

[0014] In some examples, the flag indicates an enable / disable status of sub-block-based temporal motion vector prediction at a sequence parameter set (SPS) level.

[0015] In some embodiments, the parameter is within a range that depends on a first flag indicating an enable / disable status of sub-block-based temporal motion vector prediction at a sequence parameter set (SPS) level and a second flag indicating an enable / disable status of temporal motion vector prediction at a picture header (PH) level. In some examples, the processing circuit can estimate the parameter based on a default number, the first flag indicating an enable / disable status of sub-block-based temporal motion vector prediction at an SPS level and the second flag indicating an enable / disable status of temporal motion vector prediction at a PH level in response to the parameter not being signaled in the coded video bitstream.

[0016] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for video decoding. [Brief description of the drawings]

[0017] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

[0018] [Figure 1] FIG. 2 is a schematic diagram of a current block and its surrounding spatial merging candidates in one example.

[0019] [Diagram 2] FIG. 2 is a schematic diagram of a simplified block diagram of a communication system (200) according to one embodiment.

[0020] [Diagram 3] FIG. 3 is a schematic diagram of a simplified block diagram of a communication system (300) according to one embodiment.

[0021] [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment.

[0022] [Diagram 5] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment.

[0023] [Figure 6] 4 shows a block diagram of an encoder according to another embodiment;

[0024] [Figure 7] 4 shows a block diagram of a decoder according to another embodiment;

[0025] [Figure 8A] We present affine motion models.

[0026] [Figure 8B] 1 illustrates an affine motion model.

[0027] [Figure 9] 4 shows an example of an affine motion vector field per sub-block.

[0028] [Figure 10] 13 shows an example for determining inherited control point motion vectors in affine merge mode.

[0029] [Figure 11] 1 illustrates an example of spatial and temporal neighbors according to some embodiments of the present disclosure.

[0030] [Figure 12] 1 shows a diagram illustrating motion vector usage for affine motion data inheritance in some examples.

[0031] [Figure 13] FIG. 1 shows a schematic diagram of an example of prediction refinement using optical flow methods, according to one embodiment of the present disclosure.

[0032] [Figure 14] 1 illustrates an example of a sub-block-based temporal motion vector prediction (SbTMVP) process according to some embodiments of the present disclosure.

[0033] [Figure 15] 1 illustrates an example of a sub-block-based temporal motion vector prediction (SbTMVP) process according to some embodiments of the present disclosure.

[0034] [Figure 16] 1 shows an example of a syntax table of a sequence parameter set (SPS) in some examples.

[0035] [Figure 17] 1 shows an example syntax table of an image header structure in some examples.

[0036] [Figure 18] 1 shows a flowchart outlining an example process according to some embodiments of the present disclosure.

[0037] [Figure 19] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] FIG. 2 illustrates a simplified block diagram of a communication system (200) according to an embodiment of the present disclosure. The communication system (200) includes a plurality of terminal devices that can communicate with each other, for example, via a network (250). For example, the communication system (200) includes a pair of first terminal devices (210) and (220) interconnected via the network (250). In the example of FIG. 2, the pair of first terminal devices (210) and (220) performs a one-way transmission of data. For example, the terminal device (210) may code video data (e.g., a stream of video images captured by the terminal device (210)) for transmission to another terminal device (220) via the network (250). The coded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (220) may receive the coded video data from the network (250), decode the coded video data to reconstruct the video image, and display the video image according to the reconstructed video data. Unidirectional data transmission is common, such as in media presentation applications.

[0039] In another example, the communication system (200) includes a second pair of terminal devices (230) and (240) for bidirectional transmission of coded video data, such as may occur during a video conference. For the bidirectional transmission of data, in one example, each of the terminal devices (230) and (240) may code video data (e.g., a stream of video images captured by the terminal device) for transmission to the other of the terminal devices (230) and (240) over the network (250). Each of the terminal devices (230) and (240) may also receive coded video data transmitted by the other of the terminal devices (230) and (240), decode the coded video data to reconstruct the video images, and display the video images on an accessible display device in accordance with the reconstructed video data.

[0040] In the example of FIG. 2, the terminal devices (210), (220), (230), and (240) may be depicted as a server, a personal computer, and a smartphone, although the principles of the present disclosure are not so limited. The embodiments of the present disclosure find application in laptop computers, tablet computers, media players, and / or dedicated video conferencing devices. The network (250) represents any number of networks that convey coded video data between the terminal devices (210), (220), (230), and (240), including, for example, wired and / or wireless communication networks. The communication network (250) may exchange data in circuit-switched and / or packet-switched channels. Exemplary networks include a telecommunications network, a local area network, a wide area network, and / or the Internet. For purposes of this description, the architecture and topology of the network (250) is not important to the operation of the present disclosure, unless otherwise described below.

[0041] 3 shows an arrangement of a video encoder and a video decoder in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter can be similarly applicable to other video-enabled applications including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0042] The streaming system may include a video source (301) that generates a stream of uncompressed video images (302), a capture subsystem (313) that may include, for example, a digital camera. In one example, the stream of video images (302) includes samples captured by a digital camera. The stream of video images (302), depicted as thick lines to highlight a high data volume when compared to the encoded video data (304) (or coded video bitstream), may be processed by an electronic device (320) that includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware / software, or a combination thereof, to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (304) (or encoded video bitstream (304)), depicted as thin lines to highlight a lower data volume when compared to the stream of video images (302), may be stored in a streaming server (305) for future use. One or more streaming client subsystems, such as the client subsystems (306) and (308) of FIG. 3, can access the streaming server (305) to retrieve copies (307) and (309) of the encoded video data (304). The client subsystem (306) can include a video decoder (310), for example within an electronic device (330). The video decoder (310) decodes an input copy (307) of the encoded video data and generates an output stream (311) of video images that can be rendered on a display (312) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (304), (307), and (309) (e.g., a video bitstream) can be encoded according to a particular video coding / compression standard. Examples of these standards include ITU-T Recommendation H.265.For example, a video coding standard under development is informally known as Versatile Video Coding (VVC), and the disclosed subject matter may be used in the context of VVC.

[0043] It should be noted that the electronic devices (320) and (330) can include other components (not shown). For example, the electronic device (320) can include a video decoder (not shown), and the electronic device (330) can also include a video encoder (not shown).

[0044] 4 shows a block diagram of a video decoder (410) according to one embodiment of the present disclosure. The video decoder (410) can be included in an electronic device (430). The electronic device (430) can include a receiver (431) (e.g., a receiving circuit). The video decoder (410) can be used in place of the video decoder (310) of the example of FIG. 3.

[0045] The receiver (431) may receive one or more coded video sequences to be decoded by the video decoder (410); in the same or another embodiment, one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel (401), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (431) may receive the encoded video data together with other data, such as coded audio data and / or auxiliary data streams, that may be forwarded to a respective using entity (not shown). The receiver (431) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420) (hereinafter "parser (420)"). In certain applications, the buffer memory (415) is part of the video decoder (410). In others, it may be external to the video decoder (410) (not shown). In yet others, there may be a buffer memory (not shown) external to the video decoder (410), e.g., to combat network jitter, and yet another buffer memory (415) internal to the video decoder (410), e.g., to handle playback timing. If the receiver (431) is receiving data from a store-and-forward device with sufficient bandwidth and controllability, or from an equally synchronous network, the buffer memory (415) may not be needed or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (415) may be required and may be relatively large, advantageously adaptively sized, and at least partially implemented in an operating system or similar element (not shown) external to the video decoder (410).

[0046] The video decoder (410) may include a parser (420) for reconstructing symbols (421) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (410) and potential information to control a rendering device such as a rendering device (412) (e.g., a display screen) that is not an integral part of the electronic device (430) but may be coupled to the electronic device (430) as shown in FIG. 4. The control information for the rendering device(s) may be in the form of Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (420) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (420) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroups may include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, Coding Units (CUs), blocks, Transform Units (TUs), Prediction Units (PUs), etc. The parser (420) may also extract quantization parameter values, motion vectors, etc. from the coded video sequence information, such as transform coefficients.

[0047] The parser (420) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (415) to generate symbols (421).

[0048] The reconstruction of symbols (421) can involve a number of different units, depending on the type of video image or portion thereof being coded (e.g., inter and intra images, inter and intra blocks), and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (420). The flow of such subgroup control information between the parser (420) and the following units is not shown for clarity.

[0049] In addition to the functional blocks already mentioned, the video decoder (410) may be conceptually divided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0050] The first unit is a scalar / inverse transform unit (451), which receives the quantized transform coefficients as symbol(s) (421) from the parser (420), along with control information including the transform to use, block size, quantization coefficients, quantization scaling matrix, etc. The scalar / inverse transform unit (451) can output blocks containing sample values ​​that can be input to an aggregator (455).

[0051] In some cases, the output samples of the scalar / inverse transform (451) may be associated with intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed image, but may use prediction information from a previously reconstructed portion of the current image. Such prediction information may be provided by an intra-image prediction unit (452). In some cases, the intra-image prediction unit 452 generates blocks of the same size and shape of the block being reconstructed using already reconstructed surrounding information retrieved from a current image buffer 458. The current image buffer (458) may, for example, buffer a partially reconstructed current image and / or a fully reconstructed current image. The aggregator (455) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (452) to the output sample information as provided by the scalar / inverse transform unit (451).

[0052] In other cases, the output samples of the scalar / inverse transform unit (451) may relate to an inter-coded, potentially motion compensated block. In such cases, the motion compensated prediction unit (453) may access the reference picture memory (457) to retrieve samples used for prediction. After motion compensating the retrieved samples according to the symbols (421) associated with the block, these samples may be added by the aggregator (455) to the output of the scalar / inverse transform unit (451) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (457) from which the motion compensated prediction unit (453) retrieves the prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (453), e.g., in the form of symbols (421) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory (457) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0053] The output samples of the aggregator (455) can be subject to various loop filtering techniques in the loop filter unit (456). Video compression techniques are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (456) as symbols (421) from the parser (420), but can include in-loop filter techniques that can be responsive to meta-information obtained during decoding of previous (in decoding order) portions of the coded image or coded video sequence, as well as to previously reconstructed and loop filtered sample values.

[0054] The output of the loop filter unit (456) can be a sample stream that can be output to a rendering device (412) or stored in a reference image memory (457) for use in future inter-image prediction.

[0055] Once a particular coded picture has been fully reconstructed, it can be used as a reference picture for future predictions. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (420)), the current picture buffer (458) can become part of the reference picture memory (457) and a new current picture buffer can be reallocated before starting the reconstruction of the next coded picture.

[0056] The video decoder (410) may perform decoding operations according to a given video compression technique of a standard, such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. In particular, a profile may select a particular tool from all tools available in the video compression technique or standard as the only tool available for use under that profile. Also, a requirement for compliance may be that the complexity of the coded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited, in some cases, through a hypothetical reference decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0057] In one embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence(s). The additional data may be used by the video decoder (410) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0058] 5 shows a block diagram of a video encoder (503) according to one embodiment of the present disclosure. The video encoder (503) is included in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) can be used in place of the example video encoder (303) of FIG. 3.

[0059] The video encoder (503) may receive video samples from a video source (501) (which is not part of the electronic device (520) in the example of FIG. 5) that may capture the video image(s) to be coded by the video encoder (503). In another example, the video source (501) is part of the electronic device (520).

[0060] The video source (501) may provide a source video sequence to be coded by the video encoder (503) in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media delivery system, the video source (501) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (501) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual images that convey motion when viewed in sequence. The images themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0061] According to one embodiment, the video encoder (503) may code and compress images of a source video sequence into a coded video sequence (543) in real-time or under any other time constraint required by an application. Enforcing an appropriate coding rate is one function of the controller (550). In some embodiments, the controller (550) controls and is operatively coupled to other functional units, as described below. Coupling is not shown for clarity. Parameters set by the controller (550) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (550) may be configured to have other appropriate functions associated with the video encoder (503) optimized for a particular system design.

[0062] In some embodiments, the video encoder (503) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (530) (e.g., responsible for generating symbols, such as a symbol stream, based on an input image to be coded and a reference image) and a (local) decoder (533) embedded in the video encoder (503). The decoder (533) reconstructs the symbols to generate sample data as the (remote) decoder would generate them (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference image memory (534). Since the decoding of the symbol stream results in bit-exact results that are independent of the decoder location (local or remote), the contents in the reference image memory (534) are also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference image samples that the decoder "sees" when using the prediction during decoding. This basic principle of reference image synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is used in several related technologies as well.

[0063] The operation of the "local" decoder (533) can be the same as a "remote" decoder, such as the video decoder (410), which has already been described above in connection with Figure 4. However, with brief reference also to Figure 4, the entropy decoding portion of the video decoder (410), including the buffer memory (415) and the parser (420), may not be fully implemented in the local decoder (533), since symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (545) and parser (420) can be lossless.

[0064] An observation that can be made at this point is that any decoder techniques, except for parsing / entropy decoding, present in the decoder must also be present in substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter focuses on the decoder operation. A description of the encoder techniques can be omitted, since they are the inverse of the decoder techniques described generically. Only in certain areas are more detailed descriptions necessary and are provided below.

[0065] In operation, in some examples, the source coder (530) may perform motion-compensated predictive coding, which predictively codes an input image with respect to one or more previously coded images from a video sequence designated as “reference images.” In this manner, the coding engine (532) codes differences between pixel blocks of the input image and pixel blocks of reference image(s) that may be selected as predictive reference(s) for the input image.

[0066] The local video decoder (533) may decode the coded video data of the image that may be designated as a reference image based on the symbols generated by the source coder (530). The operation of the coding engine (532) may advantageously be a lossy process. If the coded video data may be decoded in a video decoder (not shown in FIG. 5), the reconstructed video sequence may be a replica of the source video sequence, typically with some errors. The local video decoder (533) replicates the decoding process that may be performed by the video decoder on the reference image and may cause the reconstructed reference image to be stored in a reference image cache (534). In this way, the video encoder (503) may locally store (without transmission errors) a copy of the reconstructed reference image that has a common content as the reconstructed reference image obtained by the far-end video decoder.

[0067] A predictor (535) may perform a prediction search for the coding engine (532). That is, for a new image to be coded, the predictor (535) may search the reference image memory (534) for certain metadata or sample data (as candidate reference pixel blocks), such as reference image motion vectors, block shapes, etc., that may serve as suitable prediction references for the new image. The predictor (535) may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (535), the input image may have prediction references drawn from multiple reference images stored in the reference image memory (534).

[0068] The controller (550) may manage the coding operations of the source coder (530), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0069] The output of all the aforementioned functional units may be subject to entropy coding in an entropy coder (545), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0070] The transmitter (540) may buffer the coded video sequence(s) as produced by the entropy coder (545) to prepare it for transmission over a communication channel (560), which may be a hardware / software link to a storage device that stores the coded video data. The transmitter (540) may merge the coded video data from the video coder (503) with other data to be transmitted, such as coded audio data and / or ancillary data streams (not shown).

[0071] The controller (550) may manage the operation of the video encoder (503). During coding, the controller (550) may assign each coded picture a particular coded picture type, which may affect the coding technique that may be applied to the respective picture. For example, pictures are often assigned as one of the following picture types:

[0072] An intra-picture (I-picture) may be one that can be coded and decoded without using other pictures in a sequence as a source of prediction. Some video codecs allow different types of intra-pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures, as well as their respective uses and characteristics.

[0073] A predicted image (P-image) may be coded and decoded using intra prediction or inter prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0074] Bidirectionally predicted images (B-pictures) may be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted images may use more than two reference images and associated metadata for the reconstruction of a block.

[0075] A source image is usually spatially divided into several sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective image of the block. For example, blocks of an I image may be non-predictively coded or they may be predictively coded with reference to already coded blocks of the same image (spatial or intra prediction). Pixel blocks of a P image may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference image. Blocks of a B image may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference images.

[0076] The video encoder (503) may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder (503) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0077] In one embodiment, the transmitter (540) may transmit additional data along with the coded video. The source coder (530) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0078] A video may be captured as multiple source images (video images) in a time sequence. Intra-image prediction (often abbreviated as intra-prediction) exploits spatial correlation in a given image, while inter-image prediction exploits correlation (temporal or other) between images. In one example, a particular image under encoding / decoding, called the current image, is divided into blocks. If a block of the current image is similar to a reference block of a reference image of a previously coded and still buffered video, the block of the current image can be coded by a vector called a motion vector. A motion vector points to a reference block of a reference image, and may have three dimensions that identify the reference image if multiple reference images are used.

[0079] In some embodiments, bi-prediction techniques can be used in inter-image prediction. According to bi-prediction techniques, two reference images are used, such as a first reference image and a second reference image, both of which precede in decoding order (but may be past and future, respectively, in display order) the current image in a video. A block of the current image can be coded by a first motion vector that points to a first reference block of the first reference image and a second motion vector that points to a second reference block of the second reference image. A block can be predicted by a combination of the first reference block and the second reference block.

[0080] Furthermore, to improve coding efficiency, merge mode techniques can be used for inter-image prediction.

[0081] According to some embodiments of the present disclosure, predictions such as inter-image prediction and intra-image prediction are performed on a block-by-block basis. For example, according to the HEVC standard, an image in a sequence of video images is divided into coding tree units (CTUs) for compression, and the CTUs in an image have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), which are one luma CTB and two chroma CTBs. Each CTU can recursively split a quadtree into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be divided into one CU of 64×64 pixels, four CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. In general, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0082] 6 shows a diagram of a video encoder (603) according to another embodiment of the present disclosure. The video encoder (603) is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video image in a sequence of video images and to encode the processed block into a coded image that is part of a coded video sequence. In one example, the video encoder (603) is used in place of the video encoder (303) of the example of FIG. 3.

[0083] In an HEVC example, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8×8 samples. The video encoder (603) determines whether the processing block is best coded using intra-mode, inter-mode, or bi-prediction mode, for example, using rate-distortion optimization. If the processing block is to be coded in intra-mode, the video encoder (603) may use intra-prediction techniques to encode the processing block into a coded image; if the processing block is to be coded in inter-mode or bi-prediction mode, the video encoder (603) may use inter-prediction techniques or bi-prediction techniques, respectively, to encode the processing block into a coded image. In certain video coding techniques, the merge mode can be an inter-image prediction submode in which motion vectors are derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (603) includes other components, such as a mode decision module (not shown) for determining the mode of the processing block.

[0084] In the example of FIG. 6, the video encoder (603) includes an inter-encoder (630), an intra-encoder (622), a residue calculator (623), a switch (626), a residual encoder (624), a general controller (621), and an entropy encoder (625) coupled together as shown in FIG.

[0085] The inter-encoder (630) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and subsequent images), generate inter-prediction information (e.g., a description of redundant information due to inter-encoding techniques, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a prediction block) based on the inter-prediction information using any suitable technique. In some examples, the reference image is a decoded reference image that is decoded based on the encoded video information.

[0086] The intra encoder (622) is configured to receive samples of a current block (e.g., a processing block), possibly compare the block to previously coded blocks in the same image, generate transformed and quantized coefficients, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). In one example, the intra encoder (622) also calculates an intra prediction result (e.g., a prediction block) based on a reference block in the same image and the intra prediction information.

[0087] The general controller (621) is configured to determine general control data and control other components of the video encoder (603) based on the general control data. In one example, the general controller (621) determines the mode of the block and provides a control signal to the switch (626) based on the mode. For example, if the mode is intra mode, the general controller (621) controls the switch (626) to select an intra mode result to be used by the residual calculator (623) and controls the entropy encoder (625) to select intra prediction information and include the intra prediction information in the bitstream; if the mode is inter mode, the general controller (621) controls the switch (626) to select an inter prediction result to be used by the residual calculator (623) and controls the entropy encoder (625) to select inter prediction information and include the inter prediction information in the bitstream.

[0088] The residual calculator (623) is configured to calculate a difference (residual data) between the received block and a prediction result selected from the intra-encoder (622) or the inter-encoder (630). The residual encoder (624) operates on the residual data and is configured to encode the residual data to generate transform coefficients. In one example, the residual encoder (624) is configured to transform the residual data from a spatial domain to a frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra-encoder (622) and the inter-encoder (630) as appropriate. For example, the inter-encoder (630) can generate decoded blocks based on the decoded residual data and the inter-prediction information, and the intra-encoder (622) can generate decoded blocks based on the decoded residual data and the intra-prediction information. The decoded blocks are suitably processed to generate a decoded image, which can be buffered in a memory circuit (not shown) and used as a reference image in some examples.

[0089] The entropy encoder (625) is configured to format the bitstream to include the encoded blocks. The entropy encoder (625) is configured to include various information according to an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (625) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. It should be noted that when coding blocks in a merged sub-mode of either an inter mode or a bi-prediction mode in accordance with the disclosed subject matter, no residual information is present.

[0090] 7 shows a diagram of a video decoder (710) according to another embodiment of the present disclosure. The video decoder (710) is configured to receive a coded image that is part of a coded video sequence and to decode the coded image to generate a reconstructed image. In one example, the video decoder (710) is used in place of the video decoder (310) of the example of FIG. 3.

[0091] In the example of FIG. 7, the video decoder (710) includes an entropy decoder (771), an inter decoder (780), a residual decoder (773), a reconstruction module (774), and an intra decoder (772) coupled together as shown in FIG.

[0092] The entropy decoder (771) may be configured to reconstruct from a coded image certain symbols that represent syntax elements of which the coded image is composed. Such symbols may include, for example, prediction information (e.g., intra- or inter-prediction information, etc.) that may identify the mode in which the block is coded (e.g., intra- or inter-prediction mode, merged submode, or the latter two in another submode), certain samples or metadata used for prediction by the intra-decoder (772) or inter-decoder (780), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In one example, if the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter-decoder (780); if the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (772). The residual information may undergo inverse quantization and is provided to the residual decoder (773).

[0093] The inter decoder (780) is configured to receive the inter prediction information and to generate inter prediction results based on the inter prediction information.

[0094] The intra decoder (772) is configured to receive intra prediction information and to generate a prediction result based on the intra prediction information.

[0095] The residual decoder (773) is configured to perform inverse quantization to extract de-quantized transform coefficients and process the de-quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (771) (data path not shown, as it is only low volume control information).

[0096] The reconstruction module (774) is configured to combine, in the spatial domain, the residual as output by the residual decoder (773) and the prediction result (possibly as output by an inter- or intra-prediction module) to form a reconstructed block, which may be part of a reconstructed image, which may be part of a reconstructed video. It should be noted that other suitable operations, such as deblocking operations, may be performed to improve visual quality.

[0097] It should be noted that the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using any suitable technology. In one embodiment, the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using one or more processors executing software instructions.

[0098] Aspects of this disclosure provide techniques in the field of inter prediction in advanced video codecs that can be used to set a number of candidates in a candidate list, which can be referred to as a sub-block merging candidate list.

[0099] In various embodiments, for an inter-predicted CU, motion parameters including a motion vector, a reference image index, a reference image list usage index, and / or other additional information may be used for inter-predicted sample generation. Inter-prediction may include uni-prediction, bi-prediction, and / or the like. In uni-prediction, a reference image list (e.g., a first reference image list or list 0 (L0) or a second reference image list or list 1 (L1)) may be used. In bi-prediction, both L0 and L1 may be used. The reference image list usage index may indicate that the reference image list(s) include L0, L1, or L0 and L1.

[0100] The motion parameters can be signaled in an explicit or implicit manner. If a CU is coded in skip mode, the CU can be associated with one PU, and cannot contain significant residual coefficients (e.g., the residual coefficients are zero), cannot contain coded motion vector differences (MVD), or cannot contain reference picture indexes.

[0101] The merge mode can be used so that motion parameters for the current CU can be obtained from neighboring CUs, including spatial and temporal merge candidates, and optionally other merge candidates. The merge mode can be applied to inter-predicted CUs and can be used for skip mode. Alternatively, the motion parameters can be explicitly transmitted or signaled. For example, the motion vectors, corresponding reference image indexes for each reference image list, a reference image list usage flag, and other information can be explicitly signaled for each CU.

[0102] In some examples (e.g., VVC), one or more of the following inter-predictive coding tools are used: (1) enhanced merge prediction, (2) merge mode with motion vector difference (MMVD), (3) symmetric MVD (SMVD) signaling, (4) affine motion compensation prediction, (5) sub-block based temporal motion vector prediction (SbTMVP), (6) adaptive motion vector resolution (AMVR), (7) motion field storage: 1 / 16 luma sample MV storage and 8x8 motion field compression, (8) bi-prediction with CU level weights, (9) bidirectional optical flow (BDOF), (10) decoder side motion vector refinement (DMVR), (11) geometric partitioning mode (GPM), and (12) combined inter and intra prediction (CIIP).

[0103] According to one aspect of the present disclosure, some inter-predictive coding tools may operate based on a sub-block-based merge candidate list. In one example, affine motion compensation prediction may be performed in an affine merge mode (also referred to as a sub-block-based merge mode in some examples). In the affine merge mode, prediction may be performed based on an affine merge candidate list, referred to as a sub-block-based merge candidate list in some examples. In another example, sub-block-based temporal motion vector prediction (SbTMVP) may also operate based on a sub-block-based merge candidate list.

[0104] In some examples (e.g., HEVC), for affine motion compensation prediction, only translational motion model is applied to motion compensation prediction (MCP). The real world has many kinds of motion, such as zoom in / out, rotation, perspective motion, and other irregular motion. In some examples (e.g., VVC), block-based affine transformation motion compensation prediction is applied.

[0105] 8A-8B show affine motion models: Fig. 8A shows an affine motion field of a block described by motion information of two control points CP0 and CP1 (four-parameter affine model), and Fig. 8B shows an affine motion field of a block described by three control points CP0, CP1, and CP2 (six-parameter affine model).

[0106] In some embodiments, a four-parameter affine motion model is used, where the motion vector (mv x ,mv y ) can be derived as (Equation 1), and a six-parameter affine motion model, the motion vector at a sample position (x,y) within a block can be derived as (Equation 2):

number

[0107] To simplify the motion compensated prediction, block-based affine transformation prediction is applied.

[0108] FIG. 9 shows an example of an affine MV field per subblock. In one example, a current CU 910 (e.g., 16×16 luma samples) is divided into 4×4 luma subblocks (each subblock can be a 4×4 luma sample). To derive a motion vector for each 4×4 luma subblock, a motion vector for a center sample of each subblock is calculated according to the above formulas (Equation 1) and (Equation 2), as shown in FIG. 9. The motion vectors can be rounded, for example, to 1 / 16 fraction accuracy. Then, a motion compensation interpolation filter is applied to generate a prediction for each subblock with the derived motion vector. In some examples, the subblock size of the chroma components can also be set to 4×4, so that a 4×4 chroma subblock includes four corresponding 4×4 luma subblocks. The MV of a 4×4 chroma subblock is calculated as the average of the MVs of the four corresponding 4×4 luma subblocks in one example.

[0109] It should be noted that a sub-block may be defined to have any other suitable number of luma samples. It should also be noted that in some examples, a sub-block is referred to as a sub-CU.

[0110] For translational motion inter prediction, two affine motion inter prediction modes can be used, called Affine Merge (AF_MERGE) mode and Affine Advanced MVP (Affine AMVP) mode.

[0111] For affine merge prediction, in one example, AF_MERGE mode can be applied to CUs with both width and height equal to or greater than 8. In AF_MERGE mode, a control point motion vector (CPMV) of a current CU is generated based on motion information of spatially neighboring CUs. In one example, an affine merge candidate list (also called a subblock-based merge candidate list) can include up to five control point motion vector predictor (CPMVP) candidates, and an index is signaled to indicate the one to be used for the current CU. In one example, three types of CPMVP candidates are used to form the affine merge candidate list. The first type of CPMVP candidate is an inherited affine merge candidate extrapolated from the CPMV of a neighboring CU. The second type of CPMVP candidate is a constructed affine merge candidate CPMVP derived using the translation MV of a neighboring CU. The third type of CPMVP candidate uses zero MV.

[0112] In some examples, such as VVC, up to two inherited affine candidates can be used. In one example, two inherited affine candidates are derived from the affine motion models of neighboring blocks, one from the left neighboring CU (called left predictor) and one from the top neighboring CU (called top predictor). Using the neighboring blocks shown in FIG. 1 as an example, for the left predictor, the scan order is A0->A1, and for the top predictor, the scan order is B0->B1->B2. In one example, only the first inherited candidate available from both sides is selected. In some examples, no pruning check is performed between the two inherited candidates. If a neighboring affine CU is identified, the control point motion vector of the neighboring affine CU is used to derive a CPMVP candidate in the affine merge candidate list of the current CU.

[0113] Figure 10 shows an example for determining inherited control point motion vectors in affine merge mode. As shown in Figure 10, if the neighboring lower-left sub-block A is coded in affine mode, the motion vectors mv2, mv3 and mv4 of the upper-left corner, upper-right corner and lower-left corner of the CU containing sub-block A can be obtained. If sub-block A is coded in a four-parameter affine model, the two CPMVs of the current CU are calculated according to mv2 and mv3. If sub-block A is coded in a six-parameter affine model, the three CPMVs of the current CU are calculated according to mv2, mv3 and mv4.

[0114] In some examples, the constructed affine candidates are constructed by combining neighboring translational motion information for each control point, which can be derived from specified spatial and temporal neighbors.

[0115] 11 illustrates an example of spatially adjacent (e.g., sub-blocks A0-A2 and B0-B3) and temporally adjacent (e.g., T) blocks according to some embodiments of the present disclosure. In one example, CPMV k (k=1,2,3,4) represents the kth control point. For CPMV1, B2->B3->A2 blocks are checked (-> is used for checking order) and the MV of the first available block is used as CPMV1. For CPMV2, B1->B0 blocks are checked and the MV of the first available block is used as CPMV2. For CPMV3, A1->A0 blocks are checked and the MV of the first available block is used as CPMV3. For TMVP, T is checked and if the MV of block T is available it is used as CPMV4.

[0116] After obtaining the MVs of the four control points CPMV1-CPMV4, affine merge candidates are constructed based on the motion information. The following combinations of control point MVs are used to construct the candidate affine merges in the following order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}.

[0117] A combination of three CPMVs can constitute a six-parameter affine merge candidate, and a combination of two CPMVs can constitute a four-parameter affine merge candidate. In one example, to avoid the motion scaling process, related combinations of control point MVs can be discarded if the reference indices of the control points are different.

[0118] In one example, after the inherited and constructed affine merge candidates are checked, if the candidate list is not already full, a MV of zero is inserted at the end of the list.

[0119] For affine AMVP prediction, affine AMVP mode can be applied to CUs with both width and height of 16 or more. In some examples, a CU-level affine flag is signaled in the bitstream (e.g., coded video bitstream) to indicate whether affine AMVP mode is used in the CU, and then another flag is signaled to indicate whether 4-parameter affine or 6-parameter affine is used. In affine AMVP mode, the difference between the CPMV of the current CU and their predictor CPMVP can be signaled in the bitstream. The size of the affine AMVP candidate list is 2, and the affine AMVP candidate list is generated using the following four types of CPMV candidates in the following order: (1) inherited affine AMVP candidates extrapolated from the CPMV of the neighboring CU, (2) constructed affine AMVP candidates derived using the translation MV of the neighboring CU, (3) translation MV from the neighboring CU, and (4) zero MV.

[0120] In some examples, the testing order of the inherited affine AMVP candidates is the same as the testing order of the inherited affine merge candidates. In one example, the only difference between affine merge prediction and affine AMVP prediction is that only affine CUs with the same reference image as the current block are considered for AMVP candidates. In one example, no pruning process is applied when inserting the inherited affine motion predictors into the candidate list.

[0121] In some examples, the constructed AMVP candidates can be derived from the specific spatial neighbors shown in FIG. 11. In one example, the same check order is used as was done in the candidate construction for affine merge prediction. In addition, the reference image indexes of the neighboring blocks are also checked. The first block in the check order that is inter-coded and has the same reference image as the current CU is used. If the current CU is coded in a 4-parameter affine mode and the motion vectors of the two control points (e.g., {CPMV1, CPMV2}) are both available, the motion vectors of the two control points are added as one candidate to the affine AMVP list. If the current CU is coded in a 6-parameter affine mode and all three motion vectors of the control point CPMV (e.g., {CPMV1, CPMV2, CPMV3}) are available, they are added as one candidate to the affine AMVP list. Otherwise, the constructed AMVP candidates are set to disabled.

[0122] After the inherited and constructed AMVP candidates are checked, if the number of affine AMVP list candidates is still less than 2, CPMV1, CPMV2, and CPMV3 are added in order, if available, as translation MVs to predict all control point MVs of the current CU. Finally, if the affine AMVP list is not yet full, zero MVs are used to fill the affine AMVP list.

[0123] According to some aspects of the present disclosure, the motion information may be stored in an appropriate buffer, such as a local buffer, an image line buffer, or the like. The local buffer is used to store motion information at the CTU level, such as the motion vectors of 4×4 blocks in the CTU. For example, if the CUs in the CTU are decoded based on inter prediction, the motion vectors of the 4×4 blocks of each of the CUs may be stored in the local buffer and used to decode the later CU. The image line buffer is used to store motion information of the CTUs above the current CTU, such as the motion vectors of the 4×4 blocks at the bottom of the CTU above. The CTU above the current CTU may be referred to as the CTU line above.

[0124] In some examples (e.g., VVC), the CPMV of the affine CU is stored separately from the motion vector of the 4x4 block. In one example, the local buffer includes a first portion for storing the motion vector of the 4x4 block in the CTU and a second portion for storing the CPMV of the affine CU in the CTU. The CPMV stored in the second portion of the local buffer can be used to generate the inherited CPMVP in the affine merge mode and the affine AMVP mode for the recently coded CU. The sub-block MV derived from the CPMV is used for motion compensation, merging of translational MVs / MV derivation of the AMVP list and deblocking.

[0125] In some implementations, the image line buffer does not store additional CPMVs for affine CUs in the CTU line above. In some examples, affine motion data inheritance from a CU from a CTU above is treated differently than inheritance from a normal neighboring CU in the same CTU line. If a candidate CU for affine motion data inheritance is in a CTU line above, the bottom-left and bottom-right sub-block MVs in the image line buffer are used for affine MVP derivation instead of CPMVs. Thus, in some examples, CPMVs are stored only in the local buffer, not the image line buffer. In an example where a candidate CU is 6-parameter affine coded, the affine model can be reduced to a 4-parameter model.

[0126] FIG. 12 shows a diagram illustrating the use of motion vectors for affine motion data inheritance in some examples. In FIG. 12, each small square represents a 4×4 subblock, and the motion vector of the subblock can be the motion vector at the center of the subblock. Furthermore, the current CU is located at the vertex position of the current CTU. As shown in FIG. 12, in the neighboring CUs of the current CU, CU-E and CU-D are affine coded. CU-D is in the same CTU line as the current CU, and CU-E is in the CTU line above the current CU. The CPMV of CU-D can be stored in a local buffer. For example, for a four-parameter affine model, mv D0 and mv D1 is stored in the local buffer, and the CPMVs of the current CU (e.g., mv0 and mv1) are stored in the local buffer. D0 and mv D1 , as well as mv D0 and mv D1 The corresponding positions of the control points for

[0127] In one example, the image line buffer stores the motion vectors of the sub-blocks at the bottom of the top CTU line. E0 MV E1 The CPMV of the CU-E, as indicated by mvLE0 and mv LE1 The motion vectors of the lower left sub-block and the lower right sub-block of CU-E, as shown in S (For example, mv0 and mv1) are mv LE0 and mv LE1 , as well as the corresponding center positions of the two sub-blocks.

[0128] In some embodiments, prediction refinement by optical flow (PROF) (also referred to as the PROF method) may be implemented to improve sub-block-based affine motion compensation to achieve finer granularity of motion compensation without increasing memory access bandwidth for motion compensation. In one embodiment (e.g., VVC), after sub-block-based affine motion compensation is performed, differences (or refinement values, refinement, prediction refinement) derived based on the optical flow equation may be added to prediction samples (e.g., luma predicted samples or luma predicted samples) to obtain refined prediction samples.

[0129] FIG. 13 shows a schematic diagram of an example of the PROF method according to an embodiment of the present disclosure. A current block (1310) can be divided into four sub-blocks (1312, 1314, 1316, and 1318). Each of the sub-blocks (1312, 1314, 1316, and 1318) can have a size of 4×4 pixels or samples. A sub-block MV (1320) for the sub-block (1312) can be derived according to the CPMV of the current block 1310, for example, using the center position of the sub-block (1312) and an affine motion model (e.g., a four-parameter affine motion model, a six-parameter affine motion model). The sub-block MV (1320) can indicate a reference sub-block (1332) in a reference image. An initial sub-block prediction sample can be determined according to the reference sub-block (1332).

[0130] In some examples, the translational motion from the reference subblock (1332) to the subblock (1312), as described by the subblock MV (1320), may not predict the subblock (1312) with high accuracy. In addition to the translational motion described by the subblock MV (1320), the subblock (1312) may also experience non-translational motion (e.g., rotation as seen in FIG. 13). With reference to FIG. 13, a subblock (1350) in the reference image with shaded samples (e.g., sample (1332a)) corresponds to and can be used to reconstruct samples in the subblock (1312). The shaded sample (1332a) corresponds to the pixel MV (1340), samples (1312a) in the sub-block (1312) can be reconstructed with high accuracy. Thus, in some examples, when non-translational motion occurs, appropriate prediction refinement methods can be applied to the affine motion model, as described below, to improve the accuracy of the prediction.

[0131] In one example, the PROF method is implemented using the following four steps: In step (1), subblock-based affine motion compensation can be performed to generate a prediction, such as an initial subblock prediction I(i,j), for a current subblock (e.g., subblock (1312)), where i and j are coordinates corresponding to a sample at position (i,j) (also referred to as sample position or sample location) in the current subblock (1312).

[0132] In step (2), a gradient calculation can be performed, where the spatial gradient g of the initial sub-block prediction I(i,j) at each sample position (i,j) is x (i, j) and g y (i,j) can be calculated, for example, using a 3-tap filter [-1,0,1] according to Equations 3 and 4 below: g x(i,j)=I(i+1,j)-I(i-1,j) (Equation 3) g y (i,j)=I(i,j+1)-I(i,j-1) (Equation 4) The subblock prediction can be extended by one pixel on each side for gradient calculation. In some embodiments, to reduce memory bandwidth and complexity, the pixels on the extended boundary can be copied from the nearest integer pixel location in a reference image (e.g., the reference image that includes the subblock (1332)). Thus, additional interpolation for the padding region can be avoided.

[0133] In step (3), the prediction refinement ΔI(i,j) can be calculated according to Equation 5 (eg, the optical flow equation) as follows: ΔI(i,j)=g x (i,j)×Δmv x (i,j)+g y (i,j)×Δmv y (i,j) (Equation 5) Here, Δmv(i,j) (e.g., Δmv(1342)) is the pixel MV or sample MV mv(i,j) (e.g., pixel MV (1340)) for sample position (i,j) and the sub-block MV Mv(i,j) of the sub-block in which sample position (i,j) is located (e.g., sub-block (1312)). SB (e.g., sub-block MV (1320)). Δmv(i,j) may also be referred to as the MV refinement (MVR) for the sample at sample position (i,j) or sample (i,j). Δmv(i,j) may be determined using Equation 6 as follows: Δmv(i,j)=mv(i,j)-mv SB (Formula 6) Δmv x (i,j) and Δmv y (i, j) are the x component (horizontal component) and y component (vertical component) of the difference MV Δmv(i, j), respectively.

[0134] Because the pixel position and affine model parameters relative to the subblock center position do not change from one subblock to another, Δmv(i,j) can be calculated for a first subblock (e.g., subblock (1312)) and reused for other subblocks (e.g., subblocks (1314), (1316), and (1318)) within the same current block (1310). In some examples, x and y represent the horizontal and vertical shifts of sample position (i,j) relative to the center position of the subblock (1312), and Δmv(i,j) (e.g., Δmv x (i,j) and Δmv y (including (i,j)) can be derived by Equation 7 as follows:

number

[0135] For example, for a four-parameter affine motion model, parameters a to d are described by (Equation 1). For a six-parameter affine motion model, parameters a to d are described by (Equation 2) as described above.

[0136] In step (4), a prediction refinement ΔI(i,j) (e.g., luma prediction refinement) may be added to the initial sub-block prediction I(i,j) to generate another prediction, such as refined prediction I′(i,j), which may be generated for sample (i,j) using Equation 8 as follows: I'(i,j)=I(i,j)+ΔI(i,j) (Equation 8).

[0137] In some cases, PROF is not applied for affine coded CUs. In one example, all control points MV are the same, which indicates that the CU only has translational motion, and PROF is not applied. In another example, the affine motion parameters are greater than a specified limit, and then PROF is applied. In the second case, sub-block based affine motion compensation degrades to CU based motion compensation to avoid large memory access bandwidth requirements.

[0138] In some embodiments, a fast encoding method can be applied to reduce the coding complexity of affine motion estimation with PROF. In the fast encoding method, PROF is not applied in the affine motion estimation stage in the following two situations. In the first situation, if the current CU is not a root block and its parent block does not select the affine mode as the best mode, PROF is not applied because it is unlikely to select the affine mode as the best mode for the current CU. In the second situation, if the magnitudes of the four affine parameters (a-d) are all smaller than a predefined threshold and the current image is not a low-lag image, PROF is not applied because the improvement brought by PROF in this situation is small. In this way, affine motion estimation with PROF can be accelerated.

[0139] In some examples (e.g., VVC), sub-block-based temporal motion vector prediction (SbTMVP) can be used. Similar to temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field of the collocated image to improve the motion vector prediction and merge mode for the CU of the current image. In some examples, the same collocated image used in TMVP is used in SbTMVP. SbTMVP differs from TMVP in two ways. In the first aspect, TMVP predicts motion at the CU level, while SbTMVP predicts motion at the sub-CU level. In the second aspect, TMVP fetches temporal motion vectors from a collocated block in the collocated image (the collocated block is the bottom-right or center block relative to the current CU), and SbTMVP applies a motion shift before fetching the temporal motion information from the collocated image. The motion shift is obtained from a motion vector from one of the spatially neighboring blocks of the current CU.

[0140] 14-15 show an example of the SbTMVP process according to some embodiments of the present disclosure. SbTMVP predicts motion vectors of sub-CUs in a current CU in two steps. In the first step, spatially neighboring A1 shown in FIG. 14 is considered. If spatially neighboring A1 has a motion vector that uses the collocated image as its reference image, the motion vector is selected to be the motion shift to be applied. If no such motion is identified, the motion shift is set to (0,0).

[0141] In the second step, the motion shift identified in the first step is applied (i.e., added to the coordinates of the current block) to obtain sub-CU level motion information (motion vector and reference index) from the collocated image, as shown in FIG. 15. In the example of FIG. 15, the motion vector of A1 is set as the motion shift (1510). Then, for each sub-CU, the motion information of the corresponding block (the smallest motion grid covering the center sample) in the collocated image is used to derive the motion information for the sub-CU. After the motion information of the collocated sub-CU is identified, it is converted to the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC. For example, temporal motion scaling is applied to align the reference image of the temporal motion vector to the reference image of the current CU.

[0142] In some examples, such as in VVC, a subblock-based merge candidate list is used to signal the subblock-based merge mode. The subblock-based merge candidate list can include both SbTMVP candidates and affine merge candidates, and in some examples is referred to as a combined subblock-based merge candidate list. The SbTMVP mode is enabled / disabled by a flag, such as a sequence parameter set (SPS) flag. Enabling the SbTMVP mode, in one example, adds an SbTMVP predictor as the first entry in the combined subblock-based merge candidate list, followed by the affine merge candidates. In some examples (e.g., VVC), the maximum allowed size of the combined subblock-based merge candidate list is 5. Note that the maximum allowed size of the combined subblock-based merge candidate list can be any other suitable number.

[0143] In one example, the sub-CU size used in SbTMVP is fixed to 8x8, and as is done in the affine merge mode, the SbTMVP mode is only applicable to CUs whose width and height are both 8 or greater.

[0144] In some embodiments, the encoding logic for the additional SbTMVP merging candidate is the same as for other merging candidates, e.g., for each CU of a P or B slice, an additional rate-distortion check is performed to determine whether to use the SbTMVP candidate.

[0145] According to some aspects of the disclosure, a maximum number of candidates in the combined sub-block based merging candidate list can be signaled.

[0146] 16 shows an example syntax table 1600 of a sequence parameter set (SPS) in some examples. An SPS contains information that can be applied to a series of consecutive coded video pictures (also called a coded video sequence).

[0147] In the example syntax table (1600), the flag sps_temporal_mvp_enabled_flag is signaled as shown at (1610). The flag sps_temporal_mvp_enabled_flag equal to 1 specifies that temporal motion vector predictors may be used in the coded video; the flag sps_temporal_mvp_enabled_flag equal to 0 specifies that temporal motion vector predictors are not used in the coded video. In some examples, the coded video may be referred to as a coded layer video sequence (CLVS), which is a group of pictures belonging to the same layer starting with a random access point followed by pictures that may be dependent on each other and the random access point picture.

[0148] In the example of the related syntax table (1600), when the flag sps_temporal_mvp_enabled_flag is equal to 1, two flags sps_sbtmvp_enabled_flag and sps_affine_enabled_flag are signaled, as shown at (1620) and (1630). The flag sps_sbtmvp_enabled_flag equal to 1 specifies that the sub-block based temporal motion vector predictor may be used for decoding of pictures in slices having a slice type not equal to I (intra-coded) in the coded video. The flag sps_sbtmvp_enabled_flag equal to 0 specifies that the sub-block based temporal motion vector predictor is not used in the coded video. In the example, when the flag sps_sbtmvp_enabled_flag is not signaled, it can be inferred that the flag sps_sbtmvp_enabled_flag is equal to 0.

[0149] The flag sps_affine_enabled_flag specifies whether affine model-based motion compensation can be used for inter prediction. When the flag sps_affine_enabled_flag is equal to 0, in some examples, the syntax is constrained such that affine model-based motion compensation is not used in the coded video. Otherwise (sps_affine_enabled_flag is equal to 1), affine-based motion compensation can be used in the coded video.

[0150] In the example syntax table (1600), if the flag sps_affine_enabled_flag is equal to 1, then parameters such as five_minus_max_num_subblock_merge_cand may be signaled. The parameter five_minus_max_num_subblock_merge_cand specifies the maximum number of subblock-based merge candidates supported by SPS, minus 5. The value of five_minus_max_num_subblock_merge_cand ranges from 0 to 5, with some examples included. For example, if the value of five_minus_max_num_subblock_merge_cand is 2, then the maximum number of candidates in the combined subblock-based merge candidate list is 3 (5 minus 2).

[0151] In some examples, the temporal motion vector predictor can be enabled / disabled at the picture header level. Figure 17 shows an example syntax table (1700) of a picture header structure in some examples.

[0152] In the example syntax table (1700), if the SPS level flag sps_temporal_mvp_enabled_flag is equal to 1, then the flag ph_temporal_mvp_enabled_flag is signaled as shown by (1710). The flag ph_temporal_mvp_enabled_flag specifies whether a temporal motion vector predictor can be used for inter prediction of a slice associated with the picture header. If ph_temporal_mvp_enabled_flag is equal to 0, the syntax elements of the slice associated with the picture header are constrained such that a temporal motion vector predictor is not used for decoding the slice. Otherwise (ph_temporal_mvp_enabled_flag is equal to 1), a temporal motion vector predictor may be used for decoding the slice associated with the picture header. If not present, in one example, the value of ph_temporal_mvp_enabled_flag is inferred to be equal to 0. If the reference picture in the decoded picture buffer does not have the same spatial resolution as the current picture, the value of ph_temporal_mvp_enabled_flag is equal to 0.

[0153] The maximum number of subblock-based merging candidates may be derived based on the signaled or estimated flags and parameters. In one example, a variable MaxNumSubblockMergeCand is used to indicate the maximum number of subblock-based merging candidates. In one example, when sps_affine_enabled_flag is equal to 1, MaxNumSubblockMergeCand is derived according to (Equation 9), and when sps_affine_enabled_flag is equal to 0, MaxNumSubblockMergeCand is derived according to (Equation 10):

number

[0154] In some examples, the value of MaxNumSubblockMergeCand is in the range of 0 to 5, inclusive.

[0155] According to one aspect of the disclosure, when sps_affine_enabled_flag is signaled as 1, MaxNumSubblockMergeCand is derived from five_minus_max_num_subblock_merge_cand as described in (Equation 9). In some examples, a scenario is allowed where sps_affine_enabled_flag is signaled as 1 and five_minus_max_num_subblock_merge_cand is signaled equal to 5. In this scenario, the maximum number of subblock-based merge candidates MaxNumSubblockMergeCand is derived as 0, which may turn off affine merge mode as with SbTMVP, regardless of the SbTMVP enabling flag, which may cause a conflict if the SbTMVP enabling flag indicates that SbTMVP is enabled.

[0156] Aspects of the present disclosure provide techniques for setting a range of values ​​for the number of sub-block-based merging candidates (also referred to as the maximum number of sub-block-based merging candidates) according to a default number of sub-block-based merging candidates (e.g., denoted as N) and related high-level usage flags for the affine and / or SbTMVP coding tools. For example, if the SbTMVP enable flag indicates that SbTMVP is enabled, then the maximum number of sub-block-based merging candidates is not 0.

[0157] In some implementations, the parameter five_minus_max_num_subblock_merge_cand is negatively correlated with the maximum number of subblock-based merging candidates, and the upper limit of the parameter five_minus_max_num_subblock_merge_cand is determined based on the SbTMVP enable flag.

[0158] In one embodiment, the parameter five_minus_max_num_subblock_merge_cand specifies the maximum number of subblock-based merge motion vector prediction candidates supported in SPS subtracted from N, and furthermore, the value of the parameter five_minus_max_num_subblock_merge_cand is restricted to the range from 0 to N-sps_sbtmvp_enabled_flag, inclusive. The upper limit of the parameter five_minus_max_num_subblock_merge_cand depends on the flag sps_sbtmvp_enabled_flag.

[0159] In some examples, the default number N is 5, and when the flag sps_sbtmvp_enabled_flag is 0 (SbTMVP disabled), the value of the parameter five_minus_max_num_subblock_merge_cand can range from 0 to 5, inclusive. However, when the flag sps_sbtmvp_enabled_flag is 1 (SbTMVP enabled), the value of the parameter five_minus_max_num_subblock_merge_cand can range from 0 to 4, inclusive. As an example, on the encoder side, if the flag sps_sbtmvp_enabled_flag is 1 and the calculated value of the parameter five_minus_max_num_subblock_merge_cand is 5, which is above the upper limit, then the signaled value of the parameter five_minus_max_num_subblock_merge_cand in the coded video bitstream is 4, constrained to be in the range from 0 to 4, inclusive.

[0160] In some examples, if the value of the parameter five_minus_max_num_subblock_merge_cand is equal to the upper limit of the range, the parameter five_minus_max_num_subblock_merge_cand may not be signaled from the encoder side in the coded video bitstream. At the decoder side, if the decoder detects that the parameter five_minus_max_num_subblock_merge_cand is not present in the coded video bitstream, the decoder may infer that the value of the parameter five_minus_max_num_subblock_merge_cand is at the upper limit of the range. The upper limit of the range may be determined based on the SbTMVP enabled flag. For example, the value of five_minus_max_num_subblock_merge_cand is inferred to be equal to N-sps_sbtmvp_enabled_flag. In one example, the default number N is 5, and if the flag sps_sbtmvp_enabled_flag is 0 (SbTMVP is disabled), the value of the parameter five_minus_max_num_subblock_merge_cand can be inferred to be 5. However, if the flag sps_sbtmvp_enabled_flag is 1 (SbTMVP is enabled), the value of the parameter five_minus_max_num_subblock_merge_cand is inferred to be 4.

[0161] In another embodiment, the upper limit of the parameter five_minus_max_num_subblock_merge_cand is determined based on a combination of multiple SbTMVP enabled flags, such as a first flag sps_sbtmvp_enabled_flag at the SPS level and a second flag ph_temporal_mvp_enabled_flag at the picture header level. In one example, the value of the parameter five_minus_max_num_subblock_merge_cand is limited to the range from 0 to N-(sps_sbtmvp_enabled_flag && ph_temporal_mvp_enabled_flag), inclusive. If the parameter five_minus_max_num_subblock_merge_cand is not present in the coded video bitstream, then the value of five_minus_max_num_subblock_merge_cand is inferred to be equal to N-(sps_sbtmvp_enabled_flag && ph_temporal_mvp_enabled_flag).

[0162] In some examples, the default number N is 5, and if at least one of the first flag sps_sbtmvp_enabled_flag and the second flag ph_temporal_mvp_enabled_flag is 0 (SbTMVP is disabled), the value of the parameter five_minus_max_num_subblock_merge_cand can be in the range of 0 to 5, inclusive. However, if both the first flag sps_sbtmvp_enabled_flag and the second flag ph_temporal_mvp_enabled_flag are 1 (SbTMVP is enabled), the value of the parameter five_minus_max_num_subblock_merge_cand can be in the range of 0 to 4, inclusive. In one example, on the encoder side, if both the first flag sps_sbtmvp_enabled_flag and the second flag ph_temporal_mvp_enabled_flag are 1 and the calculated value of the parameter five_minus_max_num_subblock_merge_cand is 5, which is above the upper limit, then the signaled value of the parameter five_minus_max_num_subblock_merge_cand in the coded video bitstream is 4, constrained to the range of 0 to 4 (inclusive).

[0163] In some examples, if the value of the parameter five_minus_max_num_subblock_merge_cand is equal to the upper limit of the range, then the parameter five_minus_max_num_subblock_merge_cand may not be signaled from the encoder side in the coded video bitstream. At the decoder side, if the decoder detects that the parameter five_minus_max_num_subblock_merge_cand is not present in the coded video bitstream, then the decoder may infer that the value of the parameter five_minus_max_num_subblock_merge_cand is the upper limit of the range. The upper limit of the range may be determined, for example, based on a suitable combination of the first flag sps_sbtmvp_enabled_flag and the second flag ph_temporal_mvp_enabled_flag. For example, the value of five_minus_max_num_subblock_merge_cand is estimated to be equal to N-(sps_sbtmvp_enabled_flag && ph_temporal_mvp_enabled_flag). In one example, the default number N is 5, and if at least one of the first flag sps_sbtmvp_enabled_flag and the second flag ph_temporal_mvp_enabled_flag is 0 (SbTMVP is disabled), the value of the parameter five_minus_max_num_subblock_merge_cand can be estimated to be 5. However, if both the first flag sps_sbtmvp_enabled_flag and the second flag ph_temporal_mvp_enabled_flag are 1 (SbTMVP is enabled), the value of the parameter five_minus_max_num_subblock_merge_cand can be estimated to be 4.

[0164] FIG. 18 shows a flow chart outlining a process (1800) according to one embodiment of the present disclosure. The process (1800) can be used to reconstruct a block, thus generating a prediction block for the block being reconstructed. In various embodiments, the process (1800) is performed by a processing circuit, such as the processing circuit of the terminal devices (210), (220), (230), and (240), a processing circuit performing the function of the video encoder (303), a processing circuit performing the function of the video decoder (310), a processing circuit performing the function of the video decoder (410), a processing circuit performing the function of the video encoder (503), or the like. In some embodiments, the process (1800) is implemented with software instructions, and thus the processing circuit performs the process (1800) when the processing circuit executes the software instructions. The process starts at (S1801) and proceeds to (S1810).

[0165] In (S1810), a parameter (e.g., five_minus_max_num_subblock_merge_cand indicating the maximum number of candidates in a subblock-based merge candidate list) is determined based on prediction information decoded from the coded video bitstream. The parameter is within a range that depends on a flag indicating an enable / disable status of subblock-based temporal motion vector prediction. In some examples, the upper limit of the range depends on a flag indicating an enable / disable status of subblock-based temporal motion vector prediction. In one example, the flag indicates an enable / disable status of subblock-based temporal motion vector prediction at a sequence parameter set (SPS) level.

[0166] In one embodiment, the value of the parameter is signaled in the coded video bitstream. In another example, if the value of the parameter is not signaled in the coded video bitstream, the value of the parameter can be estimated to be the upper limit of the range. For example, the parameter can be estimated based on a default number and a flag indicating an enabled / disabled state of sub-block-based temporal motion vector prediction in response to the parameter not being signaled in the coded video bitstream.

[0167] In some embodiments, the parameter is within a range that depends on a first flag indicating an enable / disable status of sub-block-based temporal motion vector prediction at a sequence parameter set (SPS) level and a second flag indicating an enable / disable status of temporal motion vector prediction at a picture header (PH) level. In some examples, in response to the parameter not being signaled in the coded video bitstream, the parameter can be estimated based on a default number, a first flag indicating an enable / disable status of sub-block-based temporal motion vector prediction at an SPS level, and a second flag indicating an enable / disable status of temporal motion vector prediction at a PH level.

[0168] At (S1820), a maximum number of candidates in the sub-block based merging candidate list is calculated based on the parameter. In some examples, the maximum number of candidates in the sub-block based merging candidate list is calculated by subtracting the parameter from a default number, such as using (Equation 9). In one example, the default number is 5.

[0169] At (S1830), in response to the current block in a sub-block based prediction mode, samples of the current block are reconstructed based on candidate selection from a constructed sub-block based merging candidate list of the current block, the constructed sub-block based merging candidate list of the current block being constrained by a maximum number of candidates in the sub-block based merging candidate list.

[0170] Thereafter, the process proceeds to (S1899) and ends.

[0171] The techniques described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 19 illustrates a computer system (1900) suitable for implementing certain embodiments of the disclosed subject matter.

[0172] Computer software can be coded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or similar mechanisms to produce code including instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., directly, or through interpretation, through microcode execution, etc.

[0173] The instructions may be executed on various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0174] 19 for computer system (1900) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement related to any one or combination of components illustrated in the exemplary embodiment of computer system (1900).

[0175] The computer system (1900) may include certain human interface input devices that may be responsive to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic images).

[0176] The input human interface devices may include one or more of a keyboard (1901), a mouse (1902), a trackpad (1903), a touch screen (1910), a data glove (not shown), a joystick (1905), a microphone (1906), a scanner (1907), and a camera (1908).

[0177] The computer system (1900) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1910), data gloves (not shown), joystick (1905), although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (1909), headphones (not shown)), visual output devices (such as screens (1910), including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have touch screen input capability and each of which may or may not have haptic feedback capability - some of which may be capable of outputting three or more dimensional output via means such as two-dimensional visual output or stereoscopic output; virtual reality glasses (not shown), holographic displays and smoke tanks (not shown)), and printers (not shown).

[0178] The computer system (1900) may also include human accessible storage and associated media such as optical media including CD / DVD ROM / RW (1920) with CD / DVD or similar media (1921), thumb drives (1922), removable hard drives or solid state drives (1923), legacy magnetic media such as tape and floppy disks (not shown), specialized ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0179] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0180] The computer system (1900) may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CAN bus, etc. Certain networks generally require an external network interface adapter that is attached to a particular general-purpose data port or peripheral bus (1949) (e.g., a USB port on the computer system (1900)). Others are generally built into the core of the computer system (1900) by attaching to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1900) may communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast television), unidirectional transmit only (e.g., a CAN bus to a particular CAN bus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0181] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (1940) of the computer system (1900).

[0182] The core (1940) may include one or more central processing units (CPUs) (1941), graphics processing units (GPUs) (1942), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (1943), hardware accelerators (1944) for specific tasks, and the like. These devices may be connected via a system bus (1948), along with read only memory (ROM) (1945), random access memory (1946), internal mass storage devices (1947) such as internal non-user accessible hard drives, SSDs, and the like. In some computer systems, the system bus (1948) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the core's system bus (1948) or via a peripheral bus (1949). Peripheral bus architectures include PCI, USB, and the like.

[0183] The CPU (1941), GPU (1942), FPGA (1943), and accelerator (1944) may combine to execute certain instructions that may constitute the computer code described above. The computer code may be stored in a ROM (1945) or a RAM (1946). Transient data may also be stored in the RAM (1946), while permanent data may be stored, for example, in an internal mass storage device (1947). Rapid storage and retrieval from any memory device may be made possible through the use of a cache memory, which may be closely associated with one or more of the CPU (1941), GPU (1942), mass storage device (1947), ROM (1945), RAM (1946), etc.

[0184] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0185] As an example, and not by way of limitation, a computer system having an architecture (1900), and in particular a core (1940), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with a user-accessible mass storage device as described above, as well as a specific storage device (1940) of the core (1940) that is non-transitory in nature, such as a core-internal mass storage device (1947) or a ROM (1945). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1940). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (1940), and in particular the processor (including a CPU, GPU, FPGA, etc.) therein, to perform certain processes or certain portions of the processes described herein, including defining data structures stored in RAM (1946) and modifying such data structures according to processes defined by the software. Additionally, or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., an accelerator (1944)), which may operate in place of, or in conjunction with, software to perform particular processes or portions of particular processes described herein. Reference to software includes logic, and vice versa, where appropriate. Reference to a computer-readable medium may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software. Appendix A: Acronyms JEM: Joint exploration model VVC: General Purpose Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplemental Extended Information VUI: Video Usability Information GOP: Group of Pictures TU: conversion unit PU: Prediction Unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted block HRD: Hypothetical Reference Decoder SNR: Signal to Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD:Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CAN Bus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit CU: coding unit

[0186] While this disclosure has described several exemplary embodiments, there are alterations, substitutions, and various substitute equivalents that are within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. 1. A method for video encoding comprising: Calculating, by a processing circuit of the video encoder, a maximum number of candidates in a sub-block based merging candidate list for coding the video bitstream; determining a parameter corresponding to the coded video bitstream based on the calculated maximum number of candidates, the parameter being in a range from 0 to an upper limit determined based on sps_sbtmvp_enabled_flag, where sps_sbtmvp_enabled_flag is equal to 1 to specify that a sub-block based temporal motion vector predictor is used for decoding pictures having slices with a slice type not equal to I (intra-coding) in the coded video, and where sps_sbtmvp_enabled_flag is equal to 0 to specify that the sub-block based temporal motion vector predictor is not used in the coded video; in response to the current block being in a sub-block based prediction mode, encoding samples of the current block based on candidate selection from a constructed sub-block based merging candidate list of the current block, the constructed sub-block based merging candidate list of the current block being constrained by a maximum number of candidates in the sub-block based merging candidate list. method.

2. The determining step further comprises: determining the parameter by subtracting the calculated maximum number of candidates in the sub-block based merging candidate list from a default number; The method of claim 1.

3. The default number is 5. The method of claim 2.

4. transmitting the determined parameters in the coded bitstream.

4. The method according to claim 1 .

5. implying the parameter based on a default number, the sps_sbtmvp_enabled_flag, and not signaling the determined parameter in the coded video bitstream. The method of claim 1.

6. The sps_sbtmvp_enabled_flag flag is included at the Sequence Parameter Set (SPS) level of the coded video bitstream.

6. The method according to any one of claims 1 to 5.

7. the sps_sbtmvp_enabled_flag is included in a sequence parameter set (SPS) level of the coded video bitstream, and the upper limit of the range is further based on a flag indicating an enabled / disabled state of temporal motion vector prediction at a picture header (PH) level of the coded video bitstream.

6. The method according to any one of claims 1 to 5.

8. implying the parameter based on a default number, the sps_sbtmvp_enabled_flag, and the flag indicating the enabled / disabled state of the temporal motion vector prediction, and not signaling the parameter in the coded video bitstream. The method of claim 7.

9. and implying the determined parameter when the determined parameter is equal to an upper limit of the range and not signaling the determined parameter in the coded video bitstream. The method of claim 1.

10. 13. An apparatus for video encoding comprising: A processing circuit configured to carry out the method according to any one of claims 1 to 9. Device.

11. A non-transitory computer readable storage medium storing instructions, the instructions causing the computer to perform the method of any one of claims 1 to 9 when executed by a computer for video encoding.

Images