Method and apparatus for video encoding, decoding and storage medium

Abstract

Aspects of the disclosure provide a method and apparatus for video encoding, decoding and storage medium, the processing circuitry included in the apparatus is configured to receive an encoded video bitstream containing a current picture, the current picture including a current block, determine, based on a syntax element in the encoded video bitstream, that the current block including a plurality of sub-blocks is coded in a sub-block based temporal motion vector prediction (SbTMVP) mode. The processing circuitry determines a plurality of displacement vector predictor (DVP) candidates and receives a base index indicating a DVP among the plurality of DVP candidates and a displacement vector (DV) offset of the current block. The processing circuitry determines a DV based on the DVP and the DV offset. The DV indicates a block collocated with the current block in a collocated reference picture. The processing circuitry reconstructs a sub-block among the plurality of sub-blocks based on motion information of a corresponding sub-block in the block collocated with the current block.

Description

[0001] priority

[0002] This application claims priority to U.S. Patent Application No. 17 / 984,107, filed November 9, 2022, entitled “Subblock Level Temporal Motion Vector Prediction with Multiple Displacement Vector Predictors and an Offset,” which claims priority to U.S. Provisional Application No. 63 / 345,802, filed May 25, 2022, entitled “Subblock Based Motion Vector Prediction with Multiple Base Indexes and MV Offset.” Technical Field

[0003] This disclosure describes embodiments that generally involve video coding. Background Technology

[0004] The background description provided herein is intended to provide a general overview of the context of this disclosure. To the extent described in this background section, the work of the currently attributed inventors and aspects of the description that may not conform to the prior art at the time of filing are neither explicitly nor implicitly considered to be prior art of this disclosure.

[0005] Uncompressed digital images and / or videos can comprise a series of pictures, each with spatial dimensions, for example, 1920×1080 luminance samples and correlated chrominance samples. This series of pictures can have a fixed or variable image rate (informally also called frame rate), for example, 60 frames per second or 60Hz. Uncompressed images and / or videos have specific bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at 60Hz frame rate) requires close to 1.5 Gbit / s of bandwidth. One hour of such video would require over 600 GB of storage space.

[0006] One objective of image and / or video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression helps reduce the aforementioned bandwidth and / or storage space requirements, in some cases by two orders of magnitude or more. Although the description herein uses video encoding / decoding as an illustrative example, the same techniques can be applied to image encoding / decoding in a similar manner without departing from the spirit of this disclosure. Lossless compression and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal is useful for the intended application. In the case of video, lossy compression is widely used. (Lossy compression is widely used) The amount of distortion tolerated depends on the application; for example, users of some consumer streaming applications may tolerate higher distortion than users of television applications. The achievable compression ratio can be reflected in the fact that higher tolerance / tolerable distortion can result in a higher compression ratio.

[0007] Video encoders and decoders can utilize several major categories of techniques, including motion compensation, transform processing, quantization, and entropy coding.

[0008] Video codec techniques include a technique called intra-frame coding. In intra-frame coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the picture can be an intra-frame picture. Intra-frame pictures and their derived pictures (e.g., standalone decoder refresh pictures) can be used to reset the decoder state and thus can be used as the first picture in the encoded video bitstream and video session, or as a still image. Samples of an intra-frame block can be transformed, and the transform coefficients can be quantized before entropy coding. Intra-frame prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the transformed DC value, the smaller the AC coefficients, and the fewer bits are needed to represent the entropy-coded block at a given quantization step size.

[0009] For example, conventional intra-frame coding known in MPEG-2 generation coding techniques does not use intra-frame prediction. However, some newer video compression techniques include those that attempt to perform prediction based on surrounding sample data and / or metadata obtained, for example, during the encoding and / or decoding of data blocks. This technique is hereby referred to as "intra-frame prediction." Note that, at least in some cases, intra-frame prediction uses only reference data from the current frame in the reconstruction, and not reference data from a reference frame.

[0010] There can be many different forms of intra prediction. When multiple such techniques are available within a given video coding technique, the specific technique used can be encoded as a specific intra prediction mode employing that technique. In some cases, an intra prediction mode can have sub-modes and / or parameters, where these sub-modes and / or parameters can be encoded individually or contained within a mode codeword that defines the prediction mode used. For a given combination of modes, sub-modes, and / or parameters, the choice of which codeword is used will affect the coding efficiency gain through intra prediction, and therefore, the entropy coding technique used to convert the codeword into a bitstream will also have an impact.

[0011] H.264 introduced specific intra-prediction modes, which were improved in H.265 and further refined in newer coding techniques, such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Prediction blocks can be formed using neighboring sample values ​​from already available samples. The sample values ​​of neighboring samples are copied into the prediction block according to the orientation. The reference to the orientation being used can be encoded in the bitstream or can be predicted itself.

[0012] refer to Figure 1A The lower right corner depicts a subset of nine known prediction directions from the 33 possible prediction directions defined in H.265 (corresponding to 33 angular modes of 35 intra-frame modes). The point where the arrows converge represents the sample being predicted (101). The arrows indicate the direction of the predicted sample. For example, arrow (102) indicates that sample (101) is predicted from one or more samples at a 45° angle to the upper right of the horizontal line. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples at a 22.5° angle to the lower left of sample (101) at a 22.5° angle to the horizontal line.

[0013] Still referencing Figure 1AA 4×4 square block (104) of samples is depicted in the upper left (represented by a thick dashed line). The square block (104) comprises 16 samples, each labeled "S" with its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (counting from the top) and the first sample in the X dimension (counting from the left). Similarly, sample S44 is the fourth sample in the square block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. The reference samples are labeled with R, their Y position relative to the block (e.g., row index), and X position (column index) to the square block (104). In H.264 and H.265, predicted samples are adjacent to the blocks in the reconstruction; therefore, negative values ​​are not required.

[0014] Intra-frame image prediction works by copying reference sample values ​​from neighboring samples predicted from the prediction direction of the signaling. For example, suppose the encoded video bitstream contains signaling that indicates a prediction direction consistent with arrow (102) for this block—that is, predicting samples from the sample at a 45° angle to the upper right of the horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then sample S44 is predicted from reference sample R08.

[0015] In some cases, the values ​​of multiple reference samples can be combined, for example, by interpolation, to calculate the reference sample; especially when the direction is not divisible by 45°.

[0016] With the development of video coding technology, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and some techniques in entropy coding are used to represent those possible directions with a small number of bits. However, when accepting less likely directions, coding efficiency decreases, requiring more bits and leading to a degraded coding performance. Furthermore, the direction itself can sometimes be predicted from adjacent directions used in adjacent decoded blocks.

[0017] Figure 1B A schematic diagram (110) is shown, depicting 65 intra-frame predicted directions according to JEM, to illustrate how the number of predicted directions increases over time.

[0018] The mapping of intra-prediction direction bits in the encoded video bitstream to represent that direction can vary depending on the video coding technique. For example, this mapping can range from a simple direct mapping from prediction direction to intra-prediction mode to codewords, to complex adaptive schemes involving the most probable mode, and similar techniques. However, in most cases, some directions are statistically less likely to appear in the video content compared to others. Since the goal of video compression is to reduce redundancy, in well-functioning video coding techniques, those less likely directions will be represented by more bits than the more likely directions.

[0019] Image and / or video encoding and decoding can be performed using inter-picture prediction with motion compensation. Motion compensation can be a lossy compression technique and can involve techniques where sample data blocks from a previously reconstructed image or a portion thereof (the reference image) are spatially shifted in a direction indicated by a motion vector (MV) to predict a newly reconstructed image or portion thereof. In some cases, the reference image can be the same as the image currently being reconstructed. The MV can have two dimensions, X and Y, or three dimensions, with the third dimension being an indication of the reference image in use (which can indirectly be a temporal dimension).

[0020] In some video compression techniques, a motion vector (MV) applicable to a region of sample data can be predicted from other MVs, for example, from another region of sample data that is spatially adjacent to the region being reconstructed and precedes that MV in the decoding order. Doing so can significantly reduce the amount of data required to encode the MV, thereby eliminating redundancy and improving compression ratio. MV prediction can work efficiently, for example, when encoding an input video signal derived from a camera (called natural video), there is a statistical probability that regions larger than the region to which a single MV can be applied move in similar directions. Therefore, in some cases, similar motion vectors derived from MVs of adjacent regions can be used for prediction. This results in an MV found for a given region being similar or identical to an MV predicted from surrounding MVs, and after entropy coding, this can be represented with fewer bits than when directly encoding the MV. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself can be lossy due to rounding errors, for example, when calculating predictions from several surrounding MVs.

[0021] Various MV prediction mechanisms are described in High Efficiency Video Coding (ITU-T TRec.H.265, "High Efficiency Video Coding", abbreviated as H.265 / HEVC, December 2016). Among the many MV prediction mechanisms provided by H.265, a technique hereafter referred to as "spatial combining" is described with reference to Figure 2.

[0022] Referring to Figure 2, the current block (201) includes samples discovered by the encoder during motion search so that predictions can be made from previous blocks of the same size that have been spatially shifted. Instead of directly encoding the MV, it can be derived from metadata associated with one or more reference images, for example, from the most recent (in decoding order) reference image, using the MV associated with any of the five surrounding samples (denoted as A0, A1 and B0, B1, B2 (202 to 206 respectively)). In H.265, MV prediction can use a predictor from the same reference image being used by adjacent blocks. Summary of the Invention

[0023] This disclosure provides methods, apparatuses, and storage media for video encoding and decoding. In some examples, a video decoding apparatus includes processing circuitry that receives a base index and displacement vector (DV) offset information of a current block in a current frame from an encoded video bitstream. The current block includes multiple sub-blocks reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode. The base index indicates a displacement vector predictor (DVP) among multiple DVP candidates for the current block. The processing circuitry determines the DV of the current block based on the DVP of the current block and the DV offset of the current block indicated by the DV offset information. The DV indicates a juxtaposed block in a juxtaposed reference frame. The juxtaposed block is juxtaposed with the current block. The processing circuitry determines motion information of sub-blocks among the multiple sub-blocks based on motion information of corresponding sub-blocks in the juxtaposed block, and reconstructs the sub-blocks among the multiple sub-blocks based on the motion information of the sub-blocks among the multiple sub-blocks.

[0024] In one example, the processing circuitry determines DV as the vector sum of DVP and DV offset.

[0025] In one example, the processing circuitry determines multiple DVP candidates based on the motion vectors (MV) of the current block's neighboring blocks. The reference image for each spatial neighbor used to determine one of the multiple DVP candidates is a juxtaposed reference image.

[0026] In one example, if the number of MVs of spatial neighbors is less than a threshold, the processing circuit will insert zero MVs into multiple DVP candidates.

[0027] In one example, the processing circuitry determines multiple DVP candidates based on candidates in a merged candidate list. The candidates in the merged candidate list include at least one of (a) Spatial Motion Vector Predictor (MVP) candidates, (b) History-Based Motion Vector Predictor (HMVP) candidates, (c) Pairwise Averaged candidates, or (d) Zero Motion Vector (MV). Temporal Motion Vector Predictor (TMVP) candidates are not included. The reference image for each candidate used to determine one of the multiple DVP candidates is a juxtaposed reference image.

[0028] In one example, the processing circuitry constructs a sub-block-based merge list for the current block. The merge candidates for the sub-blocks in the merge list include multiple SbTMVP merge candidates and at least one affine merge candidate. Each of the multiple SbTMVP merge candidates corresponds to a specific one of the multiple DVP candidates.

[0029] In one example, the base index indicates which SbTMVP merge candidate from the subset of sub-block-based SbTMVP merge candidates to select as one of the multiple SbTMVP merge candidates, the selected SbTMVP merge candidate corresponding to the DVP.

[0030] In one example, a first number K0 of multiple SbTMVP merge candidates and a second number K1 of at least one affine merge candidate are transmitted in signaling form in the high-level syntax. K0 and K1 are positive integers. The base index indicates which sub-block-based merge candidate from the subset of sub-block-based merge candidates to select, the subset including (i) multiple SbTMVP merge candidates and (ii) at least one affine merge candidate. The selected sub-block-based merge candidate corresponds to one of the DVP and multiple SbTMVP merge candidates.

[0031] In one example, the processing circuitry constructs an SbTMVP merge candidate list for the current block, which includes multiple SbTMVP merge candidates. Each SbTMVP merge candidate corresponds to a corresponding one of multiple DVP candidates. The SbTMVP merge candidate list does not include affine merge candidates.

[0032] In one example, the merge mode or merged motion vector difference (MMVD) mode is applied to the current block. The processing circuitry receives a flag from the encoded video bitstream. The flag indicates that the SbTMVP mode is applied to the current block.

[0033] In one example, a merge mode or merged motion vector difference (MMVD) mode is applied to the current block. The processing circuitry receives a first flag from the encoded video bitstream. The first flag indicates that a sub-block-based merge mode is applied to the current block. The processing circuitry receives a second flag from the encoded video bitstream, indicating that the sub-block-based merge mode is SbTMVP mode.

[0034] In one example, the processing circuitry constructs a sub-block-based merge list for the current block. The sub-block-based merge list includes one SbTMVP merge candidate and at least one affine merge candidate. An SbTMVP merge candidate corresponds to a DVP (Distributed Virtualization Validation). The processing circuitry receives an index pointing to an SbTMVP merge candidate in the sub-block-based merge list. The index indicates that the SbTMVP mode is applied to the current block.

[0035] In one embodiment, the processing circuitry receives an encoded video bitstream including a current image. The current image includes a current block. The current block includes multiple sub-blocks. The processing circuitry determines, based on syntax elements in the encoded video bitstream, that the current block, including multiple sub-blocks, is encoded in SbTMVP mode. The processing circuitry determines multiple DVP candidates for the current block. Each DVP candidate is derived from one or more motion vectors (MVs). The processing circuitry receives a base index and a DV offset for the current block. The base index indicates the DVP among the multiple DVP candidates for the current block. The processing circuitry determines the DV of the current block based on the DVP indicated in the multiple DVP candidates and the DV offset of the current block. The DV indicates the block juxtaposed with the current block in a juxtaposed reference image. The processing circuitry reconstructs the sub-blocks among the multiple sub-blocks based on motion information of the corresponding sub-blocks in the juxtaposed block (the block juxtaposed with the current block in the juxtaposed reference image).

[0036] Various aspects of this disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform a method of video decoding.

[0037] Various aspects of this disclosure also provide a method for video coding, comprising: determining a displacement vector predictor (DVP) and a displacement vector (DV) offset for a current block in a current image, wherein the current block comprises multiple sub-blocks; determining a DV for the current block based on the DVP and DV offset, wherein the DV indicates a juxtaposed block in a juxtaposed reference image, the juxtaposed block in the juxtaposed reference image being juxtaposed with the current block; determining motion information of sub-blocks among the multiple sub-blocks based on motion information of corresponding sub-blocks in the juxtaposed block; and encoding the sub-blocks among the multiple sub-blocks based on the motion information of the sub-blocks among the multiple sub-blocks.

[0038] Various aspects of this disclosure also provide an apparatus for video encoding, wherein instructions, when executed by at least one processor, cause at least one processor to perform: determining the offset of a displacement vector predictor (DVP) and a displacement vector (DV) of a current block in a current image, wherein the current block comprises a plurality of sub-blocks; determining the DV of the current block based on the DVP and DV offsets, wherein the DV indicates a juxtaposed block in a juxtaposed reference image, the juxtaposed block in the juxtaposed reference image being juxtaposed with the current block; determining motion information of sub-blocks among the plurality of sub-blocks based on motion information of corresponding sub-blocks in the juxtaposed block; and encoding the sub-blocks among the plurality of sub-blocks based on the motion information of the sub-blocks among the plurality of sub-blocks. Attached Figure Description

[0039] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, wherein:

[0040] Figure 1A This is a schematic diagram of an example set of intra-frame prediction modes.

[0041] Figure 1B This is an illustration of an exemplary intra-frame prediction direction.

[0042] Figure 2 shows an example of the current block (201) and surrounding samples.

[0043] Figure 3 This is a schematic diagram of an exemplary block diagram of a communication system (300).

[0044] Figure 4 This is a schematic diagram of an exemplary block diagram of a communication system (400).

[0045] Figure 5 This is a schematic diagram of an exemplary block diagram of a decoder.

[0046] Figure 6 This is a schematic diagram of an exemplary block diagram of an encoder.

[0047] Figure 7 A block diagram of an exemplary encoder is shown.

[0048] Figure 8 A block diagram of an exemplary decoder is shown.

[0049] Figure 9 The locations of spatial merging candidates according to embodiments of this disclosure are shown.

[0050] Figure 10 Candidate pairs considered for redundancy checking of spatial merging candidates according to embodiments of this disclosure are shown.

[0051] Figure 11 An exemplary motion vector scaling of a time merging candidate is shown.

[0052] Figure 12 An exemplary candidate position for the temporal merging candidate of the current coding unit is shown.

[0053] Figure 13-14 An example of the search process in the combined motion vector difference (MMVD) mode is shown.

[0054] Figure 15 Additional refinement positions along multiple diagonals are shown in the MMVD mode.

[0055] Figure 16A An example of a 4-parameter affine motion model is shown.

[0056] Figure 16B An example of a 6-parameter affine motion model is shown.

[0057] Figure 17 An example of affine transformation prediction based on sub-blocks is shown.

[0058] Figure 18 An example of determining control point motion vector (CPMV) candidates in the current affine merging list of the CU is shown.

[0059] Figure 19 Examples of spatial and temporal neighbors for the current block are shown.

[0060] Figure 20-21 An exemplary sub-block-based temporal motion vector prediction SbTMVP process is shown in the SbTMVP mode.

[0061] Figure 22 A flowchart outlining the coding process according to some embodiments of this disclosure is shown.

[0062] Figure 23A A flowchart outlining a decoding process according to some embodiments of this disclosure is shown.

[0063] Figure 23B A flowchart outlining a decoding process according to some embodiments of this disclosure is shown.

[0064] Figure 24 This is a schematic diagram of a computer system according to one embodiment. Detailed Implementation

[0065] Figure 3 An exemplary block diagram of a communication system (300) is shown. The communication system (300) includes a plurality of terminal devices capable of communicating with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and terminal device (320) interconnected via the network (350). Figure 3In the example, the first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, terminal device (310) may encode video data (e.g., a video image stream captured by terminal device (310)) for transmission via network (350) to another terminal device (320). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. Terminal device (320) may receive the encoded video data from network (350), decode the encoded video data to recover the video images, and display the video images based on the recovered video data. Unidirectional data transmission is common in media service applications, etc.

[0066] In another example, the communication system (300) includes a second pair of terminal devices (330) and a terminal device (340) performing bidirectional transmission of encoded video. For example, data during a video conference, for bidirectional transmission of data, in one example, each terminal device of terminal devices (330) and (340) can encode video data (e.g., a stream of video images captured by the terminal device) for transmission via network (350) to the other terminal device of terminal devices (330) and (340). Each terminal device of terminal devices (330) and (340) can also receive encoded video data transmitted by the other terminal device of terminal devices (330) and (340), and can decode the encoded video data to recover the video images, and can display the video images on an accessible display device based on the recovered video data.

[0067] exist Figure 3 In the examples, terminal devices (310), (320), (330), and (340) are shown as a server, a personal computer, and a smartphone, respectively, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptops, tablets, media players, and / or dedicated video conferencing equipment. Network (350) refers to any number of networks that transmit encoded video data between terminal devices (310), (320), (330), and (340), including, for example, wired and / or wireless communication networks. Communication networks (350) can exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may be of little importance to the operation of this disclosure, unless explained below.

[0068] As an example of the application of the disclosed topic, Figure 4The diagram illustrates a video encoder and video decoder in a streaming environment. The disclosed subject matter can also be applied to other video-enabled applications, including, for example, video conferencing, digital television, streaming services, and storing compressed video on digital media such as Compact Discs (CDs), Digital Versatile Discs (DVDs), and flash memory.

[0069] The streaming system may include: a capture subsystem (413) that may include a video source (401), such as a digital camera; and a system for creating, for example, an uncompressed video picture stream (402). In one example, the video picture stream (402) includes samples captured by a digital camera. The video picture stream (402) is depicted as a thick line emphasizing the high data volume when compared to encoded video data (404) (or encoded video bitstream), and may be processed by an electronic device (420) of a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume when compared to the video picture stream (402), and may be stored on a streaming server (405) for future use. One or more streaming client subsystems (e.g., Figure 3 Client subsystems (406) and (408) can access a streaming server (405) to retrieve copies (407) and (409) of encoded video data (404). Client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the input copy (407) of the encoded video data and creates an output video picture stream (411) that can be presented on a display (412) (e.g., a screen) or other presentation device (not shown). In some streaming systems, the encoded video data (404), copies (407), and copies (409) (e.g., a video bitstream) may be encoded according to certain video coding / compression standards. Examples of such standards include ITU-T Recommendation H.265. In one example, the video coding standard under development is informally referred to as Universal Video Coding (VVC). The disclosed topics can be used in a VVC environment.

[0070] Note that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0071] Figure 5A block diagram of a video decoder (510) is shown. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used in place of... Figure 4 The video decoder (510) in the example.

[0072] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510). In one embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the decoding of other encoded video sequences. The encoded video sequences can be received from a channel (501), which can be a hardware / software link to a storage device storing the encoded video data. The receiver (531) can receive encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective user entities (not shown). The receiver (531) can separate the encoded video sequences from other data. To combat network jitter, a buffer memory (515) can be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, it can be outside the video decoder (510) (not shown). In other cases, a buffer memory (not shown) may be present outside the video decoder (510), for example, to combat network jitter. Additionally, another buffer memory (515) may be present inside the video decoder (510), for example, to handle playback timing. The buffer memory (515) may be unnecessary or small when the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network. For use on best-effort packet networks such as the Internet, a buffer memory (515) may be required. This buffer memory may be relatively large and advantageously have an adaptive size, and may be implemented at least partially outside the video decoder (510) in an operating system or similar component (not shown).

[0073] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols include information for managing the operation of the video decoder (510) and information that potentially controls a presentation device such as a display device (512) (e.g., a display screen), which is not part of the electronic device (530) but may be coupled to it. Figure 5As shown. The control information used to present the device can be in the form of Supplemental Enhancement Information (SEI message) or Video User Information (VUI) parameter set fragments (not shown). The parser (520) can parse / decode the received encoded video sequence. The encoding of the encoded video sequence can be based on video coding technology or standards and can follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) can extract a set of subgroup parameters of at least one pixel subgroup from the encoded video sequence based on at least one parameter corresponding to the group. The subgroup can include Group of Pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transforming unit (TU), prediction unit (PU), etc. The parser (520) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, MV, etc.

[0074] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) in order to create symbols (521).

[0075] Depending on the type of the encoded video picture or its portions (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (521) can involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by the parser (520). For clarity, the flow of such subgroup control information between the parser (520) and the multiple units below is not described.

[0076] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In practical implementations operating under commercial constraints, many of these units interact closely with each other and can be integrated with each other at least partially. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the following functional units.

[0077] The first unit is the scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives the quantized transform coefficients and control information, including which transform to use, block size, quantization factor, quantization scaling matrix, etc., as symbols (521) from the parser (520). The scaler / inverse transform unit (551) can output blocks containing sample values, which can be input into the aggregator (555).

[0078] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block. An intra-coded block is a block that does not use prediction information from a previously reconstructed image, but can use prediction information from a previously reconstructed portion of the current image. This prediction information can be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses surrounding reconstructed information obtained from the current picture buffer (558) to generate a block with the same size and shape as the block in the reconstruction. The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information already generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) based on each sample.

[0079] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coded blocks and may be motion-compensated. In this case, the motion compensation prediction unit (553) can access the reference picture memory (557) to obtain samples for prediction. After motion compensation of the extracted samples according to the symbols (521) associated with the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (referred to as residual samples or residual signals in this case) to generate output sample information. The addresses in the reference picture memory (557) from which the motion compensation prediction unit (553) obtains the predicted samples can be controlled by motion vectors, and the motion compensation prediction unit (553) can obtain these addresses in the form of symbols (521), which may have, for example, X, Y, and reference picture components. When using subsampled exact MV, motion compensation may also include interpolation of sampled values ​​obtained from the reference picture memory (557), motion vector prediction mechanisms, etc.

[0080] The output samples of the aggregator (555) can be processed in the loop filter unit (556) using various loop filtering techniques. Video compression techniques may include loop filtering techniques controlled by parameters contained in the encoded video sequence (also known as the encoded video bitstream) and available as symbols (521) from the parser (520) to the loop filter unit (556). Video compression may also be in response to metadata obtained during the decoding of a previous (in the order of decoding) portion of the encoded image or encoded video sequence, as well as to previously reconstructed and loop-filtered sample values.

[0081] The output of the loop filter unit (556) can be a sample stream, which can be output to a display device (512) and stored in a reference image memory (557) for future inter-frame image prediction.

[0082] Once fully reconstructed, certain encoded images can be used as reference images for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and that encoded image has been identified as a reference image (e.g., by the parser (520)), the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of the next encoded image begins.

[0083] The video decoder (510) can perform decoding operations according to a predetermined video compression technique in a standard such as ITU-T Rec.H.265. The encoded video sequence conforms to the syntax of the video compression technique or standard, and the profile documented in the video compression technique or standard specifies the syntax. Specifically, the profile may select certain tools from all available tools in the video compression technique or standard, as the only tools available under that profile. Standard conformance also requires the complexity of the encoded video sequence to be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the level can be further restricted by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer transmitted in signaling form in the encoded video sequence.

[0084] In one embodiment, the receiver (531) may receive additional (redundant) data with encoded video. The additional data may be included as part of the encoded video sequence. The video decoder (510) may use the additional data to correctly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0085] Figure 6 An exemplary block diagram of a video encoder (603) is shown. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of Figure 4 The video encoder (403) in the example.

[0086] The video encoder (603) can obtain data from the video source (601) (which is not...). Figure 6 In one example, an electronic device (620) receives video samples, and the video source (601) can capture video images to be encoded by a video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0087] The video source (601) can be provided as a digital video sample stream of a sequence of source videos to be encoded by a video encoder (603). This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601YCrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb4:2:0, YCrCb4:4:4). In a media service system, the video source (601) can be a storage device storing previously prepared video. In a video conferencing system, the video source (601) can be a camera capturing local image information as a video sequence. The video data can be provided as a series of multiple individual pictures that produce motion when viewed in sequence. The pictures themselves can be organized as a spatial array of pixels, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0088] According to one embodiment, the video encoder (603) can encode and compress images of a source video sequence into an encoded video sequence (643) in real time or under any other time constraint. Implementing an appropriate encoding rate is a function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units as described below. For clarity, coupling is not described. Parameters set by the controller (650) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other functions applicable to the video encoder (603) to optimize a particular system design.

[0089] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. To put it simply, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols in a manner similar to what a (remote) decoder would also create, to create sample data. The reconstructed sample stream (sample data) is fed into a reference image memory (634). Since decoding the symbol stream produces a bit-precise result (local or remote) regardless of the decoder's location, the contents of the reference image memory (634) are also bit-precise between the local and remote encoders. In other words, when prediction is used during decoding, the encoder's prediction portion, as a reference image sample, "sees" the exact same sample value as the decoder "sees." This basic principle of reference image synchronicity (and the resulting drift, if synchronicity cannot be maintained, for example, due to channel errors) is also used in some related techniques.

[0090] The operation of the "local" decoder (633) is the same as that of a "remote" decoder such as a video decoder (510), as described above. Figure 4 It was described in detail. However, a brief reference is provided. Figure 5 Since symbols are available and the encoding / decoding of symbols for the encoded video sequence by the entropy encoder (645) and the parser (520) can be lossless, the entropy decoding part of the video decoder (510), including the buffer memory (515) and the parser (520), does not need to be fully implemented in the local decoder (633).

[0091] In one embodiment, any decoder technique, other than the parsing / entropy decoding present in the decoder, exists in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject focuses on decoder operation. The description of encoder techniques can be simplified, as these techniques are the inverse of the fully described decoder techniques. In certain areas, a more detailed description is provided below.

[0092] In some example operations, the source encoder (630) may perform motion-compensated predictive coding, which predictively codes the input image based on one or more previously encoded images (referred to as "reference images") in the video sequence. In this way, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of the reference images, which may be selected as prediction references for the input image.

[0093] The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture, based on symbols created by the source encoder (630). The operation of the encoding engine (632) may favor a lossy process. When the encoded video data can be decoded by the video decoder (633), Figure 6 When decoded at (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that can be performed on the reference image by the video decoder, and can store the reconstructed reference image in the reference image memory (634). In this way, the video encoder (603) can locally store copies of the reconstructed reference images that have the same content as the reconstructed reference images that will be obtained by the remote video decoder (without transmission errors).

[0094] The predictor (635) can perform a prediction search on the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., which can be used as appropriate prediction references for the new image. The predictor (635) can operate on a sample block-by-pixel basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input image may have prediction references extracted from multiple reference images stored in the reference image memory (634).

[0095] The controller (650) can manage the encoding operations of the source encoder (630), including, for example, the settings of parameters and subgroup parameters for encoding video data.

[0096] The outputs of all the aforementioned functional units can be processed by the entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into an encoded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, and arithmetic coding.

[0097] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which may be a hardware / software connection to a storage device for storing the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0098] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoded picture type to each encoded picture, which can affect the encoding techniques that can be applied to the corresponding picture. For example, a picture can typically be designated as one of the following picture types:

[0099] An intra-frame picture (I-picture) can be a picture that is encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Intra-coded Picture Refresh (IDR) pictures. Those skilled in the art are aware of those variations of I-pictures and their corresponding applications and characteristics.

[0100] A predicted image (P-image) can be an image that uses at most one motion vector and a reference index to predict the sample value of each block, and is encoded and decoded using intra-frame prediction or inter-frame prediction.

[0101] A bidirectional prediction image (B-image) can be an image that uses up to two motion vectors and reference indices to predict the sample values ​​for each block, encoded and decoded using intra-frame prediction or inter-frame prediction. Similarly, a multi-prediction image can use more than two reference images and associated metadata to reconstruct the sample values ​​for a single block.

[0102] Source images can typically be spatially subdivided into multiple sample blocks (e.g., each sample block consists of 4×4, 8×8, 4×8, or 16×16 sample blocks) and encoded on a block-by-block basis. Depending on the encoding allocation applied to the respective images of each block, blocks can be predictively encoded by referencing other (already encoded) blocks. For example, blocks of image I can be nonpredictively encoded, or predictively encoded (spatial prediction or intra-frame prediction) by referencing already encoded blocks of the same image. Pixel blocks of image P can be predictively encoded by referencing a previously encoded reference image, either spatially or temporally. Blocks of image B can be predictively encoded by referencing one or two previously encoded reference images, either spatially or temporally.

[0103] The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard (e.g., ITU-T REC.H.265). In its operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard being used.

[0104] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include this data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data (e.g., redundant images and slices), SEI messages, VUI parameter set fragments, etc.

[0105] Video can be captured in chronological order as multiple source images (video pictures). Intra-frame picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given image, while inter-frame picture prediction utilizes (temporal or other) correlations between images. In one example, the specific image being encoded / decoded is divided into blocks called the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. When using multiple reference images, the motion vector points to the reference block within the reference images and can have a third dimension that identifies the reference images.

[0106] In some embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, for example, a first reference image and a second reference image, both of which precede the current image in the video in the decoding order (but can be displayed in the past and future, respectively). A block in the current image can be encoded by a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block can be predicted using a combination of the first and second reference blocks.

[0107] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0108] According to some embodiments of this disclosure, prediction is performed on a block-by-block basis, such as inter-frame picture prediction and intra-frame picture prediction. For example, according to the HEVC standard, pictures in a video picture sequence are segmented into Coding Tree Units (CTUs) for compression. The CTUs in the pictures have the same size, for example, 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU comprises three Coding Tree Blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU can be recursively divided into one or more Coding Units (CUs). For example, a 64×64 pixel CTU can be segmented into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine its prediction type, such as inter-frame prediction or intra-frame prediction. Based on temporal and / or spatial predictability, CUs are divided into one or more prediction units (PUs). Typically, each PU includes one luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in encoding (encoding / decoding) are performed on a per-prediction-block basis. Using a luma prediction block as an example, a prediction block contains a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0109] Figure 7 An exemplary diagram of a video encoder (703) is shown. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values ​​within a current video image in a video image sequence, and to encode that processing block into an encoded image as part of an encoded video sequence. In one example, the video encoder (703) is used instead of Figure 4 The video encoder (403) in the example.

[0110] In the HEVC example, the video encoder (703) receives a sample value matrix of the processed block, such as an 8×8 sample prediction block. The video encoder (703) determines the optimal encoding method for the processed block: intra-frame mode, inter-frame mode, or bidirectional prediction mode, such as rate-distortion optimized mode. When encoding the processed block in intra-frame mode, the video encoder (703) can use intra-frame prediction techniques to encode the processed block into a coded picture; and when encoding the processed block in inter-frame mode or bidirectional prediction mode, the video encoder (703) can use inter-frame prediction or bidirectional prediction techniques to encode the processed block into a coded picture, respectively. In some video coding techniques, the merging mode may be an inter-frame picture prediction sub-mode, in which motion vectors are derived from one or more motion vector predictors without benefiting from coded motion vector components outside the predictors. In some other video coding techniques, there may be motion vector components applicable to the object block. In one example, the video encoder (703) includes other components, such as a mode determination module (not shown) for determining the mode of the processed block.

[0111] exist Figure 7 In the example, the video encoder (703) includes, for example, Figure 7 The inter-frame encoder (730), intra-frame encoder (722), residual calculator (723), switch (726), residual encoder (724), general controller (721) and entropy encoder (725) are coupled together.

[0112] An inter-frame encoder (730) is configured to receive samples of the current block (e.g., the processing block), compare that block with one or more reference blocks in a reference image (e.g., blocks in previous and subsequent images), generate inter-frame prediction information (e.g., a description of redundancy information based on the inter-frame coding technique, motion vectors, merging mode information), and compute inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is a decoded reference image based on the decoded coded video information.

[0113] The intra encoder (722) is configured to receive samples of the current block (e.g., the processing block), in some cases compare the block with blocks already encoded in the same image, generate quantization coefficients after transformation, and in some cases also generate intra prediction information (e.g., intra prediction direction information based on one or more intra coding techniques). In one example, the intra encoder (722) also computes an intra prediction result (e.g., a prediction block) based on the intra prediction information in the same image and a reference block.

[0114] A general controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines the mode of a block and provides control signals to the switch (726) based on that mode. For example, when the mode is intra-frame mode, the general controller (721) controls the switch (726) to select intra-frame mode results for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream; when the mode is inter-frame mode, the general controller (721) controls the switch (726) to select inter-frame prediction results for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

[0115] A residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or inter encoder (730). A residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, which can be cached in a memory circuit (not shown) and used as a reference image in some examples.

[0116] The entropy encoder (725) is configured to format the bitstream to include coded blocks. The entropy encoder (725) is configured to include various information in the bitstream according to an appropriate standard (e.g., the HEVC standard). In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. Note that, according to the disclosed subject matter, no residual information is present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

[0117] Figure 8An exemplary diagram of a video decoder (810) is shown. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In one example, the video decoder (810) is used instead of Figure 4 The video decoder (410) in the example.

[0118] exist Figure 8 In the example, the video decoder (810) includes, for example, Figure 8 The entropy decoder (871), inter-frame decoder (880), residual decoder (873), reconstruction module (874), and intra-frame decoder (872) are coupled together.

[0119] The entropy decoder (871) can be configured to reconstruct certain symbols representing the syntax elements constituting the encoded image from the encoded image. These symbols may include, for example, the mode in which the block is encoded (e.g., intra-mode, inter-mode, bidirectional prediction mode, merged sub-mode, or the latter two of another sub-mode) and prediction information (e.g., intra-prediction information or inter-prediction information) that can identify specific samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), respectively. The symbols may also include, for example, residual information in the form of quantized transform coefficients. In one example, when the prediction mode is inter-mode or bidirectional prediction mode, inter-prediction information is provided to the inter-decoder (880); and when the prediction type is intra-prediction type, intra-prediction information is provided to the intra-decoder (872). The residual information may undergo inverse quantization and be provided to the residual decoder (873).

[0120] The inter-frame decoder (880) is configured to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

[0121] The intra-frame decoder (872) is configured to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

[0122] The residual decoder (873) is configured to perform inverse quantization to extract the dequantized transform coefficients and process them to convert the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also require some control information (to include quantizer parameters (QP)), and this information can be provided by the entropy decoder (871) (the data path is not shown because this may only be a small amount of control information).

[0123] The reconstruction module (874) is configured to combine the residual output by the residual decoder (873) and the prediction results (output by the inter-frame or intra-frame prediction module, as appropriate) in the spatial domain to form a reconstruction block, which can be part of a reconstructed image, which in turn can be part of a reconstructed video. Note that other suitable operations, such as deblocking, can be performed to improve visual quality.

[0124] It should be noted that the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using any suitable technology. In one embodiment, the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using one or more processors that execute software instructions.

[0125] Various inter-frame prediction modes can be used in VVC. For a CU in inter-frame prediction, motion parameters may include MV, one or more reference picture indices, reference picture list usage indexes, and additional information about certain coded features used to generate inter-frame prediction samples. Motion parameters can be transmitted explicitly or implicitly in signaling form. When a CU is coded in skip mode, the CU can be associated with a PU and may not have significant residual coefficients, encoded motion vector increments or MV differences (e.g., Motion Vector Derivation, or MVD), or reference picture indices. A merging mode can be specified where the motion parameters of the current CU are obtained from neighboring CUs, including spatial and / or temporal candidate parameters and optional additional information, such as information introduced in VVC. Merging modes are applicable not only to skip mode but also to CUs in inter-frame prediction. In one example, another option for merging modes is to explicitly transmit motion parameters, where the MV, the corresponding reference picture index for each reference picture list, the reference picture list usage flag, and other information are explicitly transmitted in signaling form for each CU.

[0126] In one embodiment, for example in VVC, the VVC Test Model (VTM) reference software includes one or more refined inter-frame predictive coding tools, including: Extended Combined Prediction, Combined Motion Vector Difference (MMVD) mode, Adaptive Motion Vector Prediction with Symmetric MVD Signaling (AMVP) mode, Affine Motion Compensation Prediction, Sub-Block Based Temporal Motion Vector Prediction (SbTMVP), Adaptive Motion Vector Resolution (AMVR), Motion Field Storage (1 / 16 Luminance Sample MV Storage and 8×8 Motion Field Compression), Dual Prediction with CU-Level Weights (BCW), Bidirectional Optical Flow (BDOF), Prediction Refinement Using Optical Flow (PROF), Decoder-Side Motion Vector Refinement (DMVR), Combined Inter-Frame and Intra-Frame Prediction (CIIP), Geometric Partitioning Mode (GPM), etc. Intra-frame prediction and related methods will be described in detail below.

[0127] In some examples, extended merge predictions can be used. In one example, for instance, in VTM4, the merge candidate list is constructed by sequentially including the following five categories of candidates: spatial motion vector predictors (MVPs) from spatially adjacent CUs, temporal MVPs from juxtaposed CUs, historical MVPs (HMVPs) from first-in-first-out (FIFO) tables, pairwise average MVPs, and zero MVs.

[0128] The size of the merge candidate list can be transmitted in signaling form in the slice header. For example, in VTM4, the maximum allowed size of the merge candidate list is 6. For each CU encoded in merge mode, the index of the best merge candidate (e.g., the merge index) can be encoded using truncated univariate binarization (TU). The first binary of the merge index can be encoded using context (e.g., context-adaptive binary arithmetic coding (CABAC)), and the other binary can be encoded using bypass coding.

[0129] The following provides some examples of various merge candidate generation processes. In one embodiment, the spatial candidate derivation process is as follows. The spatial merge candidate derivation process in VVC can be the same as that in HEVC. In one example, from located at Figure 9 A maximum of four merge candidates can be selected from the candidates at the indicated positions. Figure 9 A spatial merging candidate location is shown according to one embodiment of the present disclosure. Reference Figure 9 The export order is B1, A1, B0, A0, and B2. Position B2 is only considered if any of the CUs at positions A0, B0, B1, and A1 are unavailable (e.g., because the CU belongs to another slice or another tile) or if intra-frame coding is being performed. After adding the candidate at position A1, the addition of the remaining candidates undergoes a redundancy check to ensure that candidates with the same motion information are excluded from the candidate list, thereby improving coding efficiency.

[0130] To reduce computational complexity, not all possible candidate pairings are considered in the redundancy check described above. Instead, only... Figure 10 The pairs connected by arrows are added to the candidate list only if the corresponding candidates used for redundancy checking do not have the same motion information. Figure 10 The illustration shows candidate pairs considered when performing redundancy checks on spatial merging candidates according to one embodiment of this disclosure. (Reference) Figure 10 The pairs connected by arrows include A1 and B1, A1 and A0, A1 and B2, B1 and B0, and B1 and B2. Therefore, candidates at positions B1, A0, and / or B2 can be compared with candidates at position A1, and candidates at positions B0 and / or B2 can be compared with candidates at position B1.

[0131] In one embodiment, the process of deriving time candidates is as follows. In one example, only one time merging candidate is added to the candidate list. Figure 11 An exemplary example of motion vector scaling for spatiotemporal merging candidates is shown. To derive the spatiotemporal merging candidate for the current CU (1111) in the current image (1101), the scaled MV (1121) can be derived from the juxtaposed CU (1112) belonging to the juxtaposed reference image (1104). (For example, Figure 11 (As shown by the dashed line in the text). In one example, the juxtaposed reference image (also called the co-position image) is a specific reference image used for temporal motion vector prediction. The juxtaposed reference image used for spatiotemporal motion vector prediction can be represented by a reference index in the syntax, such as in high-level syntax (e.g., image header, slice header).

[0132] The list of reference images used to export the co-located CU (1112) can be explicitly transmitted in signaling form in the slice header. For example... Figure 11 As shown by the dashed line, the scaled MV (1121) of the time-merging candidate can be obtained. The scaled MV (1121) can be scaled from the MV of the juxtaposed CU (1112) using the Picture Order Count (POC) distances tb and td. The POC distance tb can be defined as the POC difference between the current reference image (1102) and the current image (1101). The POC distance td can be defined as the POC difference between the juxtaposed reference image (1104) and the juxtaposed image (1103). The reference image index of the time-merging candidate image can be set to zero.

[0133] Figure 12Exemplary candidate positions (e.g., C0 and C1) for the current CU's temporal merge candidate positions are shown. A spatiotemporal merge candidate position can be selected from candidate positions C0 and C1. Candidate position C0 is located at the lower right corner of the shared CU (1210) of the current CU. Candidate position C1 is located at the center of the juxtaposed CU (1210) of the current CU. If the CU at candidate position C0 is unavailable, intra-frame encoded, or not in the current CTU line, candidate position C1 is used to derive the temporal merge candidate. Otherwise, for example, if the CU at candidate position C0 is available, intra-frame encoded, or in the current line of the CTU, candidate position C0 is used to derive the temporal merge candidate.

[0134] Merging with Motion Vector Difference (MMVD) mode can be used in skip or merge modes with motion vector representation methods. In MMVD mode, merge candidates used in VVC can be reused. A candidate can be selected from the merge candidates as a starting point (e.g., a motion vector predictor (MVP)) and further expanded through the MMVD mode. The MMVD mode can provide a new motion vector representation of a simplified signal. The motion vector representation includes a starting point and the MVD difference (MVD). In one example, the MVD is represented by the magnitude (or motion amplitude) and direction (e.g., motion direction) of the MVD.

[0135] The MMVD mode can use a merge candidate list, such as the candidate list used in VVC. In one embodiment, the MMVD mode only considers candidates of the default merge type (e.g., MRG_TYPE_DEFAULT_N). The starting point can be indicated or defined by a basic candidate index (IDX). The basic candidate index indicates one candidate (e.g., the best candidate) among the candidates (e.g., basic candidates) in the merge candidate list. Table 1 shows an example relationship between the basic candidate index and the corresponding starting point. A basic candidate index of 0, 1, 2, or 3 indicates that the corresponding starting point is the first MVP, second MVP, third MVP, or fourth MVP. In one example, if the number of basic candidates is equal to 1, the basic candidate IDX signal will not be transmitted in signaling form.

[0136] Table 1. Basic Candidate IDX

[0137] Basic candidate IDX 0 1 2 3 The Nth MVP First MVP Second MVP 3rd MVP 4th MVP

[0138] Distance indices can indicate motion magnitude information for the MVD, such as the magnitude of the MVD. For example, a distance index represents the distance (e.g., a predefined distance) from the starting point (e.g., the MVP indicated by the base candidate index) to the ending point. In one example, this distance is one of several predefined distances, as shown in Table 2. Table 2 shows example relationships between distance indices and their corresponding distances (in samples or pixels). One pixel in Table 2 represents either a sample or a pixel. For example, a distance index of 1 indicates a distance of 1 / 2 pixel or 1 / 2 sample.

[0139] Table 2. Distance from IDX

[0140]

[0141] The direction index can represent the direction of MVD relative to the starting point. A direction index can represent one of multiple directions, such as the four directions shown in Table 3. For example, a direction index of 00 indicates that the direction of MVD is the positive x-axis.

[0142] Table 3. Directional IDX

[0143] Directional IDX 00 01 10 11 x-axis + – N / A N / A y-axis N / A N / A + –

[0144] After sending the skip and merge flags, the MMVD flag can be transmitted as signaling. If the skip and merge flags are true, the MMVD flag can be parsed. In one example, if the MMVD flag equals 1, the MMVD syntax (e.g., including the distance index and / or direction index) can be parsed. If the MMVD flag is not equal to 1, the AFFINE flag can be parsed. If the AFFINE flag equals 1, the current data block is encoded using the AFFINE mode. If the AFFINE flag is not equal to 1, the skip / merge index of the skip / merge mode (e.g., the skip / merge mode used in VTM) can be parsed.

[0145] Figures 13-14 An example of the search process in MMVD mode is shown. By performing the search process, indices, including basic candidate indices, orientation indices, and / or distance indices, can be determined for the current block (1300) in the current image (or current frame) (1301).

[0146] The first motion vector MV (1311) and the second motion vector MV (1321) belonging to the first merge candidate are shown. The first merge candidate can be a merge candidate in a merge candidate list constructed for the current block (1300). The first MV (1311) and the second MV (1321) can be associated with two reference images (1302) and (1303) in the reference image lists L0 and L1, respectively. Therefore, Figure 13-14The two starting points (1411) and (1421) can be determined at reference images (1302) and (1303), respectively.

[0147] In one example, based on the starting point (1411) and the starting point (1421), multiple predefined points extending from the starting points (1411) and (1421) in the reference images (1302) and (1303) in the vertical direction (denoted by +Y or -Y) or the horizontal direction (denoted by +X and -X) can be evaluated. Figure 14 (As shown in 1-12). In one example, a pair of points reflecting each other relative to the corresponding starting point (1411) or starting point (1421) (e.g., a pair of points (1414) and (1424), or a pair of points (1415) and (1425)) can be used to determine a pair of MVs (1314) and MV (1324) or a pair of MVs (1315) and MV (1325), which can form the MV predictor (MVP) candidates for the current block (1300). The MVP candidates determined based on predefined points around the starting point (1411) and / or point (1421) can be evaluated. Reference Figure 13 The MVD (1312) size between the first MV (1311) and MV (1314) is 1S. The MVD (1322) size between the second MV (1321) and MV (1324) is 1S. Similarly, the MVD size between the first MV (1311) and MV (1315) is 2S. The MVD size between the second MV (1321) and MV (1325) is 2S.

[0148] In addition to the first merge candidate image, other available or valid merge candidate images in the merge candidate image list of the current block (1300) can be similarly evaluated. In one example, for a single prediction merge candidate, only one prediction direction related to one of the two reference image lists is evaluated.

[0149] In one example, the best MVP candidate can be determined based on the evaluation results. Therefore, the best merge candidate corresponding to the best MVP candidate can be selected from the merge list, and the direction and distance of movement can also be determined. For example, the basic candidate index can be determined based on the selected merge candidate and Table 1. Based on the selected MVP, for example, the MVP corresponding to the predefined point (1415) (or point (1425)), the direction and distance (e.g., 2S) of point (1415) relative to the starting point (1411) can be determined. The direction index and distance index can be determined accordingly based on Tables 2 and 3.

[0150] As mentioned above, in MMVD mode, two indices (e.g., a distance index and a direction index) can be used to indicate MVD. Alternatively, in MMVD mode, a single index can also be used to indicate MVD, for example, by using a table that pairs a single index with MVD.

[0151] Candidate reordering based on template matching (TM) can be used for certain prediction patterns, such as MMVD and affine MMVD patterns. In one embodiment, the MMVD offset is extended for both MMVD and affine MMVD patterns. Figure 15 Additional refinement positions along multiple diagonals are shown, for example, k×π / 8 diagonals, where k is an integer between 0 and 15. Additional refinement positions along multiple diagonal angles can increase the number of directions, for example, from 4 directions (e.g., +X, -X, +Y, and -Y) to 16 directions (e.g., k = 0, 1, 2, ..., 15). In one example, each of the 16 directions is represented by the angle between the +X direction and the direction indicated by the center point (1500) and one of the points 1-16. For example, point 1 represents the +X direction with an angle of 0 (i.e., k = 0), point 2 represents the direction with an angle of 1×π / 8 (i.e., k = 1), and so on.

[0152] TM can be executed in MMVD mode. In one example, for each MMVD refinement location, the TM cost can be determined based on the current template of the current block and one or more reference templates. The TM cost can be determined using any method, such as the Sum of Absolute Differences (SAD) (e.g., SAD cost), the Sum of Absolute Transformed Differences (SATD), the Sum of Squared Errors (SSE), the mean, variance, partial SAD, partial SSE, partial SATD, etc., after removing the SAD / SATD / SSE.

[0153] The current template for the current block can include any suitable samples, such as a row of samples above the current block and / or a column of samples to the left of the current block. Based on the TM cost (e.g., SAD cost) between the current template and the corresponding reference template for the refinement position, the MMVD refinement positions, for example, can be reordered for all possible MMVD refinement positions (e.g., 16×6 represents 16 directions and 6 amplitudes) for each base candidate (e.g., MVP). In one example, the top-level MMVD refinement position with the minimum TM cost (e.g., minimum SAD cost) is reserved as the available MMVD refinement position for MMVD index encoding. For example, a subset (e.g., 8) of the MMVD refinement positions with the minimum TM cost are used for MMVD index encoding. For example, the MMVD index indicates which of the subset of MMVD refinement positions with the minimum TM cost is selected to encode the current block. In one example, an MMVD index of 0 indicates that the current block is encoded using the MVD (e.g., the MMVD refinement position) corresponding to the minimum TM cost. For example, MMVD indexes can be binarized using Rice encoding with a parameter of 2.

[0154] In one embodiment, in addition to the MMVD offset extension described above, such as Figure 15 As shown, the affine MMVD reordering is also extended, where additional refinement positions are added along the k×π / 4 diagonal. After reordering, the top 1 / 2 refinement positions with the smallest TM cost (e.g., SAD cost) are retained to encode the current block.

[0155] In some embodiments, block-based affine transformation motion compensation prediction is applied. Figure 16A In this context, when using a 4-parameter affine model, the affine motion field of a block is described by the motion vectors (CPMV0 and CPMV1) of two control points (CP). Figure 16B In the case of a 6-parameter affine model, the affine motion field of a block is described by three control points CPMV0, CPMV1, and CPMV3, namely CP0, CP1, and CP2.

[0156] For a 4-parameter affine motion model, the motion vector at the sampling position (x, y) in the block can be derived as follows:

[0157]

[0158] For a 6-parameter affine motion model, the motion vector at the sampling position (x, y) in the block can be derived as:

[0159]

[0160] In formula 3-4, (mv 0x ,mv 0y(mv) is the motion vector of the top-left control point. 1x ,mv 1y (mv) is the motion vector of the upper right control point. 2x ,mv 2y ) is the motion vector of the lower left control point. In addition, the coordinates (x, y) are relative to the upper left corner of the corresponding block, and W and H represent the width and height of the corresponding block.

[0161] To simplify motion compensation prediction, sub-block-based affine transformation prediction is applied in some embodiments. For example, in Figure 17 In this study, a 4-parameter affine motion model is used, and two CPMVs are determined, namely... and To derive the motion vector for each 4×4 (sample) luma sub-block (1702) divided from the current block (1710), the motion vector (1701) of the center sample of each sub-block (1702) is calculated according to Equation 3 and rounded to 1 / 16 of the accuracy. A motion-compensated interpolation filter is then applied to generate a prediction for each sub-block (1702) using the derived motion vector (1701). The sub-block size for the chroma component is set to 4×4. The MV of the 4×4 chroma sub-block is calculated as the average of the MVs of the four corresponding 4×4 luma sub-blocks.

[0162] Similar to inter-frame prediction for translational motion, some embodiments employ two affine motion inter-frame prediction modes: affine merging mode and affine AMVP mode. The affine skip mode is similar to the affine merging mode, except that residuals can be added to the predicted samples in the affine merging mode, while no residuals are added in the affine skip mode. In the affine skip mode, significant residual coefficients are not used.

[0163] In some embodiments, the affine merging pattern can be used for CUs with a width and height both greater than or equal to 8. Affine merging candidates for the current CU can be generated based on motion information of spatially adjacent CUs. There can be a maximum of five affine merging candidates, which are indexed via signaling transmission, indicating which candidate the current CU will use. For example, the following three types of affine merging candidates can be used to form the affine merging candidate list:

[0164] Inherited affine merge candidates inferred from the CPMV of adjacent CUs;

[0165] Affine merging candidates derived using the translational MV of adjacent CUs; ​​and

[0166] Zero MV.

[0167] In some embodiments, there can be up to two inherited affine candidates, which are derived from the affine motion model of adjacent blocks, one from the left-adjacent CU and the other from the upper-adjacent CU. For example, the candidate block can be located at... Figure 9 The positions are shown. For the left predictor, the scan order is A0>A1; for the top predictor, the scan order is B0>B1>B2. Only the first inheritance candidate is selected on each side. No pruning check is performed between two inheritance candidates.

[0168] When identifying neighboring affine CUs, the CPMVs of the identified neighboring affine CUs are used to derive CPMV candidates from the affine merging list of the current CU. For example... Figure 18 As shown, the lower left adjacent block A of the current CU (1810) is encoded in affine mode. The motion vectors of the upper left, upper right, and lower left corners of the CU (1820) containing block A are obtained. and When encoding block A using a 4-parameter affine model, according to and Calculate the two CPMVs of the current CU(1810). and If block A uses a 6-parameter affine model encoding, then according to and Calculate the three CPMVs of the current CU (not shown).

[0169] Affine candidates are constructed by combining the translational motion information of each control point with that of its neighbors. For example... Figure 19 As shown, the motion information of the control points is derived from specified spatial and temporal neighboring points. CPMVk (k = 1, 2, 3, 4) represents the k-th control point. For CPMV1, blocks B2>B3>A2 are checked in sequence, and the MV of the first available block is used; for CPMV2, blocks B1>B0 are checked; for CPMV3, blocks A1>A0 are checked. If the TMVP of block T is available, it is used as CPMV4.

[0170] After obtaining the motion signatures (MVs) of the four control points, affine merging candidates are constructed based on this motion information. The following combinations of control point MVs are used sequentially for construction: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}.

[0171] Combining three CPMVs constructs a 6-parameter affine merging candidate, while combining two CPMVs constructs a 4-parameter affine merging candidate. To avoid issues during motion scaling, if the reference indices of the control points are different, the relevant control point MV combination is discarded.

[0172] After checking the inherited affine merge candidates and the constructed affine merge candidates, if the list is still not full, insert zero MV at the end of the merge candidate list.

[0173] To improve coding efficiency and reduce MV transmission overhead, sub-block-level MV refinement can be applied to extend CU-level spatiotemporal motion vector prediction (TMVP). In one example, the sub-block-based TMVP (SbTMVP) mode allows the inheritance of sub-block-level motion information from the juxtaposed reference picture. As mentioned above, the juxtaposed reference picture can be indicated by a reference index in the syntax (e.g., high-level syntax (e.g., picture header, slice header)).

[0174] Each sub-block of the current CU in the current image (e.g., the current CU with the larger size) can have corresponding motion information without explicitly transmitting the block partitioning structure or the corresponding motion information. For example, in SbTMVP mode, the motion information of each sub-block can be obtained through the following three steps: In the first step, the displacement vector (DV) of the current CU is derived. The DV can indicate a block in the juxtaposed reference image; for example, the DV points from the current block in the current image to a block in the juxtaposed reference image. Therefore, the block indicated by the DV is considered to be juxtaposed with the current block and is called the juxtaposed block of the current block; In the second step, the availability of SbTMVP candidates is checked, and the center motion (e.g., the center motion of the current CU) can be derived; In the third step, the sub-block motion information can be derived from the corresponding sub-block in the juxtaposed block using the DV. These three steps can be combined into one or two steps, or the order of the three steps can be adjusted.

[0175] TMVP candidate export derives the time MV from the juxtaposed blocks in the reference frame or reference picture. In contrast, in SbTMVP mode, a DV (e.g., a DV derived from the MV of the CU to the left of the current CU) can be applied to find the corresponding sub-block in the juxtaposed reference picture for each sub-block in the current CU.

[0176] If the corresponding sub-block is not intra-coded, the motion information of the current sub-block can be set as the center motion information of the juxtaposed block.

[0177] Various video coding standards support the SbTMVP mode, including, for example, VVC. For instance, similar to the TMVP mode in HEVC, in SbTMVP mode, the motion field (also called the motion information field or MV field) in the juxtaposed reference picture can be used to improve the MV prediction and the merging pattern of CUs in the current picture. In one example, the same juxtaposed reference picture used in TMVP mode is also used in SbTMVP mode. In one example, SbTMVP mode differs from TMVP mode in the following ways: (i) TMVP mode predicts motion information at the CU level, while SbTMVP mode predicts motion information at the sub-CU level; (ii) TMVP mode obtains the temporal MV from the juxtaposed block in the juxtaposed reference picture (e.g., the juxtaposed block is the lower right or center block relative to the current CU), while SbTMVP mode can apply a motion offset before obtaining the temporal motion information from the juxtaposed reference picture. In one example, the motion offset used in SbTMVP mode is obtained from the MV of a spatially adjacent block of the current CU.

[0178] Figures 20-21 An exemplary SbTMVP process used in the SbTMVP pattern is shown. For example, the SbTMVP process can predict the MV of sub-CUs (e.g., sub-blocks) within the current CU (e.g., the current block) (2001) in the current image (2111) in two steps. In the first step, the MV of the sub-CUs (e.g., sub-blocks) within the current CU (e.g., the current block) (2001) is checked. Figure 20-21 The spatial neighbors of the current block (2001) (e.g., A1). If the spatial neighbor (e.g., A1) has an MV (2121) that uses a juxtaposed reference picture (2112) as the reference picture for the spatial neighbor (e.g., A1), then the MV (2121) can be selected as the motion offset (or DV) to be applied to the current block (2001). If no such MV is recognized (e.g., an MV that uses a juxtaposed reference picture (2112) as the reference picture), then the motion offset or DV can be set to zero MV (e.g., (0, 0)). In some examples, if no such MV is recognized for spatial neighbor A1, the MVs of other spatial neighbors (e.g., A0, B0, B1, etc.) are checked.

[0179] In the second step, the motion offset or DV (2121) identified in the first step can be applied to the current block (2001) (e.g., adding DV (2121) to the coordinates of the current block) to obtain sub-CU level motion information (e.g., including MV and reference index) from the juxtaposed reference image (2112). Figure 21In the example shown, the motion offset or DV (2121) is set to the MV of the spatial neighbor A1 (e.g., block A1) of the current block (2001). For each sub-CU or sub-block (2131) in the current block (2001), the motion information of the sub-CU or sub-block (2131) can be derived using the motion information of the corresponding juxtaposed block (2101) in the juxtaposed reference picture (2112) (e.g., motion information of the smallest motion grid covering the center sample of the juxtaposed block (2101)). After identifying the motion information of the juxtaposed sub-CU (2132) in the juxtaposed block (2101), the motion information of the juxtaposed sub-CU (2132) can be converted into motion information (e.g., MV and one or more motion information, MV and one or more reference indices), for example, using a scaling method similar to the TMVP process used in HEVC, where temporal motion scaling is applied to align the reference picture of the temporal MV with the reference picture of the current CU.

[0180] The motion field of the current block (2001) derived from DV(2121) may include motion information of each sub-block (2131) in the current block (2001), such as MV and one or more related reference indices. The motion field of the current block (2001) may also be referred to as SbTMVP candidate and corresponds to DV(2121).

[0181] Figure 21 An example of the motion field or SbTMVP candidate (also known as the SbTMVP merge candidate) for the current block (2001) is shown. The motion information for a double-predicted sub-block (2131(1)) includes a first MV, a first index indicating the first reference image in reference image list 0 (L0), and a second MV and a second index indicating the second reference image in reference image list 1 (L1). In one example, the motion information for a single-predicted sub-block (2131(2)) includes the MV and index indicating the reference image in L0 or L1.

[0182] In one example, DV (2121) is applied to the center position of the current block (2001) to locate a shifted center position in the corresponding juxtaposed reference picture (2112). If the block including the shifted center position is not inter-coded, the SbTMVP candidate is considered unavailable. Otherwise, if the block including the shifted center position (e.g., the juxtaposed block (2101)) is inter-coded, the motion information of the center position of the current block (2001) (referred to as the center motion of the current block (2001)) can be derived from the motion information of the block including the shifted center position in the juxtaposed reference picture (2112). In one example, a scaling process can be used to derive the center motion of the current block (2001) from the motion information of the block including the shifted center position in the juxtaposed reference picture (2112). When an SbTMVP candidate is available, DV (2121) can be applied to find the corresponding sub-block (2132) in the juxtaposed reference picture (2112) for each sub-block (2131) of the current block (2001). The motion information of the corresponding sub-block (2132) can be used to derive the motion information of the sub-block (2131) in the current block (2001), in the same way as deriving the center motion of the current block (2001). In one example, if the corresponding sub-block (2132) is not inter-coded, the motion information of the current sub-block (2131) is set to the center motion of the current block (2001).

[0183] In some examples, such as in VVC, a combined sub-block-based merge list (including SbTMVP candidates and affine merge candidates) is used to transmit the sub-block-based merge pattern via signaling. The SbTMVP mode can be enabled or disabled via the Sequence Parameter Set (SPS) flag. If SbTMVP mode is enabled, an SbTMVP candidate (or SbTMVP predictor) can be added as the first entry in the sub-block-based merge list (including sub-block-based merge candidates), followed by affine merge candidates. The size of the sub-block-based merge list can be transmitted via signaling in the SPS. In one example, the maximum allowed size of the sub-block-based merge list in VVC is 5. In one example, the sub-block-based merge list includes multiple SbTMVP candidates.

[0184] In some examples, such as in VVC, the sub-CU size used in SbTMVP mode is fixed at 8×8, the same size used in affine merging mode. In one example, SbTMVP mode is only applicable to CUs with a width and height greater than or equal to 8. The sub-block size (e.g., 8×8) can be configured to other sizes, for example, 4×4 in an ECM software model used to explore outside of VVC. In one example, in AMVP mode, multiple juxtaposed reference images (e.g., two juxtaposed frames) are used to provide temporal motion information for SbTMVP and / or TMVP.

[0185] For example, offset vectors (e.g., Displacement Vector Offset (DVO)) can be used in SbTMVP mode to produce better matches. DVO can be an offset of the DV used in SbTMVP mode (e.g., the initial DV as the MV of the current CU's adjacent block). In one example, DVO is added to the DV to determine the updated DV'. In some examples, DVO is referred to as Motion Vector Offset (MVO). In one example, the updated DV' is the vector sum of the DV and DVO. By using DVO, the position of the juxtaposed CU corresponding to the current CU can be adjusted, thereby adjusting the MV field within the juxtaposed CU. When DVO is not zero, the updated DV' is used as a displacement vector to indicate the position of the juxtaposed CU, thus deriving SbTMVP candidates (or SbTMVP merge candidates). Reference Figure 21 The updated DV' can be used as DV(2121) to obtain the SbTMVP candidate.

[0186] This application describes a process for deriving SbTMVP candidates (or SbTMVP merge candidates) in SbTMVP mode using multiple displacement motion vectors (e.g., a DV predictor) with DVO (or motion vector offset). Signaling indexes are used to indicate which displacement motion vector is used as the initial displacement motion vector, thereby deriving the SbTMVP merge candidate. In one example, the DVO is determined using MMVD mode.

[0187] According to one embodiment of this disclosure, multiple DV predictors (DVPs) and DVOs can be used to determine the updated displacement vector (or updated DV') of the current CU (e.g., the current block) encoded in SbTMVP mode. The multiple DVPs can also be referred to as candidate DVPs. For the current data block, index information of a base index and DV offset information (or DVO information) indicating the DVO can be parsed in signaling form to determine the updated DV'. The base index can indicate which DVP to select from the multiple DVPs. The DVO can be used as the offset of the selected DVP. The position of the juxtaposed CU (e.g., the juxtaposed block) of the current CU (e.g., the current block) can be determined by using the selected DVP indicated by the base index and the DVO indicated by the DVO information. For example, the updated DV' is determined as the vector sum of the selected DVP and DVO. The updated DV' can be used as a displacement vector to indicate the position of the juxtaposed block, thereby determining SbTMVP merging candidates.

[0188] For example, a DVO can be indicated by signaling an index from a list of DVO candidates (DVO information includes indices). A predefined list of DVOs may include candidate DVOs. One or more indices can be signaled to indicate which DVO among the candidate DVOs can be selected as the DVO, where the DVO information may include one or more indices.

[0189] In one example, the DVO is transmitted in signaling form using the MMVD mode. For instance, the DVO is an MVD indicated by a direction index and / or a distance index, as described in Table 2-3. For example, the signaling includes two indices: a distance index indicating the size of the DVO and a direction index indicating the direction of the DVO, as described in Table 2-3.

[0190] In one embodiment, DVO can be transmitted directly as signaling, for example, using any signaling method for transmitting MVD in signaling form, such as AMVP mode, AMVR mode, etc. DVO can be signaled at different resolutions, for example, using 1 / 4, 1 / 2, 1, or 4 luma sample resolutions in AMVR mode.

[0191] In one embodiment, one or more DVP candidates from a plurality of DVP candidates may be determined based on the spatial candidates (e.g., MVs) of the corresponding spatial neighbors of the current block. Spatial neighbors may include Figure 9 The spatial neighbors (or candidate blocks) in the block are A0, A1, B0, B1, and B2. Spatial neighbors may include... Figure 20 The spatial neighbors of the current block are A0, A1, B0, and B1. The spatial neighbors of the current block can be checked in any suitable order, such as... Figure 9 and Figure 20 As shown. In one example, the reference image used to determine the spatial neighbors of the current block of a DVP candidate among multiple DVP candidates is a juxtaposed reference image.

[0192] In one embodiment, when the number of available spatial candidates for a corresponding spatial neighbor is less than a threshold, multiple DVP candidates are padded with zero vectors (e.g., (0, 0)). In one example, each available spatial candidate is used to determine one of the multiple DVP candidates. For example, if the number of DVP candidates determined from the available spatial candidates is less than a threshold, a zero vector is added to the multiple DVP candidates. Spatial neighbors corresponding to the available spatial candidates can be used for inter-frame prediction. In one example, the reference image associated with each available spatial candidate is a juxtaposed reference image.

[0193] refer to Figure 20In one example, spatial neighbors A1, B0, and B1 perform inter-frame prediction. A1 uses MV1 associated with reference image 1 for unified prediction. B0 uses MV2 associated with reference image 2 for single prediction. B1 uses MV3-MV4 associated with reference images 3-4 respectively for dual prediction. Reference images 1 and 4 are juxtaposed reference images, where reference images 1, 4, and the juxtaposed reference image are the same image. Reference images 2-3 are different from the juxtaposed reference image. In this example, the number of spatial neighbors (e.g., A0, A1, B0, and B1) is 4. The available spatial candidates are those associated with A1 and B1. Multiple DVP candidates include a first DVP determined based on MV1 and a second DVP determined based on MV4. In one example, the first DVP is MV1, and the second DVP is MV4.

[0194] In one example, if the threshold is 3, multiple candidate DVPs also include zero vectors.

[0195] In one embodiment, multiple DVP candidates can be derived from MVs or candidates in a merge candidate list. In one example, the merge candidate list is a regular merge candidate list, such as a regular merge / skip candidate list. The regular merge candidate list may differ from a sub-block-based merge list or a sub-block-based merge candidate list. Candidates in the regular merge candidate list may include any suitable candidates used in the regular merge / skip mode. Candidates may include spatial candidates (e.g., spatial MVPs from spatially adjacent CUs), temporal candidates (e.g., TMVPs from co-located CUs), HMVP candidates, pairwise average candidates (e.g., pairwise average MVPs), and / or zero MVs. A pairwise average MVP can be generated using two existing candidates from the regular merge candidate list. The regular merge / skip mode may differ from other merge / skip modes, such as MMVD mode, CIIP mode, and GPM mode.

[0196] In one example, multiple DVP candidates are derived from a subset of candidates in the merged candidate list, where the candidate subset does not include time candidates (e.g., TMVP).

[0197] In one embodiment, a default MV (including, but not limited to, MVs pointing from the top-left corner of the current block to one or more positions of the juxtaposed block in the juxtaposed reference image) may be included among multiple DVP candidates. One or more positions of the juxtaposed block may include the middle position, bottom-right position, right position, left position, bottom-left position, and / or top-right position of the juxtaposed block in the juxtaposed reference image.

[0198] The current block's sub-block-based merge list (or sub-block-based merge candidate list) can include sub-block-based merge candidates or sub-block merge candidates. In one example, the sub-block-based merge candidates include multiple SbTMVP merge candidates. Each of the multiple SbTMVP merge candidates can be determined based on a corresponding one of the multiple DVP candidates. For example, the updated DV' is determined based on a corresponding one of the multiple DVP candidates and the DVO. This can be used... Figure 20-21 The SbTMVP pattern described in [the document] determines the corresponding SbTMVP merge candidate block (e.g., motion information of each sub-block in the current block corresponding to one of the multiple DVP candidate blocks). Therefore, each of the multiple SbTMVP merge candidate blocks can correspond to a corresponding candidate in the multiple DVP candidate blocks.

[0199] In one example, the sub-block-based merge list is a combined list of sub-block-based merges that also includes affine merge candidates, such as those used in affine merge / skip modes. Affine merge candidates can include affine inheritance candidates and affine build candidates.

[0200] In one embodiment, the sub-block-based merge list includes multiple SbTMVP merge candidates and affine merge candidates. Signaling flags (e.g., DVO flags) can be used to indicate that a DVO can be applied to a first subset of the sub-block-based merge candidates. In one example, the first subset includes multiple SbTMVP merge candidates but excludes affine merge candidates. Which SbTMVP merge candidate is selected from the first subset of the sub-block-based merge candidates can be indicated by a base index. Since each of the multiple SbTMVP merge candidates corresponds to a corresponding one of the multiple DVP candidates, the base index indicates the selected DVP corresponding to the selected SbTMVP merge candidate.

[0201] In one example, the DVO flag is true, indicating that the DVO will be used with the selected DVP among multiple DVP candidates. In another example, the DVO flag is the MMVD flag in MMVD mode, thus indicating that MMVD mode is used to determine the DVO from the DVO information. A first subset of subblock-based merge candidates can be specified so that the DVO is applied to the first subset of subblock-based merge candidates. In one example, the DVO is indicated using MMVD mode, as described above. The base index (e.g., subblock_mmvd_base_index) can be transmitted in signaling form to indicate which candidate in the first subset of subblock-based merge candidates is used.

[0202] In one embodiment, the sub-block-based merge list includes a second subset of sub-block-based merge candidates. This second subset includes K0 SbTMVP merge candidates (e.g., multiple SbTMVP merge candidates) and K1 affine merge candidates. A base index indicates which sub-block-based merge candidate from the second subset of sub-block-based merge candidates will be selected to encode the current block, wherein the selected sub-block-based merge candidate corresponds to the corresponding SbTMVP merge candidate among the selected DVP and multiple SbTMVP merge candidates.

[0203] In one example, the top K0 SbTMVP merge candidates in the current block are included in the second subset. In another example, the first K1 affine merge candidate of the current block is included in the second subset. K0 and K1 are both positive integers. K0 can be transmitted in signaling form using high-level (e.g., above CU level or block level) syntax, such as Sequence Parameter Set (SPS), Picture Parameter Set (PPS), tile header, tile group header, slice header, picture header, or similar headers. In one example, K0 is 2. K1 can be transmitted in signaling form using high-level syntax, such as SPS, PPS, tile header, tile group header, slice header, picture header, etc.

[0204] In one example, the selection of SbTMVP merge candidates is determined by the DVP candidates in the list derived by DV.

[0205] In one embodiment, the current block uses two sub-block-based merge lists: an affine merge candidate list and an SbTMVP merge candidate list. The affine merge candidate list and the SbTMVP merge candidate list can be independent and can be constructed separately. For example, the SbTMVP merge candidate list includes multiple SbTMVP merge candidates but does not include affine merge candidates. A base index indicates which SbTMVP merge candidate from the SbTMVP merge candidate list to select, where the selected SbTMVP merge candidate corresponds to the DVP selected from the multiple DVP candidates.

[0206] Flags indicating the merge mode or MMVD mode can be transmitted via signaling, indicating that the merge mode or MMVD mode has been applied to the current block. The affine flag indicating the affine merge mode and the SbTMVP flag indicating the SbTMVP mode can be transmitted separately via signaling. If the SbTMVP flag is true, an SbTMVP merge candidate list can be constructed.

[0207] In one example, a first flag and a second flag are used to indicate whether the SbTMVP mode is used. The first flag indicates whether a sub-block-based merge mode is applied to the current block. If the first flag (e.g., true) indicates that a sub-block-based merge mode is applied to the current block, a second flag is transmitted in signaling form to indicate whether the sub-block-based merge mode is SbTMVP mode. The second flag also indicates whether an affine merge candidate list or an SbTMVP merge candidate list is built for the current block. For example, if the second flag indicates that the sub-block-based merge mode is SbTMVP mode, an SbTMVP merge candidate list is built for the current block.

[0208] In one embodiment, the sub-block-based merge list includes a sub-block-based merge candidate list. Each sub-block-based merge candidate includes only one SbTMVP candidate, which may or may not include affine merge candidates. The number of SbTMVP candidates in the sub-block-based merge list is 1. A first index can be transmitted in signaling form to indicate which sub-block-based merge candidate is selected in the sub-block-based merge list. The selected single SbTMVP candidate block indicates that the SbTMVP pattern is applied to the current block. A second index can be transmitted in signaling form to indicate which DVP is selected from multiple DVP candidates to construct the single SbTMVP candidate. The second index can be a base index (e.g., represented by the syntax sbtMVp_mMVd_base_index). In one example, the updated DV' is obtained by the vector sum of the selected DVP and DVO. Motion information for each sub-block in the current block can be obtained from the motion information of the corresponding sub-block in the juxtaposed block determined based on the updated DV', such as... Figures 20-21 Therefore, a single SbTMVP candidate block can be determined, wherein a single SbTMVP candidate block includes motion information of each sub-block in the current block.

[0209] In one example, the value of the base index (e.g., the syntax sbtmvp_mmvd_base_index) is determined by which DVP candidate value in the list from which the DV originates. For example, the value of the base index points to the position of the expected DVP candidate in the constructed list.

[0210] Figure 22The flowchart shown outlines a process (2200) according to an embodiment of the present disclosure. The process (2200) can be used in a video encoder. The process (2200) can be executed by a means for video encoding including processing circuitry. In various embodiments, the process (2200) is executed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video encoder (403), processing circuitry performing the functions of a video encoder (603), etc. In some embodiments, the process (2200) is implemented in software instructions, so that when the processing circuitry executes the software instructions, the processing circuitry executes the process (2200). The process begins at (S2201) and proceeds to (S2210).

[0211] In (S2210), the displacement vector predictor (DVP) and the offset of the displacement vector (DV) for the current block in the current image can be determined. The current block comprises multiple sub-blocks and can be encoded using the sub-block-based temporal motion vector prediction (SbTMVP) mode.

[0212] In one example, multiple DVP candidates are determined based on the motion vectors (MVs) of the spatial neighbors of the current block, where the reference image of each spatial neighbor used to determine one of the multiple DVP candidates is a juxtaposed reference image.

[0213] If the number of MVs of spatial neighbors is less than the threshold, zero MVs can be inserted into multiple DVP candidates.

[0214] In one example, multiple DVP candidates are determined based on candidates in a merged candidate list. Candidates may include at least one of the following: (a) Spatial Motion Vector Predictor (MVP) candidates, (b) History-Based Motion Vector Predictor (HMVP) candidates, (c) Pairwise Averaged Candidates, or (d) Zero Motion Vector (MV) candidates. Temporal Motion Vector Predictor (TMVP) candidates are excluded. The reference image for each candidate used to determine one of the multiple DVP candidates may be a juxtaposed reference image.

[0215] In (S2220), the DV of the current block can be determined based on the DVP and DV offset of the current block. The DV indicates the block in the juxtaposed reference image. This block is juxtaposed with the current block and is referred to as the juxtaposed block of the current block. In one example, the DV is determined as the vector sum of the DVP and DV offset.

[0216] In (S2230), the motion information of the sub-blocks in the multiple sub-blocks is determined based on the motion information of the corresponding sub-blocks in the juxtaposed block.

[0217] In (S2240), the sub-blocks in the multiple sub-blocks are encoded based on the motion information of the sub-blocks in the multiple sub-blocks.

[0218] Then, process (2200) proceeds to (S2299) and terminates.

[0219] The process (2200) can be appropriately adjusted according to different situations, and the steps in the process (2200) can be adjusted accordingly. One or more steps in the process (2200) can be modified, omitted, repeated, and / or combined. The process (2200) can be implemented using any suitable order. Additional steps can be added.

[0220] In one example, a sub-block-based merge list is constructed for the current block. The sub-block-based merge candidates in the sub-block-based merge list can include multiple SbTMVP merge candidates and at least one affine merge candidate. Each of the multiple SbTMVP merge candidates can correspond to a specific one of the multiple DVP candidates.

[0221] In one example, the basic index of the DVP among the multiple DVP candidates for the current block is encoded and included in the bitstream.

[0222] In one example, the base index indicates which SbTMVP merge candidate from a subset of sub-block-based merge candidates to select as a merge candidate for multiple SbTMVP merge candidates, with the selected SbTMVP merge candidate corresponding to the DVP.

[0223] In one example, a first number K0 of multiple SbTMVP merge candidates and a second number K1 of at least one affine merge candidate are transmitted in signaling form in the high-level syntax. K0 and K1 are positive integers. The base index indicates which sub-block-based merge candidate from the subset of sub-block-based merge candidates to select, the subset including (i) multiple SbTMVP merge candidates and (ii) at least one affine merge candidate. The selected sub-block-based merge candidate corresponds to one of the DVP and multiple SbTMVP merge candidates.

[0224] In one example, an SbTMVP merge candidate list is constructed for the current block, comprising multiple SbTMVP merge candidates. Each SbTMVP merge candidate corresponds to a corresponding one of multiple DVP candidates. The SbTMVP merge candidate list does not include affine merge candidates.

[0225] In one embodiment, a merge mode or a merge motion vector difference (MMVD) mode is applied to the current block.

[0226] The flag is encoded in the bitstream and transmitted in signaling form, indicating that the SbTMVP mode is applied to the current block.

[0227] In one example, a first flag is encoded in the bitstream and transmitted as signaling, indicating that a sub-block-based merging mode is applied to the current block. A second flag is encoded in the encoded video bitstream and transmitted as signaling, indicating that the sub-block-based merging mode is SbTMVP mode.

[0228] In one example, a sub-block-based merge list is constructed for the current block. The sub-block-based merge list includes one SbTMVP merge candidate and at least one affine merge candidate, where one SbTMVP merge candidate corresponds to a DVP. An index pointing to an SbTMVP merge candidate in the sub-block-based merge list can be encoded in the bitstream and transmitted as signaling, where the index indicates that the SbTMVP mode is applied to the current block.

[0229] In one embodiment, the current block, comprising multiple sub-blocks, is encoded in SbTMVP mode. Multiple DVP candidates for the current block can be determined. Each DVP candidate is derived from one or more MVs. The DV of the current block can be determined based on the DVPs among the multiple DVP candidates and the DV offset of the current block. The DV indicates the block juxtaposed with the current block in the juxtaposed reference image. Sub-blocks among the multiple sub-blocks can be encoded based on the motion information of the corresponding sub-blocks in the juxtaposed block.

[0230] Figure 23A A flowchart outlining a process (2300A) according to an embodiment of the present disclosure is shown. Process (2300A) can be used in a video decoder. Process (2300A) can be executed by means of a video encoding apparatus, which may include receiving circuitry and processing circuitry. In various embodiments, process (2300A) is executed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video decoder (410), processing circuitry performing the functions of a video decoder (510), etc. In some embodiments, process (2300A) is implemented in software instructions, so that when the processing circuitry executes the software instructions, the processing circuitry executes process (2300A). The process begins at (S2301) and proceeds to (S2310).

[0231] At (S2310), the basic index and displacement vector (DV) offset information of the current block in the current picture can be received from the encoded video bitstream. The current block may include multiple sub-blocks reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode. The basic index may indicate the DV predictor (DVP) among multiple DVP candidates of the current block.

[0232] In one example, multiple DVP candidates are determined based on the motion vectors (MVs) of the spatial neighbors of the current block, where the reference image of each spatial neighbor used to determine one of the multiple DVP candidates is a juxtaposed reference image.

[0233] If the number of MVs of spatial neighbors is less than the threshold, zero MVs can be inserted into multiple DVP candidates.

[0234] In one example, multiple DVP candidates are determined based on candidates in a merged candidate list. Candidates may include at least one of the following: (a) Spatial Motion Vector Predictor (MVP) candidates, (b) History-Based Motion Vector Predictor (HMVP) candidates, (c) Pairwise Averaged Candidates, or (d) Zero Motion Vector (MV) candidates. Temporal Motion Vector Predictor (TMVP) candidates are excluded. The reference image for each candidate used to determine one of the multiple DVP candidates may be a juxtaposed reference image.

[0235] At (S2320), the DV of the current block can be determined based on the DVP of the current block and the DV offset of the current block indicated by the DV offset information. The DV can indicate a block in the juxtaposed reference picture. This block is juxtaposed with the current block and is referred to as the juxtaposed block of the current block. In one example, the DV is determined as the vector sum of the DVP and the DV offset.

[0236] In (S2330), the motion information of the sub-blocks in multiple sub-blocks can be determined based on the motion information of the corresponding sub-blocks in the juxtaposed block.

[0237] In (S2340), the sub-blocks in the multiple sub-blocks can be reconstructed based on the motion information of the sub-blocks in the multiple sub-blocks.

[0238] Then, process (2300A) proceeds to (S2399) and terminates.

[0239] The process (2300A) can be appropriately adjusted according to different situations, and the steps in the process (2300A) can be adjusted accordingly. One or more steps in the process (2300A) can be modified, omitted, repeated, and / or combined. The process (2300A) can be implemented using any suitable order. Additional steps can be added.

[0240] In one example, a merge list of sub-blocks of the current block is constructed. The merge candidates for sub-blocks in the merge list can include multiple SbTMVP merge candidates and at least one affine merge candidate. Each of the multiple SbTMVP merge candidates can correspond to a corresponding one of the multiple DVP candidates.

[0241] In one example, the base index indicates which SbTMVP merge candidate from the subset of sub-block merge candidates should be selected, which is a subset of multiple SbTMVP merge candidates, and the selected SbTMVP merge candidate corresponds to the DVP.

[0242] In one example, the high-level syntax transmits a first number K0 of multiple SbTMVP merge candidates and a second number K1 of at least one affine merge candidate in signaling form. K0 and K1 are positive integers. The base index indicates which sub-block-based merge candidate from the subset of sub-block-based merge candidates to select, the subset including (i) multiple SbTMVP merge candidates and (ii) at least one affine merge candidate, the selected sub-block-based merge candidate corresponding to one of the DVP and multiple SbTMVP merge candidates.

[0243] In one example, an SbTMVP merge candidate list is constructed for the current block, comprising multiple SbTMVP merge candidates. Each SbTMVP merge candidate corresponds to a corresponding one of multiple DVP candidates. The SbTMVP merge candidate list does not include affine merge candidates.

[0244] In one embodiment, a merge mode or a merge motion vector difference (MMVD) mode is applied to the current block.

[0245] Receive a flag from the encoded video bitstream that indicates that the SbTMVP mode is applied to the current block.

[0246] In one example, a first flag is received from the encoded video bitstream, indicating that a sub-block-based merging mode is applied to the current block. A second flag is received from the encoded video bitstream, indicating that the sub-block-based merging mode is SbTMVP mode.

[0247] In one example, a sub-block-based merge list is constructed for the current block. The sub-block-based merge list includes one SbTMVP merge candidate and at least one affine merge candidate, where the SbTMVP merge candidate corresponds to a DVP. An index pointing to an SbTMVP merge candidate in the sub-block-based merge list can be received, where the index indicates that the SbTMVP pattern is applied to the current block.

[0248] Figure 23BThe flowchart shown outlines a process (2300B) according to an embodiment of this disclosure. Process (2300B) is a variation of process (2300A). This process (2300B) can be used in a video decoder. This process (2300B) can be executed by an apparatus for video encoding, which may include receiving circuitry and processing circuitry. In various embodiments, process (2300B) is executed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video decoder (410), processing circuitry performing the functions of a video decoder (510), etc. In some embodiments, process (2300B) is implemented in software instructions, so that when the processing circuitry executes the software instructions, the processing circuitry executes process (2300B). The process begins at (S2302) and proceeds to (S2312).

[0249] In step (S2312), the encoded video bitstream including the current image is received. The current image includes the current block. The current block includes multiple sub-blocks.

[0250] In (S2322), based on the syntax elements in the encoded video bitstream, it is determined that the current block, which includes multiple sub-blocks, is encoded in the Sub-Block-Based Temporal Motion Vector Prediction (SbTMVP) mode.

[0251] In (S2332), multiple displacement vector (DV) predictor (DVP) candidates for the current block can be determined. Each DVP candidate can be derived from one or more motion vectors (MV).

[0252] In (S2342), the base index and DV offset of the current block can be received. The base index indicates the DVP among multiple DVP candidates of the current block.

[0253] In (S2352), the DV of the current block can be described based on the DVP indicated among multiple DVP candidates of the current block and the DV offset of the current block. The DV indicates the block juxtaposed with the current block in the juxtaposed reference picture.

[0254] In (S2362), the sub-blocks in multiple sub-blocks can be reconstructed based on the motion information of the corresponding sub-blocks in the juxtaposed block.

[0255] Then, process (2300B) proceeds to (S2392) and terminates.

[0256] Various embodiments used in process (2300A) can be applied to process (2300B). Process (2300B) can be appropriately adapted to different situations, and the steps in process (2300B) can be adjusted accordingly. One or more steps in process (2300B) can be modified, omitted, repeated, and / or combined. Process (2300B) can be implemented using any suitable order. Additional steps can be added.

[0257] The embodiments in this disclosure can be used individually or in any combination in any order. Furthermore, each method (or embodiment), encoder, and decoder can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0258] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 24 A computer system (2400) suitable for implementing some embodiments of the disclosed subject matter is shown.

[0259] Computer software can be coded using any suitable machine code or computer language. It can undergo assembly, compilation, linking, or similar mechanisms to create code containing instructions that can be executed directly or through interpretation, microcode execution, or other methods by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc. These instructions can execute on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things (IoT) devices.

[0260] Figure 24 The components of the computer system (2400) shown are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement on any component or combination of components shown in the exemplary embodiments of the computer system (2400).

[0261] The computer system (2400) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users via, for example, tactile input (e.g., key presses, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0262] The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard (2401), mouse (2402), touchpad (2403), touch screen (2410), data glove (not shown), joystick (2405), microphone (2406), scanner (2407), and camera (2408).

[0263] The computer system (2400) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (2410), data gloves (not shown), or joystick (2405), but may also include tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (2409), headphones (not shown)), visual output devices (e.g., screens (2410), including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light-emitting diode (OLED) screens, each with or without touchscreen input capability, each with or without tactile feedback capability—some of which are capable of outputting two-dimensional or more than three-dimensional visual outputs in a manner such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0264] The computer system (2400) may also include human-accessible storage devices and their associated media, such as optical media (2420) having CD / DVD or similar media CD / DVDROM / RW, thumb drives (2422), removable hard disk drives or solid-state drives (2423), conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated read-only memory (ROM) / application-specific integrated circuit (ASIC) / programmable logic device (PLD) devices such as security dongles (not shown).

[0265] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0266] The computer system (2400) may also include an interface (2454) to one or more communication networks (2455). The network may be wireless, wired, or fiber optic. The one or more communication networks (2455) may also be local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), vehicular and industrial networks, real-time networks, latency-tolerant networks, etc. Examples of networks include LANs such as Ethernet and wireless LANs; cellular networks including Global System for Mobile Communications (GSM), 3G, 4G, 5G, and Long-Term Evolution (LTE); cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television; and vehicular and industrial networks including Controller Area Network (CANBus), etc. Some networks typically require external network interface adapters that connect to certain general-purpose data ports or peripheral buses (2449) (e.g., the Universal Serial Bus (USB) port of a computer system (2400)); others are typically integrated into the core of the computer system (2400) via connections to system buses as described below (e.g., an Ethernet interface in a PC computer system or a cellular network interface in a smartphone computer system). Using any of these networks, the computer system (2400) can communicate with other entities. This communication can be unidirectional and receive-only (e.g., broadcast television), unidirectional and transmit-only (e.g., to a CANbus device), or bidirectional, e.g., to other computer systems using local or wide-area digital networks. As mentioned above, certain protocols and protocol stacks can be used on each of these networks and network interfaces.

[0267] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces can be attached to the core (2440) of the computer system (2400).

[0268] The core (2440) may include one or more central processing units (CPU) (2441), graphics processing units (GPUs) (2442), dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) (2443), task-specific hardware accelerators (2444), graphics adapters (2450), etc. These devices, along with read-only memory (ROM) (2445), random access memory (2446), and internal mass storage such as internal non-user-accessible hard disk drives (HDDs) and solid-state drives (SSDs) (2447), may be connected via a system bus (2448). In some computer systems, the system bus (2448) may be accessed as one or more physical connectors to allow for the expansion of additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (2448) or via a peripheral bus (2449). In one example, a screen (2410) may be connected to a graphics adapter (2450). The architecture of peripheral buses includes Peripheral Component Interconnect (PCI), USB, etc.

[0269] The CPU (2441), GPU (2442), FPGA (2443), and accelerator (2444) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (2445) or Random Access Memory (RAM) (2446). Transient data can also be stored in RAM (2446), while permanent data can be stored, for example, in internal mass storage (2447). Fast storage and retrieval of any storage device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (2441), GPUs (2442), mass storage (2447), ROM (2445), RAM (2446), etc.

[0270] Computer-readable media may contain computer code for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or may be of a type known and available to those skilled in the art of computer software.

[0271] By way of example and not limitation, a computer system having an architecture (2400), particularly a core (2440), can execute functions resulting from software contained in one or more tangible computer-readable media via a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such a computer-readable medium can be a medium associated with a user-accessible mass storage as described above, as well as some memory of the non-transitory core (2440), such as internal mass storage (2447) or ROM (2445). Software implementing various embodiments of this disclosure can be stored in such a device and executed by the core (2440). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core (2440), and particularly the processor therein (including a CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (2446) and modifying such data structures according to software-defined processes. In addition, or alternatively, the computer system may provide functionality as a result of hard-wired or otherwise incorporated logic (e.g., accelerator (2444)) that may replace or operate with software to perform the specific process or a specific portion of the specific process described herein. References to software may include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuitry storing software for execution (e.g., integrated circuit (IC)), circuitry containing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0272] Appendix A: Abbreviations

[0273] Joint Exploration Model (JEM)

[0274] Versatile Video Coding (VVC)

[0275] Benchmark Set (BMS)

[0276] Motion Vector (MV)

[0277] High Efficiency Video Coding (HEVC)

[0278] Supplemental Enhancement Information (SEI)

[0279] Video User Information (VUI)

[0280] Group of Pictures (GOP)

[0281] Transforming Unit (TUs)

[0282] Prediction Unit (PU)

[0283] Coding Tree Unit (CTU)

[0284] Coding Tree Block (CTB)

[0285] Prediction Block (PB)

[0286] Assume a Hypothetical Reference Decoder (HRD)

[0287] Signal-to-noise ratio (SNR)

[0288] Central Processing Unit (CPU)

[0289] Graphics Processing Unit (GPU)

[0290] Cathode ray tube (CRT)

[0291] Liquid Crystal Display (LCD)

[0292] Organic Light-Emitting Diode (OLED)

[0293] Compact Disc (CD)

[0294] Digital Versatile Disc (DVD)

[0295] Read-Only Memory (ROM)

[0296] Random Access Memory (RAM)

[0297] Application-Specific Integrated Circuit (ASIC)

[0298] Programmable Logic Device (PLD)

[0299] Local Area Network (LAN)

[0300] Global System for Mobile Communications (GSM)

[0301] Long-Term Evolution (LTE)

[0302] Controller Area Network (CANBus)

[0303] Universal Serial Bus (USB)

[0304] Peripheral Component Interconnect (PCI)

[0305] Field-Programmable Gate Array (FPGA)

[0306] Solid State Drive (SSD)

[0307] Integrated Circuit (IC)

[0308] Control Unit (CU)

[0309] While several exemplary embodiments have been described in this disclosure, numerous modifications, variations, and various alternative equivalents are possible and fall within the scope of this disclosure. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, although not expressly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

Claims

1. A method for video decoding in a decoder, characterized in that, include: Receive an encoded video bitstream including the current image, wherein the current image includes the current block, and the current block includes multiple sub-blocks; Based on the syntax elements in the encoded video bitstream, it is determined that the current block, which includes the plurality of sub-blocks, is encoded in the sub-block-based temporal motion vector prediction SbTMVP mode; Determine multiple displacement vector predictor (DVP) candidates for the current block, each DVP candidate being derived from one or more motion vectors (MV). Receive the base index and displacement vector DV offset of the current block, the base index indicating the DVP among multiple DVP candidates of the current block; The DV of the current block is determined based on the DVP indicated in the plurality of DVP candidates of the current block and the DV offset of the current block, wherein the DV indicates the block juxtaposed with the current block in the juxtaposed reference picture; The sub-blocks in the plurality of sub-blocks are reconstructed based on the motion information of the corresponding sub-blocks in the block juxtaposed with the current block; The method further includes: constructing a sub-block-based merge candidate list for the current block, wherein the sub-block-based merge candidates in the sub-block-based merge candidate list include a plurality of SbTMVP merge candidates and at least one affine merge candidate, and each of the plurality of SbTMVP merge candidates corresponds to a corresponding one of the plurality of DVP candidates.

2. The method according to claim 1, characterized in that, Determining the DV includes: The DV is defined as the vector sum of the DVP and the DV offset.

3. The method according to claim 1, characterized in that, Also includes: The plurality of DVP candidates are determined based on the MV of the spatial neighbors of the current block, and the reference image for each spatial neighbor of one of the plurality of DVP candidates is the juxtaposed reference image.

4. The method according to claim 3, characterized in that, Determining the plurality of DVP candidates includes: In response to the fact that the number of MVs of the spatial neighbors is less than a threshold, zero MVs are inserted into the plurality of DVP candidates.

5. The method according to claim 1, characterized in that, Also includes: The plurality of DVP candidates are determined based on candidates in a merged candidate list, which includes at least one of (a) Spatial Motion Vector Predictor (MVP) candidates, (b) History-Based Motion Vector Predictor (HMVP) candidates, (c) Pairwise Average candidates, or (d) Zero Motion Vector (MV) candidates. The merged candidate list does not include Temporal Motion Vector Predictor (TMVP) candidates. The reference image for each candidate in the merged candidate list is the juxtaposed reference image used to determine one of the plurality of DVP candidates.

6. The method according to claim 1, characterized in that, The base index indicates which SbTMVP merge candidate from the subset of the sub-block-based SbTMVP merge candidates should be selected as one of the plurality of SbTMVP merge candidates, and the selected SbTMVP merge candidate corresponds to the DVP.

7. The method according to claim 1, characterized in that, The first number K0 of the plurality of SbTMVP merging candidates and the second number K1 of the at least one affine merging candidate are transmitted in signaling form in the high-level syntax, where K0 and K1 are positive integers. as well as The basic index indicates which sub-block-based merge candidate from the subset of the sub-block-based merge candidates to select, the subset including (i) the plurality of SbTMVP merge candidates and (ii) the at least one affine merge candidate, the selected sub-block-based merge candidate corresponding to one of the DVP and the plurality of SbTMVP merge candidates.

8. The method according to claim 1, characterized in that, The method further includes: The sub-block-based merge candidate list for constructing the current block, wherein the sub-block-based merge candidates include multiple SbTMVP merge candidates and at least one affine merge candidate, and each of the multiple SbTMVP merge candidates corresponds to a corresponding one of the multiple DVP candidates, is replaced with the following steps: Construct an SbTMVP merge candidate list for the current block that includes multiple SbTMVP merge candidates, each of the multiple SbTMVP merge candidates corresponding to a corresponding one of the multiple DVP candidates, and the SbTMVP merge candidate list does not include affine merge candidates.

9. The method according to claim 8, characterized in that, Apply the merge mode to the current block; and The method includes receiving a flag from the encoded video bitstream, the flag indicating that the SbTMVP mode is applied to the current block.

10. The method according to claim 8, characterized in that, Apply the merge mode to the current block; as well as The method includes: A first flag is received from the encoded video bitstream, the first flag indicating that the sub-block-based merging mode is applied to the current block; as well as A second flag is received from the encoded video bitstream, the second flag indicating that the sub-block-based merging mode is the SbTMVP mode.

11. The method according to claim 1, characterized in that, Also includes: The method further includes: The sub-block-based merge candidate list for constructing the current block, wherein the sub-block-based merge candidates include multiple SbTMVP merge candidates and at least one affine merge candidate, and each of the multiple SbTMVP merge candidates corresponds to a corresponding one of the multiple DVP candidates, is replaced with the following steps: Construct a sub-block-based merge candidate list for the current block, the sub-block-based merge candidate list including one SbTMVP merge candidate and at least one affine merge candidate, the one SbTMVP merge candidate corresponding to the DVP; And receive an index pointing to one of the SbTMVP merge candidates in the sub-block-based merge candidate list, the index of the SbTMVP merge candidate indicating that the SbTMVP mode is applied to the current block.

12. An apparatus for video decoding, characterized in that, include: The processing circuit is configured to implement the method according to any one of claims 1-11.

13. A non-transitory computer-readable storage medium for storing instructions, characterized in that, When executed by at least one processor, the instructions cause the at least one processor to perform the method according to any one of claims 1-11.

14. A method for video encoding, characterized in that, include: Determine the displacement vector predictor (DVP) and DV offset of the current block in the current image, wherein the current block includes multiple sub-blocks; Based on the DVP and DV offset of the current block, the DV of the current block is determined, wherein the DV indicates a juxtaposed block in a juxtaposed reference image, the juxtaposed block in the juxtaposed reference image being juxtaposed with the current block; Based on the motion information of the corresponding sub-blocks in the juxtaposed block, determine the motion information of the sub-blocks among the plurality of sub-blocks; Based on the motion information of the sub-blocks in the plurality of sub-blocks, the sub-blocks in the plurality of sub-blocks are encoded, wherein the current block is used to construct a merge candidate list, the merge candidate list is a sub-block-based merge candidate list of the current block, the sub-block-based merge candidates in the sub-block-based merge candidate list include a plurality of SbTMVP merge candidates and at least one affine merge candidate, each of the plurality of SbTMVP merge candidates corresponds to a corresponding one of the plurality of DVP candidates.

15. An apparatus for video encoding, characterized in that, The instruction, when executed by at least one processor, causes the at least one processor to perform the following: Determine the displacement vector predictor (DVP) and DV offset of the current block in the current image, wherein the current block includes multiple sub-blocks; Based on the DVP and DV offset of the current block, the DV of the current block is determined, wherein the DV indicates a juxtaposed block in a juxtaposed reference image, the juxtaposed block in the juxtaposed reference image being juxtaposed with the current block; Based on the motion information of the corresponding sub-blocks in the juxtaposed block, determine the motion information of the sub-blocks among the plurality of sub-blocks; Based on the motion information of the sub-blocks in the plurality of sub-blocks, the sub-blocks in the plurality of sub-blocks are encoded, wherein the current block is used to construct a merge candidate list, the merge candidate list is a sub-block-based merge candidate list of the current block, the sub-block-based merge candidates in the sub-block-based merge candidate list include a plurality of SbTMVP merge candidates and at least one affine merge candidate, each of the plurality of SbTMVP merge candidates corresponds to a corresponding one of the plurality of DVP candidates.

Images