Method and apparatus for video encoding, decoding, and storage medium

Abstract

Aspects of the present application provide a method and apparatus for video encoding, decoding and storage medium, wherein the video decoding method comprises: receiving a coded video bitstream comprising a current block in a current picture. The current block comprises a plurality of sub-blocks and is to be predicted by a sub-block based template matching motion vector prediction (SbTMVP) mode. A respective collocated reference sub-block is determined for each sub-block based on a combination of a displacement vector (DV) and a motion vector offset (MVO) associated with the respective sub-block. A motion vector (MV) field in the respective collocated reference sub-block is determined for each sub-block in the current block. A respective reference template is derived for each sub-block based on the determined motion vector (MV) field in the respective collocated reference sub-block. The plurality of sub-blocks of the current block are reconstructed by predicting each sub-block using the respective reference template in the SbTMVP mode.

Description

[0001] priority

[0002] This application claims priority to U.S. Patent Application No. 17 / 983,866, filed November 9, 2022, entitled "Subblock-based Motion Vector Predictor with MV Offset Derived by Template Matching," which claims priority to U.S. Provisional Application No. 63 / 344,840, filed May 23, 2022, entitled "Subblock Based Motion Vector Predictor With MVOffset Derived By Template Matching." Technical Field

[0003] This disclosure describes embodiments that generally involve video coding. Background Technology

[0004] The background description provided herein is intended to provide a general overview of the context of this disclosure. To the extent described in this background section, the work of the currently attributed inventors and aspects of the description that may not conform to the prior art at the time of filing are neither explicitly nor implicitly considered to be prior art of this disclosure.

[0005] Uncompressed digital images and / or videos can comprise a series of pictures, each with spatial dimensions, for example, 1920×1080 luminance samples and correlated chrominance samples. This series of pictures can have a fixed or variable image rate (informally also called frame rate), for example, 60 frames per second or 60Hz. Uncompressed images and / or videos have specific bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at 60Hz frame rate) requires close to 1.5 Gbit / s of bandwidth. One hour of such video would require over 600 GB of storage space.

[0006] One objective of image and / or video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression helps reduce the aforementioned bandwidth and / or storage space requirements, in some cases by two orders of magnitude or more. Although the description herein uses video encoding / decoding as an illustrative example, the same techniques can be applied to image encoding / decoding in a similar manner without departing from the spirit of this disclosure. Lossless compression and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to a technique that allows for the reconstruction of an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal is useful for the intended application. In the case of video, lossy compression is widely used. The amount of distortion tolerated depends on the application; for example, users of some consumer streaming applications may tolerate higher distortion than users of television applications. The achievable compression ratio can be reflected in the fact that higher tolerance / tolerable distortion can result in a higher compression ratio.

[0007] Video encoders and decoders can utilize several major categories of techniques, including motion compensation, transform processing, quantization, and entropy coding.

[0008] Video codec techniques include a technique called intra-frame coding. In intra-frame coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the picture can be an intra-frame picture. Intra-frame pictures and their derived pictures (e.g., standalone decoder refresh pictures) can be used to reset the decoder state and thus can be used as the first picture in the encoded video bitstream and video session, or as a still image. Samples of an intra-frame block can be transformed, and the transform coefficients can be quantized before entropy coding. Intra-frame prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the transformed DC value, the smaller the AC coefficients, and the fewer bits are needed to represent the entropy-coded block at a given quantization step size.

[0009] For example, conventional intra-frame coding known in MPEG-2 generation coding techniques does not use intra-frame prediction. However, some newer video compression techniques include those that attempt to perform prediction based on surrounding sample data and / or metadata obtained, for example, during the encoding and / or decoding of data blocks. This technique is hereby referred to as "intra-frame prediction." Note that, at least in some cases, intra-frame prediction uses only reference data from the current frame in the reconstruction, and not reference data from a reference frame.

[0010] There can be many different forms of intra prediction. When multiple such techniques are available within a given video coding technique, the specific technique used can be encoded as a specific intra prediction mode employing that technique. In some cases, an intra prediction mode can have sub-modes and / or parameters, where these sub-modes and / or parameters can be encoded individually or contained within a mode codeword that defines the prediction mode used. For a given combination of modes, sub-modes, and / or parameters, the choice of which codeword is used will affect the coding efficiency gain through intra prediction, and therefore, the entropy coding technique used to convert the codeword into a bitstream will also have an impact.

[0011] H.264 introduced specific intra-prediction modes, which were improved in H.265 and further refined in newer coding techniques, such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Prediction blocks can be formed using neighboring sample values ​​from already available samples. The sample values ​​of neighboring samples are copied into the prediction block according to the orientation. The reference to the orientation being used can be encoded in the bitstream or can be predicted itself.

[0012] refer to Figure 1A The lower right corner depicts a subset of nine known prediction directions from the 33 possible prediction directions defined in H.265 (corresponding to 33 angular modes of 35 intra-frame modes). The point where the arrows converge represents the sample being predicted (101). The arrows indicate the direction of the predicted sample. For example, arrow (102) indicates that sample (101) is predicted from one or more samples at a 45° angle to the upper right of the horizontal line. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples at a 22.5° angle to the lower left of sample (101) at a 22.5° angle to the horizontal line.

[0013] Still referencing Figure 1AA 4×4 square block (104) of samples is depicted in the upper left (represented by a thick dashed line). The square block (104) comprises 16 samples, each labeled "S" with its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (counting from the top) and the first sample in the X dimension (counting from the left). Similarly, sample S44 is the fourth sample in the square block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. The reference samples are labeled with R, their Y position relative to the block (e.g., row index), and X position (column index) to the square block (104). In H.264 and H.265, predicted samples are adjacent to the blocks in the reconstruction; therefore, negative values ​​are not required.

[0014] Intra-frame image prediction works by copying reference sample values ​​from neighboring samples predicted from the prediction direction of the signaling. For example, suppose the encoded video bitstream contains signaling that indicates a prediction direction consistent with arrow (102) for this block—that is, predicting samples from the sample at a 45° angle to the upper right of the horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Sample S44 is then predicted from reference sample R08.

[0015] In some cases, the values ​​of multiple reference samples can be combined, for example, by interpolation, to calculate the reference sample; especially when the direction is not divisible by 45°.

[0016] With the development of video coding technology, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and some techniques in entropy coding are used to represent those possible directions with a small number of bits. However, when accepting less likely directions, the coding efficiency is low, requiring more bits and leading to a degraded coding performance. Furthermore, the direction itself can sometimes be predicted from adjacent directions used in adjacent decoded blocks.

[0017] Figure 1B A schematic diagram (110) is shown, depicting 65 intra-frame predicted directions according to JEM, to illustrate how the number of predicted directions increases over time.

[0018] The mapping of intra-prediction direction bits in the encoded video bitstream to represent that direction can vary depending on the video coding technique. For example, this mapping can range from a simple direct mapping from prediction direction to intra-prediction mode to codewords, to complex adaptive schemes involving the most probable mode, and similar techniques. However, in most cases, some directions are statistically less likely to appear in the video content compared to others. Since the goal of video compression is to reduce redundancy, in well-functioning video coding techniques, those less likely directions will be represented by more bits than the more likely directions.

[0019] Image and / or video encoding and decoding can be performed using inter-picture prediction with motion compensation. Motion compensation can be a lossy compression technique and can involve techniques where sample data blocks from a previously reconstructed image or a portion thereof (the reference image) are spatially shifted in a direction indicated by a motion vector (MV) to predict a newly reconstructed image or portion thereof. In some cases, the reference image can be the same as the image currently being reconstructed. The MV can have two dimensions, X and Y, or three dimensions, with the third dimension being an indication of the reference image in use (which can indirectly be a temporal dimension).

[0020] In some video compression techniques, a motion vector (MV) applicable to a region of sample data can be predicted from other MVs, for example, from another region of sample data that is spatially adjacent to the region being reconstructed and precedes that MV in the decoding order. Doing so can significantly reduce the amount of data required to encode the MV, thereby eliminating redundancy and improving compression ratio. MV prediction can work efficiently, for example, when encoding an input video signal derived from a camera (called natural video), there is a statistical probability that regions larger than the region to which a single MV can be applied move in similar directions. Therefore, in some cases, similar motion vectors derived from MVs of adjacent regions can be used for prediction. This results in an MV found for a given region being similar or identical to an MV predicted from surrounding MVs, and after entropy coding, this can be represented with fewer bits than when directly encoding the MV. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself can be lossy due to rounding errors, for example, when calculating predictions from several surrounding MVs.

[0021] Various MV prediction mechanisms are described in High Efficiency Video Coding (ITU-T TRec.H.265, "High Efficiency Video Coding", abbreviated as H.265 / HEVC, December 2016). Among the many MV prediction mechanisms provided by H.265, a technique hereafter referred to as "spatial combining" is described with reference to Figure 2.

[0022] Referring to Figure 2, the current block (201) includes samples discovered by the encoder during motion search so that predictions can be made from previous blocks of the same size that have been spatially shifted. Instead of directly encoding the MV, it can be derived from metadata associated with one or more reference images, for example, from the most recent (in decoding order) reference image, using the MV associated with any of the five surrounding samples (denoted as A0, A1 and B0, B1, B2 (202 to 206 respectively)). In H.265, MV prediction can use a predictor from the same reference image being used by adjacent blocks. Summary of the Invention

[0023] Various aspects of this disclosure provide methods and apparatus for video encoding / decoding. In some examples, the apparatus for video decoding includes processing circuitry.

[0024] According to one aspect of this disclosure, a method for performing video decoding in a video decoder is provided. In this method, an encoded video bitstream comprising a current block in a current image can be received. The current block comprises multiple sub-blocks, and the current block is predicted using a sub-block-based template matching motion vector prediction (SbTMVP) pattern. A corresponding juxtaposed reference sub-block is determined for each sub-block based on a combination of displacement vectors (DV) and motion vector offsets (MVO) associated with the corresponding sub-block. A motion vector (MV) field in the corresponding juxtaposed reference sub-block for each sub-block in the current block is determined. A corresponding reference template for each sub-block is derived based on the determined MV field of the juxtaposed reference sub-block. Multiple sub-blocks of the current block are reconstructed by predicting each sub-block using the corresponding reference template in the SbTMVP pattern.

[0025] To determine the corresponding juxtaposed reference sub-block, a search region can be defined, located in one of the current image and a reference image of the current image. One or more reference blocks of the current block can be determined based on template matching between the template of the current block and the templates of each of one or more reference blocks in the search region. The template of the current block may include samples adjacent to the current block. The template of each reference block may include samples adjacent to the corresponding reference block of its respective reference block. The corresponding juxtaposed reference sub-block of each sub-block can be determined as a sub-block juxtaposed with a corresponding sub-block in one of the one or more reference blocks.

[0026] The template matching of the current block with the template of each of one or more reference blocks can be determined based on one of the following: sum of absolute differences (SAD), sum of absolute transformation differences (SATD), sum of squared errors (SSE), subsampling SAD, and mean-removed SAD.

[0027] To identify multiple candidate reference blocks within a search area, multiple candidate reference blocks can be defined within the search area. Multiple cost values ​​can be determined based on template matching between the template of the current block and the templates of multiple candidate reference blocks. One or more reference blocks can be identified as one or more candidate reference blocks corresponding to one or more of the lowest cost values ​​among the multiple candidate reference blocks.

[0028] In one embodiment, the search area may include one of the following: (i) an area centered on a position juxtaposed with the current block in the reference image; and (ii) an area centered on the current block in the current image.

[0029] In one example, the search region can be determined based on the displacement vector (DV). The DV can be derived from either (i) the motion vectors of the spatially neighboring blocks of the current block or (ii) the motion vectors of the merge candidate list of the current block.

[0030] In one example, the search area can be defined as the region centered on the sample indicated by the DV, and the region can be one of the following: square, rectangle, or rhombus.

[0031] In one example, the search area can be defined as a set of samples centered on the sample indicated by the DV. This set of samples is located at at least one of the following locations: 0 degrees, 45 degrees, 90 degrees, or 135 degrees relative to the sample indicated by the DV.

[0032] In one example, to determine one or more reference blocks, a first reference block can be determined. The first reference block can be indicated by a first displacement vector (DV) from the template of the current block to the template of the first reference block. In one example, the first DV can be derived based on template matching, such that the first DV corresponds to a cost value associated with the difference between the template of the first reference block and the template of the current block. In one example, the first DV can be transmitted in the form of signaling.

[0033] In one example, the search region can be determined based on a first displacement vector (DV) derived prior to template matching. A first reference block among one or more reference blocks can be determined based on a second DV derived from the template of the current block to the template of the first reference block. The second DV is derived based on template matching, such that the second DV corresponds to a cost value associated with the difference between the template of the first reference block and the template of the current block.

[0034] To reconstruct sub-blocks of the current block, one or more MVs of the first sub-block among multiple sub-blocks in the current block can be determined based on one or more motion vectors (MVs) of sub-blocks juxtaposed with the first sub-block in one or more reference blocks. One or more predicted sub-blocks of the first sub-block among multiple sub-blocks can be determined based on one or more MVs of the first sub-block. A predicted sample of the first sub-block can be determined based on a combination or weighted combination of one or more predicted sub-blocks.

[0035] In some embodiments, multiple candidate reference blocks for the current block can be determined based on multiple displacement vectors (DVs). Each of the multiple candidate reference blocks can be indicated by a corresponding DV among the multiple DVs. One or more reference blocks for the current block can be determined from the multiple candidate reference blocks based on one or more cost values ​​of template matching.

[0036] According to another aspect of this disclosure, an apparatus for performing decoding is provided. The apparatus includes processing circuitry. The processing circuitry can be configured to perform the aforementioned video decoding method.

[0037] According to another aspect of this disclosure, a method for performing encoding is provided. The method includes: determining a search region located in one of a current image and a reference image of the current image, the current image including a current block; determining one or more reference blocks of the current block based on template matching between a template of the current block and templates of each of one or more reference blocks in the search region, wherein the template of the current block includes samples adjacent to the current block, and the template of each of the one or more reference blocks includes samples adjacent to the corresponding reference blocks of the one or more reference blocks; and generating sub-block prediction samples of the current block based on the determined sub-blocks of the one or more reference blocks.

[0038] According to another aspect of this disclosure, an apparatus for performing encoding is provided. The apparatus includes processing circuitry. The processing circuitry can be configured to perform the aforementioned video encoding method.

[0039] Various aspects of this disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer used for video decoding, cause the computer to perform any method of video encoding / decoding.

[0040] This disclosure also provides a computer program product, including a computer program / instructions, characterized in that, when executed by a processor, the computer program / instructions implement the steps of any of the above-described video encoding / decoding methods. Attached Figure Description

[0041] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, wherein:

[0042] Figure 1A This is a schematic diagram of an example set of intra-frame prediction modes;

[0043] Figure 1B This is an illustration of an exemplary intra-frame prediction direction;

[0044] Figure 2 is a schematic diagram of the current block and its surrounding space merging candidates in an example;

[0045] Figure 3 This is a simplified block diagram of a communication system (300) according to one embodiment;

[0046] Figure 4 This is a simplified block diagram of a communication system (400) according to one embodiment;

[0047] Figure 5 This is a simplified block diagram of a decoder according to one embodiment;

[0048] Figure 6 This is a simplified block diagram of an encoder according to one embodiment;

[0049] Figure 7 A block diagram of an encoder according to another embodiment is shown;

[0050] Figure 8 A block diagram of a decoder according to another embodiment is shown;

[0051] Figure 9 An exemplary location of a spatial merging candidate is shown in another embodiment.

[0052] Figure 10 An exemplary redundancy check for spatial merging candidates is shown.

[0053] Figure 11 A schematic diagram of motion vector scaling for time merging candidates is shown according to one embodiment.

[0054] Figure 12 An exemplary candidate position for time merging candidates according to one embodiment is shown.

[0055] Figure 13 A schematic diagram of the MMVD search process combined with motion vector difference is shown.

[0056] Figure 14A A schematic diagram of the search points in an exemplary MMVD process is shown.

[0057] Figure 14B This is a schematic diagram of the search points in an exemplary MMVD process.

[0058] Figure 15This is an additional detailed location diagram of an exemplary MMVD process.

[0059] Figure 16 A schematic diagram of an exemplary spatial neighboring block for predicting SbTMVP based on the temporal motion vector of a sub-block is shown.

[0060] Figure 17 This is a schematic diagram of an exemplary SbTMVP process.

[0061] Figure 18 A schematic diagram of an exemplary SbTMVP based on template matching is shown.

[0062] Figure 19 A flowchart outlining an exemplary decoding process according to some embodiments of this disclosure is shown.

[0063] Figure 20 This is a flowchart outlining an exemplary coding process according to some embodiments of the present disclosure.

[0064] Figure 21 This is a schematic diagram of a computer system according to one embodiment. Detailed Implementation

[0065] Figure 3 An exemplary block diagram of a communication system (300) is shown. The communication system (300) includes a plurality of terminal devices capable of communicating with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and terminal device (320) interconnected via the network (350). Figure 3 In the example, the first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, terminal device (310) may encode video data (e.g., a video image stream captured by terminal device (310)) for transmission via network (350) to another terminal device (320). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. Terminal device (320) may receive the encoded video data from network (350), decode the encoded video data to recover the video images, and display the video images based on the recovered video data. Unidirectional data transmission is common in media service applications, etc.

[0066] In another example, the communication system (300) includes a second pair of terminal devices (330) and a terminal device (340) performing bidirectional transmission of encoded video. For example, data during a video conference, for bidirectional transmission of data, in one example, each terminal device of terminal devices (330) and (340) can encode video data (e.g., a stream of video images captured by the terminal device) for transmission via network (350) to the other terminal device of terminal devices (330) and (340). Each terminal device of terminal devices (330) and (340) can also receive encoded video data transmitted by the other terminal device of terminal devices (330) and (340), and can decode the encoded video data to recover the video images, and can display the video images on an accessible display device based on the recovered video data.

[0067] exist Figure 3 In the examples, terminal devices (310), (320), (330), and (340) are shown as a server, a personal computer, and a smartphone, respectively, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptops, tablets, media players, and / or dedicated video conferencing equipment. Network (350) refers to any number of networks that transmit encoded video data between terminal devices (310), (320), (330), and (340), including, for example, wired and / or wireless communication networks. Communication networks (350) can exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may be of little importance to the operation of this disclosure, unless explained below.

[0068] As an example of the application of the disclosed topic, Figure 4 The diagram illustrates a video encoder and video decoder in a streaming environment. The disclosed subject matter can also be applied to other video-enabled applications, including, for example, video conferencing, digital television, streaming services, and storing compressed video on digital media such as Compact Discs (CDs), Digital Versatile Discs (DVDs), and flash memory.

[0069] The streaming system may include: a capture subsystem (413) that may include a video source (401), such as a digital camera; and a system for creating, for example, an uncompressed video picture stream (402). In one example, the video picture stream (402) includes samples captured by a digital camera. The video picture stream (402) is depicted as a thick line emphasizing the high data volume when compared to encoded video data (404) (or encoded video bitstream), and may be processed by an electronic device (420) of a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume when compared to the video picture stream (402), and may be stored on a streaming server (405) for future use. One or more streaming client subsystems (e.g., Figure 3 Client subsystems (406) and (408) can access a streaming server (405) to retrieve copies (407) and (409) of encoded video data (404). Client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the input copy (407) of the encoded video data and creates an output video picture stream (411) that can be presented on a display (412) (e.g., a screen) or other presentation device (not shown). In some streaming systems, the encoded video data (404), copies (407), and copies (409) (e.g., a video bitstream) may be encoded according to certain video coding / compression standards. Examples of such standards include ITU-T Recommendation H.265. In one example, the video coding standard under development is informally referred to as Universal Video Coding (VVC). The disclosed topics can be used in a VVC environment.

[0070] Note that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0071] Figure 5 A block diagram of a video decoder (510) is shown. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used in place of... Figure 4 The video decoder (510) in the example.

[0072] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510). In one embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the decoding of other encoded video sequences. The encoded video sequences can be received from a channel (501), which can be a hardware / software link to a storage device storing the encoded video data. The receiver (531) can receive encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective user entities (not shown). The receiver (531) can separate the encoded video sequences from other data. To combat network jitter, a buffer memory (515) can be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, it can be outside the video decoder (510) (not shown). In other cases, a buffer memory (not shown) may be present outside the video decoder (510), for example, to combat network jitter. Additionally, another buffer memory (515) may be present inside the video decoder (510), for example, to handle playback timing. The buffer memory (515) may be unnecessary or small when the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network. For use on best-effort packet networks such as the Internet, a buffer memory (515) may be required. This buffer memory may be relatively large and advantageously have an adaptive size, and may be implemented at least partially outside the video decoder (510) in an operating system or similar component (not shown).

[0073] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols include information for managing the operation of the video decoder (510) and information that potentially controls a presentation device such as a display device (512) (e.g., a display screen), which is not part of the electronic device (530) but may be coupled to it. Figure 5As shown. The control information used to present the device can be in the form of Supplemental Enhancement Information (SEI message) or Video User Information (VUI) parameter set fragments (not shown). The parser (520) can parse / decode the received encoded video sequence. The encoding of the encoded video sequence can be based on video coding technology or standards and can follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) can extract a set of subgroup parameters of at least one pixel subgroup from the encoded video sequence based on at least one parameter corresponding to the group. The subgroup can include Group of Pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transforming unit (TU), prediction unit (PU), etc. The parser (520) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, MV, etc.

[0074] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) in order to create symbols (521).

[0075] Depending on the type of the encoded video picture or its portions (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (521) can involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by the parser (520). For clarity, the flow of such subgroup control information between the parser (520) and the multiple units below is not described.

[0076] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as follows. In practical implementations operating under commercial constraints, many of these units interact closely with each other and can be integrated with each other at least partially. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the following functional units.

[0077] The first unit is the scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives the quantized transform coefficients and control information, including which transform to use, block size, quantization factor, quantization scaling matrix, etc., as symbols (521) from the parser (520). The scaler / inverse transform unit (551) can output blocks containing sample values, which can be input into the aggregator (555).

[0078] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block. An intra-coded block is a block that does not use prediction information from a previously reconstructed image, but can use prediction information from a previously reconstructed portion of the current image. This prediction information can be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses surrounding reconstructed information obtained from the current picture buffer (558) to generate a block with the same size and shape as the block in the reconstruction. The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information already generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) based on each sample.

[0079] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coded blocks and may be motion-compensated. In this case, the motion compensation prediction unit (553) can access the reference picture memory (557) to obtain samples for prediction. After motion compensation of the extracted samples according to the symbols (521) associated with the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (referred to as residual samples or residual signals in this case) to generate output sample information. The addresses in the reference picture memory (557) from which the motion compensation prediction unit (553) obtains the predicted samples can be controlled by motion vectors, and the motion compensation prediction unit (553) can obtain these addresses in the form of symbols (521), which may have, for example, X, Y, and reference picture components. When using subsampled exact MV, motion compensation may also include interpolation of sampled values ​​obtained from the reference picture memory (557), motion vector prediction mechanisms, etc.

[0080] The output samples of the aggregator (555) can be processed in the loop filter unit (556) using various loop filtering techniques. Video compression techniques may include loop filtering techniques controlled by parameters contained in the encoded video sequence (also known as the encoded video bitstream) and available as symbols (521) from the parser (520) to the loop filter unit (556). Video compression may also be in response to metadata obtained during the decoding of a previous (in the order of decoding) portion of the encoded image or encoded video sequence, as well as to previously reconstructed and loop-filtered sample values.

[0081] The output of the loop filter unit (556) can be a sample stream, which can be output to a display device (512) and stored in a reference image memory (557) for future inter-frame image prediction.

[0082] Once fully reconstructed, certain encoded images can be used as reference images for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and that encoded image has been identified as a reference image (e.g., by the parser (520)), the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of the next encoded image begins.

[0083] The video decoder (510) can perform decoding operations according to a predetermined video compression technique in a standard such as ITU-T Rec.H.265. The encoded video sequence conforms to the syntax of the video compression technique or standard, and the profile documented in the video compression technique or standard specifies the syntax. Specifically, the profile may select certain tools from all available tools in the video compression technique or standard, as the only tools available under that profile. Standard conformance also requires the complexity of the encoded video sequence to be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the level can be further restricted by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer transmitted in signaling form in the encoded video sequence.

[0084] In one embodiment, the receiver (531) may receive additional (redundant) data with encoded video. The additional data may be included as part of the encoded video sequence. The video decoder (510) may use the additional data to correctly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0085] Figure 6 An exemplary block diagram of a video encoder (603) is shown. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of Figure 4 The video encoder (403) in the example.

[0086] The video encoder (603) can obtain data from the video source (601) (which is not...). Figure 6 In one example, a portion of an electronic device (620) receives a video sample, the video source of which can capture video images to be encoded by a video encoder (603). In another example, a video source (601) is a portion of an electronic device (620).

[0087] The video source (601) can be provided as a digital video sample stream of a sequence of source videos to be encoded by a video encoder (603). This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601YCrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb4:2:0, YCrCb4:4:4). In a media service system, the video source (601) can be a storage device storing previously prepared video. In a video conferencing system, the video source (601) can be a camera capturing local image information as a video sequence. The video data can be provided as a series of multiple individual pictures that produce motion when viewed in sequence. The pictures themselves can be organized as a spatial array of pixels, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0088] According to one embodiment, the video encoder (603) can encode and compress images of a source video sequence into an encoded video sequence (643) in real time or under any other time constraint. Implementing an appropriate encoding rate is a function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units. For clarity, coupling is not described. Parameters set by the controller (650) may include rate control-related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other functions applicable to the video encoder (603) to optimize a particular system design.

[0089] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. To put it simply, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols in a manner similar to what a (remote) decoder would also create, to create sample data. The reconstructed sample stream (sample data) is fed into a reference image memory (634). Since decoding the symbol stream produces a bit-precise result (local or remote) regardless of the decoder's location, the contents of the reference image memory (634) are also bit-precise between the local and remote encoders. In other words, when prediction is used during decoding, the encoder's prediction portion, as a reference image sample, "sees" the exact same sample value as the decoder "sees." This basic principle of reference image synchronicity (and the resulting drift, if synchronicity cannot be maintained, for example, due to channel errors) is also used in some related techniques.

[0090] The operation of the "local" decoder (633) can be the same as that of a "remote" decoder such as a video decoder (510), as described above. Figure 4 It was described in detail. However, a brief reference is provided. Figure 5 Since symbols are available and the encoding / decoding of symbols for the encoded video sequence by the entropy encoder (645) and the parser (520) can be lossless, the entropy decoding part of the video decoder (510), including the buffer memory (515) and the parser (520), does not need to be fully implemented in the local decoder (633).

[0091] In one embodiment, any decoder technique, other than the parsing / entropy decoding present in the decoder, exists in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject focuses on decoder operation. The description of encoder techniques can be simplified, as these techniques are the inverse of the fully described decoder techniques. In certain areas, a more detailed description is provided below.

[0092] In some example operations, the source encoder (630) may perform motion-compensated predictive coding, which predictively codes the input image based on one or more previously encoded images (referred to as "reference images") in the video sequence. In this way, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of the reference images, which may be selected as prediction references for the input image.

[0093] The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture, based on symbols created by the source encoder (630). The operation of the encoding engine (632) may favor a lossy process. When the encoded video data can be decoded by the video decoder (633), Figure 6 When decoded at (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that can be performed on the reference image by the video decoder, and can store the reconstructed reference image in the reference image memory (634). In this way, the video encoder (603) can locally store copies of the reconstructed reference images that have the same content as the reconstructed reference images that will be obtained by the remote video decoder (without transmission errors).

[0094] The predictor (635) can perform a prediction search on the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., which can be used as appropriate prediction references for the new image. The predictor (635) can operate on a sample block-by-pixel basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input image may have prediction references extracted from multiple reference images stored in the reference image memory (634).

[0095] The controller (650) can manage the encoding operations of the source encoder (630), including, for example, the settings of parameters and subgroup parameters for encoding video data.

[0096] The outputs of all the aforementioned functional units can be processed by the entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into an encoded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, and arithmetic coding.

[0097] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which may be a hardware / software connection to a storage device for storing the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0098] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoded picture type to each encoded picture, which can affect the encoding techniques that can be applied to the corresponding picture. For example, a picture can typically be designated as one of the following picture types:

[0099] An intra-frame picture (I-picture) can be a picture that is encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Intra-coded Picture Refresh (IDR) pictures. Those skilled in the art are aware of those variations of I-pictures and their corresponding applications and characteristics.

[0100] A predicted image (P-image) can be an image that uses at most one motion vector and a reference index to predict the sample value of each block, and is encoded and decoded using intra-frame prediction or inter-frame prediction.

[0101] A bidirectional prediction image (B-image) can be an image that uses up to two motion vectors and reference indices to predict the sample values ​​for each block, encoded and decoded using intra-frame prediction or inter-frame prediction. Similarly, a multi-prediction image can use more than two reference images and associated metadata to reconstruct the sample values ​​for a single block.

[0102] Source images can typically be spatially subdivided into multiple sample blocks (e.g., each sample block consists of 4×4, 8×8, 4×8, or 16×16 sample blocks) and encoded on a block-by-block basis. Depending on the encoding allocation applied to the respective images of each block, blocks can be predictively encoded by referencing other (already encoded) blocks. For example, blocks of image I can be nonpredictively encoded, or predictively encoded (spatial prediction or intra-frame prediction) by referencing already encoded blocks of the same image. Pixel blocks of image P can be predictively encoded by referencing a previously encoded reference image, either spatially or temporally. Blocks of image B can be predictively encoded by referencing one or two previously encoded reference images, either spatially or temporally.

[0103] The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard (e.g., ITU-T REC.H.265). In its operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard being used.

[0104] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include this data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data (e.g., redundant images and slices), SEI messages, VUI parameter set fragments, etc.

[0105] Video can be captured in chronological order as multiple source images (video pictures). Intra-frame picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given image, while inter-frame picture prediction utilizes (temporal or other) correlations between images. In one example, the specific image being encoded / decoded is divided into blocks called the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. When using multiple reference images, the motion vector points to the reference block within the reference images and can have a third dimension that identifies the reference images.

[0106] In some embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, for example, a first reference image and a second reference image, both of which precede the current image in the video in the decoding order (but can be displayed in the past and future, respectively). A block in the current image can be encoded by a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block can be predicted using a combination of the first and second reference blocks.

[0107] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0108] According to some embodiments of this disclosure, prediction is performed on a block-by-block basis, such as inter-frame picture prediction and intra-frame picture prediction. For example, according to the HEVC standard, pictures in a video picture sequence are segmented into Coding Tree Units (CTUs) for compression. The CTUs in the pictures have the same size, for example, 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU comprises three Coding Tree Blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-segmented into one or more Coding Units (CUs). For example, a 64×64 pixel CTU can be segmented into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine its prediction type, such as inter-frame prediction or intra-frame prediction. Based on temporal and / or spatial predictability, CUs are divided into one or more prediction units (PUs). Typically, each PU includes one luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in encoding (encoding / decoding) are performed on a per-prediction-block basis. Using a luma prediction block as an example, a prediction block contains a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0109] Figure 7 An exemplary diagram of a video encoder (703) is shown. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values ​​within a current video image in a video image sequence, and to encode that processing block into an encoded image as part of an encoded video sequence. In one example, the video encoder (703) is used instead of Figure 4 The video encoder (403) in the example.

[0110] In the HEVC example, the video encoder (703) receives a sample value matrix of the processed block, such as an 8×8 sample prediction block. The video encoder (703) determines the optimal encoding method for the processed block: intra-frame mode, inter-frame mode, or bidirectional prediction mode, such as rate-distortion optimized mode. When encoding the processed block in intra-frame mode, the video encoder (703) can use intra-frame prediction techniques to encode the processed block into a coded picture; and when encoding the processed block in inter-frame mode or bidirectional prediction mode, the video encoder (703) can use inter-frame prediction or bidirectional prediction techniques to encode the processed block into a coded picture, respectively. In some video coding techniques, the merging mode may be an inter-frame picture prediction sub-mode, in which motion vectors are derived from one or more motion vector predictors without benefiting from coded motion vector components outside the predictors. In some other video coding techniques, there may be motion vector components applicable to the object block. In one example, the video encoder (703) includes other components, such as a mode determination module (not shown) for determining the mode of the processed block.

[0111] exist Figure 7 In the example, the video encoder (703) includes, for example, Figure 7 The inter-frame encoder (730), intra-frame encoder (722), residual calculator (723), switch (726), residual encoder (724), general controller (721) and entropy encoder (725) are coupled together.

[0112] An inter-frame encoder (730) is configured to receive samples of the current block (e.g., the processing block), compare that block with one or more reference blocks in a reference image (e.g., blocks in previous and subsequent images), generate inter-frame prediction information (e.g., a description of redundancy information based on the inter-frame coding technique, motion vectors, merging mode information), and compute inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is a decoded reference image based on the decoded coded video information.

[0113] The intra encoder (722) is configured to receive samples of the current block (e.g., the processing block), in some cases compare the block with blocks already encoded in the same image, generate quantization coefficients after transformation, and in some cases also generate intra prediction information (e.g., intra prediction direction information based on one or more intra coding techniques). In one example, the intra encoder (722) also computes an intra prediction result (e.g., a prediction block) based on the intra prediction information in the same image and a reference block.

[0114] A general controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines the mode of a block and provides control signals to the switch (726) based on that mode. For example, when the mode is intra-frame mode, the general controller (721) controls the switch (726) to select intra-frame mode results for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream; when the mode is inter-frame mode, the general controller (721) controls the switch (726) to select inter-frame prediction results for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

[0115] A residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or inter encoder (730). A residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, and the decoded image can be cached in a memory circuit (not shown) and used as a reference image in some examples.

[0116] The entropy encoder (725) is configured to format the bitstream to include coded blocks. The entropy encoder (725) is configured to include various information in the bitstream according to an appropriate standard (e.g., the HEVC standard). In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. Note that, according to the disclosed subject matter, no residual information is present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

[0117] Figure 8An exemplary diagram of a video decoder (810) is shown. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In one example, the video decoder (810) is used instead of Figure 4 The video decoder (410) in the example.

[0118] exist Figure 8 In the example, the video decoder (810) includes, for example, Figure 8 The entropy decoder (871), inter-frame decoder (880), residual decoder (873), reconstruction module (874), and intra-frame decoder (872) are coupled together.

[0119] The entropy decoder (871) can be configured to reconstruct certain symbols representing the syntax elements constituting the encoded image from the encoded image. These symbols may include, for example, the mode in which the block is encoded (e.g., intra-mode, inter-mode, bidirectional prediction mode, merged sub-mode, or the latter two of another sub-mode) and prediction information (e.g., intra-prediction information or inter-prediction information) that can identify specific samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), respectively. The symbols may also include, for example, residual information in the form of quantized transform coefficients. In one example, when the prediction mode is inter-mode or bidirectional prediction mode, inter-prediction information is provided to the inter-decoder (880); and when the prediction type is intra-prediction type, intra-prediction information is provided to the intra-decoder (872). The residual information may undergo inverse quantization and be provided to the residual decoder (873).

[0120] The inter-frame decoder (880) is configured to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

[0121] The intra-frame decoder (872) is configured to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

[0122] The residual decoder (873) is configured to perform inverse quantization to extract the dequantized transform coefficients and process them to convert the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also require some control information (to include quantizer parameters (QP)), and this information can be provided by the entropy decoder (871) (the data path is not shown because this may only be a small amount of control information).

[0123] The reconstruction module (874) is configured to combine the residual output by the residual decoder (873) and the prediction results (output by the inter-frame or intra-frame prediction module, as appropriate) in the spatial domain to form a reconstruction block, which can be part of a reconstructed image, which in turn can be part of a reconstructed video. Note that other suitable operations, such as deblocking, can be performed to improve visual quality.

[0124] Note that the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using any suitable technology. In one embodiment, the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using one or more processors that execute software instructions.

[0125] This disclosure includes embodiments related to sub-block-based temporal motion vector prediction (SbTMVP). SbTMVP can be derived using displacement motion vectors with motion vector offsets.

[0126] ITU-TVCEG (Q6 / 16) and ISO / IEC MPEG (JTC1 / SC29 / WG11) released the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4), respectively. In 2015, the two standards organizations jointly established the Joint Video Exploration Team (JVET) to explore the possibility of developing a next video coding standard beyond HEVC. In October 2017, the two standards organizations issued a "Call for Proposals" (CfP) on video compression capabilities beyond HEVC. As of February 15, 2018, 22 CfP responses were submitted regarding Standard Dynamic Range (SDR), 12 regarding High Dynamic Range (HDR), and 12 regarding 360 video categories. In April 2018, all received CfP responses were evaluated at the 122nd MPEG / 10th JVET meeting. As an outcome of this meeting, JVET officially launched the standardization process for next-generation video coding beyond HEVC. The new standard was named Multifunctional Video Coding (VVC), and JVET was renamed the Joint Video Experts Group. In 2020, ITU-T TVCEG (Q6 / 16) and ISO / IEC MPEG (JTC1 / SC29 / WG11) released the VVC video coding standard (first version).

[0127] In inter-frame prediction, each inter-frame prediction coding unit (CU) requires motion parameters to implement the coding functions of VVC, such as generating inter-frame prediction samples. Motion parameters may include motion vectors, reference picture indices, reference picture list usage indices, and / or other information. Motion parameters can be passed explicitly or implicitly in signaling form. When encoding a CU using skip mode, the CU can be associated with a PU and may not require significant residual coefficients, encoded motion vector increments, and / or reference picture indices. When encoding a CU using merge mode, the motion parameters of the CU can be obtained from neighboring CUs. Neighboring CUs may include spatial and temporal candidates as well as additional time schedules (or additional candidates), as introduced in VVC. Merge mode can be applied to any inter-frame prediction CU, not just skip mode. Another option for merge mode is to explicitly transmit motion parameters, where motion vectors, the corresponding reference picture index for each reference picture list, reference picture list usage flags, and / or other required information can be explicitly passed to each CU in signaling form.

[0128] In VVC, the VVC Versatile Test Model (VTM) reference software can include many new and improved inter-frame predictive coding tools, which may include one or more of the following tools:

[0129] (1) Extended merge forecast

[0130] (2) Merge Motion Vector Difference (MMVD)

[0131] (3) Advanced Motion Vector Prediction (AMVP) mode with symmetric MVD signaling

[0132] (4) Motion-simulated compensation prediction

[0133] (5) Sub-block-based temporal motion vector prediction (SbTMVP)

[0134] (6) Adaptive Motion Vector Resolution (AMVR)

[0135] (7) Motion field storage: 1 / 16 luminance sample MV storage and 8×8 motion field compression

[0136] (8) Bi-prediction with CU-level Weights (BCW)

[0137] (9) Bidirectional Optical Flow (BDOF)

[0138] (10) Decoder Side Motion Vector Refinement (DMVR)

[0139] (11) Combined Inter and Intra Prediction (CIIP)

[0140] (12) Geometric Partitioning Mode (GPM)

[0141] The merge candidate list can be constructed by including five candidate types, for example, VTM 4. The order in which the merge candidate list is constructed is as follows:

[0142] (1) Spatial MVP from spatially adjacent CUs

[0143] (2) Time MVP from shared CU

[0144] (3) A history-based MVP derived from a first-in-first-out (FIFO) table.

[0145] (4) Paired average MVP, and

[0146] (5) Zero MV.

[0147] The size of the merge list can be transmitted in signaling form in the slice header. For example, in VTM 4, the maximum allowed size of the merge list is 6. For each CU encoded in merge mode, the index of the best merge candidate can be encoded, for example, using truncated unary binarization. The first binary of the merge index can be context-coded, and the other binary can be bypass-coded.

[0148] In spatial candidate derivation, taking VVC as an example, the derivation of spatial merge candidates can be the same as or similar to the derivation of spatial merge candidates in HEVC. For example, it can be derived from the spatial merge candidates located at... Figure 9 Choose the maximum number of merge candidates from the candidates shown (e.g., four merge candidates). Figure 9 As shown, the current block (901) can include adjacent blocks (902) and adjacent blocks (906) located at positions A0, A1, B0, B1, and B2, respectively. The derivation order of the spatial merging candidate blocks can be B1, A1, B0, A0, and B2. Position B2 is considered only if any CU (or block) located at positions A0, B0, B1, or A1 is unavailable (e.g., because the CU belongs to another slice or tile) or belongs to internal coding. After adding a candidate (or block) at position A1, a redundancy check can be performed on the remaining added candidates (or blocks). Redundancy checks ensure that candidate blocks with the same motion information are excluded from the merging list, thereby improving coding efficiency. To reduce computational complexity, redundancy checks can disregard all possible candidate pairs. Instead, only those pairs can be considered. Figure 10 Candidate pairs are connected by arrows. For example, redundancy checking can be applied to five candidate pairs, such as the candidate pair A1 and B1, and the candidate pair A1 and A0. A candidate is added to the merge list only if the corresponding candidate used for redundancy checking does not contain the same motion information. For example, candidate B0 is added to the merge list only if the corresponding candidate B1 does not contain the same motion information.

[0149] In the time-based candidate export, only one candidate may be added to the merge list. For example, such as Figure 11As shown, when exporting the timing merge candidates for the current CU (1114), the scaling motion vector can be exported based on the corresponding CU (1104) belonging to the juxtaposed reference image (1112). The list of reference images used to export the corresponding CU (1104) can be transmitted in signaling form in the slice header. Figure 11 As shown by the dashed line (1102), the scaled motion vector of the temporal merge candidate is obtained, which is scaled from the motion vector of the co-located CU (1104) using Picture Order Count (POC) distances tb and td. tb can be defined as the POC difference between the reference image (e.g., Curr_ref) (1106) of the current image and the current image (e.g., Curr_pic) (1108). td can be defined as the POC difference between the reference image (1110) of the co-located image (e.g., Col_ref) and the co-located image (e.g., Col_pic) (1112). The reference image index of the temporal merge candidate can be set to zero.

[0150] like Figure 12 As shown, the position of the temporal candidate image can be selected between candidate images C0 and C1. For example, if the CU at position C0 is unavailable, intra-coded, or not in the current CTU line, position C1 can be used. Otherwise, position C0 can be used when exporting temporal merge candidates.

[0151] Motion Vector Difference Merging (MMVD) can be used for certain prediction modes, such as skip modes or merging modes using motion vector representation methods. For example, in VVC, MMVD can reuse merge candidate data. Among the merge candidates, one can be selected and further expanded (or refined) using motion vector representation methods. MMVD can provide new motion vector representations of the signal that are simplified. Motion vector representations can include the origin, amplitude, and direction of motion.

[0152] For example, in VVC, MMVD can use a list of merge candidates. When expanding MMVD, candidates of the default merge type (e.g., MRG_TYPE_DEFAULT_N) can be considered. In MMVD, a base candidate index can define a starting point. For example, the base candidate index (IDX) can indicate the best candidate in the list, as shown in Table 1.

[0153] Table 1. Basic Candidate IDX

[0154] Basic candidate IDX 0 1 2 3 The Nth MVP First MVP Second MVP 3rd MVP 4th MVP

[0155] If the candidate base is equal to 1, the basic candidate IDX may not be transmitted in signaling form. The distance index provides motion amplitude information. The distance index indicates a predetermined distance from the starting point. Table 2 below provides predetermined distances based on the distance index:

[0156] Table 2. Distance from IDX

[0157] Distance from IDX 0 1 2 3 4 5 6 7 Pixel distance 1 / 4-pixel 1 / 2-pixel 1-pixel 2-pixel 4-pixel 8-pixel 16-pixel 32-pixel

[0158] The direction index can represent the direction of the MVD relative to the starting point. As shown in Table 3, the direction index can represent four directions. The MMVD flag can be transmitted in signaling form when the jump flag and merge flag are sent. If the jump flag and merge flag are true, the MMVD flag can be parsed. If the MMVD flag equals 1, the MMVD syntax can be parsed. However, if the MMVD flag is not 1, the AFFINE flag can be parsed. If the AFFINE flag equals 1, the AFFINE mode can be applied. However, if the AFFINE flag is not 1, the skip / merge index for the skip / merge mode can be parsed.

[0159] Table 3. Directional IDX

[0160]

[0161]

[0162] Figure 13 An example search process for MMVD is shown. Figure 13 As shown, the starting MV can be represented by (1311) (e.g., based on direction IDX and basic candidate IDX), the offset can be represented by (1312) (e.g., based on distance IDX and direction IDX), and the final MV predictor can be represented by (1313). In another example, the starting MV can be represented by Figure 13 In (1321), it is indicated (e.g., based on the direction IDX and the basic candidate IDX), that the offset can be used... Figure 13 In (1322), it is indicated (e.g., based on distance IDX and orientation IDX), that the final MV predictor can be used Figure 13 (1323) in the text represents this.

[0163] Figure 14A and 14B Example search points for MMVD are shown. Figure 14A As shown, the starting point MV can be represented by (1411) in the first reference list L0 (e.g., based on the direction IDX and the basic candidate IDX). In Figure 14AIn the example, four search directions are used, such as +Y, -Y, +X, and -X, which can be indexed using 0, 1, 2, and 3. Distances can be indexed using 0 (distance to the starting point MV is 0), 1 (distance to the starting point MV is 1s (or 1 sample)), 2 (distance to the starting point MV is 2s), 3 (distance to the starting point is 3s), and so on. Therefore, when the direction IDX is 3 and the distance IDX is 2, the final MV predictor is displayed as (1415).

[0164] In another example, search direction and distance can be combined for indexing. For example, the starting point MV is shown as (1421) in the second reference list L1 (e.g., based on direction IDX and basic candidate IDX). Figure 14B As shown, the search direction and distance are combined into a 0-12 index.

[0165] In template matching candidate reordering based on MMVD and affine MMVD, the MMVD offset can be extended for both the MMVD and affine MMVD modes. In one example, additional refinement positions can be added first along the k×π / 8 diagonal. An example additional refinement position is shown below. Figure 15 As shown, the number of directions can be increased from 4 to 16. Secondly, based on the SAD cost between the template at each refinement location (e.g., the row above and the column to the left of the current block) and the template reference, all possible MMVD refinement locations (e.g., 16×6) for each basic candidate can be reordered. Finally, the top 1 / 8 refinement locations with the lowest template SAD costs are reserved as available locations for MMVD index encoding. The MMVD index can be binarized using encoding, such as Rice encoding with a parameter of 2.

[0166] In another example, building upon the MMVD extension described above, the affine MMVD can be further extended with a reordering, where additional refinement positions can be added along a k×π / 4 diagonal. After reordering, the first half of the refinement positions with the lowest template SAD cost can be retained.

[0167] To improve coding efficiency and reduce motion vector transmission overhead, sub-block level motion vector refinement can be applied to extend CU-level temporal motion vector prediction (TMVP). Sub-block-based temporal motion vector prediction (SbTMVP) inherits motion information from juxtaposed reference images (or reference images juxtaposed with the current image) at the sub-block level. Each sub-block of a large CU can have its own motion information without explicitly specifying the transport block partitioning structure or motion information. SbTMVP obtains the motion information for each sub-block in three steps. The first step may include deriving the displacement vector (DV) of the current CU. In the second step, the availability of SbTMVP candidates is checked, and the center motion is derived. In the third step, the sub-block motion information can be derived from the corresponding sub-block indicated by the DV. Unlike TMVP candidate derivation, which derives temporal motion vectors from juxtaposed blocks in a reference frame, SbTMVP applies the DV derived from the MV of the current CU's left neighbor CU to find the corresponding sub-block in the juxtaposed image for each sub-block of the current CU. If the corresponding sub-block is not inter-coded, the motion information of the current sub-block can be set to the center motion.

[0168] SbTMVP is supported in related coding standards such as VVC. For example, similar to TMVP in HEVC, SbTMVP can use motion fields from the juxtaposed image of the current image to improve motion vector prediction of the CU in the current image (e.g., merging patterns). The juxtaposed image used by TMVP can also be used in SbTMVP. SbTMVP differs from TMVP in one or more of the following ways:

[0169] (1) TMVP predicts motion at the CU level, while SbTMVP predicts motion at the subCU level; and

[0170] (2) TMVP obtains the temporal motion vector from the juxtaposed blocks in the juxtaposed image (e.g., the juxtaposed block can be the bottom right or middle block relative to the current CU). SbTMVP may apply a motion offset before obtaining the temporal motion information from the juxtaposed image, wherein the motion offset may be obtained from the motion vector of a spatially adjacent block of the current CU.

[0171] An example spatial adjacency block applied to SbTMVP is as follows: Figure 16 As shown. SbTMVP can predict the motion vectors of sub-CUs (not shown) within the current CU (1602) in two steps. In the first step, the motion vectors of the sub-CUs (not shown) can be checked. Figure 16In the context of spatial neighbor A1(1604), if A1(1604) has a motion vector that uses the juxtaposed image of the current image as a reference image, then the motion vector of A1(1604) can be selected as the motion displacement (or displacement vector) of SbTMVP to find the corresponding sub-block for each sub-block of the current CU in the juxtaposed image. If no such motion vector is determined, the motion displacement can be set to (0, 0).

[0172] In the second step, the motion offset determined in the first step can be applied (e.g., added to the coordinates of the current CU) to obtain sub-CU level motion information (e.g., sub-CU level motion vectors and reference indices) from the juxtaposed image. Figure 17 As shown, the current CU (1704) can be contained within the current image (1702). The current CU (1704) can include multiple sub-CUs (or sub-blocks), for example, sub-CU (1706). The adjacent block A1 (1708) can be located to the lower left of the current CU (1704). Figure 17 In one example, the motion offset (or DV) (1710) can be set as the motion vector of the neighboring block A1 (1708). Based on the DV (1710), the reference block A1' (1718) of the neighboring block A1 (1708) can be determined. The reference block (1714) adjacent to the reference block A1' (1718) can be determined as the reference block of the current block (1704). For each sub-CU (e.g., sub-CU (1706)) in the current block (1704), the motion information of the corresponding sub-CU can be derived using the motion information of the corresponding block (or corresponding sub-CU) (e.g., sub-CU (1716)) in the reference block (1714) in the juxtaposed picture (1712) of the current picture (1702) (which may be the smallest motion grid covering the center sample of the corresponding block). After determining the motion information of the juxtaposed sub-CU (e.g., sub-CU (1716)), the motion information can be converted into the motion vector and reference index of the current sub-CU (e.g., sub-CU (1706)). The transformation of motion information can be similar to the TMVP process of HEVC, in which time motion scaling is applied to align the time motion vector of the reference image with the time motion vector of the current CU.

[0173] For example, in VVC, a merge-based sub-block candidate list containing SbTMVP candidates and affine merge candidates can be used for signaling the sub-block merge mode (e.g., SbTMVP mode) for use in signaling. The SbTMVP mode can be enabled / disabled via the Sequence Parameter Set (SPS) flag. If SbTMVP mode is enabled, the SbTMVP predictor can be added as the first entry in the sub-block merge candidate list, followed by affine merge candidates. For example, in VVC, the size of the sub-block merge list can be signaled in the SPS, and the maximum allowed size of the sub-block merge list can be 5.

[0174] The sub-CU size used in SbTMVP mode can be fixed at 8×8, which is the same as the sub-CU size in affine merge mode. In one example, SbTMVP mode may only apply to CUs with a width and height greater than or equal to 8 (or 8 pixels). The sub-block size in SbTMVP mode can be configured to other sizes, for example, 4×4 in an ECM software model used to explore beyond VVC.

[0175] In relevant coding standards (e.g., VVC and ECM), a sub-block-based TMVP (or SbTMVP) can be derived from a DV derived from the MV of the current CU's neighboring CUs. However, the derived SbTMVP and the derived DV may not be an optimal or most suitable match.

[0176] Additional motion offsets for the DV can be transmitted in signaling form to derive a sub-block-based TMVP. However, the additional motion offset signal can be expensive due to the need for extra bits.

[0177] In this disclosure, SbTMVP based on template matching (TM) can be applied. The template for the current block can indicate the region adjacent to the current block, such as a predetermined neighboring reconstruction region of the current block. In one example, the template may include the top N rows of the neighboring reconstruction sample above the current block, and / or the left M columns of the neighboring reconstruction sample to the left of the current block. Example values ​​for M and N may include, but are not limited to, 1, 2, 3, 4, or similar values.

[0178] According to the TM-based SbTMVP, SbTMVP can derive DVs using template matching instead of using DVs derived from neighboring CUs of the current CU, or further, by transmitting additional motion offsets of the DVs in signaling form. The template of the current coded block (or current block) in the current image can be compared with one or more templates (or reference templates) of multiple blocks located at specified candidate locations (or multiple reference blocks in a juxtaposed reference image of the current image). A cost value C can be calculated for each candidate location (or each reference block) and associated with the difference between the template of the current CU and the templates of each reference block. Motion information in sub-blocks of the K blocks (e.g., the K reference blocks in a juxtaposed image) associated with the K minimum cost values ​​can be used to derive the SbTMVP (e.g., motion vectors and reference indices of sub-blocks in the current block). K can be the number of possible DVs.

[0179] Each template of multiple reference blocks can include an M×N sub-block (or region) adjacent to the corresponding reference block in the juxtaposed image. The motion vectors of each M×N sub-block can be derived from the MV field of the current block's juxtaposed block (or juxtaposed reference block), which is adjacent to that block (or reference block in the juxtaposed image) and associated with the reference template. The localization block can be derived using a motion vector offset (MVO) from one of K displacement motion vectors (DV). The derived motion vector SbMV(i,j) of the current block's sub-blocks... Lx It can be used to indicate the position of a sub-block template on a reference frame (or juxtaposed image).

[0180] An example of TM-based SbTMVP is as follows: Figure 18 As shown. The current image (1804) may contain the current block (1802). The current block (1802) may include a template Tc (1806). The template Tc (1806) may include neighboring samples adjacent to the top and / or left side of the current block (1802). Multiple candidate reference blocks may be identified within a search area in the reference image (1822). Multiple cost values ​​may be determined by matching the template Tc (1806) of the current block (1802) with the templates of the multiple candidate reference blocks. Each of the multiple cost values ​​may be based on the difference between the template of the current block and the template of the corresponding candidate reference block. One or more candidate reference blocks may be selected from the multiple candidate reference blocks, which correspond to one or more of the lowest cost values ​​among the multiple cost values. The selected one or more candidate reference blocks may be represented by one or more DVs. For example, a template T0 (1820) and a template T1 (1816) may be selected, which correspond to the two lowest cost values ​​among all candidate templates. Candidate reference block (1818) can be associated with template T0 (1820) and derived from template T. cThe DV0(1812) representation is derived from template T0. A candidate reference block (1814) can be associated with template T1(1816) and derived from template T. c The DV1 (1810) representation of template T1 is obtained. Furthermore, the motion vectors of the sub-blocks of candidate reference block (1814) and candidate reference block (1818) can be used for the TMVP of the sub-blocks of the current block (1802). For example, the motion information of sub-block (1816) in candidate reference block (1814) and sub-block (1824) in candidate reference block (1818) can be transformed (e.g., based on time motion scaling) into the motion vector and / or reference index of sub-block (1808) in the current block (1802). The predicted sample of sub-block (1808) can be further determined based on the motion vector and / or reference index.

[0181] In one embodiment, the cost value C can be determined based on the difference between the template of the current block (e.g., the current block (1802)) and the template of each candidate reference block (e.g., candidate reference block (1818)). In one example, the cost value C can be one of the following: Sum of Absolute Differences (SAD), Sum of Absolute Transformed Differences (SATD), Sum of Squared Errors (SSE), Subsampling SAD, and Mean Removal SAD.

[0182] In one embodiment, the selected one or more candidate reference blocks can be indicated by KDV (e.g., DV1 (1810) or DV0 (1812)). K can be, but is not limited to, 1, 2, 3, 4 or similar values.

[0183] In one embodiment, the search region for finding one or more candidate reference blocks (e.g., candidate reference block (1814) and candidate reference block (1818)) can be defined and centered at the juxtaposition position of the current block in the reference image / frame. For example, the search region can be determined as the area centered at the position in the reference image (1822) juxtaposed with the current block (1802).

[0184] In one embodiment, the search region may be determined based on the encoded region of the same image (e.g., the current image). For example, the search region may be determined as the region in the current image (1804) centered on the current block (1802).

[0185] In one embodiment, a candidate location (or candidate reference block) can be specified by a search region determined by a displacement vector. Therefore, the search region can be determined by a first DV (not shown). Each candidate location (or candidate reference block) within the search region can be determined by a corresponding second DV. For example, a candidate reference block (1814) can be represented by DV1 (1810).

[0186] In one embodiment, the DV can be derived from the motion vectors of spatially adjacent coded blocks of the current block in the current image. For example, the DV can be derived based on the MVP list of the current block, where adjacent coded blocks can be candidates in the MVP list of the current block.

[0187] In one embodiment, the displacement vector can be derived from the motion vector of the regular (e.g., no sub-blocks) merge candidate list for the current block in the current image. Therefore, the Temporal Motion Vector Predictor (TMVP) candidate can be excluded from the regular merge candidate list.

[0188] In one embodiment, the search area can be a specific region (or a specific area) centered on the sample to which the displacement vector points (or indicates). Examples of a specific region (or a specific area) may include, but are not limited to, a square region, a rectangular region, a rhombus region, etc.

[0189] In one embodiment, the search area may be a set of samples centered on the sample to which the displacement vector points (or indicates).

[0190] In one example, the sample group could be located at the same location as the MVD search point, for example, Figure 14A and 14B The MMVD search point is shown in the figure.

[0191] In one example, a sample group may include samples located in the horizontal (or 0-degree) direction, vertical (or 90-degree) direction, 45-degree direction, or 135-degree direction of the sample to which the displacement vector points.

[0192] In one embodiment, TM-based DV export (or TM-based SbTMVP) can be used in conjunction with another method (e.g., inter-frame prediction mode or affine mode). Information transmitted in signaling form (e.g., flags) can be transmitted in signaling form or implicitly exported to indicate whether the DV is exported via template matching or transmitted in signaling form by other methods.

[0193] In one embodiment, TM-based DV export can be used in conjunction with another method (e.g., inter-frame prediction mode or affine mode). For example, the first DV can be exported by another method, while the second DV can be exported by template matching within the search region defined by the first DV.

[0194] When multiple blocks are determined (e.g., K is greater than 1), for each sub-block (e.g., sub-block (1808)) of the current coded block (e.g., coded block (1802)), multiple prediction blocks (or multiple prediction sub-blocks) can be derived using motion vectors associated with juxtaposed sub-blocks in the multiple K blocks, and motion compensation can be achieved based on a combination of multiple prediction blocks (e.g., a weighted sum). For example, as Figure 18 As shown, candidate reference blocks (1814) and reference blocks (1818) can be determined for the current block (1802) based on TM-based SbTMVP. The motion vectors of sub-blocks (1826) in candidate reference block (1814) and sub-blocks (1824) in candidate reference block (1818) can be used to derive a first predicted sub-block and a second predicted sub-block for sub-block (1808) in the current block (1802). The predicted sample of sub-block (1808) can be determined based on the weighted sum of the first predicted sub-block and the second predicted sub-block.

[0195] In one embodiment, up to S derived DVs with the lowest template matching cost can be used as candidate DVs for the signal. S can be, but is not limited to, 1, 2, 3, 4, or similar values. Therefore, the maximum number of codewords used to transmit candidate DVs in signaling form may be limited. For example, the encoder can transmit up to S DV candidates in signaling form, and then the decoder selects to use one or more of the S DV candidates transmitted in signaling form.

[0196] In one embodiment, the best (or selected) DV with the lowest template matching cost can be used. Therefore, no additional signaling is required to transmit the best DV in signaling form. On the decoder side, the decoder can perform the same template matching process as the encoder to derive the best (or selected) DV.

[0197] Figure 19 The flowchart shown outlines an exemplary decoding process (1900) according to some embodiments of this disclosure. Figure 20 The flowcharts shown outline exemplary encoding processes (2000) according to some embodiments of this disclosure. The proposed processes can be used individually or in combination in any order. Furthermore, each process (or embodiment), encoder, and decoder can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0198] The operations of processes (e.g., processes (1900) and (2000)) can be combined or set in any number or order as needed. In embodiments, two or more operations of processes (e.g., processes (1900) and (2000)) can be performed in parallel.

[0199] These processes (such as processes (1900) and (2000)) can be used to reconstruct and / or encode a data block in order to generate a predictive data block for the data block in the reconstruction. In various embodiments, the processes (e.g., processes (1900) and (2000)) are executed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video encoder (403), processing circuitry performing the functions of a video decoder (410), processing circuitry performing the functions of a video decoder (510), processing circuitry performing the functions of a video encoder (603), etc. In some embodiments, the processes (e.g., processes (1900) and (2000)) are implemented in software instructions, so that when the processing circuitry executes the software instructions, the processing circuitry executes the processes (e.g., processes (1900) and (2000)).

[0200] like Figure 19 As shown, process (1900) can start from (S1901) and proceed to (S1910). At (S1910), the encoded video bitstream including the current block in the current picture is received. The current block includes multiple sub-blocks, and the current block is predicted by the motion vector prediction SbTMVP mode based on template matching of the sub-blocks.

[0201] At (S1920), a corresponding juxtaposition reference sub-block is determined for each sub-block based on the combination of displacement vector (DV) and motion vector offset (MVO) associated with the corresponding sub-block.

[0202] At (S1930), determine the motion vector (MV) field in the corresponding juxtaposed reference subblock for each subblock in the current block.

[0203] At (S1940), the corresponding reference template for each sub-block is derived based on the MV field of the determined juxtaposed reference sub-block.

[0204] At (S1950), multiple sub-blocks of the current block are reconstructed by predicting each sub-block using the corresponding reference template in the SbTMVP mode.

[0205] To determine the corresponding juxtaposed reference sub-block, a search region is defined, located in the current image and one of its reference images. One or more reference blocks are determined based on template matching between the template of the current block and the templates of each of the one or more reference blocks in the search region. The template of the current block includes samples adjacent to the current block. The template of each of the one or more reference blocks includes samples adjacent to the corresponding reference block of the one or more reference blocks. The corresponding juxtaposed reference sub-block of each sub-block is determined as the sub-block juxtaposed with the corresponding sub-block in one of the one or more reference blocks.

[0206] The template matching of the current block with the template of each of one or more reference blocks is determined based on one of the following: SAD, SATD, SSE, subsampling SAD, and mean-removed SAD.

[0207] To determine one or more reference blocks, multiple candidate reference blocks are identified within a search area. Multiple cost values ​​are determined based on template matching between the current block's template and the templates of the multiple candidate reference blocks. One or more reference blocks are then identified as one or more candidate reference blocks corresponding to one or more of the lowest cost values ​​among the multiple candidate reference blocks.

[0208] In one embodiment, the search area includes one of the following: (i) an area centered on the position juxtaposed with the current block in the reference image; and (ii) an area centered on the current block in the current image.

[0209] In one example, the search region can be determined based on the DV. The DV can be derived from either (i) the motion vectors of the spatially neighboring blocks of the current block or (ii) the motion vectors of the merge candidate list of the current block.

[0210] In one example, the search area is defined as the region centered on the sample indicated by the DV, and the region is one of the following: square, rectangle, and rhombus.

[0211] In one example, the search area is defined as a set of samples centered on the sample indicated by the DV. This set of samples is located at at least one of the following locations: 0 degrees, 45 degrees, 90 degrees, or 135 degrees relative to the sample indicated by the DV.

[0212] To determine one or more reference blocks, a first reference block can be determined within the one or more reference blocks. The first reference block is indicated by a first displacement vector (DV) from the template of the current block to the template of the first reference block. In one example, the first DV may be derived based on template matching, such that the first DV corresponds to a cost value associated with the difference between the template of the first reference block and the template of the current block. In one example, the first DV may be transmitted in the form of signaling.

[0213] The search area can be determined based on a first DV derived before template matching. A first reference block among one or more reference blocks is determined based on a second DV derived from the template of the current block to the template of a first reference block. The second DV is derived based on template matching, such that the second DV corresponds to a cost value associated with the difference between the template of the first reference block and the template of the current block.

[0214] To reconstruct sub-blocks of the current block, one or more MVs of the first sub-block among multiple sub-blocks in the current block are determined based on one or more motion vectors (MVs) of sub-blocks juxtaposed with the first sub-block in one or more reference blocks. One or more predicted sub-blocks of the first sub-block among multiple sub-blocks are determined based on the one or more MVs of the first sub-block. A predicted sample of the first sub-block is determined based on a combination or weighted combination of one or more predicted sub-blocks.

[0215] In some embodiments, multiple candidate reference blocks for the current block are determined based on multiple displacement vectors (DVs). Each of the multiple candidate reference blocks is indicated by a corresponding DV among the multiple DVs. One or more reference blocks for the current block are determined from the multiple candidate reference blocks based on one or more cost values ​​of template matching.

[0216] After (S1940), the process proceeds to (S1999) and terminates.

[0217] Procedure (1900) can be adjusted as appropriate. Steps in procedure (1900) can be modified and / or omitted. Additional steps can be added. Any suitable execution order can be used.

[0218] like Figure 20 As shown, process (2000) can start from (S2001) and proceed to (S2010). At (S2010), a search area is determined, which is located in one of the current image and the reference image of the current image.

[0219] At (S2020), one or more reference blocks are determined based on template matching between the template of the current block and the templates of each of the one or more reference blocks in the search region. The template of the current block includes samples adjacent to the current block. The template of each of the one or more reference blocks includes samples adjacent to the reference blocks corresponding to the one or more reference blocks.

[0220] At (S2030), a sub-block prediction sample for the current block is generated based on the sub-blocks of one or more determined reference blocks.

[0221] Then, the process proceeds to (S2099) and terminates.

[0222] Procedure (2000) can be adjusted as appropriate. Steps in procedure (2000) can be modified and / or omitted. Additional steps can also be added. Any suitable execution order can be used.

[0223] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 21 A computer system (2100) suitable for implementing some embodiments of the disclosed subject matter is shown.

[0224] Computer software can be encoded using any suitable machine code or computer language. It can be assembled, compiled, linked, or similarly processed to create code containing instructions that can be executed directly or through interpretation, microcode execution, or other means by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0225] The instructions can be executed on various types of computers or their components, including personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0226] Figure 21 The components of the computer system (2100) shown are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement on any component or combination of components shown in the exemplary embodiments of the computer system (2100).

[0227] The computer system (2100) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users via, for example, tactile input (e.g., key presses, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0228] Human-machine interface input devices may include one or more of the following (only one of each is shown): keyboard (2101), mouse (2102), touchpad (2103), touch screen (2110), data glove (not shown), joystick (2105), microphone (2106), scanner (2107), and camera (2108).

[0229] The computer system (2100) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (2110), data gloves (not shown), or joystick (2105), but may also include tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (2109), headphones (not shown)), visual output devices (e.g., screens (2110), including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light-emitting diode (OLED) screens, each with or without touchscreen input capability, each with or without tactile feedback capability—some of which are capable of outputting two-dimensional or more than three-dimensional visual outputs in a manner such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0230] The computer system (2100) may also include human-accessible storage devices and their associated media, such as optical media (2120) including CD / DVDROM / RW with CD / DVD or similar media (2120), thumb drives (2122), removable hard disk drives or solid-state drives (2123), conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated read-only memory (ROM) / application-specific integrated circuits (ASICs) / programmable logic devices (PLDs) such as security dongles (not shown).

[0231] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0232] The computer system (2100) may also include an interface (2154) to one or more communication networks (2155). The network may be wireless, wired, or fiber optic. The one or more communication networks (2155) may also be local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), vehicular and industrial networks, real-time networks, latency-tolerant networks, etc. Examples of networks include LANs such as Ethernet and Local Area Networks (LANs), cellular networks including Global System for Mobile Communications (GSM), 3G, 4G, 5G, and Long-Term Evolution (LTE), cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, and vehicular and industrial networks including Controller Area Network (CANBus), etc. Some networks typically require external network interface adapters that connect to certain general-purpose data ports or peripheral buses (2149) (e.g., the Universal Serial Bus (USB) port of a computer system (2100)); others are typically integrated into the core of the computer system (2100) via connections to system buses such as Ethernet interfaces in PC computer systems or cellular network interfaces in smartphone computer systems. Using any of these networks, the computer system (2100) can communicate with other entities. This communication can be unidirectional and receive-only (e.g., broadcast television), unidirectional and transmit-only (e.g., to a CANbus device), or bidirectional, such as to other computer systems using local or wide-area digital networks. As described above, certain protocols and protocol stacks can be used on each of these networks and network interfaces.

[0233] The aforementioned human-machine interface device, human-accessible storage device, and network interface can be attached to the core (2140) of the computer system (2100).

[0234] The core (2140) may include one or more central processing units (CPU) (2141), graphics processing units (GPUs) (2142), dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) (2143), task-specific hardware accelerators (2144), graphics adapters (2150), etc. These devices, along with read-only memory (ROM) (2145), random access memory (2146), and internal mass storage such as internal non-user-accessible hard disk drives (SDs) (2147), may be connected via a system bus (2148). In some computer systems, the system bus (2148) may be accessed as one or more physical connectors to allow for the expansion of additional CPUs, GPUs, etc. Peripheral devices may be connected directly or via a peripheral bus (2149) to the core's system bus (2148). In one example, a screen (2110) may be connected to a graphics adapter (2150). The architecture of peripheral buses includes Peripheral Component Interconnect (PCI), USB, etc.

[0235] The CPU (2141), GPU (2142), FPGA (2143), and accelerator (2144) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (2145) or Random Access Memory (RAM) (2146). Transient data can also be stored in RAM (2146), while permanent data can be stored, for example, in internal mass storage (2147). Fast storage and retrieval of any storage device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (2141), GPUs (2142), mass storage (2147), ROM (2145), RAM (2146), etc.

[0236] Computer-readable media may contain computer code for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or may be of a type known and available to those skilled in the art of computer software.

[0237] By way of example and not limitation, a computer system having an architecture (2100), particularly a core (2140), can perform functions resulting from software contained in one or more tangible computer-readable media via a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such a computer-readable medium can be a medium associated with a user-accessible mass storage as described above, as well as some memory of the non-transitory core (2140), such as internal mass storage (2147) or ROM (2145). Software implementing various embodiments of this disclosure can be stored in such a device and executed by the core (2140). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core (2140), and particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (2146) and modifying such data structures according to software-defined processes. In addition, or alternatively, the computer system may provide functionality as a result of hard-wired or otherwise incorporated logic (e.g., an accelerator (2144)) that may replace or operate with software to perform the specific process or a specific portion of the specific process described herein. References to software may include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry containing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0238] Appendix A: Abbreviations

[0239] Joint Exploration Model (JEM)

[0240] Versatile Video Coding (VVC)

[0241] Motion Vector (MV)

[0242] Benchmark Set (BMS)

[0243] High Efficiency Video Coding (HEVC)

[0244] Supplemental Enhancement Information (SEI)

[0245] Video User Information (VUI)

[0246] Group of Pictures (GOP)

[0247] Transforming Unit (TUs)

[0248] Prediction Unit (PU)

[0249] Coding Tree Unit (CTU)

[0250] Coding Tree Block (CTB)

[0251] Prediction Block (PB)

[0252] Assume a Hypothetical Reference Decoder (HRD)

[0253] Signal-to-noise ratio (SNR)

[0254] Central Processing Unit (CPU)

[0255] Graphics Processing Unit (GPU)

[0256] Cathode ray tube (CRT)

[0257] Liquid Crystal Display (LCD)

[0258] Organic Light-Emitting Diode (OLED)

[0259] Compact Disc (CD)

[0260] Digital Versatile Disc (DVD)

[0261] Read-Only Memory (ROM)

[0262] Random Access Memory (RAM)

[0263] Application-Specific Integrated Circuit (ASIC)

[0264] Programmable Logic Device (PLD)

[0265] Local Area Network (LAN)

[0266] Global System for Mobile Communications (GSM)

[0267] Long-Term Evolution (LTE)

[0268] Controller Area Network (CANBus)

[0269] Universal Serial Bus (USB)

[0270] Peripheral Component Interconnect (PCI)

[0271] Field-Programmable Gate Array (FPGA)

[0272] Solid State Drive (SSD)

[0273] Integrated Circuit (IC)

[0274] Control Unit (CU)

[0275] While several exemplary embodiments have been described in this disclosure, numerous modifications, variations, and various alternative equivalents are possible and fall within the scope of this disclosure. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, although not expressly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

Claims

1. A decoding method executed in a decoder, characterized in that, include: Receive an encoded video bitstream including the current block in the current image, wherein the current block includes multiple sub-blocks, and the current block is predicted by motion vector prediction SbTMVP mode based on template matching of the sub-blocks; The corresponding juxtaposition reference sub-block is determined for each sub-block based on the combination of displacement vector DV and motion vector offset MVO associated with the corresponding sub-block. Determine the motion vector MV field in the corresponding juxtaposed reference sub-block for each sub-block in the current block; Based on the determined MV field of the juxtaposed reference sub-blocks, derive the corresponding reference template for each sub-block; and The multiple sub-blocks of the current block are reconstructed by predicting each sub-block using the corresponding reference template in the SbTMVP mode.

2. The method according to claim 1, characterized in that, Determining the corresponding juxtaposed reference sub-block further includes: Define a search area, which is located in one of the current image and a reference image of the current image; Based on template matching between the template of the current block and the template of each of one or more reference blocks in the search region, one or more reference blocks of the current block are determined, wherein the template of the current block includes samples adjacent to the current block, and the template of each of the one or more reference blocks includes samples adjacent to the corresponding reference block of the one or more reference blocks; and The corresponding juxtaposed reference sub-block for each sub-block is determined as the sub-block juxtaposed with the sub-block corresponding to one of the one or more reference blocks.

3. The method according to claim 2, characterized in that, The template matching between the template of the current block and the template of each of the one or more reference blocks is determined based on one of the following: absolute difference sum (SAD), absolute transform difference sum (SATD), sum of squared errors (SSE), subsampling SAD, and mean-removed SAD.

4. The method according to claim 3, characterized in that, The determination of the one or more reference blocks further includes: Identify multiple candidate reference blocks within the search region; Multiple cost values ​​are determined based on the template matching between the template of the current block and the templates of the plurality of candidate reference blocks; and The one or more reference blocks are determined as the one or more candidate reference blocks that correspond to one or more of the lowest cost values ​​among the plurality of candidate reference blocks.

5. The method according to claim 2, characterized in that, The search area includes one of the following: (i) an area centered on the position juxtaposed with the current block in the reference image; and (ii) an area centered on the current block in the current image.

6. The method according to claim 2, characterized in that, Determining the search area further includes: The search region is determined based on the displacement vector DV, which is derived from one of (i) the motion vectors of the spatially adjacent blocks of the current block and (ii) the motion vectors of the merge candidate list of the current block.

7. The method according to claim 6, characterized in that, Determining the search area further includes: The search area is defined as the region centered on the sample indicated by the DV, and the region is one of the following: rectangle and rhombus.

8. The method according to claim 6, characterized in that, Determining the search area further includes: The search area is defined as a set of samples centered on the sample indicated by the DV, which is located at at least one of the following locations: 0 degrees, 45 degrees, 90 degrees, or 135 degrees relative to the sample indicated by the DV.

9. The method according to claim 2, characterized in that, The determination of the one or more reference blocks further includes: A first reference block among the one or more reference blocks is determined by a first displacement vector DV between the template of the current block and the template of the first reference block, the first DV being one of the following: (i) derived based on the template matching such that the first DV corresponds to a cost value associated with the difference between the template of the first reference block and the template of the current block; and (ii) transmitted in the form of signaling.

10. The method according to claim 2, characterized in that: Determining the search area further includes: The search region is determined based on a first displacement vector DV derived prior to the template matching; and The determination of the one or more reference blocks further includes: A first reference block is determined from the template of the current block to the template of the first reference block by a second DV derived from the template matching of the template, such that the second DV corresponds to a cost value associated with the difference between the template of the first reference block and the template of the current block.

11. The method according to claim 2, characterized in that, The plurality of sub-blocks that reconstruct the current block also include: Based on one or more motion vectors MV of the sub-block juxtaposed with the first sub-block in the one or more reference blocks, determine one or more MVs of the first sub-block among the plurality of sub-blocks in the current block; Based on the one or more MVs of the first sub-block, determine one or more predicted sub-blocks of the first sub-block among the plurality of sub-blocks; and The prediction sample of the first sub-block is determined based on a combination or weighted combination of one or more prediction sub-blocks.

12. The method according to claim 2, characterized in that, Also includes: Multiple candidate reference blocks are determined based on multiple displacement vectors (DVs) for the current block, and each candidate reference block is indicated by a corresponding DV among the multiple DVs. as well as Based on one or more cost values ​​matched by the template, the one or more reference blocks of the current block are determined from the plurality of candidate reference blocks.

13. An apparatus for performing decoding, characterized in that, include: A processing circuit configured to perform the method as described in any one of claims 1 to 12.

14. An encoding method for execution, characterized in that, The method includes: Define a search region, which is located in one of the current image and a reference image of the current image, wherein the current image includes the current block; Based on template matching between the template of the current block and the template of each of the one or more reference blocks in the search area, one or more reference blocks of the current block are determined, wherein the template of the current block includes samples adjacent to the current block, and the template of each of the one or more reference blocks includes samples adjacent to the reference blocks corresponding to the one or more reference blocks; Based on the determined sub-blocks of the one or more reference blocks, generate sub-block prediction samples for the current block.

15. An encoding apparatus for execution, characterized in that, The device includes: Processing circuit, the processing circuit being configured to: Define a search region, which is located in one of the current image and a reference image of the current image, wherein the current image includes the current block; Based on template matching between the template of the current block and the template of each of the one or more reference blocks in the search area, one or more reference blocks of the current block are determined, wherein the template of the current block includes samples adjacent to the current block, and the template of each of the one or more reference blocks includes samples adjacent to the reference blocks corresponding to the one or more reference blocks; Based on the determined sub-blocks of the one or more reference blocks, generate sub-block prediction samples for the current block.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed by a processor, performs the method according to any one of claims 1 to 12 or 14.

17. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 12 or 14.

Images