Improvements in merge candidate derivation using intra template matching

Abstract

The various implementations described herein include methods and systems for encoding and decoding video. In one aspect, the method includes: receiving a video bitstream comprising a current image, the current image being composed of multiple blocks including a current block; identifying a reference block by applying a template matching technique to the current block; identifying a reference vector of the reference block, wherein the reference vector is a motion vector or a block vector; populating a merge candidate list of the current block with the reference vector; and reconstructing the current block using information from the merge candidate list.

Description

[0001] Related applications

[0002] This application is a continuation to U.S. Patent Application No. 19 / 039,629, filed January 28, 2025, which claims priority to U.S. Provisional Patent Application No. 63 / 627,685, filed January 31, 2024, entitled “Merge Candidate Derivation Improvement By Using Intra Template-Matching,” each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The disclosed implementations generally relate to video encoding and decoding, including but not limited to systems and methods for using template matching to identify motion vector candidates. Background Technology

[0004] Digital video is supported by various electronic devices such as digital televisions, laptop or desktop computers, tablet computers, digital camera devices, digital recording devices, digital media players, video game consoles, smartphones, video conferencing equipment, video streaming devices, etc. Electronic devices send and receive or otherwise transmit digital video data across communication networks, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video codecs can be used to compress video data according to one or more video codec standards before transmission or storage. Video codecs can be performed by hardware and / or software on electronic / client devices or servers providing cloud services.

[0005] Video codecs typically employ prediction methods that leverage the inherent redundancy in video data (e.g., inter-frame prediction, intra-frame prediction, etc.). Video codecs aim to compress video data into a form using a lower bitrate while avoiding or minimizing video quality degradation. Several video codec standards have been developed. For example, High-Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. The ITU-T and ISO / IEC released the HEVC / H.265 standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Versatile Video Coding (VVC / H.266) is a video compression standard designed as a successor to HEVC. The ITU-T and ISO / IEC released the VVC / H.266 standard in 2020 (Revision 1) and 2022 (Revision 2). AOMedia Video 1 (AV1) is an open video codec format designed as an alternative to HEVC. On January 8, 2019, a validation version 1.0.0 with errata table 1 was released. The Enhanced Compression Model (ECM) is a video codec standard currently under development. ECM aims to significantly improve compression efficiency, surpassing existing standards such as HEVC / H.265 and VVC, essentially allowing for higher quality video at lower bitrates. ECM version 13 was released in MPEG 146 on July 7, 2024. Summary of the Invention

[0006] Furthermore, this disclosure describes a set of methods for video (image) compression, such as using intra-frame template matching to populate a merge candidate list used to reconstruct the current block. For example, template matching processing can be applied to the currently reconstructed image to find adjacent and / or non-adjacent blocks that have motion vector (MV) information and / or block vector (BV) information that can be further used by the current block. The MV and / or BV from adjacent or non-adjacent blocks can be used to construct the merge candidate list. Some implementations include applying template matching processing in the currently reconstructed image to find adjacent and / or non-adjacent blocks with sub-block-level MV and / or BV information. Some implementations include applying chained MVs to a specific intra-frame prediction pattern (e.g., within a P / B slice). This intra-frame prediction pattern can employ template matching methods within a predefined region in the current image to build a list of N candidates, where N is a positive number. These candidate predictors can be ranked in ascending order by template cost and indicated by N BVs respectively. An index can be signaled to indicate which candidate in the list is used as an intra-frame predictor.

[0007] The techniques described in the preceding paragraphs can be used to improve encoding / decoding accuracy, for example, by increasing the diversity of the merge list and using MV information from reference blocks that more closely match the current block. Additionally, signaling overhead can be reduced by using a merge candidate list and by signaling the index of the list (instead of directly signaling the MV). Compression efficiency can also be improved by using the methods described herein to populate the merge candidate list more efficiently.

[0008] According to some implementations, a method for video decoding is provided. The method includes (i) receiving a video bitstream (e.g., an encoded / decoded video sequence) comprising a current image, the current image being composed of multiple blocks including a current block; and (ii) identifying a reference block by applying a template matching technique to the current block. The method further includes (iii) identifying a reference vector for the reference block, wherein the reference vector is an MV or a BV; (iv) populating a merge candidate list for the current block using the reference vector; and (v) reconstructing the current block using information from the merge candidate list.

[0009] According to some implementations, a video encoding method includes (i) receiving video data (e.g., a source video sequence) including a current image, the current image comprising multiple blocks, the multiple blocks including the current block. The method includes (ii) identifying a reference block by applying a template matching technique to the current block; (iii) identifying a reference vector for the reference block, the reference vector being an MV or BV; (iv) populating a merge candidate list for the current block with the reference vector; and (v) encoding the current block using information from the merge candidate list.

[0010] According to some embodiments, a computing system, such as a streaming system, server system, personal computer system, or other electronic device, is provided. The computing system includes a control circuitry system and a memory storing one or more sets of instructions. The one or more sets of instructions include instructions for performing any of the methods described herein. In some embodiments, the computing system includes encoder components and decoder components (e.g., a transcoder).

[0011] According to some embodiments, a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more sets of instructions executable by a computing system. The one or more sets of instructions include instructions for performing any of the methods described herein.

[0012] Therefore, methods, apparatus, and systems for encoding and decoding video are disclosed. Such methods, apparatus, and systems can supplement or replace conventional methods, apparatus, and systems for video encoding / decoding.

[0013] The features and advantages described in this specification are not necessarily all included, and in particular, some additional features and advantages will be apparent to those skilled in the art in light of the accompanying drawings, specification, and claims provided in this disclosure. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes and is not necessarily intended to depict or limit the subject matter described herein. Attached Figure Description

[0014] To provide a more detailed understanding of this disclosure, a more specific description can be made by referring to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the drawings only show relevant features of this disclosure and are therefore not necessarily intended to be limiting, as those skilled in the art will understand upon reading this disclosure that other valid features may be permissible.

[0015] Figure 1 This is a block diagram illustrating an example communication system according to some implementations.

[0016] Figure 2A This is a block diagram illustrating example elements of an encoder component according to some embodiments.

[0017] Figure 2B This is a block diagram illustrating example elements of a decoder component according to some embodiments.

[0018] Figure 3 This is a block diagram illustrating an example server system according to some implementation methods.

[0019] Figure 4AAn example intra-frame block copying technique according to some implementations is shown.

[0020] Figure 4B An example of generating a list of merge candidates according to some implementation methods is shown.

[0021] Figure 4C Example predefined locations in a block are shown according to some implementations.

[0022] Figure 4D An example chained MV is shown according to some implementation methods.

[0023] Figure 4E An example of a merge candidate build with sub-block level MVs is shown according to some implementations.

[0024] Figure 4F Another example of generating a list of merge candidates according to some implementation methods is shown.

[0025] Figure 5A This illustrates the use of template matching processing to identify the BV of the current block according to some implementations.

[0026] Figure 5B This illustrates how, according to some implementations, intra-frame modes are determined based on the most frequent patterns found within a block.

[0027] Figure 5C An example of intra-frame template matching refinement according to some implementations is shown.

[0028] Figure 6A An example video decoding process according to some implementation methods is shown.

[0029] Figure 6B An example video encoding process according to some implementation methods is shown.

[0030] By convention, the various features shown in the accompanying drawings are not necessarily drawn to scale, and similar reference numerals may be used to indicate similar features throughout the specification and the drawings. Detailed Implementation

[0031] This disclosure describes a video / image compression technique that includes using template matching to populate a merge candidate list used to reconstruct the current block. For example, a reference block can be identified by applying a template matching technique to the current block. A reference vector (e.g., MV or BV) for the reference block can be identified, and the reference vector can be used to generate / populate a merge candidate list for the current block. The current block can then be reconstructed using information from the merge candidate list. In another example, a list of candidate predictors for the current block is generated by applying a template matching technique to the current block. In this example, at least one candidate predictor from the list of candidate predictors can be used to reconstruct the current block.

[0032] Populating the candidate list for the current block based on reference blocks identified using template matching can offer advantages including increased codec accuracy. For example, codec accuracy can be increased by using MV information from reference blocks identified using intra-frame template matching that more closely match the current block (rather than relying solely on neighboring blocks). Signaling overhead can be reduced when using a candidate list and signaling to the list's index (rather than directly signaling MV or other codec information). Codec efficiency can also be improved by, for example, populating the candidate list more efficiently based on template matching costs, thus providing better MV information for the current block.

[0033] Example systems and devices

[0034] Figure 1 This is a block diagram illustrating a communication system 100 according to some embodiments. The communication system 100 includes a source device 102 and a plurality of electronic devices 120 (e.g., electronic devices 120-1 to 120-m) communicatively coupled to each other via one or more networks. In some embodiments, the communication system 100 is a streaming system, for example, for use with video-enabled applications such as video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0035] Source device 102 includes a video source 104 (e.g., a camera device component or media storage) and an encoder component 106. In some embodiments, the video source 104 is a digital camera device (e.g., configured to create an uncompressed video sample stream). The encoder component 106 generates one or more encoded video bitstreams from the video stream. The video stream from video source 104 can have a higher data volume compared to the encoded video bitstream 108 generated by encoder component 106. Because the encoded video bitstream 108 has a lower data volume (less data) compared to the video stream from video source 104, it requires less bandwidth to transmit and less storage space to store compared to the video stream from video source 104. In some embodiments, source device 102 does not include encoder component 106 (e.g., configured to transmit uncompressed video to network 110).

[0036] One or more networks 110 represent any number of networks that transmit information between source device 102, server system 112, and / or electronic device 120, including, for example, wired (connected) and / or wireless communication networks. One or more networks 110 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet.

[0037] One or more networks 110 include server systems 112 (e.g., distributed / cloud computing systems). In some embodiments, server system 112 is or includes streaming servers (e.g., configured to store and / or distribute video content such as encoded video streams from source device 102). Server system 112 includes codec components 114 (e.g., configured to encode and / or decode video data). In some embodiments, codec components 114 include encoder components and / or decoder components. In various embodiments, codec components 114 are instantiated as hardware, software, or a combination thereof. In some embodiments, codec components 114 are configured to decode encoded video bitstream 108 and re-encode the video data using different encoding standards and / or methods to generate encoded video data 116. In some embodiments, server system 112 is configured to generate multiple video formats and / or encodings based on encoded video bitstream 108. In some embodiments, server system 112 serves as a Media-Aware Network Element (MANE). For example, server system 112 can be configured to trim the encoded video bitstream 108 to tailor potentially different bitstreams for one or more of the electronic devices 120. In some implementations, MANE is provided separately from server system 112.

[0038] Electronic device 120-1 includes a decoder component 122 and a display 124. In some embodiments, the decoder component 122 is configured to decode encoded video data 116 to generate an outgoing video stream that can be displayed on a display or other type of presentation device. In some embodiments, one or more of the electronic devices 120 do not include a display component (e.g., communicatively coupled to an external display device and / or includes media storage). In some embodiments, electronic device 120 is a streaming client. In some embodiments, electronic device 120 is configured to access server system 112 to obtain encoded video data 116.

[0039] The source device and / or multiple electronic devices 120 are sometimes referred to as “terminal devices” or “user devices”. In some implementations, one or more of the electronic devices 120 and / or the source device 102 are instances of server systems, personal computers, portable devices (e.g., smartphones, tablets, or laptops), wearable devices, video conferencing equipment, and / or other types of electronic devices.

[0040] In an example operation of communication system 100, source device 102 sends an encoded video bitstream 108 to server system 112. For example, source device 102 may encode and decode a stream of images captured by the source device. Server system 112 receives the encoded video bitstream 108 and may decode and / or encode the encoded video bitstream 108 using codec component 114. For example, server system 112 may apply encoding to the video data that is better for network transmission and / or storage. Server system 112 may send encoded video data 116 (e.g., one or more encoded video bitstreams) to one or more electronic devices 120. Each electronic device 120 may decode the encoded video data 116 and optionally display video images.

[0041] Figure 2AThis is a block diagram illustrating example elements of an encoder component 106 according to some embodiments. The encoder component 106 receives video data (e.g., a source video sequence) from a video source 104. In some embodiments, the encoder component includes a receiver (e.g., transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component 106 receives the video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder component 106). The video source 104 may provide the source video sequence in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 YCrCb or RGB), and any suitable sampling structure (e.g., YCrCb 4:2:0 or YCrCb 4:4:4). In some embodiments, the video source 104 is a storage device storing previously captured / prepared video. In some embodiments, the video source 104 is a camera device that captures local image information as a video sequence. Video data can be provided as multiple individual images that are given motion when viewed sequentially. Each image can be organized as a spatial array of pixels, where, depending on the sampling structure, color space, etc., each pixel may include one or more samples. The relationship between pixels and samples will be readily understood by those skilled in the art.

[0042] Encoder component 106 is configured to encode, decode, and / or compress images of a source video sequence into an encoded video sequence 216 in real time or under other time constraints required by the application. In some embodiments, encoder component 106 is configured to perform a conversion between the source video sequence and a bitstream of visual media data (e.g., a video bitstream). Implementing an appropriate encoding / decoding speed is a function of controller 204. In some embodiments, controller 204 controls and is functionally coupled to other functional units as described below. Parameters set by controller 204 may include rate control-related parameters (e.g., image skipping, quantizer, and / or the λ value of rate-distortion optimization techniques), image size, group of pictures (GOP) layout, maximum MV search range, etc. Other functions of controller 204 can be readily identified by those skilled in the art, as these functions may belong to encoder component 106 optimized for a particular system design.

[0043] In some implementations, encoder component 106 is configured to operate within an encoding / decoding loop. In a simplified example, the encoding / decoding loop includes a source codec 202 (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder 210. Decoder 210 reconstructs the symbols in a manner similar to that of the (remote) decoder to create sample data (when compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory 208. Since decoding the symbol stream produces bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference image memory 208 are also bit-accurate between the local and remote encoders. In this way, the encoder's prediction portion interprets the same sample values ​​as the sample values ​​that the decoder interprets during prediction as reference image samples.

[0044] The operation of decoder 210 can be combined with a remote decoder, for example, as shown below. Figure 2B The operation of the decoder component 122 is the same as described in the detailed description. However, a brief reference is provided. Figure 2B Since symbols are available and encoding / decoding symbols into a codec video sequence by entropy codec 214 and parser 254 can be lossless, the entropy decoding portion of decoder component 122, including buffer memory 252 and parser 254, does not need to be fully implemented in local decoder 210.

[0045] Besides parsing / entropy decoding, the decoder techniques described in this paper can exist in the corresponding encoders in essentially the same form. For this reason, the subject matter focuses on decoder operations. Furthermore, the description of the encoder techniques can be simplified, as the encoder techniques are inverses of the decoder techniques.

[0046] As part of the operation of source codec 202, source codec 202 can perform motion-compensated predictive coding, which predictively codes the input frame with reference to one or more previously codecped frames from the video sequence designated as reference frames. In this manner, codec engine 212 encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frame, which can be selected as the prediction reference for the input frame. Controller 204 can manage the coding and decoding operations of source codec 202, including, for example, setting parameters and subgroup parameters for encoding video data.

[0047] Decoder 210 decodes encoded video data based on symbols created by source codec 202, which can be designated as reference frames. The operation of codec engine 212 can advantageously support lossy processing. When encoded video data is processed by video decoder (… Figure 2AWhen decoded at (not shown), the reconstructed video sequence can be a copy of the source video sequence with some errors. Decoder 210 can replicate the decoding process performed on the reference frame by a remote video decoder, and the reconstructed reference frame can be stored in reference image memory 208. In this way, encoder component 106 locally stores a copy of the reconstructed reference frame that shares common content (no transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.

[0048] Predictor 206 can perform a prediction search against codec engine 212. That is, for a new frame to be encoded or decoded, predictor 206 can search the reference image memory 208 for sample data (as candidate reference pixel blocks) or specific metadata such as reference image MV, block shape, etc., that can be used as appropriate prediction references for the new image. Predictor 206 can operate pixel-by-pixel based on the sample blocks to find appropriate prediction references. As determined by the search results obtained by predictor 206, the input image can have prediction references obtained from multiple reference images stored in reference image memory 208.

[0049] The outputs of all the aforementioned functional units can undergo entropy encoding and decoding in entropy codec 214. Entropy codec 214 converts the symbols generated by the various functional units into a codec video sequence by lossless compression of the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, and / or arithmetic coding).

[0050] In some implementations, the output of entropy codec 214 is coupled to a transmitter. The transmitter can be configured to buffer the encoded video sequence created by entropy codec 214 in preparation for transmission via communication channel 218, which can be a hardware / software link to a storage device storing the encoded video data. The transmitter can be configured to combine the encoded video data from source codec 202 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown). In some implementations, the transmitter can send additional data along with the encoded video. Source codec 202 can include such data as part of the encoded video sequence. Additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant images and slices, Supplementary Enhancement Information (SEI) messages, fragments of Visual Usability Information (VUI) parameter sets, etc.

[0051] Controller 204 can manage the operation of encoder component 106. During encoding and decoding, controller 204 can assign a specific codec image type to each encoded image, which may affect the encoding and decoding techniques applied to the corresponding image. For example, images can be assigned as intra-frame images (I-images), prediction images (P-images), or bidirectional prediction images (B-images). Intra-frame images can be encoded and decoded without using any other frames in the sequence as prediction sources. Some video codecs allow different types of intra-frame images, including, for example, Independent Decoder Refresh (IDR) images. Those skilled in the art are familiar with those variations of I-images and their corresponding applications and characteristics, and therefore they will not be repeated here. Predictive images can be encoded and decoded using inter-frame prediction or intra-frame prediction, which utilizes at most one MV and reference index to predict sample values ​​for each block. Bidirectional prediction images can be encoded and decoded using inter-frame prediction or intra-frame prediction, which utilizes at most two MVs and reference indexes to predict sample values ​​for each block. Similarly, multiple prediction images can use more than two reference images and associated metadata to reconstruct a single block.

[0052] Source images can typically be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and encoded / decoded block by block. Predictive encoding / decoding of these blocks can be performed with reference to other (already encoded / decoded) blocks, determined by the encoding / decoding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be nonpredictively encoded / decoded, or blocks of an I-image can be predictively encoded / decoded (spatial prediction or intra-frame prediction) with reference to already encoded / decoded blocks of the same image. Pixel blocks of a P-image can be nonpredictively encoded / decoded with reference to a previously encoded / decoded reference image via spatial prediction or temporal prediction. Blocks of a B-image can be nonpredictively encoded / decoded with reference to one or two previously encoded / decoded reference images via spatial prediction or temporal prediction.

[0053] Video can be captured as multiple source images (video images) in a time-series manner. Intra-frame image prediction (often simply called intra-prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In the example, a specific image in the encoding / decoding process (referred to as the current image) is segmented into blocks. Where a block in the current image is similar to a reference block in a previously encoded and decoded reference image still buffered in the video, the block in the current image can be encoded and decoded using a vector called a motion vector. The MV points to the reference block in the reference image, and when using multiple reference images, the MV can have a third dimension identifying the reference images.

[0054] Encoder component 106 can perform encoding and decoding operations according to any predetermined video codec technology or standard described herein. In operation, encoder component 106 can perform various compression operations, including predictive codec operations utilizing temporal and spatial redundancy in the input video sequence. Therefore, the encoded and decoded video data can conform to the syntax specified by the video codec technology or standard used.

[0055] Figure 2B This is a block diagram illustrating example elements of a decoder component 122 according to some embodiments. Figure 2B The decoder component 122 is coupled to channel 218 and display 124. In some embodiments, the decoder component 122 includes a transmitter coupled to loop filter 256 and configured to transmit data to display 124 (e.g., via a wired or wireless connection).

[0056] In some embodiments, decoder component 122 includes a receiver coupled to channel 218 and configured to receive data from channel 218 (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more codec video sequences decoded by decoder component 122. In some embodiments, the decoding of each codec video sequence is independent of other codec video sequences. Each codec video sequence may be received from channel 218, which may be a hardware / software link to a storage device storing the encoded video data. The receiver may receive the encoded video data along with other data, such as codec audio data and / or auxiliary data streams, which may be forwarded to their respective user entities (not depicted). The receiver may separate the codec video sequences from other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. The additional data may be included as part of the codec video sequence. Decoder component 122 may use the additional data to decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0057] According to some embodiments, the decoder component 122 includes a buffer memory 252, a parser 254 (sometimes also referred to as an entropy decoder), a scaler / inverse transform unit 258, an intra-frame image prediction unit 262, a motion compensation prediction unit 260, an aggregator 268, a loop filter unit 256, a reference image memory 266, and a current image memory 264. In some embodiments, the decoder component 122 is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuit systems. The decoder component 122 can be implemented at least partially in software.

[0058] Buffer memory 252 is coupled between channel 218 and parser 254 (e.g., to combat network jitter). In some embodiments, buffer memory 252 is separate from decoder component 122. In some embodiments, a separate buffer memory is provided between the output of channel 218 and decoder component 122. In some embodiments, a separate buffer memory is provided outside decoder component 122 (e.g., to combat network jitter) in addition to buffer memory 252 inside decoder component 122 (e.g., buffer memory 252 is configured to handle playback timing). Buffer memory 252 may not be necessary or may be small when receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network. Buffer memory 252 may be required to make the best use of packet networks such as the Internet; buffer memory 252 may be relatively large and / or have an adaptive size and may be implemented at least partially in an operating system or similar component outside decoder component 122.

[0059] Parser 254 is configured to reconstruct symbols 270 from the encoded video sequence. Symbols may include, for example, information for managing the operation of decoder component 122, and / or information for controlling presentation devices such as display 124. Control information for the presentation device may be in the form of, for example, supplementary enhancement information (SEI) messages or video availability information (VUI) parameter set fragments (not depicted). Parser 254 parses (entropy decodes) the encoded video sequence. The encoding and decoding of the encoded video sequence may be performed according to video codec techniques or standards, and may follow principles known to those skilled in the art, including variable-length codecs, Huffman codecs, arithmetic codecs with or without context sensitivity, etc. Parser 254 may extract a subgroup parameter set from the encoded video sequence for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to a group. Subgroups may include picture groups (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser 254 can also extract information such as transform coefficients, quantizer parameter values, and MV from the encoded and decoded video sequence.

[0060] Depending on the type of the encoded video picture or a portion of the encoded video picture (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 270 may involve multiple different units. Which units are involved and how these units are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 254. For simplicity, such subgroup control information flow between parser 254 and the following multiple units is not depicted.

[0061] The decoder component 122 can be conceptually subdivided into multiple functional units, and in some implementations, these units interact closely with each other and can be at least partially integrated with each other. However, for the sake of brevity, this paper retains the conceptual subdivision of the functional units.

[0062] The scaler / inverse transform unit 258 receives quantized transform coefficients and control information (such as which transform to use, block size, quantization factor, and quantization scaling matrix) as symbols 270 from the parser 254. The scaler / inverse transform unit 258 can output blocks including sample values, which can be input to the aggregator 268. In some cases, the output samples of the scaler / inverse transform unit 258 belong to intra-frame codec blocks; that is, blocks that do not use prediction information from previously reconstructed images, but can use prediction information from previously reconstructed portions of the current image. Such prediction information can be provided by the intra-frame image prediction unit 262. The intra-frame image prediction unit 262 can use surrounding reconstructed information obtained from the current (partially reconstructed) image from the current image memory 264 to generate blocks of the same size and shape as the blocks in the reconstruction. The aggregator 268 can add the prediction information already generated by the intra-frame image prediction unit 262 to the output sample information provided by the scaler / inverse transform unit 258 based on each sample.

[0063] In other cases, the output samples of the scaler / inverse transform unit 258 belong to the inter-frame encoding and decoding and potentially to the motion-compensated block. In such cases, the motion-compensated prediction unit 260 can access the reference image memory 266 to obtain samples for prediction. After motion compensation of the obtained samples according to the symbol 270 belonging to the block, these samples can be added by the aggregator 268 to the output of the scaler / inverse transform unit 258 (referred to in this case as residual samples or residual signals) to generate output sample information. The address in the reference image memory 266 from which the motion-compensated prediction unit 260 obtains the predicted samples can be controlled by MV. MV can be used by the motion-compensated prediction unit 260 in the form of symbol 270, which can have, for example, an X component, a Y component, and a reference image component. Motion compensation can also include, for example, interpolation of sample values ​​obtained from the reference image memory 266 when using subsampled exact MV, and MV prediction mechanisms.

[0064] The output samples of aggregator 268 can undergo various loop filtering techniques in loop filter unit 256. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and available to loop filter unit 256 as symbols 270 from parser 254. However, video compression techniques may also respond to metadata obtained during decoding of previous (in decoding order) portions of the encoded / decoded picture or encoded / decoded video sequence, and to sample values ​​obtained from previous reconstruction and loop filtering. The output of loop filter unit 256 may be a sample stream that can be output to a presentation device such as display 124 and stored in reference picture memory 266 for future inter-frame picture prediction.

[0065] Once a certain codec image is reconstructed, it can be used as a reference image for future predictions. Once a codec image is reconstructed and has been identified as a reference image (e.g., by parser 254), the current reference image can become part of the reference image memory 266, and a new current image memory can be reallocated before reconstructing subsequent codec images begins.

[0066] Decoder component 122 can perform decoding operations according to a predetermined video compression technique that may be recorded in any standard such as those described herein. As specified in a video compression technique document or standard, and particularly in a configuration file therein, the encoded / decoded video may conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded / decoded video sequence follows the syntax of the video compression technique or standard. Furthermore, the complexity of the encoded / decoded video sequence can be within a range limited by the level of the video compression technique or standard in order to conform to some video compression techniques or standards. In some cases, the level limits the maximum image size, maximum frame rate, maximum reconstruction sample rate (measured, for example, megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the level can be further restricted by the Hypothetical Reference Decoder (HRD) specification and metadata managed by the HRD buffer used for signaling in the encoded / decoded video sequence.

[0067] Figure 3This is a block diagram illustrating a server system 112 according to some embodiments. The server system 112 includes a control circuitry system 302, one or more network interfaces 304, a memory 314, a user interface 306, and one or more communication buses 312 for interconnecting these components. In some embodiments, the control circuitry system 302 includes one or more processors (e.g., CPU, GPU, and / or DPU). In some embodiments, the control circuitry system includes a field-programmable gate array, a hardware accelerator, and / or an integrated circuit (e.g., an application-specific integrated circuit).

[0068] Network interface 304 can be configured to interface with one or more communication networks (e.g., wireless networks, wired networks, and / or optical networks). These communication networks can be local, wide area, metropolitan area, vehicle-mounted, industrial, real-time, latency-tolerant, etc. Examples of communication networks include: local area networks such as Ethernet; wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; vehicle and industrial networks including CANbus, etc. Such communication can be one-way receiving (e.g., broadcast TV), one-way transmitting (e.g., to a CANbus device), or bidirectional (e.g., to other computer systems using local or wide area digital networks). Such communication can include communication to one or more cloud computing networks.

[0069] User interface 306 includes one or more output devices 308 and / or one or more input devices 310. Input devices 310 may include one or more of the following: keyboard, mouse, touchpad, touchscreen, data glove, joystick, microphone, scanner, camera, etc. Output devices 308 may include one or more of the following: audio output devices (e.g., speakers), visual output devices (e.g., displays or monitors), etc.

[0070] Memory 314 may include high-speed random access memory (e.g., DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices) and / or non-volatile memory (e.g., one or more disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state memory devices). Memory 314 may optionally include one or more storage devices remote from the control circuitry system 302. Memory 314, or alternatively, the non-volatile solid-state memory devices within memory 314 include non-transitory computer-readable storage media. In some embodiments, memory 314 or the non-transitory computer-readable storage media of memory 314 stores programs, modules, instructions, and data structures, or subsets or supersets thereof:

[0071] • Operating system 316, which includes procedures for handling various basic system services and for performing hardware-related tasks;

[0072] • A network communication module 318 for connecting the server system 112 to other computing devices via one or more network interfaces 304 (e.g., via wired and / or wireless connections);

[0073] • Codec module 320, which performs various functions related to encoding and / or decoding data such as video data. In some embodiments, codec module 320 is an instance of codec component 114. Codec module 320 includes, but is not limited to, one or more of the following:

[0074] Decoding module 322, which performs various functions related to decoding encoded data, such as those previously described with respect to decoder component 122; and

[0075] Encoding module 340, which performs various functions related to encoding data, such as those previously described with respect to encoder component 106; and

[0076] • For example, an image memory 352 for storing images and image data, used in conjunction with the encoding / decoding module 320. In some embodiments, the image memory 352 includes one or more of the following: a reference image memory 208, a buffer memory 252, a current image memory 264, and a reference image memory 266.

[0077] In some implementations, the decoding module 322 includes a parsing module 324 (e.g., configured to perform various functions previously described with respect to the parser 254), a transformation module 326 (e.g., configured to perform various functions previously described with respect to the scaler / inverse transformation unit 258), a prediction module 328 (e.g., configured to perform various functions previously described with respect to the motion compensation prediction unit 260 and / or the intra-frame picture prediction unit 262), and a filter module 330 (e.g., configured to perform various functions previously described with respect to the loop filter 256).

[0078] In some embodiments, the encoding module 340 includes a code module 342 (e.g., configured to perform various functions previously described with respect to source codec 202 and / or codec engine 212) and a prediction module 344 (e.g., configured to perform various functions previously described with respect to predictor 206). In some embodiments, the decoding module 322 and / or the encoding module 340 includes... Figure 3 A subset of the modules shown. For example, a shared prediction module is used by both the decoding module 322 and the encoding module 340.

[0079] Each of the modules identified above and stored in memory 314 corresponds to a set of instructions for performing the functions described herein. The modules identified above (e.g., instruction sets) do not need to be implemented as separate software programs, processes, or modules, and therefore various subsets of these modules can be combined or otherwise rearranged in various implementations. For example, the codec module 320 may optionally not include separate decoding and encoding modules, but instead use the same set of modules to perform both sets of functions. In some implementations, memory 314 stores a subset of the modules and data structures identified above. In some implementations, memory 314 stores additional modules and data structures not described above.

[0080] although Figure 3 A server system 112 according to some embodiments is shown, but Figure 3 This is intended more as a functional description of various features that can exist in one or more server systems than as a structural diagram of the implementation described herein. In practice, items shown individually may be combined and some items may be separated. For example, Figure 3 Some items shown individually can be implemented on a single server, and a single item can be implemented by one or more servers. The actual number of servers used to implement server system 112 and how features are distributed among them will vary depending on the implementation and, optionally, in part, depend on the amount of data traffic processed by the server system during peak usage periods and during average usage periods.

[0081] Example encoding and decoding techniques

[0082] The encoding and decoding processes and techniques described below can be performed at the aforementioned devices and systems (e.g., source device 102, server system 112, and / or electronic device 120). According to some implementations, an example method for populating a candidate list (for reconstructing the current block) using template matching techniques is described below.

[0083] As is known to those skilled in the art, intra-block copying is a technique that uses block characterization (BV) to identify the predicted block of the current block. BV is used to identify another block in the same image as the current block, whether adjacent or non-adjacent. BV can be explicitly signaled or implicitly derived. When BV is explicitly signaled, it is generally referred to as the intra-block copy (IBC) method. When BV is implicitly derived, for example, by comparing the template region (a set of neighboring reconstructed samples located adjacent to the block) between the current block and the candidate predicted block, it is generally referred to as the intra-template matching method or template-based intra-mode derivation (TIMD) method.

[0084] In some cases, the current codec block and its neighboring samples share similar texture characteristics. In such scenarios, the reconstructed samples of the current block's neighbors (collectively referred to as "templates") can be used to predict the current block. Template matching can be used in inter-frame prediction to derive a predicted block by calculating the distortion between the template of the current block and the template of the predicted block in the reference image. Template matching can also be applied to intra-frame prediction over the reconstructed region of the current image, a process known as "intra-frame template matching."

[0085] "Superblock size" or "coding tree unit (CTU)" can refer to the maximum block size applied to encoding / decoding images / video pictures or video sequences. Additionally, block size (or region size) can refer to the block / region width, height, area, number of samples in the block / region, the maximum (or minimum) value between the block / region width and height, and / or the block / region aspect ratio.

[0086] Figure 4A An example of an intra-block copying technique according to some implementations is shown. In this example, the intra-block copying technique includes identifying a predicted block 404 within the same picture 400 as the current block 402 via a predicted BV 408. Figure 4A An example is shown where the predicted block 404 is not adjacent to the current block 402. In some embodiments, the predicted block 404 is adjacent to the current block 402. In some embodiments, the predicted BV 408 is selected from a list of candidate BVs. For example, the list of candidate BVs may be populated by BVs used in neighboring blocks and / or BVs from a BV library. The BV predictor index can be signaled to indicate which candidate BV in the candidate list was used to predict the BV of the current block 402.

[0087] The block vector difference (BVD) 410 represents the difference between the predicted BV and the actual BV 406, where the actual BV 406 is the vector between the corresponding portions of the current block (e.g., current block 402) and the predicted block (e.g., predicted block 404). Although Figure 4A The BVD 410 depicted has components along both the horizontal and vertical dimensions, but the BVD may extend along a single dimension or span two or more dimensions. In some implementations, the BVD 410 includes information representing the magnitude and / or direction of the BVD, one or both of which may be signaled in the bitstream as intra-frame block copy information or syntax.

[0088] The reconstructed neighbor samples of the current block (collectively referred to as "templates") can be used to predict the current block. In some implementations, template matching is applied to the current image to find at least one neighboring and / or non-neighboring block with MV and / or BV information that can be used for the current block. The MV and / or BV from the neighboring and / or non-neighboring block can be used to construct a merge candidate list for the current block. Figure 4B An example of generating a list of merge candidates according to some implementation methods is shown. Figure 4B In the current image 420, the current block 412 is included. According to some implementations, template matching processing (e.g., intra-frame template matching) is used to identify the BV 418 of the current block 412. For example, the current block 412 has a template 416 comprising multiple reconstructed samples. The distortion between template 416 within the current image 420 and other templates (e.g., templates formed from reconstructed samples in the current image 420) is calculated, and a predicted block 414 associated with a template 429 having smaller or minimum distortion and / or minimum template matching cost (e.g., based on the sum of absolute difference (SAD), sum of absolute transformed difference (SATD), sum of squared error (SSE), or another metric) can be identified, thereby identifying the BV 418. Figure 4B In the example, prediction block 414 has an MV 424 (also represented as MV) pointing to reference block 426 in reference image 428. A' Candidate 422 (also denoted as MV) can be merged by adding BV418 and MV424 (e.g., MV=MV). A' +BV) or set merge candidate 422 to equal MV 424 (e.g., MV=MV A'The result is obtained by taking the merge candidate 422 as MV 424. For example, when the merge candidate 422 is set to MV 424, the merge candidate 422 does not take into account the non-adjacency of the predicted block 414 and the current block 412 (e.g., ignoring the displacement of the predicted block 414 and the current block 412 via BV).

[0089] In some implementations, candidate 422 (e.g., MV) is merged. A' The motion field is derived from a predefined motion field within a prediction block 414 with dimensions W×H. For example, prediction block 414 (e.g., 32×32, the same size as the current block 412) can include multiple sub-blocks (e.g., each sub-block is 4×4, resulting in an arrangement of 8×8 different sub-blocks containing 64 motion fields). For example, as... Figure 4C As shown. Figure 4B Prediction block 414 in the middle can correspond to Figure 4C The larger block 430 (e.g., sometimes referred to as the "predefined sports field") comprises various sub-blocks, such as the upper right sub-block 432, lower right sub-block 434, lower left sub-block 436, upper left sub-block 438, and center sub-block 440. In some implementations, the availability of the MV within the sports field is checked by scanning (optionally in a predefined scan order when checking multiple locations) for the availability of the MV at one or more locations. In some implementations, for example, the center sub-block 440 (e.g., at location [W / 2, H / 2]) is checked first to determine if the MV information is available. In some implementations, based on the determination that the MV information is unavailable at the center sub-block 440, the MV information is checked sequentially at the upper right sub-block 432, lower right sub-block 434, lower left sub-block 436, and upper left sub-block 438 to determine if the MV information is available. If the MV at the center location is unavailable (e.g., sub-block 440 is intra-coded), the MV information at the four corners (e.g., the intra-mode of top-left sub-block 438, top-right sub-block 432, bottom-right sub-block 434, and bottom-left sub-block 436) is determined in clockwise order or using a different order. In some implementations, if no MV is available at these predefined locations (center, top-left, top-right, bottom-right, bottom-left, where sub-blocks at the predefined locations are intra-coded), candidate 422 (e.g., MV...) is merged. A' This is not available for the current block 412 (for example, other sub-blocks in the non-shaded portion of block 430 have not been checked).

[0090] The distortion or template matching cost between template 416 and other templates within the current image 420 can be calculated to select a template with a low (e.g., lowest) template matching cost. For example, a template with an associated cost below a predetermined threshold can be selected. In some implementations, the template matching cost may be based on the sum of absolute differences (SAD), the sum of absolute transform differences (SATD), the sum of squared errors (SSE), or another metric.

[0091] In some implementations, prediction blocks are searched within a smaller, predefined search region within the reconstructed region of the current image 420. For example, search range restrictions can be applied to search within a fixed-size search range, search within the current CTU line, search within the current CTU line and / or within N previously encoded / decoded CTU lines, etc. In some implementations, instead of finding a single prediction block 414 (e.g., within the current image 420), additional prediction blocks (e.g., N distinct prediction blocks) with the lowest N template matching costs are searched within the current image 420, and the corresponding MVs within these N blocks are derived.

[0092] In some implementations, instead Figure 4B The L-shaped templates shown (e.g., templates 416 and 429) are used for intra-frame template matching, with templates of different shapes or sizes. For example, a left-only template (e.g., the left portion of L-shaped template 416), or a top-only template (e.g., the horizontal portion of L-shaped template 416), or a top-left template (e.g., L-shaped template 416) can be selected / used. For example, for smaller blocks (e.g., blocks less than or equal to a size threshold (e.g., 64, 32, or different values), a template with two rows of reconstructed samples is used, while for blocks larger than the size threshold (e.g., greater than 64, 32, or different values), a template with four rows of reconstructed samples is used. Therefore, the template size may depend on the block size. In some implementations, templates with different template matching types and shapes are adaptively selected at the block level (e.g., using any suitable method to determine which template type is optimal when encoding and decoding the current block), and optionally, the encoder signals the syntax to the bitstream to indicate which template type and / or shape to use.

[0093] In some implementations, pixel subsampling within the template is used in the template matching process to compute the template matching cost. For example, for a 32-pixel (e.g., 16×2 size) template, instead of calculating the pixel difference (e.g., absolute difference) for every pixel in the 32 pixels (e.g., all samples within the template), the pixel difference is calculated only partially (e.g., for even-numbered positions only). Pixel subsampling allows for reduced computation time and can aid in hardware design. In some implementations, the template matching process is performed with a coarser step size. For example, the search step size is changed to two samples per iteration instead of searching every sample (e.g., a step size of 1). For example, instead of searching every position in the 32×32 block, the search is performed only at even-numbered positions or only at odd-numbered positions, such that the search is performed only in a subset of regions.

[0094] In some implementations, merge candidates (e.g., merge candidate 422) are inserted into the merge candidate list before merge candidates derived from non-adjacent neighboring blocks and / or based on historical motion information (e.g., MVs from previous blocks placed in a data buffer, such as a first-in-first-out (FIFO) buffer). For example, merge candidates generated from the prediction block (e.g., obtained via intra-frame template matching) may be closer to and / or more similar to the current block and may be more accurate than merge candidates constructed from other non-adjacent neighboring blocks.

[0095] Figure 4D An example chained MV, including BV and / or MV, is shown according to some implementations for deriving merge candidates. Figure 4D In this context, current image 442 includes current block 444. In some implementations, template matching processing (e.g., intra-frame template matching) is used to identify the BV 452 (e.g., also denoted as BV) of current block 444. Current block 444 has template 450, and the distortion between template 450 within current image 442 and other templates (e.g., templates formed by reconstructed samples in current image 442) can be calculated, and the predicted block 446 associated with the template having the minimum distortion and / or the lowest template matching cost (e.g., based on SAD, SATD, SSE, or another metric) can be identified, similar to... Figure 4B The identification of BV418 in the current block 412 shown. Related to MV 424. Figure 4B Compared to prediction block 414, in Figure 4D Another BV 454 (also represented as BV) is derived from the BV field of prediction block 446 in the prediction block. A'→A'' BV 454 (BV A'→A'' Prediction block 446 points to another prediction block 448. Prediction block 448 has an MV 456 (also represented as MV) pointing to reference block 460 in reference image 462. A'' Merging MV or merging candidate 458 is done by merging BV 452 (e.g., BV), BV 454 (e.g., BV), and so on. A'→A'' ) and MV 456 (e.g., MV A'' The chained MV is obtained by adding the values ​​of BV and BV, for example, MV = BV + BV. A'→A'' +MV A'' In some implementations, the depth of chained music video propagation is limited. In scenarios such as, for example, Figure 4D The depth of the chained MV propagation described in the text will be 2. In some implementations, adding BV is not considered to increase the depth of the chained MV propagation. In such scenarios, for example, Figure 4DThe depth of the chained MV propagation described herein will be 1 (or zero). In some implementations, flags are signaled using high-level syntax (e.g., SPS, PPS, APS, image headers, or slice headers) to indicate whether BV is used for chained MV construction.

[0096] In some implementations, a chained MV construction is applied using a cropping operation to ensure that the MV (e.g., MV 456) remains within a predefined region (e.g., the central portion or any other specific region) in the reference image (e.g., reference image 462).

[0097] Figure 4E An example of a merge candidate build with sub-block-level MVs is shown according to some implementation methods. Figure 4E In this process, template matching identifies at least one neighboring and / or non-neighboring block with sub-block-level MV and / or BV information within the reconstructed region 463 of the current image 464. For the current block 466, template matching is used within the reconstructed region 463 of the current image 464 to derive a predicted block 468 and a BV 472. Within the predicted block 468, the associated sub-block-level MV is derived from the corresponding motion field covered by the predicted block 468. For example, Figure 4E The diagram shows that the lower-left sub-block of prediction block 468 has an MV pointing to the left (e.g., to the reference sub-block in the reference image). The final sub-block MV (MVi, j) has each sub-block (i, j) in the current block 466. i, j The constructed merge candidate can be obtained by combining the corresponding sub-blocks MV (MV) of BV 472 and prediction block 468. Ai, j Add (e.g., MV) i, j =MV Ai, j +BV) or the final sub-block MV (MV) with each sub-block (i, j) in the current block 466. i, j The constructed merge candidate is simply set as the corresponding sub-block MV of predicted block 468. Ai, j (For example, MV) i, j =MV Ai, j This can be derived from...

[0098] In some implementations, when the sub-block MV at the center position (e.g., Figure 4EWhen the lower right sub-block (as shown, or a sub-block at another location) is unavailable, the constructed or derived merge candidate with sub-block-level MVs is unavailable. For example, the center position of prediction block 468 of size W×H could be (W / 2, H / 2), similar to Subblock-based Temporal Motion Vector Prediction (SBTMVP). In some implementations, if the associated sub-block-level MV at any sub-block (i, j) is unavailable in the corresponding motion field, the sub-block MV at the center position is used for that sub-block (e.g., filling the missing information at that sub-block with information from the center sub-block).

[0099] In some implementations, when the reference image pointed to by the corresponding sub-block level MV is not used in the current block, MV time scaling can be applied to the corresponding sub-block level MV. For example, the sub-block can point to different reference images and implement time scaling. In some implementations, the reference... Figures 4A to 4D The described method can be applied to reference Figure 4E The method described.

[0100] In some implementations, refer to Figure 4D The described chained MV construction process is applied to a specific intra-prediction mode within a P-slice and / or B-slice. In some implementations, the specific intra-prediction mode constructs a list of N candidate predictors, where N is a positive number, within a predefined region in the current image using a template matching method. These candidate predictors (e.g., prediction blocks) are ranked in ascending order according to their respective template costs, and each candidate predictor is pointed to by one of the N BVs from the current block. In some implementations, an index is signaled to indicate which candidate in the list is used as the intra-prediction mode. In some implementations, the chained MV construction process is used to extend the candidate predictors to reference images beyond the current image. The extended predictors can be defined by MV=MV A' +BV points to MV A It is a music video (MV) that spans across images (e.g., from the current image to a reference image), and a picture video (BV) that is a BV within the same image, similar to a reference image. Figure 4D As stated above.

[0101] In some implementations, the candidate list employs a two-step strategy. In the first step, a candidate list is constructed using template matching constraints to include only candidates within the current image. In the second step, the first M (M<=N) candidates in the list are examined to determine the chained MV. i '(For example, MV) i =MV i +BV iWhether the motion vector MV (where i is between 1 and M) constitutes a new candidate. i Up to M new candidate MVs are inserted into a list of candidates only within the current image. For example, the insertion of candidates constructed using chained MVs is appended to the candidate list, and a larger candidate list with L candidates is built, where L>N. In some implementations, the insertion of new candidates constructed using chained MVs is based on template cost, and the list size remains constant. For example, the chained MV with the highest template cost is removed from the list to include chained MVs with lower template costs.

[0102] In some implementations, refinement is applied to new candidate predictors pointed to by the chained MVs (e.g., MV=(MVx, MVy)) to determine if they can be better candidates when further refinement is applied (e.g., MV'=(MVx+x0, MVy+y0)). In some implementations, the refinement offset (x0, y0) can have multiple instances and is predefined within the search range. For example, based on the best position found, a first template search can be performed using a coarser step size, refinement can be applied by searching with a finer step size (e.g., to find a better block close to the global minimum of the template cost), and merge candidates can be constructed using the MVs associated with the block at the refinement position. In some implementations, the BV is not signaled for refinement processing.

[0103] Figure 4F An example of merging candidate construction using an additional reference image is shown according to some implementations. Current image 474 includes current block 476. According to some implementations, template matching processing (e.g., intra-frame template matching) is used to identify the BV 484 (e.g., also denoted as BV) of current block 476. Current block 476 has a template 480 comprising multiple reconstructed samples. The distortion between template 480 within current image 474 and other templates (e.g., templates formed from reconstructed samples in current image 474) is calculated, and a prediction block 478 is identified associated with the template having the minimum distortion and / or the lowest template matching cost (e.g., based on sum of absolute differences (SAD), sum of absolute transform differences (SATD), sum of squared errors (SSE), or another metric). Prediction block 478 has an MV 486 (also denoted as MV) pointing to reference block 490 in reference image 492. A' In some embodiments, reference block 490 has an MV 491 (also denoted as MV) pointing to another reference block 494 in another reference picture 496. A''For example, the current image could be a picture order count (POC) 6 image, and reference image 492 could be a POC 4 image, which has a reference block 494 that further points to another POC image. Merge candidates are obtained by adding BV 484, MV486, and MV 491 (e.g., also represented as MV) (e.g., MV=MV). A'' +MV A' +BV).

[0104] Figure 5A An example is shown where the BV of a prediction block can be determined from a reconstructed region using intra-template matching. For instance, in intra-template matching, the BV is derived by identifying the prediction block with the template that minimizes distortion between the template of the current block and the template of the prediction block. The current block can be copied from the prediction block or derived by applying a filter to the prediction block. Intra-template matching is a type of intra-prediction method different from intra-prediction methods that predict the current block based on neighboring reference samples. Intra-prediction methods that rely on neighboring reference samples can use intra-prediction modes that indicate some angular characteristics of the prediction block (e.g., neighboring reference samples of the current block). As an example, VVC defines various directional intra-prediction modes (e.g., 45 degrees, 22.5 degrees, etc.) and non-angular modes (e.g., planar mode and DC mode) that can also be used when the prediction block (e.g., neighboring reference samples of the current block) has a smooth texture.

[0105] In some cases, when constructing the most probable mode (MPM) list, intra-prediction can be improved by providing intra-prediction mode information at non-adjacent locations, rather than being limited to adjacent locations (e.g., including top row, top right, top left, left row, top two rows, left two rows, top three rows, and left three rows). In some implementations, several predefined non-adjacent locations are checked sequentially (e.g., one after another) to derive intra-prediction mode information at each non-adjacent location.

[0106] Figure 5AThe diagram illustrates the use of template matching processing (e.g., intra-frame template matching) to identify the BV of the current block according to some implementations. The current image 502 includes a current block 504 and a reconstructed region 508. The current block 504 is outside the reconstructed region 508. The current block 504 has a template 510 that includes multiple reconstructed samples (e.g., they are within the reconstructed region 508). The distortion between the template 510 and other templates within the reconstructed samples is calculated, and a predicted block 506 can be identified associated with the template 512 that has the minimum distortion or the lowest template matching cost (e.g., based on the sum of absolute differences (SAD), sum of absolute transform differences (SATD), sum of squared errors (SSE), or another metric). Figure 5A An example is shown where the predicted block 506 is not adjacent to the current block 504. In some implementations, the predicted block is adjacent to the current block 504. In some implementations, the predicted block can be searched within a smaller, predefined search area within the reconstructed region 508, instead of searching the entire reconstructed region 508. In some implementations, search range restrictions can be applied to intra-frame template matching to search within a fixed-size search range, search within the current CTU line, search within the current CTU line and / or search within N previously encoded / decoded CTU lines, etc.

[0107] The BV 516 of the current block 504 is derived via a vector pointing from the current block 504 to the prediction block 506 (e.g., from a portion of the template 510 of the current block 504 to a corresponding portion of the template 512 of the prediction block 506). In some embodiments, the prediction block 506 includes intra-frame prediction information, such as an intra-frame prediction mode. Figure 5A In the example shown, the intra-prediction mode of prediction block 506 is a directional intra-prediction mode (e.g., mode 61 in the vertical set, pointing to the lower left corner). In some implementations, the intra-prediction mode information x of prediction block 506 (e.g., intra-prediction mode) is derived and used for the prediction of the current block 504 (e.g., the intra-prediction mode of the current block 504 is set to mode 61).

[0108] In some implementations, instead of finding a single prediction block 506 (e.g., within the reconstructed region 508 or within a predefined search area within the reconstructed region 508), additional prediction blocks (e.g., N prediction blocks) with the lowest N template matching costs are searched, and the corresponding intra-frame prediction modes within these N blocks are derived.

[0109] In some implementations, prediction block 506 may include multiple blocks, for example, as referred to above. Figure 4C Described. For example, prediction block 506 could correspond to Figure 4CThe larger block 430 in the prediction block 506 includes various blocks such as the top-right block 432, the bottom-right block 434, the bottom-left sub-block 436, the top-left sub-block 438, and the center sub-block 440. The larger block 430 can be determined based on blocks from predefined positions within the prediction block 506. Figure 5A The intra-frame mode information of the current block 504 is described. In some implementations, predefined positions are checked in a predefined order, and the intra-frame mode is derived from the first available intra-frame mode in a predefined scan order (e.g., starting from the center position, moving to the upper right position and proceeding clockwise). For example, prediction block 506 (and block 430) may have a width W and a height H, and sub-block 440 at the center position (e.g., [W / 2, H / 2]) is checked first to determine if its intra-frame prediction mode is available. If the intra-frame mode at the center position is not available (e.g., sub-block 440 is inter-frame encoded), the intra-frame mode information at the four corners (e.g., the intra-frame modes of the upper left sub-block 438, upper right block 432, lower right block 434, and lower left sub-block 436) is determined in a clockwise order or using a different order. In some implementations, if no intra-frame mode information is available at these predefined locations (center, top left, top right, bottom right, bottom left, where the blocks are inter-frame encoded / decoded), then the intra-frame mode is not available for the current block 504 (e.g., other blocks in the non-shaded portion of block 430 are not checked).

[0110] In some implementations, the intra-frame mode information derived from one of the predefined locations (e.g., the intra-frame mode of the center sub-block 440, the top-left sub-block 438, the top-right sub-block 432, and the bottom-right sub-block 434) includes another BV, and additional intra-frame mode information is derived from the corresponding block pointed to by the BV.

[0111] In some implementations, the availability of intra-prediction modes is determined by scanning all locations within the intra-mode information field covered by the found blocks (e.g., including...). Figure 4C The intra-prediction mode of the current block 504 can be determined by the non-shaded portion of block 430. For example, the intra-prediction mode field corresponds to entering block 430. The intra-prediction mode of the current block 504 can be set to any intra-prediction mode found within block 430. Otherwise, the intra-prediction mode is not available for block 430. For example, the found block can be partially inter-predicted and partially intra-predicted. In such a scenario, the intra-prediction mode used for the found block is considered not yet found.

[0112] Figure 5BThe diagram illustrates how, according to some implementations, an intra-frame mode is determined based on the most frequent pattern within the found block. In some implementations, when a similar block 518 is found using a template matching method, its intra-frame mode is determined as the most frequent pattern within the found block (e.g., similar block 518). For example, for a minimum intra-frame prediction block size of 4×4, if the size of similar block 518 is 16×8, the found similar block 518 is divided into a (16 / 4)×(8 / 4)=4×2 grid. For each grid cell, an intra-frame mode is derived, and the most frequent intra-frame mode within the grid (IPM0, which is more frequent than mode IPM1, as well as modes IPM2 and IPM3, where IPM0 is the intra-frame mode for four grid cells in similar block 518, mode IPM1 is the intra-frame mode for two grid cells, and modes IPM2 and IPM3 are each the intra-frame modes for one grid cell) is selected as the intra-frame mode for the found similar block 518. In some implementations, intra-frame modes may be placed in the MPM list, but may not be used.

[0113] In some implementations, the above references are used. Figure 5A and Figure 5B The described template matching method is applied to a most probable mode (MPM) list. In some implementations, the MPM list is constructed at both the encoder and decoder using predefined rules to collect the most probable modes for the current block (e.g., current block 504). In some implementations, the encoder signals the index of the MPM list in the bitstream, and the decoder can use the MPM list and the signaled index to derive the final intra-mode. The encoder and decoder can use the same rules to obtain the same intra-mode information. For example, intra-modes derived using the template matching method (e.g., based on intra-prediction information x from predicted block 506) are added to the MPM list before intra-mode candidates from non-adjacent blocks. In some cases, intra-modes derived using the template matching method are added to the MPM list and sorted together with intra-mode candidates from non-adjacent blocks.

[0114] In some implementations, instead Figure 5AThe L-shaped templates shown (e.g., templates 510 and 512) are used for intra-frame template matching. For example, a left-only template (e.g., the left portion of L-shaped template 510), or a top-only template (e.g., the horizontal portion of L-shaped template 510), or a top-left template (e.g., L-shaped template 510) may be used. For example, the current block may be more relevant in one direction (e.g., horizontal or vertical), and a left or top template may provide more accurate results. For example, for smaller blocks (e.g., blocks less than or equal to a size threshold (e.g., 64, 32, or different values), a template with two rows of reconstructed samples is used, while for blocks larger than the size threshold (e.g., greater than 64, 32, or different values), a template with four rows of reconstructed samples is used. In some implementations, templates with different template matching types and shapes are adaptively selected at the block level, and the encoder signals the syntax to the bitstream to indicate which template type and / or shape is used.

[0115] In some implementations, as a complexity reduction method, intra-template pixel subsampling is used in the template matching process to compute the template matching cost. For example, template matching can be time-consuming, and pixel subsampling can reduce the latency of template matching. In some implementations, template matching is performed with a coarser step size. For example, the search step size is changed to two samples per iteration instead of searching each sample (e.g., a step size of 1). In some implementations, a signaling flag is used with a high-level syntax (e.g., SPS, PPS, APS, image header, or slice header) to indicate whether intra-frame template matching is used to derive intra-frame prediction mode information.

[0116] Figure 5C An example of intra-frame template matching refinement for finding the location of a predicted block with minimum template matching cost, according to some implementations, is shown. The non-adjacent location p in current image 520 relative to current block 522 is shown. i It is used to derive intra-frame prediction patterns through refinement. Within the template matching search range of 526, at non-adjacent positions p... i The surrounding template matching refinement is applied to find the prediction block P with the minimum template matching cost within the template matching search range 526 (e.g., the minimum distortion between template 524 of the current block 522 and template 528 of the prediction block P). The associated intra-prediction mode of the prediction block P is used for the current block 522.

[0117] In some implementations, the MPM list is constructed by applying refinement to non-adjacent locations of the current block 412. In some implementations, intra-prediction modes derived at locations determined by refinement are applied to the construction of the MPM list if the template matching cost associated with a location in the refinement process is less than or equal to a predefined threshold. In some implementations, intra-prediction modes derived at locations determined by refinement are not applied to the construction of the MPM list if the template matching cost associated with a location in the refinement process is greater than a predefined threshold. In some implementations, the predefined threshold is a value related to the quantization parameter (QP).

[0118] In some implementations, based on the determination that the prediction block with the lowest template matching cost does not have associated intra-prediction mode information (e.g., the prediction block with the lowest template matching cost is inter-frame encoded / decoded), a second low template matching cost (e.g., a second low cost) is used to derive the intra-prediction mode (e.g., optionally, traversing a ranking list of template matching costs until intra-prediction mode information is determined).

[0119] In some implementations, the blocks used for template matching (e.g., predicted block P or non-adjacent positions p) i The position of ) can be anywhere within a similar block, for example, such as Figure 5C As shown. For example, predicting block P or non-adjacent location p. i Corresponding to Figure 4C Block 430, or non-adjacent position p i The position is the center position [W / 2, H / 2] of the block used for template matching or any corner position of the block, such as Figure 4C As shown. In some embodiments, refer to Figure 5A and Figure 5B The described methods and techniques can be applied to reference Figure 5C The method described.

[0120] Figure 6A This is a flowchart illustrating a method 600 for decoding video according to some embodiments. Method 600 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuitry and a memory storing instructions executed by the control circuitry. In some embodiments, method 600 is executed by executing instructions stored in the computing system's memory (e.g., memory 314).

[0121] The system receives (602) a video bitstream (e.g., a encoded video sequence) including the current image, which consists of multiple blocks including the current block. The system identifies (604) a reference block by applying template matching to the current block and identifies (606) a reference vector for the reference block, where the reference vector is an MV or BV. The system populates (608) a merge candidate list for the current block using the reference vector and reconstructs (610) the current block using information from the merge candidate list. In this way, template matching can be applied to the currently reconstructed image to find at least one adjacent and / or non-adjacent block with MV and / or BV information. The MV and / or BV from the block can then be used to construct the merge candidate list. Figure 4B An example is shown. For instance, the BV of block A' can be determined through template matching processing, and another motion vector MV can be derived from the encoded / decoded block A'. A' MV A' This points to a reference image used for encoding and decoding the current block A. Finally, by using intra-frame template matching, a merge list can be constructed with MV (which is equal to MV). A' Constructed merge candidates (+BV), such as Figure 4B As shown. Figure 4B Another example shows that the resulting constructed merge candidate MV can be equal to MV. A' .

[0122] In some implementations, the motion vector MV is derived from a predefined motion field within the W×H block A'. A' The availability of MVs within the sports field is checked by scanning the availability of MVs at at least one location, and if multiple locations are checked, the scanning order is a predefined order. Figure 4CAn example of MV derivation is shown. The system determines whether the MV at the center location [W / 2, H / 2] is available. If the MV at the center location is unavailable, the MV information at the four corners is determined sequentially. In some implementations, if no available MV is found from the locations listed above, the resulting motion vector MVA' is unavailable. In some implementations, the template matching cost is SAD, SATD, SSE, etc. In some implementations, the search area for intra-frame template matching is within the reconstructed current image. In some implementations, search range restrictions are applied to intra-frame template matching, for example, within a search range of a fixed size, within the current CTU line, within the current CTU line and N previously encoded / decoded CTU lines, etc. In some implementations, more than one MV is derived by sorting the template matching costs in ascending order. For example, MVs from N blocks that have N minimum template matching costs during template matching processing can be sorted. In some implementations, different template matching types are used for intra-frame template matching. In some implementations, only left templates, top templates, or top-left templates are used. In some implementations, the template size is 2 rows for smaller blocks (e.g., less than or equal to 64), and 4 rows for others. In some implementations, these different template matching types can be used adaptively at the block level, and the syntax is signaled to indicate which template type to use. In some implementations, pixel subsampling within the template can be used during template matching processing to compute the template matching cost. In some implementations, template matching processing is performed with a coarser step size. For example, the search step size is changed to two samples per iteration instead of one sample per iteration. In some implementations, constructed merge candidates are inserted before merge candidates derived from non-adjacent neighbor blocks and / or historical motion information.

[0123] In some implementations, chained MVs including BV and / or MV can be used to derive merge candidates by using intra-frame template matching. Figure 4D An example of merging MV using chained MV and BV is shown. Intra-template matching is used to derive block A'. The associated BV... A'→A'' It is derived from the corresponding BV field within block A', and the block vector BV A'→A'' Pointing to block A''. Another motion vector MV A'' It is derived from block A''. And then, the chained MV (BV) A'→A'' +MV A'' It can be used with BV to build merged MV.

[0124] In some implementations, the depth of chained music video propagation is limited. In some implementations, adding a blog post (BV) is not considered to increase the depth of chained music video propagation.

[0125] In some implementations, high-level syntax (e.g., SPS, PPS, APS, image header, slice header) is used to signal flags to indicate whether BV is used for chain MV predictor construction.

[0126] In some implementations, the chained MV method and system described herein are applied using a cropping operation, for example, to restrict the MV to a predefined restricted area in a reference image.

[0127] The disclosed methods and systems include applying template matching processing to the currently reconstructed image to find at least one adjacent and / or non-adjacent block with sub-block level MV and / or BV information. Figure 4E The image illustrates an example of constructing a merge candidate with sub-block-level motion vectors (MVs). Template matching is used in the currently reconstructed image to derive block A and its bounding v (BV). Within block A, associated sub-block-level MVs are derived from the corresponding motion fields covered by block A. Finally, the constructed merge candidates are constructed, and the final sub-block motion vector MV for each sub-block (i, j) is obtained. i, j It can be set to equal MV Ai, j +BV. Furthermore, the final sub-block motion vector MV for each sub-block (i, j) i, j It can also be set to equal MV Ai, j In some implementations, when the sub-block MV at the center location is unavailable, the derived merge candidate with the sub-block-level MV is invalid. The center location of block A of size W×H can be, but is not limited to, (W / 2, H / 2). In some implementations, if the associated sub-block-level MV at sub-block (i, j) is unavailable in the corresponding motion field, the sub-block MV at the aforementioned center location is used for that sub-block.

[0128] In some implementations, when the reference image pointed to by the sub-block-level MV is not used in the current block, MV time scaling is applied to each sub-block-level MV.

[0129] The method and system described in this paper involve applying the chained MV described above to a specific intra-prediction mode (within a P / B slice). This intra-prediction mode constructs a list of N candidates, where N is a positive number, within a predefined region in the current image using a template matching method. These candidate predictors are ranked in ascending order by template cost and are pointed to by N BVs respectively. An index is signaled to indicate which candidate in the list is used as the intra-prediction model.

[0130] In some implementations, a chained motion vector (MV) is used to extend the candidate predictor to reference images beyond the current image. This extended predictor is pointed to by MV = MVA' + BV, where MVA' is the MV across images, and BV is the BV within the same image. In some implementations, populating the candidate list involves a two-step strategy: in the first step, a candidate list is constructed using template matching constraints to include only candidates within the current image. In the second step, the top M (M <= N) candidates in the list are selected, and they are examined to see if they correspond to images with chained motion vectors (MV). i =MV i +BV i A new candidate, where i is an iterator between 1 and M. (This is derived from MV...) i The new candidates pointed to by '(up to M)' are inserted into a list that includes only candidates within the current image. In some implementations, the insertion of candidates with chained MV is appended to the candidate list, and a large candidate list with L candidates is built, where L>N. In some implementations, the insertion is based on template cost, and the list size remains constant.

[0131] In some implementations, refinement is applied to the new candidate predictor pointed to by the chained MV. Assuming a new candidate predictor pointed to by MV(MVx, MVy) is found using the method described above, further refinement is applied using MV'(MVx+x0, MVy+y0) to search for better candidates. The refinement offset (x0, y0) can have multiple instances and is predefined within the search range.

[0132] Figure 6B This is a flowchart illustrating a method 650 for encoding video according to some embodiments. Method 650 can be executed at a computing system (e.g., server system 112, source device 102, or electronic device 120) having a control circuitry and memory storing instructions executed by the control circuitry. In some embodiments, method 650 is executed by executing instructions stored in the computing system's memory (e.g., memory 314). In some embodiments, method 650 is executed by the same system as method 600 described above.

[0133] The system receives (652) video data (e.g., a source video sequence) including the current image, which comprises multiple blocks, including the current block. The system identifies (652) a reference block by applying template matching to the current block and identifies (656) a reference vector for the reference block, where the reference vector is an MV or BV. The system populates (658) a merge candidate list for the current block with the reference vector and encodes the current block using information from the merge candidate list (660). As previously mentioned, the encoding process can be a mirror image of the decoding process described herein (e.g., template matching and merge candidate list construction as described above). For brevity, these details are not repeated here.

[0134] although Figure 6A and Figure 6B Multiple logical stages are shown in a specific order, but stages that are not dependent on the order can be reordered, and other stages can be combined or split. Some reorderings or other groupings not specifically mentioned will be obvious to those skilled in the art, and therefore the orderings and groupings presented herein are not exhaustive. Furthermore, it should be recognized that the stages can be implemented in hardware, firmware, software, or any combination thereof.

[0135] Now let's turn to some example implementations.

[0136] (A1) In one aspect, some implementations include a method for video decoding (e.g., method 600). In some implementations, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some implementations, the method is performed at a codec module (e.g., codec module 320). In some implementations, the method is performed at a source codec component (e.g., source codec 202), a codec engine (e.g., codec engine 212), and / or an entropy codec (e.g., entropy codec 214). The method includes (i) receiving a video bitstream (e.g., a codec video sequence) comprising a current picture, the current picture being composed of a plurality of blocks including a current block, and (ii) identifying a reference block by applying a template matching technique to the current block. The method further includes (iii) identifying a reference vector of the reference block, wherein the reference vector is MV or BV, (iv) populating a merge candidate list of the current block with the reference vector, and (v) reconstructing the current block using information from the merge candidate list. For example, template matching is applied to the currently reconstructed image to find at least one adjacent and / or non-adjacent block with MV and / or BV information. At least one MV and / or BV from the block is used to construct a merge candidate list. As an example, a BV is determined via template matching. An MV is derived from the reference block pointed to by the BV. The MV points to a reference image used in the current block. The constructed merge candidates with MVs are added to the merge list of the current block. The constructed merge candidates may consist of MVs or may include MVs combined with BVs. In some implementations, the current block corresponds to a P or B slice. In some implementations, reconstructing the current block using information from the merge candidate list includes selecting a vector from the merge candidate list and reconstructing the current block using the vector. In some implementations, an index is signaled in the video bitstream to indicate which candidate vector from the merge candidate list will be used to reconstruct the current block.

[0137] (A2) In some embodiments of A1, filling the merge candidate list with reference vectors includes: filling the merge candidate list with combined vectors, which are generated by combining the BV obtained according to template matching techniques with the reference vector of the reference block. In some embodiments, the combined vectors are cropped to fit within a predefined region. For example, a chained MV is applied using a cropping operation to ensure that the MV is within a predefined restricted region in the reference image. In some embodiments, refinement is applied to the combined vectors before insertion into the merge candidate list. In some embodiments, the refinement amount is signaled in the video bitstream. In some embodiments, the refinement amount is predefined for the current block.

[0138] (A3) In some implementations of A2, the combined vector is inserted before at least one of the following: merge candidates derived from non-adjacent neighbor blocks, and historical motion information. For example, the constructed merge candidates are inserted before merge candidates derived from non-adjacent neighbor blocks and / or historical motion information.

[0139] (A4) In some implementations of any of A1 through A3, filling the merge candidate list with a reference vector includes: filling the merge candidate list with a reference vector. In some implementations, an indicator is parsed from the video bitstream indicating whether the BV will be combined with the reference vector according to the template matching technique. For example, a signaling flag is used in a high-level grammar to indicate whether the BV is used in the construction of the predictive (chained) MV predictor.

[0140] (A5) In some embodiments of any of A1 to A4, the reference vector is identified by examining at least one location of the reference block. For example, the resulting MV is derived from a predefined motion field within the reference block. The availability of the MV within the motion field can be examined by scanning the availability of the MV at at least one location, and if multiple locations are examined, the scanning order can be a predefined order.

[0141] (A6) In some implementations of A5, at least one location includes the center location of a reference block. For example, the center location (e.g., the location at width / 2, height / 2) is checked first to determine if its MV is available. In the example, if the MV at the center location is unavailable, the MV information at the four corners is checked sequentially. In some implementations, when the sub-block MV at the center location is unavailable, the derived merge candidate with the sub-block-level MV is invalid.

[0142] (A7) In some implementations of A5 or A6, the reference vector is identified by checking the center position of the reference block and one or more corner positions of the reference block according to a predefined scan order.

[0143] (A8) In some embodiments of any of A5 to A7, when at least one location of the reference block does not have a usable reference vector, the method includes: abandoning the use of the vector corresponding to the reference block to fill the merge candidate list. For example, if there is no usable MV derived from the aforementioned location, the resulting MV is unusable.

[0144] (A9) In some implementations of any of A1 to A8, applying the template matching technique includes deriving the template matching cost. For example, the template matching cost may be the sum of absolute differences (SAD), the sum of absolute transformation differences (SATD), and / or the sum of squared errors (SSE).

[0145] (A10) In some embodiments of any of A1 to A9, template matching is applied to a search region corresponding to a portion of the reconstructed current image. For example, the search region for intra-frame template matching is within the reconstructed current image. In some embodiments, a search range limitation is applied to intra-frame template matching. For example, the search range may be a fixed size, such as within the current CTU line or within the current CTU line and N previously encoded / decoded CTU lines.

[0146] (A11) In some embodiments of any of A1 to A10, the method further includes: identifying a second reference block by applying a template matching technique to a reference block or the current block; identifying a second reference vector for the second reference block; and populating a merge candidate list using the second reference vector. For example, more than one MV can be derived by using the template matching costs in ascending order. As an example, MVs from N blocks having N minimum template matching costs are identified during the template matching process.

[0147] (A12) In some embodiments of any of A1 to A11, the method further includes: identifying template matching types from a set of template matching types, and applying template matching techniques using the template matching types. For example, different template matching types can be used for intra-frame template matching processing. As an example, a set of template matching types may include left-only template, top-only template, and / or top-left template. As an example, a 2-line template size can be used for smaller blocks (e.g., less than or equal to 64 samples), while a 4-line template size can be used for larger blocks. In some embodiments, different template matching types are used adaptively, for example, at the block level. In some embodiments, a signaling syntax is used to indicate which template type to use.

[0148] (A13) In some implementations of any of A1 to A12, the template matching technique uses a subsampled template. For example, pixel subsampling within the template can be used to compute the template matching cost during the template matching process. In some implementations, the template matching process is performed with a coarse step size. For example, the search step size is changed to two samples per iteration instead of one sample per iteration.

[0149] (A14) In some embodiments of any of A1 to A13, the reference vector is BV; and the method further includes: identifying a second reference block using BV; and identifying a second reference vector for the second reference block, and populating a merge candidate list using the second reference vector. For example, a chained MV including BV and / or MV can be used to derive merge candidates by using intra-frame template matching. Reference block A' is derived by using intra-frame template matching. The associated BV of reference block A' is derived from the corresponding BV field within block A', and the second reference block A'' is pointed to by BV. Another motion vector MVA'' is derived from block A''. As an example, equal to BV and motion vector MV A The chained MV can be used with BV identified using template matching technology to build merged MVs.

[0150] (A15) In some implementations of A14, the second reference block is identified based on the allowed chain depth. For example, the depth of chained MV propagation is limited (e.g., limited to 1, 2, or 3 BVs). In some implementations, adding a BV is not considered to increase the depth of chained MV propagation. In some implementations, subsequent reference blocks are identified based on the corresponding BV until the chain depth limit is reached. In some implementations, the merge candidate list is not populated using reference vector chains if the chain depth limit is reached.

[0151] (A16) In some implementations of any of A1 through A15, the reference vector of the reference block includes a sub-block level MV. For example, template matching is applied in the currently reconstructed image to find at least one neighboring and / or non-neighboring block with sub-block level MV information and / or BV information. Block A and BV are derived by using template matching in the currently reconstructed image. Within block A, the associated sub-block level MV is derived from the corresponding motion field covered by block A. In this example, a constructed merge candidate is built, and the final sub-block MV of each sub-block may be equal to the sub-block MV plus the BV. In another example, the final sub-block MV of each sub-block may be equal to the sub-block MV (e.g., without BV).

[0152] (A17) In some implementations of A16, the sub-block level MV of the reference block corresponds to the center position of the reference block. For example, if the associated sub-block level MV at a sub-block is not available in the corresponding motion field, the sub-block MV at the center position is used for a different sub-block.

[0153] (A18) In some embodiments of any of A1 to A17, the reference vector includes an MV pointing to a reference picture that is not used by the current block; and the method further includes applying time scaling to the MV. For example, when a reference picture pointed to by a sub-block level MV is not used by the current block, time scaling of the MV can be applied to each sub-block level MV.

[0154] (B1) In another aspect, some implementations include a video encoding method (e.g., method 650). In some implementations, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some implementations, the method is performed at an encoding / decoding module (e.g., encoding / decoding module 320). The method includes (i) receiving video data (e.g., a source video sequence) including a current image, the current image comprising multiple blocks, the multiple blocks including the current block. The method includes (ii) identifying a reference block by applying a template matching technique to the current block; (iii) identifying a reference vector for the reference block, the reference vector being MV or BV; (iv) populating a merge candidate list for the current block with the reference vector; and (v) encoding the current block using information from the merge candidate list. Some implementations of B1 include applying any of the techniques described above in A2 through A18.

[0155] (C1) In another aspect, some implementations include a method for processing visual media data. In some implementations, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some implementations, the method is performed at a codec module (e.g., codec module 320). The method includes: (i) obtaining a source video sequence comprising multiple frames; and (ii) performing a conversion between a video bitstream of visual media data and the source video sequence according to format rules, the video bitstream comprising a current block corresponding to a current image, and the format rules specifying: (a) identifying a reference block by applying a template matching technique to the current block; (b) identifying a reference vector of the reference block, the reference vector being MV or BV; (c) populating a merge candidate list of the current block with the reference vector; and (d) reconstructing the current block using information from the merge candidate list.

[0156] (D1) In another aspect, some implementations include a method for video decoding. In some implementations, the method is performed at a computing system (e.g., server system 112) having memory and control circuitry. In some implementations, the method is performed at a codec module (e.g., codec module 320). In some implementations, the method is performed at a source codec component (e.g., source codec 202), a codec engine (e.g., codec engine 212), and / or an entropy codec (e.g., entropy codec 214). The method includes (i) receiving a video bitstream comprising a current picture, the current picture being composed of a plurality of blocks including a current block; (ii) generating a list of candidate predictors for the current block by applying a template matching technique to the current block; and (iii) reconstructing the current block using at least one candidate predictor from the list of candidate predictors. As an example, an intra-frame prediction mode uses a template matching method within a predefined region in the current picture to construct a list of N candidates, where N is a positive integer. These candidate predictors may be ranked in ascending order by template cost and are pointed to by N BVs respectively. The index can be signaled to indicate which candidate in the list is used as the intra-frame predictor.

[0157] (D2) In some implementations of D1, candidate predictors are derived from a list of candidate predictors based on the BV identified via template matching and the MV of the reference block indicated by the BV. In some implementations, chained MVs are used to expand the candidate predictors to reference images beyond the current image. The expanded predictors can be derived from MV = MV A' +BV points to MV A' It is a music video that spans multiple images, and the BV is a BV within the same image.

[0158] (D3) In some implementations of D2, candidate predictors are derived by refining the combined vector that combines BV and MV. For example, refinement is applied to a new candidate predictor pointed to by the chained MV. Example candidate predictors pointed to by MV(MVx, MVy) are found, and further refinement is performed using MV'(MVx+x0, MVy+y0) to search for better candidates. The refinement offset (x0, y0) can have multiple instances and can be predefined within the search range. In some implementations, candidate predictors are used to identify different predictors within the current image (e.g., based on cost calculation). For example, a new candidate predictor pointed to by the chained MV can be used to refine the original predictor pointed to by BV. The cost can be calculated between the reconstructed blocks around the new predictor and the original predictor using a search window.

[0159] (D4) In some implementations of any of D1 to D3, generating a list of candidate predictors includes: inserting a first set of candidate predictors constrained to the current image into the list of candidate predictors; and inserting a second set of candidate predictors corresponding to the chained motion vectors (MVs) into the list of candidate predictors. For example, the candidate list employs a two-step strategy: in the first step, a candidate list (e.g., of size N) is built using template matching constraints to include only candidates within the current image. In the second step, the first M (M<=N) candidates in the list are selected, and it is checked whether they contain candidates with chained motion vectors (MVs). i' =MV i +BV i A new candidate, where i is an iterator between 1 and M. (This is derived from MV...) i' Up to M new candidates are pointed to and inserted into a list that includes only candidates within the current image. In one example, the insertion of candidates with chained MV is appended to the candidate list, and a large candidate list with L candidates is built, where L > N. In another example, the insertion is based on template cost, and the list size remains constant.

[0160] On the other hand, some implementations include a computing system (e.g., server system 112) that includes a control circuitry system (e.g., control circuitry system 302) and a memory (e.g., memory 314) coupled to the control circuitry system, the memory storing one or more sets of instructions configured to be executed by the control circuitry system, the set of one or more sets of instructions including instructions for performing any of the methods described herein (e.g., A1 to A18, B1, C1 and D1 to D4 above).

[0161] In another aspect, some embodiments include a non-transitory computer-readable storage medium storing one or more sets of instructions for execution by a control circuitry of a computing system, said set of instructions including instructions for performing any of the methods described herein (e.g., A1 to A18, B1, C1, and D1 to D4 above). In some embodiments, the non-transitory computer-readable storage medium stores a video bitstream generated by any video coding method described herein.

[0162] Unless otherwise stated, any syntax element described herein (e.g., indicator) can be a high-level syntax (HLS). As used herein, the HLS is signaled at a level higher than the block level. For example, the HLS can correspond to the sequence level, frame level, slice level, or tile level. As another example, HLS elements can be signaled in the Video Parameter Set (VPS), Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Adaptation Parameter Set (APS), slice header, picture header, tile header, and / or CTU header.

[0163] It will be understood that although the terms “first,” “second,” etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of embodiments and appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or,” as used herein, refers to and covers any and all possible combinations of one or more associated listed items. It will also be understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0164] As used herein, the term "if..." can be interpreted, depending on the context, as meaning "when the prerequisite is true," or "after the prerequisite is true," or "in response to determining that the prerequisite is true," or "based on determining that the prerequisite is true," or "in response to detecting that the prerequisite is true." Similarly, the phrases "if it is determined [that the prerequisite is true]," "if [that the prerequisite is true]," or "when [that the prerequisite is true]" can be interpreted, depending on the context, as meaning "after determining that the prerequisite is true," or "in response to determining that the prerequisite is true," or "based on determining that the prerequisite is true," or "after detecting that the prerequisite is true," or "in response to detecting that the prerequisite is true."

[0165] For illustrative purposes, the foregoing description has been described with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the claims to the precise forms disclosed. In view of the above teachings, many modifications and variations are possible. These embodiments were chosen and described in order to best illustrate the operating principles and practical applications, thereby enabling others skilled in the art to implement them.

Claims

1. A method for video decoding performed at a computing system having a memory and one or more processors, the method comprising: Receive a video bitstream including the current image, which is composed of multiple blocks including the current block; Reference blocks are identified by applying template matching techniques to the current block; Identify the reference vector of the reference block, wherein the reference vector is a motion vector or a block vector; The current block's merge candidate list is populated with the reference vector; and The current block is reconstructed using information from the merge candidate list.

2. The method according to claim 1, wherein, Populating the merge candidate list with the reference vector includes: populating the merge candidate list with a combined vector, the combined vector being generated by combining a block vector obtained according to the template matching technique with the reference vector of the reference block.

3. The method according to claim 2, wherein, The combined vector is inserted before at least one of the following: merge candidates from non-adjacent neighboring blocks, and motion information based on history.

4. The method according to claim 1, wherein, Populating the merge candidate list using the reference vector includes: populating the merge candidate list using the reference vector.

5. The method according to claim 1, wherein, The reference vector is identified by examining at least one location of the reference block.

6. The method according to claim 5, wherein, The at least one location includes the center location of the reference block.

7. The method according to claim 5, wherein, The reference vector is identified by examining the center position of the reference block and one or more corner positions of the reference block according to a predefined scan order.

8. The method according to claim 5, further comprising: When at least one position of the reference block does not have an available reference vector, the use of the vector corresponding to the reference block to fill the merge candidate list is abandoned.

9. The method according to claim 1, wherein, Applying the template matching technique includes deriving the template matching cost.

10. The method according to claim 1, wherein, The template matching technique is applied to the search area corresponding to the reconstructed portion of the current image.

11. The method according to claim 1, further comprising: The second reference block is identified by applying the template matching technique to the reference block; Identify the second reference vector of the second reference block; as well as The merge candidate list is populated using the second reference vector.

12. The method of claim 1, further comprising identifying a template matching type from a set of template matching types, wherein, The template matching technique is applied using the template matching type.

13. The method according to claim 1, wherein, The template matching technique uses subsampled templates.

14. The method according to claim 1, wherein, The reference vector is a block vector; as well as The method further includes: The second reference block is identified using the block vector; and Identify a second reference vector for the second reference block, wherein the second reference vector is used to populate the merge candidate list.

15. The method according to claim 14, wherein, The second reference block is identified based on the allowed chain depth.

16. The method according to claim 1, wherein, The reference vector of the reference block includes sub-block level motion vectors.

17. The method according to claim 16, wherein, The sub-block level motion vector of the reference block corresponds to the center position of the reference block.

18. The method according to claim 1, wherein: The reference vector includes a motion vector pointing to a reference image that is not used by the current block; and The method also includes applying time scaling to the motion vector.

19. A method for video encoding performed at a computing system having a memory and one or more processors, the method comprising: Receive video data including the current image, wherein the current image comprises multiple blocks, and the multiple blocks include the current block; Reference blocks are identified by applying template matching techniques to the current block; Identify the reference vector of the reference block, wherein the reference vector is a motion vector or a block vector; The current block's merge candidate list is populated with the reference vector; and The current block is encoded using information from the merge candidate list.

20. A non-transitory computer-readable storage medium storing a video bitstream generated by a video coding method, the video coding method comprising: Receive video data including the current image, wherein the current image comprises multiple blocks, and the multiple blocks include the current block; Reference blocks are identified by applying template matching techniques to the current block; Identify the reference vector of the reference block, wherein the reference vector is a motion vector or a block vector; The current block's merge candidate list is populated with the reference vector; and The current block is encoded using information from the merge candidate list.

Images