Block Vector Improvement Method for Intra-Block Copy

Abstract

Aspects of the present disclosure provide methods and apparatuses for video encoding / decoding. In some examples, an apparatus for video decoding includes a receiving circuit and a processing circuit. The processing circuit extracts from a bitstream a signal indicating the application of template matching-based block vector refinement for coding a current block in a current picture reference (CPR) mode. The processing circuit derives a first refined block vector associated with the current block according to the template matching-based block vector refinement. The first refined block vector includes a first block vector refinement offset applied to a first block vector associated with the current block. The processing circuit reconstructs the current block according to a first reference block in the same picture as the current block, and the first reference block is indicated by the first refined block vector.

Description

Technical Field

[0001] [Incorporation by Reference] This application claims the benefit of priority to U.S. Patent Application No. 17 / 983,1437, filed on November 8, 2022, entitled "BLOCK VECTOR REFINEMENT METHOD FOR INTRA BLOCK COPY", which claims the benefit of priority to U.S. Provisional Application No. 63 / 388,852, filed on July 13, 2022, entitled "Block Vector Refinement Method for Intra Block Copy". The entire disclosure of the prior application is incorporated herein by reference.

[0002] [Technical Field] The present disclosure generally describes embodiments related to video encoding (coding).

Background Art

[0003] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the inventors named in the present application, to the extent that the work is within the scope described in this background section, and aspects of this description that may not be eligible as prior art at the time of filing in other respects, are not admitted as prior art to the present disclosure, either expressly or implicitly.

[0004] Uncompressed digital images and / or videos can include a series of pictures, each picture having spatial dimensions, for example, of 1920×1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (informally also known as the frame rate), for example, a picture rate of 60 pictures per second or 60 Hz. Uncompressed images and / or videos have specific bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at a frame rate of 60 Hz) requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires a storage space of over 600 GB.

[0005] One purpose of image and / or video encoding and decoding can be the reduction of redundancy in the input image and / or video signal by compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, sometimes by more than two orders of magnitude. Although the description herein uses video encoding / decoding as an illustrative example, the same techniques can be equally applied to image encoding / decoding without departing from the spirit of the present disclosure. Both reversible compression and irreversible compression, as well as combinations thereof, can be used. Reversible compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal. When using irreversible compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal useful for its intended purpose. In the case of video, irreversible compression is widely used. The amount of acceptable distortion depends on the application. For example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect the fact that higher acceptable / tolerable distortion can result in a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform processing, quantization, and entropy coding.

[0007] Video codec techniques can include techniques known as intra coding. In intra coding, sample values are represented without referring to samples from previously reconstructed reference pictures or other data. In some video codecs, a picture is spatially divided into blocks of samples. If all blocks of samples are coded in an intra mode, that picture can be an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture in an encoded video bitstream (coding video bitstream) and video session or as a still image. Samples of an intra block can be subjected to a transform, and the transform coefficients can be quantized prior to entropy coding. Intra prediction can be a technique that minimizes sample values in a pre-transform region. In some cases, the smaller the post-transform DC value and the smaller the AC coefficients, the fewer bits are required for a given quantization step size to represent the block after entropy coding.

[0008] For example, traditional intra coding used in MPEG-2 generation coding techniques does not use intra prediction. However, some newer video compression techniques include techniques that attempt to perform prediction based on, for example, surrounding sample data and / or metadata obtained during encoding / decoding of a block of data. Such techniques are hereinafter referred to as "intra prediction" techniques. Note that in at least some cases, intra prediction uses only reference data from the current picture being reconstructed and does not use reference data from reference pictures.

[0009] There can be various forms of intra prediction. In a given video coding technology, if two or more such technologies can be used, the specific technology used can be encoded as a specific intra prediction mode that uses that specific technology. In certain cases, the intra prediction mode can have sub - modes and / or parameters, and the sub - modes and / or parameters can be encoded individually, or can be included in the mode codeword that defines the prediction mode. Which codeword to use for a given combination of mode, sub - mode and / or parameter can affect the coding efficiency gain through intra prediction, and the entropy coding technology used to convert the codeword into the bitstream can similarly have an impact.

[0010] A certain mode of intra prediction was introduced in H.264, refined in H.265, and further refined in newer coding technologies such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). The predictor block can be formed using sample values of neighboring samples that are already available. The sample values of the neighboring samples are copied into the predictor block according to a certain direction. The reference to the direction used can be encoded in the bitstream, or it can be predicted itself.

[0011] Referring to Figure 1A, in the lower right, a subset of 9 predictor directions known from 33 possible predictor directions defined in H.265 (corresponding to 33 of the 35 intra - modes, the angular modes) is depicted. The point (101) where the arrows converge represents the sample to be predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from the sample(s) in the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from the sample(s) in the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] Continuing to refer to FIG. 1A, in the upper left, a square block (104) of 4×4 samples is depicted (shown by the thick dashed line). The square block (104) contains 16 samples, and each sample is labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample from the top in the Y dimension and the first sample from the left in the X dimension. Similarly, sample S44 is the fourth sample within block (104) in both the Y and X dimensions. Since the block is of size 4×4 samples, S44 is in the lower right. Further, reference samples following a similar numbering scheme are shown. The reference samples are labeled with "R" and its Y position (e.g., row index) and X position (column index) relative to block (104). In both H.264 and H.265, the predicted samples are in the vicinity of the block being reconstructed, and thus there is no need to use negative values.

[0013] Intra-picture prediction can function by copying the reference sample value from the neighboring samples indicated by the signaling predicted direction. For example, assume that the encoded video bitstream includes signaling indicating the predicted direction that aligns with arrow (102) for this block. That is, the samples are predicted from the upper right sample at an angle of 45 degrees from the horizontal. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the direction is not divisible by 45 degrees, the values of multiple reference samples can be combined, for example, by interpolation, to calculate the reference sample.

[0015] With the development of video coding technology, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments are conducted to identify the most likely directions, and certain techniques in entropy coding are used to represent those likely directions with a small number of bits while accepting a penalty for some of the less likely directions. Furthermore, the direction itself may be predictable from neighboring directions already decoded in neighboring blocks.

[0016] Figure 1B shows a schematic diagram (110) depicting 65 intra prediction directions by JEM to show the number of prediction directions increasing over time.

[0017] The mapping of intra prediction direction bits in the encoded video bitstream representing the direction can be different for each video coding technology. Such mappings can range from a simple straightforward mapping to codewords, complex adaptive schemes related to the most probable mode, and similar techniques. However, in most cases, in video content, there may be certain directions that are statistically less likely to occur than other specific directions. Since the goal of video compression is to reduce redundancy, in a well-functioning video coding technology, such less likely methods are represented by a larger number of bits than the more likely directions.

[0018] Image and / or video encoding and decoding can be performed using inter-picture prediction with motion compensation. Motion compensation can be an irreversible compression technique, and a block of sample data from a previously reconstructed picture or a part thereof (reference picture) is spatially shifted in the direction indicated by a motion vector (hereinafter, MV) and then used for prediction of a newly reconstructed picture or a part thereof. In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV can have two dimensions of X and Y, or three dimensions, and the third dimension is an indication of the reference picture used (which can be indirectly the temporal dimension).

[0019] In some video compression techniques, the MV applicable to a certain region of sample data can be predicted from other MVs, for example, from an MV associated with another region of sample data that is spatially adjacent to the region being reconstructed and that precedes that MV in decoding order. By doing so, the amount of data required for encoding the MV can be significantly reduced, thereby removing redundancy and increasing compression. MV prediction can function effectively because, for example, when encoding an input video signal derived from a camera (known as natural video), a larger region than the region where a single MV is applicable moves in a similar direction, and thus, in certain cases, there is a statistical likelihood that a similar motion vector derived from the MVs of neighboring regions can be used for prediction. As a result, the MV found for a given region will be similar or identical to the MV predicted from the surrounding MVs, and it can be represented with fewer bits than would be used if the MV were directly encoded after entropy encoding. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating predictors from some surrounding MVs.

[0020] H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016) describes various motion vector (MV) prediction mechanisms. Among the many MV prediction mechanisms provided by H.265, the one described with reference to FIG. 2 is hereinafter a technique called "spatial merge".

[0021] Referring to FIG. 2, the current block (201) includes samples found by the encoder during the motion search process that it is predictable from a previous block of the same size that has been spatially shifted. Instead of directly encoding the MV, the MV can be derived from metadata associated with one or more reference pictures, for example, from the latest reference picture (in decoding order), using an MV associated with any of five surrounding samples denoted as A0, A1, and B0, B1, B2 (202 to 206 respectively). In H.265, MV prediction can use predictors from the same reference picture that neighboring blocks are using. SUMMARY OF THE INVENTION

[0022] Aspects of the present disclosure provide methods and apparatuses for video encoding / decoding. In some examples, an apparatus for video decoding includes a receiving circuit and a processing circuit. The processing circuit receives a coded video bitstream that includes a current picture, and the current picture includes a current block. The processing circuit determines that the current block is in a current picture referencing (CPR) mode based on a syntax element in the coded video bitstream. The processing circuit extracts from the bitstream a signal indicating the application of template matching-based block vector refinement for coding the current block in the current picture referencing (CPR) mode. The processing circuit derives a first refined block vector associated with the current block according to the template matching-based block vector refinement. The first refined block vector includes a first block vector refinement offset applied to a first block vector associated with the current block. The processing circuit reconstructs the current block according to a first reference block in the same picture as the current block, and the first reference block is indicated by the first refined block vector.

[0023] In some embodiments, the CPR mode is an intra block copy (IBC) mode. The processing circuit decodes a block vector difference associated with the current block at a first accuracy indicated by an adaptive motion vector resolution (AMVR) syntax from the coded video bitstream. The first block vector refinement offset is finer than or equal to the first accuracy.

[0024] In some examples, the first block vector is the block vector predictor (BVP) of the current block. The processing circuit applies a template matching-based block vector refinement to the BVP of the current block to generate a refined BVP using a first block vector refinement offset applied to the BVP. The processing circuit combines the refined BVP with a block vector difference to determine a first refined block vector.

[0025] In some examples, the processing circuit combines a block vector predictor (BVP) with a block vector difference to determine a first block vector, and applies a template matching-based block vector refinement to the first block vector of the current block to generate a first refined block vector using a first block vector refinement offset applied to the first block vector.

[0026] In some examples, the search step size in the template matching-based block vector refinement is smaller than a first accuracy.

[0027] In some examples, the processing circuit determines a first block vector having a first resolution and applies a template matching-based block vector refinement to the first block vector using a search step size finer than the first resolution.

[0028] In some examples, the processing circuit performs a hash-based search to determine whether there is a match for the current template of the current block within a reconstructed region in the same picture as the current block, and performs a template matching-based block vector refinement in response to the absence of a match in the hash-based search.

[0029] In some examples, to perform hash-based search, the processing circuit applies a hash function to a first sub-block within the reconstruction region to generate an entry in the hash map, applies the hash function to a second sub-block within the current template to generate a hash value, and determines whether a match exists based on the hash map and the hash value.

[0030] In some examples, template matching-based block vector refinement determines a first block vector refinement offset according to a template matching cost calculated based on a distortion between a current template of a current block and a reference template of a first reference block.

[0031] In one example, the processing circuit determines that a second refined block vector has the same minimum template matching cost value as a first refined block vector, calculates a first sum of a first horizontal absolute value of a horizontal component of the first refined block vector and a first vertical absolute value of a vertical component of the first refined block vector, calculates a second sum of a second horizontal absolute value of a horizontal component of the second refined block vector and a second vertical absolute value of a vertical component of the second refined block vector, and selects the first refined block vector in response to the first sum being smaller than the second sum.

[0032] In another example, the processing circuit determines that one or more other refined block vectors have the same minimum template matching cost value as the first refined block vector, constructs a block vector candidate list including the first refined block vector and the one or more other refined block vectors, and decodes a signal indicating the first refined block vector from the block vector candidate list from a coded video bitstream.

[0033] In some examples, the processing circuit applies template matching-based block vector refinement to IBC merge candidates in a candidate list to determine refined block vectors to add to the candidate list, determines that a refinement value associated with a particular IBC merge candidate is the same value as an existing refinement value in the candidate list, and prunes the refinement value associated with the particular IBC merge candidate. In one example, the processing circuit adds the unrefined block vector of a particular IBC merge candidate to the candidate list. In another example, the processing circuit deletes a particular IBC merge candidate from the candidate list.

[0034] In some examples, the processing circuit decodes a flag indicating application of template matching-based block vector refinement to an IBC merge candidate, the flag being at least one of a block level flag, a sequence level flag, a picture level flag, a slice level flag, a tile level flag, and / or a tile group level flag.

[0035] In some embodiments, the CPR mode is an Intra Template Matching Prediction (IntraTMP) mode. In some examples, the processing circuit performs intra template matching prediction on a current block to derive a first block vector, and performs template matching-based block vector refinement on the first block vector to obtain a first refined block vector.

[0036] In some examples, the CPR mode is an intra block copy (IBC) mode. The processing circuit determines that an adjacent block of the current block is in an intra template matching prediction (IntraTMP) mode, and the adjacent block is at least one of a spatial adjacent or a temporal adjacent of the current block. The processing circuit uses a block vector obtained from the adjacent block as a candidate for a block vector predictor (BVP) of the current block.

[0037] In some embodiments, the processing circuit receives a coded video bitstream that includes a current picture, and the current picture includes a current block. The processing circuit determines that the current block is in an intra template matching prediction (IntraTMP) mode based on a syntax element in the coded video bitstream. The processing circuit derives a first block vector associated with the current block in the intra template matching prediction (IntraTMP) mode, decodes a block vector difference associated with the current block from the coded video bitstream, determines a final block vector according to the first block vector and the block vector difference, reconstructs the current block according to a first reference block in the same picture as the current block, and the first reference block is indicated by the final block vector.

[0038] In some examples, the processing circuit determines that an adjacent block of the current block is in an Intra Template Matching Prediction (IntraTMP) mode, and the adjacent block is at least one of a spatial adjacent or a temporal adjacent of the current block. The processing circuit uses a block vector obtained from the adjacent block as a starting point of the Intra Template Matching Prediction to determine a first block vector.

[0039] In some examples, the processing circuit determines that an adjacent block of the current block is in an Intra Block Copy mode, and the adjacent block is at least one of a spatial adjacent or a temporal adjacent of the current block. The processing circuit uses a block vector obtained from the adjacent block as a starting point of the Intra Template Matching Prediction to determine a first block vector.

[0040] Aspects of the present disclosure also provide a non - transitory computer - readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform a method for video decoding.

Brief Description of the Drawings

[0041] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 11

Figure 12

Figure 13

Figure 14A

Figure 14B

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

DETAILED DESCRIPTION OF THE INVENTION

[0042] Figure 3 shows an exemplary block diagram of a communication system (300). The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, the terminal device (310) may encode video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video pictures, and display the video pictures according to the restored video data. Unidirectional data transmission may be common in media service applications and the like.

[0043] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, for example, during a video conference. For bidirectional data transmission, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) may receive the encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to restore the video pictures, and display the video pictures on an accessible display device according to the restored video data.

[0044] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) are shown as a server, a personal computer, and a smartphone, respectively, but the principles of the present disclosure may not be limited thereto. Embodiments of the present disclosure find application in laptop computers, tablet computers, media players, and / or dedicated video conferencing facilities. The network (350) represents any number of networks that transmit encoded video data between the terminal devices (310), (320), (330), and (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of the discussion herein, the architecture and topology of the network (350) may not be important for the operation of the present disclosure, unless otherwise described below.

[0045] FIG. 4 shows a video encoder and a video decoder in a streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0046] The streaming system can include a video source (401), such as a digital camera, and may include a capture subsystem (413) that generates a stream (402) of, for example, uncompressed video pictures. In one example, the stream of video pictures (402) includes samples captured by the digital camera. The stream of video pictures (402), depicted as a thick line to emphasize the high data volume when compared to the encoded video data (404) (or encoded video bitstream), can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as will be described in more detail below. The encoded video data (404) (or encoded video bitstream), depicted as a thin line to emphasize the lower data volume when compared to the stream of video pictures (402), can be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) within an electronic device (430). The video decoder (410) decodes an input copy (407) of the encoded video data and generates an output stream (411) of video pictures that can be rendered on a display (412), such as a display screen, or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) can be encoded according to a particular video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265.In one example, the video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0047] Note that electronic devices (420) and (430) can include other components (not shown). For example, electronic device (420) can include a video decoder (not shown), and electronic device (430) can also include a video encoder (not shown).

[0048] FIG. 5 shows an exemplary block diagram of a video decoder (510). The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (310) in the example of FIG. 4.

[0049] Receiver (531) may receive one or more encoded video sequences to be decoded by video decoder (510). In certain embodiments, one encoded video sequence is received at a time, and decoding of each encoded video sequence is independent of decoding of other encoded video sequences. The encoded video sequences may be received from channel (501), which may be a hardware / software link to a storage device storing the encoded video data. Receiver (531) may receive the encoded video data together with other data, such as encoded audio data and / or auxiliary data streams, and these data may be transferred to their respective using entities (not shown). Receiver (531) can separate the encoded video sequences from other data. As a network jitter countermeasure, buffer memory (515) may be coupled between receiver (531) and entropy decoder / parser (520) (hereinafter “parser”). In certain applications, buffer memory (515) is part of video decoder (510). In other applications, it can be external to video decoder (510) (not shown). In still other applications, for example, to counter network jitter, there may be a buffer memory (not shown) external to video decoder (510), and further, for example, to handle playback timing, there may be another buffer memory (515) internal to video decoder (510). If receiver (531) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous network, buffer memory (515) may not be required, or may be small. For use in a best effort packet network such as the Internet, buffer memory (515) may be required, may be relatively large, and advantageously may be of an adaptable size and may be implemented, at least in part, in an operating system or similar element (not shown) external to video decoder (510).

[0050] Video decoder (510) may include a parser (520) for reconstructing symbols (521) from an encoded video sequence. The categories of these symbols include information used to manage the operation of the video decoder (510) and potentially information for controlling a rendering device such as a render device (512) (e.g., a display screen). The rendering device can be coupled to the electronic device (530) rather than being an integral part of the electronic device (530) as shown in FIG. 5. The control information for the rendering device(s) may be in the form of Supplementary Enhancement Information (SEI message) or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) can parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence can follow video encoding techniques or standards and can follow various principles including variable length encoding, Huffman encoding, arithmetic encoding with or without context sensitivity, etc. The parser (520) can extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the group. The subgroups can include Group of Pictures (GOP), picture, tile, slice, macroblock, Coding Unit (CU), block, Transform Unit (TU), Prediction Unit (PU), etc. The parser (520) can also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0051] The parser (520) can perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (515), thereby generating symbols (521).

[0052] The reconstruction of the symbols (521) can involve multiple different units depending on the type of the encoded video picture or its parts (e.g., inter and intra pictures, inter and intra blocks) and other factors. How each unit is involved can be controlled by subgroup control information parsed by the parser (520) from the encoded video sequence. Such a flow of subgroup control information between the parser (520) and the multiple units below is not depicted for clarity.

[0053] In addition to the functional blocks already described, the video decoder (510) can conceptually be divided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units can interact closely with each other and can be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate.

[0054] The first unit is the scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives from the parser (520) the quantized transform coefficients and control information as symbols (singular or plural) (521). The control information includes which transform to use, block size, quantization coefficient, quantization scaling matrix, etc. The scaler / inverse transform unit (551) can output a block containing sample values that can be input to the aggregator (555).

[0055] In some cases, the output samples of the scaler / inverse transform (551) can relate to intra-coded blocks. An intra-coded block is a block that does not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed part of the current picture. Such prediction information can be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) can use surrounding already reconstructed information taken from the current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) can, in some cases, add, sample by sample, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0056] In other cases, the output samples of the scaler / inverse transform unit (551) can relate to inter-coded and potentially motion-compensated blocks. In such cases, the motion-compensation prediction unit (553) can access the reference picture memory (557) to fetch the samples used for prediction. After motion-compensating the fetched samples according to the symbols (521) related to the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (in this case called the residual samples or residual signal), thereby generating output sample information. The address in the reference picture memory (557) from which the motion-compensation unit (553) fetches the prediction samples can be controlled by the motion vectors available to the motion-compensation unit (553) in the form of symbols (521). The symbols can have, for example, X, Y, and reference picture components. Motion compensation can include interpolation of the sample values fetched from the reference picture memory (557) when exact motion vectors below the sample level are used, a motion vector prediction mechanism, etc.

[0057] The output samples of the aggregator (555) can be subjected to various loop filtering techniques within the loop filter unit (556). Video compression techniques can include in-loop filter techniques. The in-loop filter techniques are controlled by parameters included in the encoded video sequence (also referred to as the encoded video bitstream) and are made available to the loop filter unit (556) as symbols (521) from the parser (520). Video compression can also respond to meta information obtained during the decoding of a previous portion (in decoding order) of the encoded picture or encoded video sequence and can respond to previously reconstructed and loop-filtered sample values.

[0058] The output of the loop filter unit (556) can be a sample stream, which can be output to the renderer device (512) and can be stored in the reference picture memory (557) for use in future inter-picture prediction.

[0059] Once an encoded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, when the encoded picture corresponding to the current picture is fully reconstructed and the encoded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a fresh current picture buffer can be reallocated before starting the reconstruction of subsequent encoded pictures.

[0060] The video decoder (510) can perform a decoding operation according to a predetermined video compression technology or standard such as ITU-T Recommendation H.265. The encoded video sequence can conform to the syntax of the video compression technology or standard and the profile documented in the video compression technology or standard, meaning that the encoded video sequence can comply with the syntax defined by the video compression technology or standard being used. Specifically, the profile can select specific tools from all the tools available in the video compression technology or standard as the tools that are only available for use under that profile. For compliance, it may also be necessary that the complexity of the encoded video sequence is within the range defined by the level of the video compression technology or standard. In some cases, the level restricts the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level can, in some cases, be further restricted through the virtual reference decoder (Hypothetical Reference Decoder, HRD) specifications and metadata for HRD buffer management signaled in the encoded video sequence.

[0061] In one embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data can be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0062] FIG. 6 shows an exemplary block diagram of a video encoder (603). The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0063] The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6) that can capture a video image to be encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0064] The video source (601) can provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits,...), any color space (e.g., BT.601 YCrCB, RGB,...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) may be a storage device storing pre - prepared videos. In a video conferencing system, the video source (601) may be a camera that locally captures image information as a video sequence. The video data may be provided as a plurality of individual pictures that impart motion when viewed in sequence. Each picture itself may be organized as a spatial array of pixels, and each pixel can include one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0065] According to one embodiment, a video encoder (603) can encode and compress pictures of a source video sequence in real time or under any other required temporal constraints to produce an encoded video sequence (643). Enforcing an appropriate encoding speed is one function of a controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units as described below. Such couplings are not drawn for clarity. Parameters set by the controller (650) can include parameters related to rate control (picture skip, quantizer, lambda value of rate-distortion optimization techniques, …), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other suitable functions related to the video encoder (603) optimized for a particular system design.

[0066] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. As a simplified example for illustration purposes, in one example, the encoding loop can include a source encoder (630) (e.g., responsible for generating symbols such as a symbol stream based on an input picture and reference picture(s) to be encoded), and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a manner similar to what a (remote) decoder would also generate. The reconstructed sample stream (sample data) is input into the reference picture memory (634). Since the decoding of the symbol stream results in bit-exact results regardless of the decoder location (local or remote), the content of the reference picture memory (634) is also bit-exact between the local encoder and the remote encoder. In other words, the prediction unit of the encoder "sees" the same sample values as reference picture samples that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift in case synchronization cannot be maintained, e.g., due to channel errors) is also used in some related arts.

[0067] The operation of the "local" decoder (633) may be the same as that of the "remote" decoder, such as the video decoder (410), which has been described in detail above in relation to FIG. 5. However, referring briefly to FIG. 5 as well, since symbols are available and the encoding / decoding of symbols into an encoded video sequence by the entropy encoder (645) and the parser (420) can be reversible, the entropy decoding part of the video decoder (410) including the buffer memory (415) and the parser (420) may not be fully implemented in the local decoder (633).

[0068] In some embodiments, decoder techniques, excluding parse / entropy decoding that exists within the decoder, exist in the same or substantially the same functional form within the corresponding encoder. Accordingly, the disclosed subject matter focuses on decoder operations. The description of encoder techniques can be omitted since it is the reverse of the decoder techniques described comprehensively. In certain areas, more detailed descriptions are provided below.

[0069] During operation, in some examples, the source encoder (630) can perform motion-compensated predictive encoding that predictively encodes an input picture by referring to one or more previously encoded pictures from the video sequence designated as "reference pictures". In this way, the encoding engine (632) encodes the difference between a pixel block of the input picture and a pixel block of the reference picture(s) that can be selected as a prediction reference for the input picture.

[0070] The local video decoder (633) can decode the encoded video data of a picture that can be designated as a reference picture based on the symbols generated by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the encoded video data can be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence can typically be a reproduction of the source video sequence with some errors. The local video decoder (633) can replicate the decoding process that can be performed on the reference picture by the video decoder and cause the reconstructed reference picture to be stored in the reference picture memory (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture that has (in the absence of transmission errors) the common content as the reconstructed reference picture that would be obtained by a remote video decoder.

[0071] Predictor (635) can perform prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) can search the reference picture memory (634) to find sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., that can function as appropriate prediction references for the new picture. The predictor (635) can operate on a sample block-by-pixel block basis to find an appropriate prediction reference. In some cases, depending on what is determined by the search results obtained by the predictor (635), the input picture can have prediction references drawn from a plurality of reference pictures stored in the reference picture memory (634).

[0072] The controller (650) may manage the encoding operation of the source encoder (630), including, for example, setting parameters and subgroup parameters used for encoding video data.

[0073] The outputs of all the above functional units can undergo entropy encoding in the entropy encoder (645). The entropy encoder (645) converts the symbols generated by various functional units into an encoded video sequence by applying lossless compression to the symbols according to techniques such as Huffman encoding, variable-length encoding, arithmetic encoding, etc.

[0074] The transmitter (640) can buffer the encoded video sequence generated by the entropy encoder (645) and prepare it for transmission via the communication channel (660). The communication channel (660) may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) can merge the encoded video data from the video encoder (630) with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0075] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) can assign a certain encoded picture type to each encoded picture. The encoded picture type can affect the encoding technique that can be applied to each picture. For example, a picture may often be assigned as one of the following picture types.

[0076] An intra picture (I picture) can be encoded and decoded without using other pictures in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, an Independent Decoder Refresh (IDR) picture. Those skilled in the art recognize these variations of I pictures, as well as their respective uses and characteristics.

[0077] A predicted picture (P picture) can be encoded and decoded using intra prediction or inter prediction that uses at most one motion vector and a reference index to predict the sample values of each block.

[0078] Bidirectional predicted pictures (B pictures) can be encoded and decoded using intra prediction or inter prediction that uses up to two motion vectors and reference indices to predict the sample values of each block. Similarly, multi-predicted pictures can use three or more reference pictures and associated metadata for the reconstruction of a single block.

[0079] The source picture can typically be spatially divided into a plurality of sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and encoded block by block. The blocks can be predictively encoded by referring to other (already encoded) blocks, as determined by the encoding assignment applied to each picture of the block. For example, blocks of an I picture may be encoded non-predictively, or may be predictively encoded by referring to already encoded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture may be predictively encoded via spatial prediction or via temporal prediction by referring to one previously encoded reference picture. Blocks of a B picture may be predictively encoded via spatial prediction or via temporal prediction by referring to one or two previously encoded reference pictures.

[0080] The video encoder (603) can perform an encoding operation according to a predetermined video encoding technology or standard such as ITU-T Recommendation H.265. In that operation, the video encoder (603) can perform various compression operations, including a predictive encoding operation that utilizes the temporal and spatial redundancy in the input video sequence. Thus, the encoded video data can conform to the syntax specified by the video encoding technology or standard used.

[0081] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0082] Video may be captured as a plurality of source pictures (video pictures) in a temporal sequence. Intra picture prediction (often abbreviated as intra prediction) utilizes the spatial correlation in a given picture, and inter picture prediction utilizes the (temporal or other) correlation between pictures. In one example, a particular picture to be encoded / decoded, called the current picture, is divided into blocks. If a block within the current picture is similar to a reference block within a reference picture that has been previously encoded and is still in the buffer in the video, that block within the current picture can be encoded by a vector called a motion vector. The motion vector points to the reference block within the reference picture and can have a third dimension that identifies the reference picture when multiple reference pictures are used.

[0083] In some embodiments, dual prediction techniques can be used in inter picture prediction. According to the dual prediction technique, two reference pictures such as a first reference picture and a second reference picture that both precede the current picture in decoding order (but may be past and future respectively in display order) in the video are used. A block within the current picture can be encoded by a first motion vector pointing to a first reference block within the first reference picture and a second motion vector pointing to a second reference block within the second reference picture. The block can be predicted by a combination of the first reference block and the second reference block.

[0084] Furthermore, to improve the encoding efficiency, merge mode techniques can be used in inter picture prediction.

[0085] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and those CTUs in a picture have the same size such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs) which are one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree divided into one or more coding units (CUs). For example, a 64×64 pixel CTU can be divided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine a prediction type for that CU, such as an inter prediction type or an intra prediction type. The CU is divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In some embodiments, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values (e.g., luma values) for pixels such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0086] FIG. 7 shows an exemplary diagram of a video encoder (703). The video encoder (703) receives a processing block (e.g., a prediction block) of sample values in a current video picture within a sequence of video pictures, and is configured to encode the processing block into an encoded picture that is part of an encoded video sequence. In one example, the video encoder (703) is used instead of the video encoder (403) in the example of FIG. 4.

[0087] In an example of HEVC, the video encoder (703) receives a matrix of sample values for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) determines, for example using rate-distortion optimization, which of the intra mode, inter mode, or bi-prediction mode the processing block is best encoded using. If the processing block is encoded in the intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into the encoded picture. If the processing block is encoded in the inter mode or bi-prediction mode, the video encoder (703) may use inter prediction techniques or bi-prediction techniques, respectively, to encode the processing block into the encoded picture. In certain video encoding techniques, the merge mode may be an inter-picture prediction sub-mode in which the motion vector is derived from one or more motion vector predictors but there is no benefit of the encoded motion vector components outside the predictors. In certain other video encoding techniques, there may be motion vector components applicable to the target block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the mode of the processing block.

[0088] In the example of FIG. 7, the video encoder (703) includes an inter encoder (730), an intra encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in FIG. 7.

[0089] The inter-encoder (730) receives samples of a current block (e.g., a processing block), compares the block with one or more reference blocks (e.g., blocks in a previous picture and a subsequent picture) in a reference picture, generates inter-prediction information (e.g., a description of redundant information by an inter-encoding technique, a motion vector, merge mode information), and is configured to calculate an inter-prediction result (e.g., a predicted block) using any suitable technique based on the inter-prediction information. In some examples, the reference picture is a decoded reference picture decoded based on encoded video information.

[0090] The intra-encoder (722) receives samples of a current block (e.g., a processing block), and in some cases, compares the block with blocks already encoded within the same picture, generates quantized coefficients after transformation, and in some cases, is also configured to generate intra-prediction information (e.g., intra-prediction direction information by one or more intra-encoding techniques). In one example, the intra-encoder (722) also calculates an intra-prediction result (e.g., a predicted block) based on the intra-prediction information and reference blocks within the same picture.

[0091] The overall controller (721) is configured to determine overall control data and control other components of the video encoder (703) based on the overall control data. In one example, the overall controller (721) determines the mode of a block and provides a control signal to the switch (726) based on that mode. For example, when the mode is the intra mode, the overall controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723), selects the intra prediction information, and controls the entropy encoder (725) to include the intra prediction information in the bitstream. When the mode is the inter mode, the overall controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723), selects the inter prediction information, and controls the entropy encoder (725) to include the inter prediction information in the bitstream.

[0092] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) is configured to encode the residual data based on the residual data to generate conversion coefficients. In one example, the residual encoder (724) is configured to convert the residual data from the spatial domain to the frequency domain and generate conversion coefficients. Next, the conversion coefficients are subjected to quantization processing to obtain quantized conversion coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform inverse conversion to generate decoded residual data. The decoded residual data can be preferably used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and the intra prediction information. The decoded block is preferably processed to generate a decoded picture, and the decoded picture is buffered in a memory circuit (not shown) and can be used as a reference picture in some examples.

[0093] The entropy encoder (725) is configured to format the bitstream to include the encoded block. The entropy encoder (725) is configured to include various information in the bitstream according to a suitable standard such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other suitable information in the bitstream. Note that according to the disclosed subject matter, there is no residual information when encoding a block in either the merge submode of the inter mode or the bi-prediction mode.

[0094] FIG. 8 shows an exemplary diagram of a video decoder (810). The video decoder (810) is configured to receive an encoded picture that is part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In one example, the video decoder (810) is used in place of the video decoder (410) in the example of FIG. 4.

[0095] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) coupled together as shown in FIG. 8.

[0096] The entropy decoder (871) can be configured to reconstruct from the encoded picture specific symbols that represent the syntax elements that the encoded picture is composed of. Such symbols can include, for example, the mode in which a block is encoded (e.g., the latter two in intra mode, inter mode, bi-prediction mode, merge sub-mode, or another sub-mode), and prediction information (e.g., intra prediction information or inter prediction information, etc.) that can identify specific samples or metadata used for prediction by the intra decoder (872) or the inter decoder (880), respectively. The symbols can also include, for example, residual information in the form of quantized transform coefficients. In one example, when the prediction mode is inter or bi-prediction mode, the inter prediction information is provided to the inter decoder (880). When the prediction type is intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can undergo inverse quantization and is provided to the residual decoder (873).

[0097] The inter decoder (880) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0098] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0099] The residual decoder (873) is configured to perform inverse quantization to extract dequantized transform coefficients, process the dequantized transform coefficients, and convert the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (including quantization parameter (QP)), and such information may be provided by the entropy decoder (871) (since this is only low-volume control information, the data path is not depicted).

[0100] The reconstruction module (874) is configured to combine, in the spatial domain, the residual information output by the residual decoder (873) and the prediction result (output by the intra or inter prediction module as appropriate) to form a reconstructed block, which may be part of a reconstructed picture, and the reconstructed picture may be part of a reconstructed video. Note that other suitable operations such as a deblocking operation can be performed to improve visual quality.

[0101] Note that the video encoders (403), (603), (703) and the video decoders (410), (510), (810) can be implemented using any suitable technology. In one embodiment, the video encoders (403), (603), (703) and the video decoders (410), (510), (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), (703) and the video decoders (410), (510), (810) can be implemented using one or more processors that execute software instructions.

[0102] Aspects of the present disclosure provide techniques for improving block vectors in the current picture reference mode.

[0103] In some examples, current picture reference, also known as intra block copy (IBC), has applications in various video codecs such as High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and AOMedia Video 1 (AV1). IBC coding tools are used in the IBC mode for image / video coding. Different video codecs may have specific features or IBC coding tools. Current picture reference can include various modes in different standards such as intra block copy (IBC) mode, intra block copy (IntraBC) mode, intra template matching prediction (IntraTMP), etc.

[0104] Some IBC coding tools are used as current picture referencing (CPR) in the High Efficiency Video Coding Screen Content Coding (HEVC SCC) extension. The IBC mode can use coding techniques where the current picture is used as a reference picture in the inter prediction in the IBC mode. The advantage of using the IBC mode is the reference structure of the IBC mode where a two-dimensional (2D) spatial vector can be used as the representation of the addressing mechanism for referencing samples. The advantage of the IBC mode architecture is that the integration of IBC requires relatively small changes to the specification, and if a manufacturer has already implemented a specific inter prediction technology such as HEVC version 1, the implementation burden can be reduced. CPR in the HEVC SCC extension can be a special inter prediction mode, resulting in the same syntax structure as the inter prediction mode and a decoding process similar to the decoding process of the inter prediction mode.

[0105] The IBC mode can be integrated into the inter prediction process. In some examples, the IBC mode (or CPR) is an inter prediction mode, and only intra-only prediction slices become prediction slices that enable the use of the IBC mode. When the IBC mode is applicable, the coder can extend the reference picture list by only one entry for the pointer to the current picture. For example, the current picture uses a buffer of one picture size in the shared decoded picture buffer (DPB). The signaling of the IBC mode can be done implicitly. For example, if the selected reference picture points to the current picture, the CU can use the IBC mode. In various embodiments, the reference samples used in the IBC process are not filtered, which is different from normal inter prediction. The corresponding reference picture used in the IBC process is a long-term reference. To minimize memory requirements, the coder can release the buffer after reconstructing the current picture. For example, the coder releases the buffer immediately after reconstructing the current picture. The filtered version of the reconstructed picture can be returned to the DPB by the coder as a short-term reference if the reconstructed picture is a reference picture.

[0106] In block vector (BV) coding, the reference to the reconstruction area can be performed via a 2D BV similar to inter prediction. The prediction and coding of the BV can reuse the MV prediction and coding in the inter prediction process. In some examples, the luma BV has an integer resolution instead of a quarter of the accuracy of the MV used for normal inter-coded CTUs.

[0107] FIG. 9 shows the BV associated with the current CU (901) according to an embodiment of the present disclosure. Each square (900) can represent a CTU. The gray shaded area represents an already coded area (e.g., an already encoded area), and the non-shaded white area represents an area to be coded (e.g., an area to be encoded). The current CTU (900(4)) during reconstruction includes the current CU (901), the coded area (902), and the area to be coded (903). In one example, after coding the current CU (901), the area (903) is coded.

[0108] In an example such as HEVC, the gray shaded area excluding the two CTUs (900(1) - 900(2)) above and to the right of the current CTU (900(4)) can be used as a reference area in the IBC mode to enable wavefront parallel processing (WPP). The BV permitted in HEVC can refer to blocks within the reference area (e.g., the gray shaded area (900(1) - 900(2)) excluding the two CTUs). For example, the BV (905) permitted in HEVC refers to the reference block (911).

[0109] In an example such as VVC, as a reference area in the IBC mode, in addition to the current CTU (900(4)), only the left adjacent CTU (900(3)) to the left of the current CTU (900(4)) is permitted. In one example, the reference area used in the IBC mode in VVC is within the dotted area (915) and includes coded samples. For example, the BV (906) permitted in VVC refers to the reference block (912).

[0110] In some examples, the decoded motion vector difference of the BV (also called BV difference (BVD)) can be left-shifted by 2 before adding it to the corresponding BV predictor to reconstruct the final BV.

[0111] In some embodiments, due to implementation and performance reasons, special processing of the IBC mode may be required, and the IBC mode and the inter prediction mode (e.g., the normal inter prediction mode) may be different as described below. In one example, the reference samples used in the IBC mode are not filtered (e.g., the reconstructed samples before the in-loop filtering process such as the DBF and sample adaptive offset (SAO) filter is applied). Other inter prediction modes of HEVC (e.g., the normal inter prediction mode) can use filtered samples, e.g., reference samples filtered by the in-loop filtering process.

[0112] In some examples, luma sample interpolation is not performed in the IBC mode. Chroma sample interpolation can be performed in the IBC mode. In some examples, chroma sample interpolation is only required when the chroma BV is non-integer when the chroma BV is derived from the corresponding luma BV. In some examples, luma sample interpolation and chroma sample interpolation can be performed in the normal inter prediction mode.

[0113] In the IBC mode, special cases may occur when the chroma BV is a non-integer BV and the reference block is near the boundary of the available area (e.g., the reference area). For example, in order to perform chroma interpolation, the surrounding reconstructed samples can be outside the boundary. In one example, the BV pointing to the line next-to-border of a single boundary can result in the surrounding reconstructed samples being outside the boundary.

[0114] According to an aspect of the present disclosure, the IBC architecture in VVC has specific features.

[0115] In the HEVC SCC extension, the valid reference region for the IBC mode can, as described in FIG. 9, with some exceptions for the purpose of parallel processing, include the entire already reconstructed region of the current picture. A drawback of the reference region used in HEVC can include additional memory requirements in the DPB which may use external memory in a hardware implementation. Additional accesses to external memory can increase the memory bandwidth and the use of the DPB may not be attractive. In some embodiments, on-chip fixed memory (e.g., a fixed-size memory) that can be implemented on-chip for the IBC mode can be used in VVC. The on-chip fixed memory in the IBC mode can significantly reduce the complexity of implementing the IBC mode in a hardware architecture. In one example, the on-chip fixed memory in the IBC mode can reduce latency. In some examples, as in the HEVC SCC extension, the changes address signaling concepts that are separate from the integration within the inter-prediction process.

[0116] In the examples shown in FIGS. 10A to 10D, fixed memory can be allocated to store the reference region used in the IBC mode. The fixed memory can be referred to as a reference sample memory (RSM). A part of the RSM can be updated at different intermediate times during the coding process (e.g., the encoding process or the reconstruction process). FIGS. 10A to 10D show the RSM update process at various intermediate times during the coding process (e.g., the encoding process or the reconstruction process) according to an embodiment of the present disclosure. FIGS. 10A to 10D show the reference region for the IBC mode in VVC and the configuration in VVC.

[0117] Referring to FIGS. 10A to 10D, currently the CTU (1020) is adjacent to a CTU (e.g., the left adjacent CTU) (1010) on the left of the current CTU (1020). In some examples, the current CTU (1020) includes four regions (1021) to (1024). The left adjacent CTU (1010) can include four regions (1011) to (1014) corresponding to the regions (1021) to (1024) respectively. The positions of the regions (1011) to (1014) are shifted left by the width of the CTU (1020) from the positions of the regions (1021) to (1024) respectively. The RSM can include a part of the current CTU (1020) and / or a part of the left adjacent CTU (1010). In the example shown in FIGS. 10A to 10D, the size of the RSM is equal to the size of the CTU. The lightly shaded regions can include reference samples of the left adjacent CTU (1010), the darkly shaded regions can include reference samples of the current CTU (1020), and the unshaded white regions can represent regions to be coded (e.g., future coding regions).

[0118] Referring to FIG. 10A, at the first intermediate point of the coding process which is the start of the coding process of the current CTU (1020), the RSM includes the entire left adjacent CTU (1010), and at the start of the coding process of the current CTU (1020), the entire left adjacent CTU (1010) can function as a reference region in the IBC mode. The RSM at the start of the coding process of the current CTU (1020) does not include any of the regions (1021) to (1024).

[0119] Referring to FIG. 10B, region (1021) includes sub-regions (1031) to (1033). Sub-region (1031) has already been coded (e.g., encoded or reconstructed), sub-region (1032) is the current CU being coded (e.g., during encoding or reconstruction), and sub-region (1033) will be coded later. At the second intermediate point in the coding process of the current CTU (1020) where the sub-region (1032) of the current CTU (1020) is being coded, the RSM is updated to include a part of the left adjacent CTU (1010) and a part of the current CTU (1020). For example, the RSM includes regions (1012) to (1014) of the left adjacent CTU (1010) and the sub-region (1031) of the current CTU (1020). The reference region at the second intermediate point can include regions (1012) to (1014) of the left adjacent CTU (1010) and the sub-region (1031) of the current CTU (1020).

[0120] Referring to FIG. 10C, region (1022) includes sub-regions (1041) to (1043). Sub-region (1041) (dark gray shaded) has already been coded (e.g., encoded or reconstructed), sub-region (1042) is the current CU being coded (e.g., during encoding or reconstruction), and sub-region (1043) (white) will be coded later. At the third intermediate point in the coding process of the current CTU (1020) where the sub-region (1042) of the current CTU (1020) is being coded, the RSM is updated to include (i) regions (1013) to (1014) of the left adjacent CTU (1010) and (ii) regions (1021) and sub-region (1041) of the current CTU (1020). In the RSM, region (1012) is replaced by sub-region (1041). The reference region at the third intermediate point can include (i) regions (1013) to (1014) of the left adjacent CTU (1010) and (ii) regions (1021) and sub-region (1041) of the current CTU (1020).

[0121] Referring to FIG. 10D, region (1024) includes sub-regions (1051) to (1053). Sub-region (1051) (dark gray shaded) has already been coded (e.g., encoded or reconstructed), sub-region (1052) is the current CU being coded (e.g., in the process of encoding or reconstructing), and sub-region (1053) (white) will be coded later. At the fourth intermediate point of the coding process of the current CTU (1020) where the sub-region (1052) of the current CTU (1020) is being coded, the RSM is updated to include regions (1021) to (1023) and sub-region (1051) of the current CTU (1020). The RSM at the fourth intermediate point does not include the regions within the left adjacent CTU (1010). The reference region at the fourth intermediate point can include regions (1021) to (1023) and sub-region (1051) of the current CTU (1020).

[0122] According to one aspect of the present disclosure, VVC has a specific syntax and semantics for the IBC mode.

[0123] In VVC, the IBC architecture can form a dedicated coding mode where the IBC mode is the third prediction mode, in addition to the intra prediction mode and the inter prediction mode (e.g., the normal inter prediction mode). The bitstream can include an IBC syntax element indicating the IBC mode of a CU when the size of the CU is 64×64 or less, for example. In some examples, the maximum CU size for which the IBC mode can be utilized is 64×64 in order to implement the continuous memory update mechanism of RSM as described with reference to FIGS. 10A to 10D. In one example, the reference sample addressing mechanism remains the same as that used in the HEVC SCC extension by indicating a 2D offset and reusing the vector (e.g., MV) coding process of the inter prediction mode. In one example, when the chroma separate tree (CST) is active, the coder cannot derive the chroma BV from the corresponding luma BV, and as a result, the IBC mode is used only for the luma CB.

[0124] The IBC design in VVC can use a fixed memory size (e.g., 128×128) for each color component to store reference samples. As described above, the fixed memory size can enable the on-chip placement of memory (e.g., RSM) in a hardware implementation. In an example such as VVC, the fixed memory size for the maximum CTU size and the IBC mode is 128×128. In one example, when the maximum CTU size configuration is equal to the fixed memory size of the IBC mode (e.g., 128×128), the RSM includes the samples of a single CTU.

[0125] The feature of the RSM is a continuous update mechanism that replaces the reconstructed samples of the left adjacent CTU with the reconstructed samples of the current CTU, as described in FIGS. 10A to 10D. FIGS. 10A to 10D show examples of a simplified RSM of the update mechanism at four intermediate time points during the coding process (e.g., the reconstruction process). The lightly shaded areas in FIGS. 10A to 10C can include the reference samples of the left adjacent CTU (1010), and the darkly shaded areas in FIGS. 10B to 10D can include the reference samples of the current CTU (1020). Referring to FIG. 10A, at the first intermediate time point representing the start of the coding (e.g., encoding or reconstruction) of the current CTU (1020), the RSM is composed only of the reference samples of the left adjacent CTU (1010). At the other three intermediate time points shown in FIGS. 10B to 10D, the coding process (e.g., the encoding process or the reconstruction process) replaces the samples of the left adjacent CTU (1010) with the samples of the current CTU (1020).

[0126] In some examples, the RSM is implicitly divided into four regions, e.g., divided into four separate regions of 64×64. Resetting of regions in the RSM may occur when the coder processes the first CU in the corresponding region within the current CTU, facilitating the hardware implementation work. For example, the RSM is mapped to regions within the CTU (e.g., the left adjacent CTU and the current CTU). FIG. 11 shows the spatial continuous update process (1100) of the RSM. The left adjacent CTU (1010) and the current CTU (1020) are shown in FIGS. 10A - 10D. The left adjacent CTU (1010) can include regions (1011)-(1014). The current CTU (1020) can include regions (1021)-(1024). Region (1023) within the current CTU (1020) can include the currently coded CU (1152), the already coded sub-region (1151), and the sub-region (1153) to be coded. The gray shaded regions can include the samples stored in the RSM, and the unshaded white regions can include the replaced samples or the uncoded samples (e.g., the non-reconstructed samples).

[0127] At the coding time (e.g., reconstruction time) shown in FIG. 11, the RSM update process replaces the samples covered by the unshaded white regions (e.g., regions (1011)-(1013)) within the left adjacent CTU (1010) with the gray shaded regions (e.g., regions (1021)-(1022) and sub-region (1151)) of the current CTU (1020). In FIG. 11, the RSM can include (i) region (1014) within the left adjacent CTU (1010) and (ii) regions (1021)-(1022) and sub-region (1051) of the current CTU (1020).

[0128] In some examples, when the maximum CTU size is less than the RSM size (e.g., 128×128), the RSM can include more than a single left-adjacent CTU, and in the IBC mode, multiple adjacent CTUs can be used as the reference region. For example, when the maximum CTU size is 32×32, an RSM having a size of 128×128 can include samples of 15 adjacent CTUs.

[0129] In VVC, BV coding in the IBC mode can use the process specified for inter prediction (e.g., normal inter prediction). BV coding can use a rule that is simpler than the rule used for inter prediction (e.g., normal inter prediction) to construct the candidate list.

[0130] For example, the candidate list for inter prediction can include five spatial candidates, one temporal candidate, and six history-based candidates. To avoid duplicate entries in the final candidate list for inter prediction, comparison of multiple candidates can be used for the history-based candidates. The candidate list for inter prediction can include pairwise-averaged candidates.

[0131] The candidate list for the IBC mode can include two BVs from each spatial adjacent and five history-based BVs (HBVPs, history-based BVs). In one example, the candidate list for the IBC mode is limited to two BVs from each spatial adjacent and five history-based BVs (HBVPs). In one embodiment, in the IBC mode, when the first HBVP is added to the candidate list, only the first HBVP is compared with the spatial candidates.

[0132] The normal inter prediction mode can use two different candidate lists. For example, one candidate list is for the merge mode, and the other candidate list is for the normal mode (e.g., the inter prediction mode that is not the merge mode). The candidate list in the IBC mode can be the same for both IBC modes (e.g., the merge IBC mode and the normal IBC mode). In the IBC mode, the merge mode may use up to six candidates of the candidate list, and the normal mode uses only the first two candidates of the candidate list.

[0133] Block vector difference (BVD) coding can use the MVD process used in the normal inter prediction mode, and the final BV can have any size. The determined BV (e.g., the reconstructed BV) may refer to a region outside the reference sample region. In one example, a modulo operation based on the width and / or height of the RSM can be used to apply a correction to the absolute offset in each direction.

[0134] According to one aspect of the present disclosure, in some examples, the block vector of the chroma block can be derived from the block vector of the luma block.

[0135] In some examples, when the current coding tree type is SINGLE_TREE, the chroma block always has a corresponding luma block. In the IBC mode, the BV of the chroma block can be derived from the BV of the corresponding luma block using appropriate scaling according to the chroma sampling format (e.g., 4:2:0, 4:2:2) and the chroma BV precision.

[0136] In some examples, the derivation process is used to derive the BV for the chroma block from the BV of the corresponding luma block. The input to the derivation process includes a luma block vector at 1 / 16 fractional sample accuracy (where bvL represents the luma block vector, bvL[0] represents the x component, and bvL[1] represents the y component), and the output of the derivation process includes a chroma block vector at 1 / 32 fractional sample accuracy (where bvC represents the chroma block vector, bvC[0] represents the x component, and bvL[1] represents the y component).

[0137] In some examples, the chroma block vector is derived from the corresponding luma block vector according to Equations (1) and (2). bvC[0] = ((bvL[0]>>(3 + SubWidthC)) × 32 Equation (1) bvC[1] = ((bvL[1]>>(3 + SubHeightC)) × 32 Equation (2) Here, the variables SubWidthC and SubHeightC are specified in Table 1. The variables SubWidthC and SubHeightC depend on the chroma format sampling structure specified through sps_chroma_format_idc.

Table 1

[0138] For example, when sps_chroma_format_idc is equal to 0, the chroma format is monochrome sampling format, and there is only one sample array that can be nominally considered as a luma array. When sps_chroma_format_idc is equal to 1, the chroma format is 4:2:0 sampling format, and each of the two chroma arrays has half the height and half the width of the luma array. When sps_chroma_format_idc is equal to 2, the chroma format is 4:2:2 sampling format, and each of the two chroma arrays has the same height as the luma array and half the width. When sps_chroma_format_idc is equal to 3, the chroma format is 4:4:4 sampling, and each of the two chroma arrays has the same height and width as the luma array.

[0139] In some examples, the number of bits required for the representation of each sample in the luma and chroma arrays within a video sequence is in the range of 8 or more and 16 or less.

[0140] In some examples such as AV1, the IBC mode is called the IntraBC mode, and the BV is used to find the prediction block within the same picture of the current block. The BV can be signaled in the bitstream, and the precision of the signaled BV can be made an integer point. The prediction process in the IBC mode can be the same as the prediction process in the inter prediction mode (e.g., inter picture prediction). The differences between the IBC mode and the inter picture prediction are described below. In the IBC mode, the prediction block can be formed from the reconstructed samples of the current picture (e.g., before applying loop filtering). The IBC mode can be considered as "motion compensation" within the current picture using the BV as the MV.

[0141] In AV1, a flag indicating whether the IBC mode is valid for the current block can be transmitted in the bitstream. If the IBC mode is valid for the current block, the BV difference can be derived by subtracting the predicted BV from the current BV, and the BV difference can be classified into four types according to the horizontal and vertical components of the BV difference value. The type information can be signaled to the bitstream, and the BV difference values of the two components (e.g., the horizontal and vertical components) can be signaled following the type information.

[0142] In AV1, the IBC mode can be effective in coding screen content. The IBC mode may pose challenges to the hardware design. To facilitate the hardware design, some changes can be adopted in the IBC mode, e.g., in AV1.

[0143] In the example of the first change, when the IBC mode is permitted, the loop filter can be disabled. The loop filter can include a deblocking filter, a Constrained Directional Enhancement Filter (CDEF), and a loop restoration (LR) filter. By disabling the loop filter, a second picture buffer dedicated to enabling the IBC mode can be avoided.

[0144] In the example of the second change, to facilitate parallel decoding, the prediction cannot exceed a restricted area. The coordinates of the upper left position of the superblock are (x0, y0). In the superblock, the prediction at position (x, y) can be accessed in the IBC mode when the vertical coordinate is less than y0 and the horizontal coordinate is less than (x0 + 2(y0 - y)). In one example, the prediction at position (x, y) can be accessed in the IBC mode only when the vertical coordinate is less than y0 and the horizontal coordinate is less than (x0 + 2(y0 - y)). In one example, the prediction at position (x, y) can be accessed in the IBC mode only when the vertical coordinate is less than or equal to y0 and the horizontal coordinate is less than (x0 + 2(y0 - y)).

[0145] In the example of the third change, in order to enable hardware write-back delay, the most recent reconstruction area cannot be accessed in IBC mode. The restricted most recent reconstruction area can include 1 to N super blocks, where N is a positive integer. In addition to the second change, if the coordinates of the upper left position of the super block (1210) being reconstructed are (x0, y0), the prediction at the position (x, y) can be accessed in IBC mode when the vertical coordinate is less than or equal to y0 and the horizontal coordinate is less than (x0 + 2(y0 - y) - D). D can indicate the size of the most recent reconstruction area restricted for the IBC mode. FIG. 12 shows an example of the restricted most recent reconstruction area. The areas shaded in gray include the permitted search areas that can be accessed in IBC mode for each current super block (1210) being reconstructed. The areas shaded in black include the prohibited search areas that cannot be accessed in IBC mode for each current super block (1210). The unshaded white areas include the super blocks to be coded (e.g., reconstructed). For the current super block (1210(1)), the most recent reconstruction area includes two super blocks (1221) to (1222) to the left of the current super block (1210(1)) (e.g., N is 2). D indicates the size of 2W, where W is the width of each super block. The super blocks (1221) to (1222) are not accessible to the current super block (1210(1)). The area (1230) is accessible to the current super block (1210(1)).

[0146] According to one aspect of the present disclosure, AV1 can use a local reference range defined in the IBC mode. For example, a part of the on-chip memory having a size of M×M (e.g., 128×128) (e.g., a memory manufactured on the same chip as the processor) can be allocated to store reference samples used in the IBC mode, and that part of the on-chip memory is called the RSM. The RSM can store reconstructed samples that can be used as reference samples. The reconstructed samples stored in the RSM are updated according to an update process, and the range of available reference samples in the RSM can be called the local reference range. In one embodiment, the size of the RSM is equal to the size of the superblock. Based on L×L (e.g., 64×64), a memory reuse mechanism can be applied to the RSM. The RSM can be divided into I RSM units, and I is equal to the ratio of M×M to L×L. For example, when M×M is 128×128 and L×L is 64×64, I is 4 (128×128 / (64×64)). Due to the local reference range, several changes can be made to the IBC mode.

[0147] In the example of the first change, the maximum block size in the IBC mode is limited to L×L (e.g., 64×64).

[0148] In the example of the second change, the reference block and the corresponding current block within the current superblock (SB) can be assumed to be in the same SB row. In one example, the reference block is located only in the current SB or the left adjacent SB to the left of the current SB.

[0149] In the example of the third change, when a unit having a size of L×L (e.g., 64×64) of the RSM unit starts to be updated with the reconstructed samples of the current SB, the previously stored reference samples (e.g., the reference samples of the left adjacent SB) within the entire L×L unit can be marked as unavailable for generating the predicted samples used in the IBC mode.

[0150] FIG. 13 shows an exemplary memory reuse mechanism (1300) in which a memory (e.g., RSM (1310)) is updated during the coding (e.g., encoding or decoding) of the current SB (1301) in the current picture according to an embodiment of the present disclosure. The upper block shows the RSM (1310) in state (0). The upper row shows the RSM (1310) in states (1)-(4). The lower row shows the currently coded current SB (1301) and the left adjacent SB (1302) within the current picture in states (0)-(4). The left adjacent SB (1302) can be to the left of the current SB (1301). In the example of FIG. 13, quadtree partitioning is used at the SB root, and the SB can include four regions. In one example, the size of each of the four regions is 64×64. In one example, the current SB (1301) includes four regions 4-7, and the left adjacent SB (1302) includes four regions 0-3.

[0151] In the initial state (0) when starting to code each SB such as the current SB (1301), the RSM (1310) can store samples of previously coded SBs (e.g., the left adjacent SB (1302)). When the current block is located in one of the four regions (e.g., four 64×64 regions) within the current SB (1301), the corresponding region within the RSM (1310) is emptied and can be used to store samples of the current coding region (e.g., the current 64×64 coding region). The samples within the RSM (1310) can be gradually updated by the samples within the current SB (1301).

[0152] Referring to state (1), the current block (1311) is located in region 4 within the current SB (1301), the corresponding region (e.g., the upper left region) in the RSM (1310) is emptied, and can be used to store samples of region 4 which is the currently coded region. Referring to the line below, the BV (e.g., the coded BV or the decoded BV) (1321) can point from the current block (1311) to a reference block (1331) within the search range (1341) of the current block (1311) (the boundary of the search range (1341) is marked by a dashed line). Referring to the line above, the corresponding offset (1351) in the RSM (1310) can point from the current block (1311) to the reference block (1331) in the RSM (1310). Referring to state (1), the search range (1341) includes regions 1 - 3 in the left adjacent SB (1302) and the coded sub-region (1361) within region 4. The search range (1341) does not include region 0 in the left adjacent SB (1302).

[0153] Referring to state (2), the current block (1312) is located in region 5 within the current SB (1301), the corresponding region (e.g., the upper right region) in the RSM (1310) is emptied, and can be used to store samples of region 5 which is the currently coded region. The BV (e.g., the coded BV or the decoded BV) (1322) can point from the current block (1312) to a reference block (1332) within the search range (1342) of the current block (1312) (the boundary of the search range (1342) is marked by a dashed line). The corresponding offset (1352) in the RSM (1310) can point from the current block (1312) to the reference block (1332) in the RSM (1310). Referring to state (2), the search range (1342) includes (i) regions 2 - 3 in the left adjacent SB (1302) and (ii) the coded sub-region (1362) within regions 4 and 5 within the current SB (1301). The search range (1342) does not include regions 0 - 1 in the left adjacent SB (1302).

[0154] Referring to state (3), the current block (1313) is located in region 6 within the current SB (1301), the corresponding region (e.g., the lower left region) in the RSM (1310) is emptied, and it can be used to store samples of region 6 which is the currently coded region. The BV (e.g., the coded BV or the decoded BV) (1323) can point from the current block (1313) to a reference block (1333) within the search range (1343) of the current block (1313) (the boundaries of the search range (1343) are marked by dashed lines). The corresponding offset (1353) in the RSM (1310) can point from the current block (1313) to the reference block (1333) in the RSM (1310). Referring to state (3), the search range (1343) includes (i) region 3 in the left adjacent SB (1302) and (ii) the coded sub-regions (1363) within regions 4 - 5 and region 6 within the current SB (1301). The search range (1343) does not include regions 0 - 2 in the left adjacent SB (1302).

[0155] Referring to state (4), the current block (1314) is located in region 7 within the current SB (1301), the corresponding region (e.g., the lower right region) in the RSM (1310) is emptied, and it can be used to store samples of region 7 which is the currently coded region. The BV (e.g., the coded BV or the decoded BV) (1324) can point from the current block (1314) to a reference block (1334) within the search range (1344) of the current block (1314) (the boundaries of the search range (1344) are marked by dashed lines). The corresponding offset (1354) in the RSM (1310) can point from the current block (1314) to the reference block (1334) in the RSM (1310). Referring to state (4), the search range (1344) includes the coded sub-regions (1364) within regions 4 - 6 and region 7 within the current SB (1301). The search range (1344) does not include regions 0 - 3 in the left adjacent SB (1302).

[0156] When the current SB (1301) is fully coded, the entire RSM (1310) can be filled with all samples of the current SB (1301).

[0157] In the example shown in FIG. 13, the current SB (1301) is partitioned using a quadtree split. The coding order of the four regions within the current SB (1301) can be the upper left region (e.g., region 4), the upper right region (e.g., region 5), the lower left region (e.g., region 6), and the lower right region (e.g., region 7). In other block split decisions such as those shown in FIGS. 14A - 14B, the RSM update process can be made similar to that shown in FIG. 13 by, for example, replacing each region in the RSM using the reconstructed samples within the current SB.

[0158] FIGS. 14A - 14B show an exemplary memory update process in the RSM during the coding (e.g., encoding or decoding) of the current SB (1401). In FIGS. 14A - 14B, the left - adjacent SB (1402) is to the left of the current SB (1401) being coded (e.g., encoded or decoded). In one example, the size of each of the current SB (1401) and the left - adjacent SB (1402) is 128×128. Each of the current SB (1401) and the left - adjacent SB (1402) can include four regions (e.g., four blocks) of size 64×64. The current SB (1401) can include blocks 4 - 7, and the left - adjacent SB (1402) can include blocks 0 - 3.

[0159] In FIG. 14A, a horizontal split is performed at the SB root, followed by a vertical split. The SB (e.g., the current SB (1401)) can include four blocks: the upper left block (e.g., block 4), the lower left block (e.g., block 6), the upper right block (e.g., block 5), and the lower right block (e.g., block 7). The coding order of the current SB (1401) can be the upper left block (state 1), the upper right block (state 2), the lower left block (state 3), and the lower right block (state 4).

[0160] In FIG. 14B, vertical splitting is performed on the SB route, followed by horizontal splitting. The coding order of the current SB (1401) can be the upper left block (state 1), the lower left block (state 2), the upper right block (state 3), and the lower right block (state 4).

[0161] Depending on the position of the current block (e.g., (1431)) with respect to the current SB (1401), the following can be applied.

[0162] (i) Referring to state (1) in FIGS. 14A - 14B, the current block (1431) is in the upper left block (e.g., block 4) of the current SB (1401). The RSM can include, in addition to the already reconstructed samples in the block (1461) in block 4, the reference samples in the lower right block (e.g., block 3), the lower left block (e.g., block 2), and the upper right block (e.g., block 1) of the left adjacent SB (1402).

[0163] (ii) Referring to state (2) in FIG. 14A or state (3) in FIG. 14B, the current block (1432) is in the upper right block (e.g., block 5) of the current SB (1401).

[0164] As shown in state (2) in FIG. 14A, if the luma sample located at the upper left corner of block 6 (e.g., (0, 64) with respect to the current SB (1401)) has not been reconstructed yet, in addition to the already reconstructed samples in block 4 and the block (1462) in block 5, the current block (1432) can refer to the reference samples in the lower left block (e.g., block 2) and the lower right block (e.g., block 3) of the left adjacent SB (1402). The corresponding RSM can include, in addition to the blocks in block 4 and block 5 (1462), the reference samples in the lower left block (e.g., block 2) and the lower right block (e.g., block 3) of the left adjacent SB (1402).

[0165] Otherwise, as shown in state (3) in FIG. 14B, when the luma sample located at the upper left corner of block 6 (for example, (0, 64) with respect to the current SB (1401)) is reconstructed, the current block (1432) can refer to the reference samples in the lower right block of the left adjacent SB (1402) (for example, block 3). The corresponding RSM can include the already reconstructed samples in blocks 4 and 6, the block (1462) in block 5, and in addition, the reference samples in the lower right block of the left adjacent SB (1402) (for example, block 3).

[0166] (iii) Referring to state (3) in FIG. 14A or state (2) in FIG. 14B, the current block (1433) is in the lower left block of the current SB (1401) (for example, block 6).

[0167] As shown in state (2) in FIG. 14B, when the luma sample located at the upper left corner of block 5 (for example, (64, 0) with respect to the current SB (1401)) has not yet been reconstructed, in addition to the already reconstructed samples in block 4 and the block (1463) in the current SB (1401), the current block (1433) can refer to the reference samples in the upper right block (for example, block 1) and the lower right block (for example, block 3) of the left adjacent SB (1402). The corresponding RSM can include the block 4, the block (1463) in the current SB (1401), and in addition, the reference samples in the upper right block (for example, block 1) and the lower right block (for example, block 3) of the left adjacent SB (1402).

[0168] Otherwise, as shown in state (3) in FIG. 14A, when the luma sample located at the upper left corner of block 5 (for example, (64, 0) with respect to the current SB (1401)) is reconstructed, the current block (1433) can refer to the reference samples within the lower right block of the left adjacent SB (1402) (for example, block 3). The corresponding RSM can include the reference samples within the lower right block of the left adjacent SB (1402) (for example, block 3) in addition to the already reconstructed samples within blocks 4 - 5 and the block (1463) in the current SB (1401).

[0169] (iv) Referring to state (4) in FIGS. 14A - 14B, the current block (1434) is in the lower right block of the current SB (1401) (for example, block 7). The current block (1434) can refer to the already reconstructed samples within the current SB (1401), such as the already reconstructed samples within blocks 4 - 6 and the block (1464). The corresponding RSM can include the reference samples within blocks 4 - 6 and the block (1464). In one example, when the current block (1434) corresponds to the lower right block of the current SB (1401), the current block can only refer to the already reconstructed samples within the current SB (1401).

[0170] According to one aspect of the present disclosure, in some examples (for example, ECM software), template - matching - based prediction techniques can be used for intra - prediction.

[0171] Template matching (TM) technology can be used in video / image coding. To further improve the compression efficiency of the VVC standard, for example, TM can be used to improve the motion vectors (MVs). In one example, TM is used on the decoder side. Using the TM mode, the MV is improved by constructing a template (e.g., the current template) of a block (e.g., the current block) within the current picture, and the closest match between the template of the block within the current picture and multiple possible templates (e.g., multiple possible reference templates) within the reference picture can be determined. In one embodiment, the template of a block within the current picture can include the left adjacent reconstructed samples of the block and the upper adjacent reconstructed samples of the block. TM can be used in video / image coding beyond VVC.

[0172] Figure 15 shows an example of template matching (1500). TM can be used to derive the motion information of the current coding unit (CU) (e.g., the current block) (1501) by determining the closest match between a template (e.g., the current template) (1521) of the current CU within the current picture (1510) and a template (e.g., the reference template) among multiple possible templates within the reference picture (1511) (e.g., one of the multiple possible templates is template (1525)). The template (1521) of the current CU (1501) can have any suitable shape and any suitable size.

[0173] In one embodiment, the template (1521) of the current CU (1501) includes an upper template (1522) and a left template (1523). Each of the upper template (1522) and the left template (1523) can have any suitable shape and any suitable size.

[0174] The upper template (1522) can include samples within one or more upper adjacent blocks of the current CU (1501). In one example, the upper template (1522) includes samples of four rows within one or more upper adjacent blocks of the current CU (1501). The left template (1523) can include samples within one or more left adjacent blocks of the current CU (1501). In one example, the left template (1523) includes samples of four columns within one or more left adjacent blocks of the current CU (1501).

[0175] Each of the plurality of possible templates (e.g., template (1525)) within the reference picture (1511) corresponds to the template (1521) within the current picture (1510). In one embodiment, the initial MV (1502) points to the reference block (1503) within the reference picture (1511) from the current CU (1501). Each of the plurality of possible templates (e.g., template (1525)) within the reference picture (1511) and the template (1521) within the current picture (1510) can have the same shape and the same size. For example, the template (1525) of the reference block (1503) includes the upper template (1526) within the reference picture (1511) and the left template (1527) within the reference picture (1511). The upper template (1526) can include samples within one or more upper adjacent blocks of the reference block (1503). The left template (1527) can include samples within one or more left adjacent blocks of the reference block (1503).

[0176] The TM cost can be determined based on pairs of templates such as a template (e.g., the current template) (1521) and a template (e.g., the reference template) (1525). The TM cost can indicate the match between the template (1521) and the template (1525). The optimized MV (or the final MV) can be determined based on a search around the initial MV (1502) of the current CU (1501) within the search range (1515). The search range (1515) can have any appropriate shape and any appropriate number of reference samples. In one example, the search range (1515) within the reference picture (2111) includes a [-L, L]-pel range, where L is a positive integer such as 8 (e.g., 8 samples). For example, a difference (e.g., [0, 1]) is determined based on the search range (1515), and an intermediate MV is determined by the sum of the initial MV (1502) and the difference (e.g., [0, 1]). The intermediate reference block and the corresponding template within the reference picture (1511) can be determined based on the intermediate MV. The TM cost can be determined based on the template (1521) and the intermediate template within the reference picture (1511). The TM cost can correspond to a difference (e.g., [0, 0], [0, 1], etc. corresponding to the initial MV (1502)) determined based on the search range (1515). In one example, the difference corresponding to the minimum TM cost is selected, and the optimized MV is the sum of the difference corresponding to the minimum TM cost and the initial MV (1502). As described above, TM can derive the final motion information (e.g., the optimized MV) from the initial motion information (e.g., the initial MV 1502).

[0177] In the example of FIG. 15, a better MV can be searched around the initial motion vector of the current CU within a search range such as [-8 pels, +8 pels].

[0178] The above TM technique may be appropriately modified. In one example, the search step size can be determined based on the adaptive motion vector resolution (AMVR) mode, and TM can be cascaded with a bilateral matching process in the merge mode.

[0179] In some examples, in an advanced motion vector prediction (AMVP) mode, to select a particular MVP candidate that reaches the minimum difference between the current block template and the reference block template, a particular MVP candidate (also referred to as an AMVP candidate) is determined based on a template matching error, and then TM is performed only for the particular MVP candidate for MV refinement. TM uses iterative diamond search to start from full pixel MVD accuracy within the [-8, +8] pixel search range (or 4 pixels in the case of 4-pixel AMVR mode) to refine a particular MVP candidate. The AMVP candidate is further refined by using cross search with full pixel MVD accuracy (or 4 pixels in the case of 4-pixel AMVR mode), and then half pixel and quarter pixel accuracies sequentially follow depending on the AMVR mode as specified in Table 2. The search process ensures that the MVP candidate still maintains the same MV accuracy as indicated by the AMVR mode after the TM process. In the search process, if the difference between the previous minimum cost and the current minimum cost in an iteration is smaller than a threshold equal to the area of the block, the search process ends.

Table 2

[0180] In some examples, in merge mode, a similar template matching-based search method is applied to the merge candidates indicated by the merge index. As shown in Table 2, TM may execute all the way to 1 / 8 pel MVD accuracy or skip them beyond half pel MVD accuracy, depending on whether an alternative interpolation filter (used when AMVR is in half pel mode) is used according to the merge motion information. Additionally, when the TM mode is enabled, template matching can function as an independent process or an extra MV refinement process between the block-based bilateral matching (BM) method and the sub-block-based BM method, depending on whether BM can be enabled according to its enabling condition check.

[0181] According to an aspect of the present disclosure, a template matching-based candidate reordering technique can be used to reduce signaling overhead. For example, a technique called adaptive reordering of merge candidates with template matching (ARMC-TM) can be used.

[0182] In some examples, using ARMC-TM, merge candidates are adaptively reordered using template matching (TM). ARMC-TM can be applied to normal merge mode, template matching (TM) merge mode, and affine merge mode (excluding SbTMVP candidates). For TM merge mode, merge candidates are reordered before the refinement process.

[0183] In some examples, after the merge candidate list is constructed using ARMC-TM, the merge candidates are split into several subgroups. In one example, the subgroup size is set to 5 for both the normal merge mode and the TM merge mode. In another example, the subgroup size is set to 3 for the affine merge mode. The merge candidates within each subgroup are sorted in ascending order according to the cost value based on template matching. For simplicity, in some examples, the merge candidates in the last subgroup rather than the first subgroup are not sorted.

[0184] The template matching cost of the merge candidate is measured by the sum of absolute differences (SAD) between the samples of the template of the current block and the corresponding reference samples for the template (also called the reference template in one example). The template includes a set of reconstructed samples adjacent to the current block. The reference samples of the template are positioned according to the motion information of the merge candidate.

[0185] When the merge candidate utilizes bidirectional prediction, the reference samples of the template of the merge candidate are also generated by bidirectional prediction.

[0186] FIG. 16 shows a reference sample of a template of a current block for a merge candidate of dual prediction. In FIG. 16, a current picture (1610) includes a current block for coding. When a merge candidate is a dual prediction merge candidate, the MV of the merge candidate can point to a first reference block in a reference picture (1620) and a second reference block in a second reference picture (1630). The template of the current block is indicated by (T), and the template includes a set of reconstructed samples adjacent to the current block. The reference samples of the first set of the template are in the first reference picture (1620) adjacent to the first reference block, and the reference samples of the second set of the template are in the second reference picture (1630) adjacent to the second reference block. In one example, the template matching cost of the dual prediction merge candidate is calculated by adding a first sum of absolute differences (SAD) between the samples of the template of the current block and the reference samples of the first set of the template and a second sum of absolute differences (SAD) between the samples of the template of the current block and the reference samples of the second set of the template.

[0187] In some examples, the merge candidate can be a sub-block based merge candidate. In one example, for a sub-block based merge candidate having a sub-block size equal to Wsub×Hsub, the top template can include several sub-templates having a size of Wsub×1, and the left template can include several sub-templates having a size of 1×Hsub. Wsub can be the width of the sub-block, and Hsub can be the height of the sub-block.

[0188] An exemplary derivation of a template for a current block and a reference sample of the template using sub-block-based merge candidates can be shown in FIG. 17. As shown in FIG. 17, the current block 1702 can be included in the current picture 1704. The current block (1702) can include sub-blocks A - G in the first row and the first column. The current block (1702) can include a template (1706) adjacent to the upper side and the left side of the current block (1702). The block (1708) at the same position of the current block (1702) is within the reference picture (1710). The block (1708) at the same position can include sub-blocks A - G in the first row and the first column corresponding to sub-blocks A - G within the current block (1702). The sub-block motion information (e.g., corresponding to an affine motion vector) of sub-blocks A - G in the first row and the first column of the current block (1702) can be used to derive a reference sample of a sub-template (or sub-reference template) of the block (1708) at the same position. For example, the motion information of sub-blocks A, E, F, and G of the current block 1702 can be applied to derive a reference sample of a sub-template located adjacent to the left side of sub-blocks A, E, F, and G of the block (1708) at the same position. The sub-template adjacent to the left side of sub-blocks A, E, F, and G of the block (1708) at the same position can form a left reference template of the block (1708) at the same position. The motion information of sub-blocks A, B, C, and D of the current block (1702) can be applied to derive a reference sample of a sub-template located adjacent to the upper side of sub-blocks A, B, C, and D of the block (1708) at the same position. The sub-template adjacent to the upper side of sub-blocks A, B, C, and D of the block (1708) at the same position can further form an upper reference template of the block (1708) at the same position.

[0189] In some examples, MV candidate type-based ARMC can be used. For example, the merge candidates of one single candidate type, e.g., TMVP or non-adjacent MVP (NA-MVP), can be sorted based on the ARMC TM cost value. Then, the sorted candidates are added to the merge candidate list. For example, the TMVP candidate type ARMC can add more TMVP candidates with more temporal positions and different inter-prediction directions to perform sorting and selection. Further, the NA-MVP candidate type ARMC extends to non-adjacent MVPs with more spatially non-adjacent positions. The target reference picture of the TMVP candidate can be selected from any one of the reference pictures in the list according to the scaling factor. For example, the selected reference picture is the one with the scaling factor closest to 1.

[0190] According to one aspect of the present disclosure, template matching-based candidate sorting can be performed for MMVD and affine MMVD.

[0191] In some examples, the MMVD offset is extended to more positions for the MMVD mode and the affine MMVD mode.

[0192] FIG. 18 shows a diagram indicating directions in which improved positions can be added for MMVD. In FIG. 18, additional improved positions along the diagonal of k×π / 8 are added, where k is an integer. The position (1801) corresponds to the base candidate and can be used as the starting point, and the positions (1811) to (1814) are in the directions of 0, π / 2, π, and 3π / 2, respectively. Further directions are added. For example, the positions (1821) to (1824) are in the directions of π / 4, 3π / 4, 5π / 4, and 7π / 4, respectively, and the positions (1831) to (1838) are in the directions of π / 8, 3π / 8, 5π / 8, 7π / 8, 9π / 8, 11π / 8, 13π / 8, and 15π / 8, respectively. Therefore, the number of directions increases from 4 to 16. Further, in one example, each direction can have 6 MMVD improved positions. The total number of possible MMVD improved positions is 16×6.

[0193] According to one aspect of the present disclosure, the SAD cost between the current template (e.g., one row above and one column to the left of the current block) and the reference template can be calculated for each improvement position. Based on the SAD cost of the improvement positions, all possible MMVD improvement positions (16×6) for each base candidate are sorted. Then, the upper part of the improvement positions, such as the upper 1 / 8 of the improvement positions (e.g., 12) like those having the minimum template SAD cost, is retained as available positions for the resulting MMVD index coding. The MMVD index is binarized by a rice code having a parameter equal to 2.

[0194] In some examples, the improvement positions for the affine MMVD can be increased, and the template matching-based candidate sorting can be applied to the affine MMVD sorting. For example, the affine MMVD improvement positions are in the directions along the diagonals of k×π / 4, such as each of the eight directions of 0, π / 4, π / 2, 3π / 4, π, 5π / 4, 3π / 2, and 7π / 4. Each direction can have six affine MMVD improvement positions. The total number of possible affine MMVD improvement positions is 8×6. In one example, the SAD cost between the current template (e.g., one row above and one column to the left of the current block) and the reference template can be calculated for each improvement position. Based on the SAD cost of the improvement positions, all possible affine MMVD improvement positions (8×6) for each base candidate are sorted. Then, the upper part of the improvement positions, such as the upper 1 / 2 of the improvement positions (e.g., 24) like those having the minimum template SAD cost, is retained as available positions for the resulting affine MMVD index coding.

[0195] Note that different video codecs may include further features. For example, the IBC mode in ECM-5.0 has several improvements in addition to the IBC method in VVC. According to one aspect of the present disclosure, the construction of the IBC merge / AMVP list in ECM-5.0 is updated with several changes. In an example of the first change, the IbC merge / AMVP candidate can be inserted into the IbC merge / AMVP candidate list only when the IbC merge / AMVP candidate is valid. In an example of the second change, the upper-right, lower-left, and upper-left spatial candidates and one pairwise average candidate can be added to the IBC merge / AMVP candidate list. In an example of the third change, template-based adaptive reordering (ARMC-TM) is applied to the IBC merge list.

[0196] In ECM-5.0, history-based motion vector prediction (HMVP) can be used in the IBC mode, and the HMVP table size for IBC increases to 25. In some examples, after up to 20 IBC merge candidates are derived using full pruning, they are sorted together. After sorting, the first six candidates with the lowest template matching cost are selected as the final candidates in the IBC merge list.

[0197] In ECM-5.0, the candidate of the zero vector for padding the IBC merge / AMVP list is replaced with a set of BVP candidates located within the IBC reference region. The set of BVP candidates is called the padding candidates. The zero vector is invalid as a block vector in the IBC merge mode, and thus the zero vector is discarded as a BVP in the IBC candidate list.

[0198] FIG. 19 shows a diagram illustrating a set of padding candidates in some examples. In the example of FIG. 19, a buffer called the IBC buffer is allocated for reconstruction in IBC mode. During reconstruction, the IBC buffer is allocated to buffer a part (1900) of the picture. The part (1900) includes reconstruction samples within the reconstruction area (shown hatched), the current block (shown by the black solid line), and the area to be reconstructed (shown by the white space). In FIG. 19, W represents the width of the current block, and H represents the height of the current block. In the example of FIG. 19, a part of the reconstruction area can be used for reference to reconstruct the current block, and a part of it is called the reference area (1910). The reference area (1910) includes sub-areas (e.g., A, B, and C). In FIG. 19, three padding candidates (1901)-(1903) are located at the closest corners of the reference area (1910), and three further padding candidates (1904)-(1906) are determined at the centers of the three sub-areas (A, B, and C). The coordinates of the padding candidates (1901)-(1906) are determined by the width and height of the current block, and the ΔX and ΔY parameters, as shown in FIG. 19.

[0199] According to one aspect of the present disclosure, template matching can be appropriately used in the current picture reference, such as in the IBC merge mode, the IBC AMVP mode, etc. In some examples, a merge list called the IBC-TM merge list is used. The IBC-TM merge list is changed from the normal merge list used in the normal IBC merge mode. For example, candidates are selected according to a pruning method such as a pruning method using the motion distance between candidates in the normal TM merge mode. The end zero vector can be replaced by padding candidates to the left (-W,0) (shown as (1901) in FIG. 19), up (0,-H) (shown as (1902) in FIG. 19), and upper left (-W,-H) (shown as (1903) in FIG. 19), etc., where W is the width of the current CU and H is the height of the current CU.

[0200] In IBC-TM merge mode, in one example, the selected candidate is refined using template matching before rate distortion optimization (e.g., on the encoder side) or the decoding process (e.g., on the decoder side). In some examples, when both IBC-TM merge mode and normal IBC merge mode can be used, a TM merge flag can be signaled to indicate whether IBC-TM merge mode or normal IBC merge mode is used.

[0201] In IBC-TM AMVP mode, up to three candidates are selected from the IBC-TM merge list. Each of the three selected candidates is refined using template matching refinement and sorted according to the template matching cost. Only the first two candidates with lower template matching costs are considered in the motion estimation process as normal.

[0202] The template matching refinement for both IBC-TM merge mode and AMVP mode follows the IBC motion vector constraints that (i) it is an integer and (ii) it is within the reference region.

[0203] FIG. 20 shows a diagram illustrating an IBC reference region that depends on the position of the current CU position. In the example of FIG. 20, the IBC buffer is allocated the CTU size. Each CTU contains four sub-regions, and the IBC buffer is configured to be updated according to the sub-regions. In the example of FIG. 20, the left CTU includes sub-regions 0 to 3, and the current CTU includes sub-regions 4 to 7. During reconstruction, when the current block is in sub-region 4, as shown by state (1), the IBC reference region includes sub-regions 1, 2, and 3; when the current block is in sub-region 5, as shown by state (2), the IBC reference region includes sub-regions 2, 3, and 4; when the current block is in sub-region 6, as shown by state (3), the IBC reference region includes sub-regions 3, 4, and 5; when the current block is in sub-region 7, as shown by state (4), the IBC reference region includes sub-regions 4, 5, and 6.

[0204] In some examples, in the IBC-TM merge mode, template matching improvements are performed with integer precision. In some examples, in the IBC-TM AMVP mode, the template matching improvements are performed with either integer precision or quarter precision depending on the AMVR value. In some examples, the template matching improvements in the IBC-TM merge mode and the IBC-TM AMVR mode are applied to samples without interpolation. In both the IBC-TM merge mode and the IBC-TM AMVR mode, the improvement motion vector (also referred to as the improvement block vector) and the template used in each improvement step are required to satisfy the constraints of the reference region (e.g., it is required to be within the IBC reference region shown by FIG. 20).

[0205] In some examples, the reference region for IBC is extended to two CTU rows above the current CTU.

[0206] FIG. 21 shows a diagram (2100) showing a reference region for coding a CTU (m, n) in some examples. For the current CTU (m, n) to be coded, the reference region includes CTUs having indices (m-2, n-2)...(W, n-2), (0, n-1)...(W, n-1), (0, n)...(m, n), where W represents the maximum horizontal index within the current tile, slice, or picture, such as the lightly shaded CTU in FIG. 21. The upper reference region setting can ensure that when the CTU size is 128, IBC does not require extra memory in the current ETM platform. In some examples, the block vector search (or local search, also called) range per sample is limited to [-(C<<1), C>>2] in the horizontal direction and [-C, C>>2] in the vertical direction to adapt to the reference region extension, where C represents the CTU size.

[0207] Template matching-based intra prediction is called intra template matching prediction (IntraTMP). In the IntraTMP mode, the best prediction block for the current block is determined from the reconstructed part of the current picture based on the matching of the L-shaped reference template of the best prediction block for the current template of the current block. In some examples, the encoder searches for a block having a template most similar to the current template of the current block within a predetermined search range in the reconstructed part of the current frame, and uses that block as the prediction block for the current block. Then, the encoder signals the use of the IntraTMP mode for the prediction of the current block, and the same prediction operation can be performed on the decoder side.

[0208] FIG. 22 shows a diagram illustrating a search region for intra template matching prediction in some examples. In FIG. 22, the current picture (2200) is partitioned into CTUs as shown by the horizontal CTU boundary (22101) and the vertical CTU boundary (2202) in FIG. 22. The current block (2210) is within the current CTU. The adjacent samples of the current block form the current template (2215) which is L-shaped. FIG. 22 shows a predetermined search region for intra template matching prediction including four regions R1 to R4. R1 is within the current CTU, R2 is within the upper left CTU, R3 is within the upper CTU, and R4 is within the left CTU.

[0209] In one example, within each region, for each potential matching (coincident) block (2220), the L-shaped adjacent samples of the potential matching block form a potential template (2225) which is L-shaped. The sum of absolute differences (SAD) between the potential template and the current template is calculated as the template matching cost of the potential matching block (2220).

[0210] The encoder or decoder can search a predetermined area to determine a matching block having the lowest template matching cost, and the matching block is used as the predicted block of the current block (2210).

[0211] In some examples, the dimensions of a predetermined area such as R1 to R4 are defined in proportion to the dimensions of the current block in order to have a fixed number of comparisons per pixel. In one example, the area R2 can have a size of (SearchRange_w, SearchRange_h), where SearchRange_w is the width of the area R2 and SearchRange_h is the height of the area R2. The current block (2210) can have a size of (BlkW, BlkH), where BlkW is the width of the current block (2210) and BlkH is the height of the current block (2210). The width and height of the area R2 can be set according to Equations (3) and (4). SearchRange_w = a × BlkW Equation (3) SearchRange_h = a × BlkH Equation (4) Here, "a" is a constant that can control the trade-off between gain and complexity. In one example, "a" is equal to 5.

[0212] In some examples, the IntraTMP tool is enabled for CUs having sizes with a width and height of 64 or less. In some examples, the maximum CU size for IntraTMP is configurable.

[0213] In some examples, the IntraTMP mode is signaled at the CU level through a dedicated flag when decoder-side intra mode derivation (DIMD) is not used for the current CU.

[0214] According to one aspect of the present disclosure, the BV predictor of IBC and the BV accuracy of the BV difference in the current ECM software are restricted by the BV accuracy signaled by the AMVR syntax value. Even when using the template matching method and using a rough BV accuracy signal transmission to save bits for BVD, it is impossible to find a finer BV value. Furthermore, the intra-template matching mode does not require BV signaling, but the template matching-based search process may have restrictions on the search range (for example, search range restrictions based on equations (3) and (4)), and as a result, it is also affected by the search pattern, which may not converge to an optimal result.

[0215] Note in the following description that the template of a block can indicate any suitable part of the adjacent samples of the block, such as the adjacent samples above, to the left, to the right, and below the block.

[0216] FIG. 23 shows an example of the current block and the template of the current block. The template (indicated by the gray area) includes the upper and left adjacent reconstructed samples.

[0217] FIG. 24 shows a diagram showing another example of the current block and the template of the current block. The template (indicated by the gray area) includes the upper and left adjacent reconstructed samples.

[0218] In addition to the template patterns in FIGS. 23 and 24, other suitable patterns can also be used in the technology disclosed in the present disclosure.

[0219] Aspects of the present disclosure provide techniques for use in current picture reference modes such as IBC mode, IntraIBC mode, IntraTMP mode. In some embodiments, template matching based BV improvement is used in IBC mode (including IBC and IntraIBC) to achieve better BV (e.g., reduced template matching cost). In some embodiments, BVD is introduced into IntraTMP to relax the restrictions on the search range.

[0220] Aspects of the present disclosure provide techniques for enabling template matching (TM) based BV improvement by reduced signaling cost for IBC mode. The template matching based BV improvement for IBC can search for reconstructed samples within the available reference region of the current picture.

[0221] FIG. 25 shows a diagram illustrating template matching-based BV improvement in some examples. FIG. 25 shows a current block (2510) using a current template (2515). The block vector (BV, block crcotr) of the current block (2510) is determined within the search range of the reconstruction area. The block vector points to a reference block (2520) using a reference template (2525) within the search range. The template matching-based BV improvement is applied to the block vector (BV, block vector). For example, during the process of template matching-based BV improvement, an improvement offset is applied to the BV in small steps (e.g., in a predetermined direction and position pattern) to generate potential improved BV values that point to potential improved reference blocks around the reference block (2520). The potential improved reference blocks each have a respective potential improved reference template. The template matching cost can be calculated between the current template (2515) and each potential improved reference template. In some examples, a potential improved reference template having the lowest template matching cost is determined, and the potential improved BV associated with the potential improved reference template having the lowest template matching cost is determined to be the improved BV output from the process of template matching-based BV improvement.

[0222] In some embodiments, for the IBC mode using BVD signal transmission, template matching-based BV improvement can be applied without following the constraints on BV accuracy (also called BV resolution in some examples) indicated by the AMVR syntax value. Thus, in some examples, the BVD signal transmission can use a coarser accuracy to save bits for transmitting the BVD value, and the final block vector with a finer accuracy can be obtained by template matching-based BV improvement, such as using an improvement step size smaller than the coarser accuracy.

[0223] In some examples, a block vector predictor (BVP) is improved without accuracy limitations (e.g., the constraints indicated by the AMVR syntax value). The improved BVP can be combined with the signaled block vector difference (BVD) to determine the final BV in the IBC mode, and the signaling of the BVD still follows the BV accuracy according to the AMVR syntax value.

[0224] In some examples, both the BVP and the BVD follow the BV accuracy according to the AMVR syntax value. The BV value generated by BVP + BVD can be used as a starting point for template matching-based BV improvement, and the template matching-based BV improvement can have a step size smaller than the BV accuracy indicated by the AMVR syntax value.

[0225] In some examples, the search step size in the template matching-based improvement can be set to be smaller than the BV resolution signaled by the AMVR syntax value. In one example, when AMVR indicates a 4-pel BV accuracy, the search step size in the template matching-based BV improvement is set to 1 pel or 2 pels. In one example, when the BVP is improved using template matching-based BV improvement, the BVP may have a value that does not match the BV accuracy indicated by the AMVR syntax value. In one example, to calculate the final BV value for IBC, the improved BVP is added together with the signaled BVD value (which is consistent with the BV accuracy signaled by the AMVR syntax).

[0226] In some embodiments, template matching-based BV improvement can be applied to the decoded or derived (merge mode) IBC BV as an improvement. For example, when the decoded or derived (merge mode) BV has a resolution higher than 1 pel (1 sample integer), the template matching-based BV improvement can be performed around the BV with a reduced step size to find a better position that can minimize the matching cost.

[0227] In some examples, the decoded BV is at 4-pel resolution. For example, when the BV value is (-40, -40), the adjacent BV values that may be signaled can be (-36, -40), (-36, -36), (-40, -36), (-44, -44), etc. The template matching-based BV improvement can use a reduced step size such as 1 sample, and the search range in the template matching-based BV improvement is set to be ±1 or ±2 around the decoded BV.

[0228] In some embodiments, hash-based search is applied to accelerate the template matching search process for IBC. In some examples, the hash-based search can find an exact match of the template. In some examples, the template matching-based search process includes a first search step that is a hash-based search and a second search step that is a template matching-based BV improvement. The first search step is a sub-block-based search (e.g., sub-block comparison), and the second search step is a sample-based search (e.g., sample comparison).

[0229] In some examples, in hash-based search, each sub-block having size W×H in the luma samples within the reconstruction area available for reference can be hashed to generate an entry in the hash map, where W is the width of each sub-block and H is the height of each sub-block. In one example, both W and H are equal to 4. The template for IBC improvement (e.g., the template of the current block, also referred to as the current template) also includes a plurality of sub-blocks (also referred to as template sub-blocks). The sub-blocks within the template for IBC improvement can be hashed to generate hash values. The hash-based search operates based on the hash values of the template sub-blocks.

[0230] In one example, in the first search step, when the hash-based search finds a BV having a matching template for the current template, the template matching-based search ends. When the first search step cannot find a BV having a matching template for the current template, a second search step, which is a BV improvement based on template matching, can then be applied.

[0231] In some embodiments, the template matching cost is used as a metric for BV improvement. In some examples, for candidates in template matching-based BV improvement, the candidate represents a reference block, and then the template matching cost is calculated by the distortion between the template of the current block (referred to as the current template) and the template of the reference block (referred to as the reference template). The distortion can be the sum of absolute differences (SAD) of samples in the current template and the reference template, the sum of absolute transformed differences (SATD), the mean squared error (MSE), or the sum of squared error (SSE). The BV value with the minimum template matching cost can be used as the final BV for the prediction of the current block in the IBC mode.

[0232] In some examples, multiple searched BVs may have the same minimum template matching cost value, and one of the searched BVs is appropriately selected to be the final BV for the prediction of the current block. In one example, when multiple BVs have the same minimum TM cost, a specific BV having the minimum sum of the absolute value of the horizontal component and the absolute value of the vertical component is selected as the final BV for the current block. In another example, when multiple BVs have the same minimum template matching cost value, the multiple BVs are used to construct a BV candidate list, and one index is signaled to indicate which BV should be used for the prediction of the current block in the IBC mode.

[0233] In some embodiments, template matching based BV improvement is applied to IBC merge candidates. In some examples, the template matching based BV improvement is performed on the IBC merge candidates in the candidate list in a specific order. For a specific IBC merge candidate in the candidate list, when the improved BV value of the specific IBC merge candidate is the same as that of an existing IBC merge candidate that has already been improved, the specific IBC merge candidate with the improved BV value can be pruned. In one example, the specific IBC merge candidate is restored to its original unimproved BV value, and a specific IBC merge candidate with the original unimproved BV value is added to the candidate list.

[0234] In another example, a specific IBC merge candidate with an improved BV value is considered unavailable and discarded (e.g., removed from the candidate list).

[0235] In some examples, for blocks signaled in IBC merge mode, an additional flag is signaled at the block level to indicate whether template matching based BV improvement is applied to the IBC merge candidates.

[0236] In some examples, a flag is signaled at a higher level (e.g., a level higher than the block level) to indicate whether template matching based BV improvement is applied to the IBC merge candidates of a block. For example, the flag can be signaled at the sequence level, picture level, slice level, tile / tile group level, etc.

[0237] In some embodiments, when the IntraTMP mode is valid for a block, the BV information obtained for the block by IntraTMP can be further improved. For example, the BV information is used as a starting point for template matching-based BV improvement. In one example, the step size in template matching-based BV improvement is finer than the accuracy of IntraTMP, and thus a better BV (e.g., having a lower template matching cost) can be achieved by template matching-based BV improvement.

[0238] In some embodiments, when the spatial or temporal neighboring blocks of the current block are coded using IntraTMP, the BV information of the neighboring blocks obtained from IntraTMP can be used as a BVP candidate for the prediction of the current block in the IntraBC mode.

[0239] According to one aspect of the present disclosure, for a current block coded in the IntraTMP mode, the BVD can be signaled in addition to the BV derived using template matching in the IntraTMP mode. The final BV for fetching the reference block to be the current block can be generated by the sum of the signaled BVD and the BV derived from the IntraTMP mode.

[0240] In some examples, when the spatial or temporal neighboring blocks are coded in the IntraTMP mode, the BV information of the neighboring blocks is used as a starting point for template matching of the current block that is also coded in IntraTMP.

[0241] In some examples, when the spatial or temporal neighboring blocks are coded in the IntraBC mode or the IBC mode, the BV information of the neighboring blocks is used as a starting point for template matching of the current block that is also coded by IntraTMP.

[0242] FIG. 26 shows a flowchart outlining a process (2600) according to an embodiment of the present disclosure. The process (2600) can be used in a video encoder. In various embodiments, the process (2600) is executed by processing circuitry such as in terminal devices (310), (320), (330), and (340), processing circuitry that executes the functions of video encoder (403), processing circuitry that executes the functions of video encoder (603), processing circuitry that executes the functions of video encoder (703), and the like. In some embodiments, the process (2600) is implemented in software instructions, and thus, when the processing circuitry executes the software instructions, the processing circuitry executes the process (2600). The process begins at (S2601) and proceeds to (S2610).

[0243] (S2610), it is determined whether to use template matching-based block vector refinement for coding the current block in the current picture referencing (CPR) mode.

[0244] (S2620), a first refined block vector associated with the current block is derived according to the template matching-based block vector refinement, and the first refined block vector includes a first block vector refinement offset applied to a first block vector associated with the current block.

[0245] (S2630), a signal indicating whether to use template matching-based block vector refinement for coding the current block in the CRP mode is encoded into the bitstream carrying the video including the current picture.

[0246] In some embodiments, the CPR mode is an intra block copy (IBC) mode. The block vector difference associated with the current block is encoded in the bitstream. The block vector difference has a first accuracy indicated by an adaptive motion vector resolution (AMVR) syntax, and the first block vector refinement offset is finer or equal to the first accuracy.

[0247] In some examples, the first block vector is the block vector predictor (BVP) of the current block. Template matching-based block vector refinement is applied to the BVP of the current block to generate a refined BVP using the first block vector refinement offset applied to the BVP. The refined BVP is combined with the block vector difference to determine the first refined block vector.

[0248] In some examples, a block vector predictor (BVP) is combined with the block vector difference to determine the first block vector. Template matching-based block vector refinement is applied to the first block vector of the current block to generate a first refined block vector using the first block vector refinement offset applied to the first block vector.

[0249] In some examples, the search step size in the template matching-based block vector refinement is finer than the first accuracy.

[0250] In some examples, the CPR mode is an intra block copy (IBC) mode, and the first block vector is encoded or derived at a first resolution. A template matching-based block vector refinement is applied to the first block vector using a search step size finer than the first resolution.

[0251] In some examples, the CPR mode is an intra block copy (IBC) mode. A hash-based search is performed. The hash-based search determines whether there is a match for the current template of the current block within a reconstruction area in the same picture as the current block. In response to the absence of a match in the hash-based search, a template matching-based block vector refinement is performed.

[0252] In some examples, to perform the hash-based search, a hash function is applied to a first sub-block within the reconstruction area to generate an entry in a hash map. A hash function is applied to a second sub-block within the current template to generate a hash value. Whether a match exists is determined based on the hash map and the hash value.

[0253] In some embodiments, the template matching-based block vector refinement determines a first block vector refinement offset according to a template matching cost calculated based on the distortion between the current template of the current block and the reference template of the first reference block. In some examples, a second refined block vector having the same minimum template matching cost value as the first refined block vector is detected. A first sum of a first horizontal absolute value of the horizontal component of the first refined block vector and a first vertical absolute value of the vertical component of the first refined block vector is calculated. A second sum of a second horizontal absolute value of the horizontal component of the second refined block vector and a second vertical absolute value of the vertical component of the second refined block vector is calculated. The first refined block vector is selected in response to the first sum being smaller than the second sum.

[0254] In some examples, one or more other refined block vectors having the same minimum template matching cost value as the first refined block vector are determined. A block vector candidate list is constructed to include the first refined block vector and the one or more other refined block vectors. A signal indicating the first refined block vector from the block vector candidate list is encoded into a bitstream.

[0255] In some embodiments, the CPR mode is an intra block copy (IBC) mode. Template matching-based block vector refinement is applied to IBC merge candidates in a candidate list to determine refined block vectors to be added to the candidate list. It is determined that the refinement value associated with a particular IBC merge candidate is the same value as an existing refinement value in the candidate list. The refinement value associated with the particular IBC merge candidate is pruned. In one example, the unrefined block vector of the particular IBC merge candidate is added to the candidate list. In another example, the particular IBC merge candidate is removed from the candidate list.

[0256] In some examples, a flag is encoded in the bitstream, and the flag indicates the application of template matching based block vector refinement for IBC merge candidates. The flag is at least one of a block level flag, a sequence level flag, a picture level flag, a slice level flag, a tile level flag, and / or a tile group level flag.

[0257] In some embodiments, the CPR mode is an Intra Template Matching Prediction (IntraTMP) mode. Intra template matching prediction is performed on the current block to derive a first block vector. Template matching based block vector refinement is applied to the first block vector to obtain a first refined block vector.

[0258] In some embodiments, the CPR mode is an Intra Block Copy (IBC) mode. In the Intra Template Matching Prediction (IntraTMP) mode, an adjacent block of the current block is determined. The adjacent block is at least one of a spatial adjacent or a temporal adjacent of the current block. The block vector obtained from the adjacent block is used as a Block Vector Predictor (BVP) candidate for the current block.

[0259] Next, the process proceeds to (S2699) and ends.

[0260] The process (2600) can be suitably adapted. The steps in the process (2600) can be changed and / or omitted. Further steps can be added. Any suitable order of implementation can be used.

[0261] FIG. 27 shows a flowchart outlining a process (2700) according to an embodiment of the present disclosure. The process (2700) can be used in a video decoder. In various embodiments, the process (2700) is executed by a processing circuit such as the processing circuits within terminal devices (310), (320), (330), and (340), the processing circuit that executes the functions of video decoder (410), the processing circuit that executes the functions of video decoder (510), and the like. In some embodiments, the process (2700) is implemented by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (2700). The process starts at (S2701) and proceeds to (S2710).

[0262] (S2710), a signal indicating the application of template matching-based block vector refinement for coding the current block in the current picture referencing (CPR) mode is extracted from the coding video bitstream. In some examples, the coding video bitstream is received, the coding video bitstream includes a current picture, and the current picture includes the current block. The current block is determined in the CPR mode based on syntax elements within the coding video bitstream.

[0263] (S2720), a first refined block vector associated with the current block is derived according to the template matching-based block vector refinement. The first refined block vector includes a first block vector refinement offset applied to a first block vector associated with the current block.

[0264] (S2730), the current block is reconstructed according to a first reference block within the same picture as the current block, and the first reference block is indicated by the first refined block vector.

[0265] In some embodiments, the CPR mode is an intra block copy (IBC) mode. A block vector difference associated with a current block having a first accuracy indicated by an adaptive motion vector resolution (AMVR) syntax is decoded from a coded video bitstream. A first block vector refinement offset is finer than or equal to the first accuracy.

[0266] In some examples, the first block vector is a block vector predictor (BVP) of the current block. A template matching-based block vector refinement is applied to the BVP of the current block to generate a refined BVP using the first block vector refinement offset applied to the BVP. The refined BVP is combined with the block vector difference to determine a first refined block vector.

[0267] In some examples, a block vector predictor (BVP) is combined with the block vector difference to determine a first block vector. A template matching-based block vector refinement is applied to the first block vector of the current block to generate a first refined block vector using the first block vector refinement offset applied to the first block vector.

[0268] In some examples, a search step size in the template matching-based block vector refinement is finer than the first accuracy.

[0269] In some embodiments, the CPR mode is an intra block copy (IBC) mode. The first block vector is decoded or derived (e.g., merge mode) from the coded video bitstream and has a first resolution. A template matching-based block vector refinement is applied to the first block vector using a search step size finer than the first resolution.

[0270] In some embodiments, the CPR mode is an intra block copy (IBC) mode. A hash-based search is performed. The hash-based search determines whether there is a match for the current template of the current block within a reconstructed region in the same picture as the current block. In response to the absence of a match in the hash-based search, a template matching-based block vector refinement is performed.

[0271] In some examples, to perform the hash-based search, a hash function is applied to a first sub-block within the reconstructed region to generate an entry in a hash map. A hash function is applied to a second sub-block within the current template to generate a hash value. Whether a match exists is determined based on the hash map and the hash value.

[0272] In some embodiments, the template matching-based block vector refinement determines a first block vector refinement offset according to a template matching cost calculated based on the distortion between the current template of the current block and the reference template of the first reference block. In some examples, a second refined block vector having the same minimum template matching cost value as the first refined block vector is detected. A first sum of a first horizontal absolute value of the horizontal component of the first refined block vector and a first vertical absolute value of the vertical component of the first refined block vector is calculated. A second sum of a second horizontal absolute value of the horizontal component of the second refined block vector and a second vertical absolute value of the vertical component of the second refined block vector is calculated. The first refined block vector is selected according to the first sum being smaller than the second sum.

[0273] In some examples, one or more other refined block vectors having the same minimum template matching cost value as the first refined block vector are determined. A block vector candidate list is constructed to include the first refined block vector and the one or more other refined block vectors. A signal indicating the first refined block vector from the block vector candidate list is decoded from the coding bitstream.

[0274] In some embodiments, the CPR mode is an intra block copy (IBC) mode. Template matching-based block vector refinement is applied to IBC merge candidates in the candidate list to determine a refined block vector to be added to the candidate list. It is determined that an improvement value associated with a particular IBC merge candidate is the same value as an existing improvement value in the candidate list. The improvement value associated with the particular IBC merge candidate is pruned. In one example, the unrefined block vector of the particular IBC merge candidate is added to the candidate list. In another example, the particular IBC merge candidate is removed from the candidate list.

[0275] In some examples, a flag indicating the application of template matching-based block vector refinement for IBC merge candidates is decoded from the coded video bitstream. The flag is at least one of a block level flag, a sequence level flag, a picture level flag, a slice level flag, a tile level flag, and / or a tile group level flag.

[0276] In some examples, the CPR mode is an Intra Template Matching Prediction (IntraTMP) mode. Intra template matching prediction is applied to the current block to derive a first block vector. Template matching-based block vector refinement is applied to the first block vector to obtain a first refined block vector.

[0277] In some examples, the CPR mode is an Intra Block Copy (IBC) mode. In the Intra Template Matching Prediction (IntraTMP) mode, adjacent blocks of the current block are determined, and the adjacent blocks are at least one of the spatial adjacency and the temporal adjacency of the current block. Then, the block vector obtained from the adjacent blocks is used as a Block Vector Predictor (BVP) candidate for the current block.

[0278] Then, the process proceeds to (S2799) and ends.

[0279] The process (2700) can be appropriately adapted. The steps in the process (2700) can be changed and / or omitted. Further steps can be added. Any appropriate order of implementation can be used.

[0280] FIG. 28 shows a flowchart outlining a process (2800) according to an embodiment of the present disclosure. The process (2800) can be used in a video encoder. In various embodiments, the process (2800) is executed by a processing circuit such as the processing circuits in the terminal devices (310), (320), (330), and (340), the processing circuit that executes the functions of the video encoder (403), the processing circuit that executes the functions of the video encoder (603), the processing circuit that executes the functions of the video encoder (703), and the like. In some embodiments, the process (2800) is implemented by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (2800). The process starts at (S2801) and proceeds to (S2810).

[0281] (S2810), a first block vector associated with the current block is derived in an IntraTMP (intra template matching prediction) mode.

[0282] (S2820), the block vector difference associated with the current block is encoded into the bitstream carrying the video. The block vector difference is the difference between the final block vector and the first block vector, and the final block vector indicates a reference block in the same picture as the current block for prediction of the current block.

[0283] In some examples, the neighboring blocks of the current block are determined in an IntraTMP (intra template matching prediction) mode. The neighboring blocks are at least one of the spatial neighbors or the temporal neighbors of the current block. Then, the block vector obtained from the neighboring blocks is used as the starting point of the intra template matching prediction to determine the first block vector.

[0284] In some examples, the neighboring blocks of the current block are determined in an intra-block copy mode, and the neighboring blocks are at least one of the spatial or temporal neighbors of the current block. Then, the block vector obtained from the neighboring blocks is used as a starting point for intra-template matching prediction to determine a first block vector.

[0285] Then, the process proceeds to (S2899) and ends.

[0286] The process (2800) can be suitably adapted. The steps in the process (2800) can be changed and / or omitted. Further steps can be added. Any suitable order of implementation can be used.

[0287] FIG. 29 shows a flowchart outlining a process (2900) according to an embodiment of the present disclosure. The process (2900) can be used in a video decoder. In various embodiments, the process (2900) is executed by a processing circuit such as a processing circuit in terminal devices (310), (320), (330), and (340), a processing circuit that executes the functions of video decoder (410), a processing circuit that executes the functions of video decoder (510), etc. In some embodiments, the process (2900) is implemented by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (2900). The process starts at (S2901) and proceeds to (S2910).

[0288] (In S2910), a first block vector related to the current block is derived in an intra-template matching prediction (IntraTMP) mode. In some examples, a coded video bitstream is received, the coded video bitstream includes a current picture, and the current picture includes the current block. The current block is determined in the IntraTMP mode based on syntax elements in the coded video bitstream.

[0289] (S2920) In this step, the block vector difference related to the current block is decoded from the coded video bitstream.

[0290] (S2930) In this step, the final block vector is determined according to the first block vector and the block vector difference. In one example, the first block vector and the block vector difference are added to obtain the final block vector.

[0291] (S2940) In this step, the current block is reconstructed according to the first reference block within the same picture as the current block. The first reference block is indicated by the final block vector.

[0292] In some examples, the adjacent blocks of the current block are determined in the Intra Template Matching Prediction (IntraTMP) mode. The adjacent blocks are at least one of the spatial adjacent or temporal adjacent of the current block. The block vector obtained from the adjacent blocks is used as the starting point of the Intra Template Matching Prediction to determine the first block vector.

[0293] In some examples, the adjacent blocks of the current block are determined in the Intra Block Copy mode. The adjacent blocks are at least one of the spatial adjacent or temporal adjacent of the current block. The block vector obtained from the adjacent blocks is used as the starting point of the Intra Template Matching Prediction to determine the first block vector.

[0294] Next, the process proceeds to (S2999) and ends.

[0295] The process (2900) can be appropriately adapted. The steps in the process (2900) can be changed and / or omitted. Further steps can be added. Any appropriate execution order can be used.

[0296] The above technology can be implemented as computer software using computer-readable instructions and can be physically stored in one or more computer-readable media. For example, FIG. 30 shows a computer system (3000) suitable for implementing certain embodiments of the disclosed subject matter.

[0297] The computer software can be coded using any suitable machine code or computer language and be the subject of assembly, compilation, linking, or similar mechanisms to create code containing instructions executable directly or through interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), and the like.

[0298] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game devices, Internet of Things devices, and the like.

[0299] The components shown in FIG. 30 for the computer system (3000) are illustrative in nature and are not intended to suggest any limitations on the use or functionality of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be construed as having any dependencies or requirements with respect to any one or combination of the components shown in the exemplary embodiments of the computer system (3000).

[0300] The computer system (3000) can include a specific human interface input device. Such a human interface input device can respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement of a data glove), voice input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). Also, the human interface device can be used to capture specific media that is not necessarily directly related to conscious human input, such as voice (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), video (e.g., 2D video, 3D video including stereoscopic video).

[0301] The input human interface device may include one or more of a keyboard (3001), a mouse (3002), a trackpad (3003), a touch screen (3010), a data glove (not shown), a joystick (3005), a microphone (3006), a scanner (3007), a camera (3008) (only one of each is shown).

[0302] The computer system (3000) may also include certain human interface output devices. Such human interface output devices may, for example, stimulate the senses of one or more human users through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback by a touch screen (3010), a data glove (not shown), or a joystick (3005); however, there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (3009), headphones (not shown)), visual output devices (e.g., a screen (3010) including a CRT screen, an LCD screen, a plasma screen, an OLED screen; each may or may not have a touch screen input function, each may or may not have a tactile feedback function, and some of them can output higher-than-three-dimensional output through means such as two-dimensional visual output or stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and may also include a printer (not shown).

[0303] The computer system (3000) may also include an optical medium including a CD / DVD ROM / RW (3020) together with a human-accessible memory device and related media, such as a CD / DVD or similar media (3021), a thumb drive (3022), a removable hard drive or solid state drive (3023), legacy magnetic media such as tapes and floppy disks (not shown), specialized ROM / ASIC / PLD-based devices such as security dongles (not shown), etc.

[0304] One of ordinary skill in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include a transmission medium, a carrier wave, or other transient signals.

[0305] The computer system (3000) can also include an interface (3054) to one or more communication networks (3055). The network can be, for example, wireless, wired, or optical. The network can further be local, wide area, metropolitan area, in-vehicle and industrial, real-time, delay tolerant, etc. Examples of networks include Ethernet®, wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., cellular networks, cable TV, satellite TV, TV wired or wireless wide area digital networks including terrestrial broadcast TV, in-vehicle and industrial including CANBus, etc. Certain networks typically require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (3049) (e.g., a USB port of the computer system (3000), etc.). Others are typically integrated into the core of the computer system (3000) by attachment to a system bus as described later (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (3000) can communicate with other entities. Such communication can be unidirectional, receive-only (e.g., broadcast TV), dedicated unidirectional transmission (e.g., CANbus to a specific CANbus device), or bidirectional to other computer systems using, for example, a local or wide area digital network. For each of the networks and network interfaces as described above, specific protocols and protocol stacks can be used.

[0306] The aforementioned human interface device, human-accessible memory device, and network interface can be attached to the core (3040) of the computer system (3000).

[0307] The core (3040) can include one or more central processing units (CPUs) (3041), graphics processing units (GPUs) (3042), specialized programmable processing devices in the form of field programmable gate arrays (FPGAs) (3043), hardware accelerators (3044) for specific tasks, graphics adapters (3050), etc. These devices can be connected through a system bus (3048) together with internal mass storage devices such as read-only memory (ROM) (3045), random access memory (3046), internal hard drives that are not user-accessible, solid state drives (SSDs), etc. (3047). In some computer systems, the system bus (3048) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (3048) or through a peripheral bus (3049). In one example, a screen (3010) can be connected to the graphics adapter (3050). Architectures for peripheral buses include PCI, USB, etc.

[0308] The CPU (3041), GPU (3042), FPGA (3043), and accelerator (3044) can execute specific instructions that can, in combination, constitute the above-mentioned computer code. That computer code can be stored in the ROM (3045) or RAM (3046). Temporary data can also be stored in the RAM (3046), while persistent data can be stored, for example, in the internal mass storage device (3047). By using cache memory that can be closely associated with one or more CPUs (3041), GPUs (3042), mass storage devices (3047), ROM (3045), RAM (3046), etc., fast storage and retrieval to any of the memory devices can be enabled.

[0309] A computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and the computer code may be specially designed and constructed for the purposes of this disclosure, or they may be of the kind well-known and available to those having skill in the art of computer software.

[0310] By way of example and not limitation, a computer system having an architecture (3000), specifically a core (3040), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied on one or more tangible computer-readable media. Such computer-readable media can be media related to user-accessible mass storage as introduced above and specific storage of the core (3040) of a non-transitory nature such as a mass storage device (3047) inside the core or ROM (3045). The software implementing various embodiments of the present disclosure can be stored on such devices and executed by the core (3040). The computer-readable media can include one or more memory devices or chips according to specific needs. The software defines data structures stored in RAM (3046) and modifies such data structures according to processes defined by the software, causing the core (3040) and specifically the processors (including CPU, GPU, FPGA, etc.) therein to execute specific processes or specific parts described herein. Additionally or alternatively, the computer system can provide functionality as a result of logic wired within a circuit (e.g., an accelerator (3044)) or otherwise embodied, which can operate instead of or in conjunction with software for executing specific processes or specific parts of specific processes described herein. References to software include logic and, where appropriate, vice versa. References to computer-readable media can, where appropriate, include circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits embodying logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0311] Appendix A: Acronyms JEM: joint exploration model VVC: versatile video coding BMS: benchmark set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI: Video Usability Information GOP: Group of Pictures TU: Transform Unit PU: Prediction Unit CTU: Coding Tree Unit CTB: Coding Tree Block PB: Prediction Block HRD: Hypothetical Reference Decoder SNR: Signal Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Areas SSD: solid-state drive IC: Integrated Circuit CU: Coding Unit

[0312] Although the present disclosure has described several exemplary embodiments, there are changes, substitutions, and various alternative equivalents that fall within the scope of the present disclosure. Thus, it will be understood by those skilled in the art that many systems and methods, although not explicitly shown or described herein, embody the principles of the present disclosure and are thus within the spirit and scope of the present disclosure, can be devised.

Claims

1. A method for video processing in a decoder, comprising: receiving a coded video bitstream including a current picture, wherein the current picture includes a current block; determining that the current block is coded in a current picture reference (CPR) mode based on a syntax element in the coded video bitstream; extracting, from the coded video bitstream, a signal indicating whether a template matching-based block vector refinement is applied to the current block in the CPR mode; deriving a first refined block vector related to the current block according to the template matching-based block vector refinement, wherein the first refined block vector includes a first block vector refinement offset applied to a first block vector related to the current block; reconstructing the current block according to a first reference block in the same picture as the current block, wherein the first reference block is indicated by the first refined block vector; and a method comprising the steps of.

2. The CPR mode is an intra block copy (IBC) mode, and the method further comprises: decoding, from the coded video bitstream, a block vector difference related to the current block with a first accuracy indicated by an adaptive motion vector resolution (AMVR) syntax, wherein the first block vector refinement offset is finer than or equal to the first accuracy. The method according to claim 1.

3. The first block vector is a block vector predictor (BVP) of the current block, and the step of deriving the first refined block vector comprises: applying the template matching-based block vector refinement to the BVP of the current block to generate a refined BVP using the first block vector refinement offset applied to the BVP; combining the refined BVP with the block vector difference to determine the first refined block vector; and the method according to claim 2.

4. The step of deriving the first improved block vector comprises: combining a block vector predictor (BVP) with the block vector difference to determine the first block vector; and applying the template matching-based block vector improvement to the first block vector of the current block, and generating the first improved block vector using the first block vector improvement offset applied to the first block vector. The method according to claim 2, further comprising. **Claim 5** The method according to claim 2, wherein a search step size in the template matching-based block vector improvement is smaller than the first accuracy. **Claim 6** The CPR mode is an intra-block copy (IBC) mode, and the step of deriving the first improved block vector comprises: determining the first block vector having a first resolution; and applying the template matching-based block vector improvement to the first block vector using a search step size finer than the first resolution. The method according to claim 1, comprising. **Claim 7** The CPR mode is an intra-block copy (IBC) mode, and the step of deriving the first improved block vector comprises: performing a hash-based search to determine whether there is a match for the current template of the current block within a reconstruction area in the same picture as the current block; and performing the template matching-based block vector improvement in response to the absence of a match in the hash-based search. The method according to claim 1, comprising. **Claim 8** The step of performing the hash-based search comprises: applying a hash function to a first sub-block in the reconstruction area to generate an entry in a hash map; applying the hash function to a second sub-block in the current template to generate a hash value; and determining whether there is a match based on the hash map and the hash value. The method according to claim 7, further comprising. **Claim 9** The method according to claim 1, wherein the template matching-based block vector improvement determines the first block vector improvement offset according to a template matching cost calculated by distortion between the current template of the current block and the reference template of the first reference block.

10. Determining that a second improved block vector has the same minimum template matching cost value as the first improved block vector; Calculating a first sum of a first horizontal absolute value of a horizontal component of the first improved block vector and a first vertical absolute value of a vertical component of the first improved block vector; Calculating a second sum of a second horizontal absolute value of a horizontal component of the second improved block vector and a second vertical absolute value of a vertical component of the second improved block vector; Selecting the first improved block vector in response to the first sum being smaller than the second sum; The method according to claim 9, further comprising:

11. Determining that one or more other improved block vectors have the same minimum template matching cost value as the first improved block vector; Constructing a block vector candidate list including the first improved block vector and the one or more other improved block vectors; Decoding, from the coded video bitstream, a signal indicating the first improved block vector from the block vector candidate list; The method according to claim 9, further comprising:

12. The CPR mode is an intra-block copy (IBC) mode, and the method comprises: Applying the template matching-based block vector improvement to IBC merge candidates in a candidate list to determine an improved block vector to be added to the candidate list; Determining that an improvement value associated with a specific IBC merge candidate is the same value as an existing improvement value in the candidate list; Pruning the improvement value associated with the specific IBC merge candidate; The method according to claim 1, further comprising:

13. Adding an unimproved block vector of the specific IBC merge candidate to the candidate list; Deleting the specific IBC merge candidate from the candidate list; The method according to claim 12, further comprising at least one of them.

14. Decoding a flag indicating application of the template matching based block vector improvement to the IBC merge candidate, wherein the flag is at least one of a block level flag, a sequence level flag, a picture level flag, a slice level flag, a tile level flag, and / or a tile group level flag, the method according to claim 12, further comprising the step.

15. The CPR mode is an Intra Template Matching Prediction (IntraTMP) mode, and the method Performing an Intra Template Matching Prediction on the current block to derive the first block vector; Performing the template matching based block vector improvement on the first block vector to obtain the first improved block vector; The method according to claim 1, further comprising.

16. The CPR mode is an Intra Block Copy (IBC) mode, and the method Determining that an adjacent block of the current block is in an Intra Template Matching Prediction (IntraTMP) mode, wherein the adjacent block is at least one of a spatial adjacent or a temporal adjacent of the current block; Using the block vector obtained from the adjacent block as a Block Vector Predictor (BVP) candidate of the current block; The method according to claim 1, further comprising.

17. A method of video processing in a decoder, Receiving a coded video bitstream including a current picture, wherein the current picture includes a current block; Determining that the current block is coded in an Intra Template Matching Prediction (IntraTMP) mode based on a syntax element in the coded video bitstream; Deriving a first block vector associated with the current block in the IntraTMP mode; Decoding a block vector difference associated with the current block from the coded video bitstream; Determining a final block vector according to the first block vector and the block vector difference; Reconstructing the current block according to a first reference block within the same picture as the current block, wherein the first reference block is indicated by the final block vector; A method including the above steps.

18. Determining that an adjacent block of the current block is in an Intra Template Matching Prediction (IntraTMP) mode, wherein the adjacent block is at least one of spatial adjacency or temporal adjacency of the current block; Determining the first block vector by using a block vector obtained from the adjacent block as a starting point for Intra Template Matching Prediction; The method according to claim 17, further including the above steps.

19. Determining that an adjacent block of the current block is in an Intra Block Copy mode, wherein the adjacent block is at least one of spatial adjacency or temporal adjacency of the current block; Determining the first block vector by using a block vector obtained from the adjacent block as a starting point for Intra Template Matching Prediction; The method according to claim 17, further including the above steps.

20. An apparatus for video decoding including a processing circuit, wherein the processing circuit is configured to execute the method according to any one of claims 1 to 19.

21. A program for causing a computer to execute the method according to any one of claims 1 to 19.

22. A method for video processing in an encoder, Determining the use of template matching based block vector refinement for coding a current block in a Current Picture Reference (CPR) mode; Deriving a first refined block vector related to the current block according to the template matching based block vector refinement, wherein the first refined block vector includes a first block vector refinement offset applied to a first block vector related to the current block; Encoding a signal indicating use of the template matching based block vector improvement for coding the current block in the CRP mode into a bitstream carrying a video including the current picture A method comprising.

Images