Template matching for IBC merge candidates

Abstract

Aspects of the present disclosure provide methods and apparatuses for video encoding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit determines an initial block vector for predicting the current block in the current coding tree unit (CTU) in response to the current block being predicted in intra-block copy (IBC) mode. The processing circuit performs template matching based on the initial block vector to determine the current block as a refined block vector pointing to a reference block within the same picture; and reconstructs the current block based on the reference block.

Description

Technology Field

[0001] Integration by reference

[0002] The present application claims the benefit of priority to U.S. Provisional Application No. 63 / 239,767, "TEMPLATE MATCHING ON IBC MERGE CANDIDATES," filed September 1, 2021, and the benefit of priority to U.S. Patent Application No. 17 / 898,454, "TEMPLATE MATCHING ON IBC MERGE CANDIDATES," filed August 29, 2022. The entire disclosures of the prior applications are incorporated herein by reference in their entirety.

[0003] Technology field

[0004] The present disclosure describes embodiments generally related to video coding. Background Technology

[0005] The background description provided in this specification is intended to provide the context of the present disclosure in general. The work of the presently named inventors to the extent that work is described in this background section, as well as modes of description that may not otherwise qualify as prior art at the time of filing, are not recognized as prior art to the present disclosure, either explicitly or implicitly.

[0006] Uncompressed digital video may contain a series of pictures, each having, for example, 1920x1080 spatial dimensions of luminance samples and associated chrominance samples. The series of pictures may have, for example, 60 pictures per second or a fixed or variable picture rate of 60Hz (informally also known as the frame rate). Uncompressed video has specific bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920x1080 luminance sample resolution at a 60Hz frame rate) requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 GBytes of storage space.

[0007] One objective of video coding and decoding may be to reduce the redundancy of the input video signal through compression. Compression can help reduce the aforementioned bandwidth and / or storage requirements by more than double orders of magnitude in some cases. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application. For video, lossy compression is widely used. The amount of acceptable distortion depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. Achievable compression ratios may reflect that higher acceptable distortion can yield higher compression ratios.

[0008] Video encoders and decoders can utilize several broad categories of techniques, including, for example, motion compensation, transformation, quantization, and entropy coding.

[0009] Video codec technologies may include techniques known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples from previously reconstructed reference pictures or other data. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture may be an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as a first picture or a still image within a coded video bitstream and video session. Samples in an intra-block may be exposed to transformation, and transformation coefficients may be quantized before entropy coding. Intra-prediction may be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the post-transform DC value and the smaller the AC coefficients, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0010] For example, traditional intra-coding, such as that known from MPEG-2 generative coding techniques, does not use intra-prediction. However, some newer video compression techniques include methods that attempt to use, for example, surrounding sample data and / or metadata acquired during the encoding and / or decoding of spatially neighboring and preceding data blocks in the decoding order. These techniques are hereinafter referred to as "intra-prediction" techniques. It should be noted that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, rather than from reference pictures.

[0011] Many different forms of intra-prediction may exist. When more than one of these techniques can be used in a given video coding technique, the technique in use may be coded in an intra-prediction mode. In certain cases, modes may have submodes and / or parameters, which can be coded individually or included in a mode codeword. Which codeword to use for a given combination of mode, submode, and / or parameter can affect the coding efficiency gain through intra-prediction, and the same may apply to the entropy coding technique used to convert the codewords into a bitstream.

[0012] Intra prediction of specific modes was introduced in H.264, improved in H.265, and further enhanced in newer coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Predictor blocks can be formed using neighbor sample values ​​belonging to already available samples. The sample values ​​of neighbor samples are copied into the predictor block according to direction. A reference to the direction of use can be coded within the bitstream or predicted itself.

[0013] Referring to FIG. 1a, a subset of nine known predictor directions from the 33 possible predictor directions of H.265 (corresponding to 33 angle modes out of 35 intra modes) is shown in the lower right. The point where the arrows converge (101) represents the sample being predicted. The arrows represent the direction in which the sample is being predicted. For example, arrow (102) indicates that the sample (101) is predicted to the upper right at an angle of 45 degrees from the horizontal from the sample or samples. Likewise, arrow (103) indicates that the sample (101) is predicted to the lower left of the sample (101) at an angle of 22.5 degrees from the horizontal from the sample or samples.

[0014] Referring still to FIG. 1a, a square block (104) of 4x4 samples is shown in the upper left (indicated by a bold dashed line). The square block (104) contains 16 samples, each labeled "S," its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Likewise, sample S44 is the fourth sample of the block (104) in both the Y and X dimensions. Since the block is 4x4 sample size, S44 is located in the lower right. Further reference samples following a similar numbering scheme are shown. The reference samples are labeled R, its Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, the predicted samples are adjacent to the block being reconstructed; Therefore, there is no need to use negative values.

[0015] Intra-picture prediction can be performed by copying reference sample values ​​from appropriate neighbor samples by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating a prediction direction corresponding to the arrow (102) for this block, that is, samples are predicted from the prediction sample or samples to the upper right at a 45-degree angle from the horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Next, sample S44 is predicted from reference sample R08.

[0016] In certain cases, particularly when directions cannot be divided equally into 45 degrees, multiple reference sample values ​​may be combined, for example, through interpolation, to calculate reference samples.

[0017] As video coding technology has advanced, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and JEM / VVC / BMS at the time of release could support up to 65 directions. Experiments have been conducted to identify the most likely directions, and specific techniques of entropy coding are used to represent these possible directions with a small number of bits while allowing for specific penalties for less likely directions. Additionally, the directions themselves can sometimes be predicted from neighboring directions used in already decoded neighboring blocks.

[0018] FIG. 1b shows a schematic diagram (110) illustrating 65 intra-predicted directions according to JEM to illustrate an increasing number of predicted directions over time.

[0019] The mapping of intra-predicted direction bits within a coded video bitstream representing direction can vary depending on the video coding technique; for example, it can range from simple direct mappings of predicted directions to complex adaptive schemes involving intra-predicted modes, codewords, and most likely modes, as well as similar techniques. However, in all cases, there may exist specific directions that are statistically less likely to occur in video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a well-functioning video coding technique, less likely directions will be represented by a greater number of bits than more likely directions.

[0020] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Motion compensation may be a lossy compression technique and may relate to techniques used for prediction of a newly reconstructed picture or part of a picture after a block of sample data from a previously reconstructed picture or part thereof (reference picture) has been spatially shifted in the direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be identical to the picture currently being reconstructed. MVs may have two or three dimensions of X and Y, where the third dimension is an indication of the reference picture in use (the latter may indirectly be a time dimension).

[0021] In some video compression techniques, applicable MVs for specific regions of sample data can be predicted from other MVs, for example, from those related to other regions of sample data that are spatially adjacent to the region being reconstructed and precede that MV in the decoding order. Doing so can significantly reduce the amount of data required to code the MVs, thereby eliminating redundancy and increasing compression. MV prediction can work effectively because, for example when coding an input video signal derived from a camera (known as natural video), regions larger than the area where a single MV can be applied move in similar directions, and therefore, in some cases, there is a statistical possibility that they can be predicted using similar motion vectors derived from the MVs of neighboring regions. This results in the MV found for a given region becoming similar or identical to the MV predicted from surrounding MVs, ultimately leading to a result where it can be represented with fewer bits than when directly coding the MV after entropy coding. In some cases, MV prediction can be an example of lossless compression of signals (i.e., MVs) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself may be lost, for example, due to rounding errors when calculating the predictor from several surrounding MVs.

[0022] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms proposed by H.265, a technique referred to hereafter as "spatial merge" is described here.

[0023] Referring to FIG. 2, the current block (201) contains samples found to be predictable from a previous block of the same size that has been spatially shifted by the encoder during the motion search process. Instead of directly coding the MV, the MV may be derived from metadata associated with one or more reference pictures, for example from the most recent reference picture (in the decoding order), using an MV associated with any one of five neighboring samples denoted as A0, A1 and B0, B1, B2 (202 to 206, respectively). In H.265, the MV prediction may use predictors of the same reference picture being used by neighboring blocks.

[0024] Aspects of the present disclosure provide methods and apparatuses for video encoding and / or decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit determines an initial block vector for predicting the current block in the current coding tree unit (CTU) in response to the current block being predicted in intra-block copy (IBC) mode. The processing circuit performs template matching based on the initial block vector to determine a refined block vector pointing to a reference block in the picture as the current block, and reconstructs the current block based on the reference block. In some examples, the processing circuit determines the initial block vector based on a merge index contained in the coded video bitstream. The merge index represents a block vector candidate in an IBC merge candidate list containing multiple IBC candidates in IBC merge mode.

[0025] In the example, the processing circuit parses a first flag indicating an IBC merge mode from the coded video bitstream and parses a second flag indicating whether template matching is applied to a block vector candidate among a plurality of IBC candidates in the IBC merge mode indicated by a merge index from the coded video bitstream.

[0026] In another example, the processing circuit parses a flag indicating the IBC merge mode from the coded video bitstream, and template matching is applied to block vector candidates for each block within the CTU predicted in the IBC merge mode.

[0027] In some examples, the processing circuit constructs an IBC merge candidate list including at least a first block vector candidate pointing to a first position outside the IBC reference area.

[0028] In the example, the processing circuit determines the position closest to the first position within the IBC reference area in response to the merge index indicating the first block vector candidate, and determines an initial block vector to point to the closest position.

[0029] In the example, to construct an IBC merge candidate list, the processing circuit inserts a first block vector candidate into the IBC merge candidate list in response to the determination that the template matching search area of ​​the first block vector candidate overlaps at least partially with the IBC reference area.

[0030] In some examples, the IBC reference area includes a reconfigured portion of the current CTU and an area of ​​the left CTU, which are cached in a memory space sized to store the CTU.

[0031] In some examples, the processing circuit excludes parts of the template matching search area outside the IBC reference area from template matching.

[0032] Aspects of the present disclosure also provide a non-transient computer-readable medium storing instructions that cause a computer to perform a method for video decoding when executed by a computer for video decoding. Brief explanation of the drawing

[0033] Additional features, characteristics, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. Figure 1a is a schematic illustration of an exemplary subset of intra-prediction modes. Fig. 1b is a city of exemplary intra-predicted directions. Figure 2 is a schematic city of the current block and its surrounding space merger candidates in one example. FIG. 3 is a schematic illustration of a simplified block diagram of a communication system (300) according to an embodiment. FIG. 4 is a schematic illustration of a simplified block diagram of a communication system (400) according to an embodiment. FIG. 5 is a schematic illustration of a simplified block diagram of a decoder according to an embodiment. FIG. 6 is a schematic illustration of a simplified block diagram of an encoder according to an embodiment. FIG. 7 shows a block diagram of an encoder according to another embodiment. FIG. 8 shows a block diagram of a decoder according to another embodiment. FIG. 9 shows an example of intra-block copying according to an embodiment of the present disclosure. FIG. 10 shows an example of intra-block copying according to an embodiment of the present disclosure. FIG. 11 shows an example of intra-block copying according to an embodiment of the present disclosure. FIGS. 12a to 12d illustrate examples of intra-block copying according to embodiments of the present disclosure. FIG. 13 illustrates an example of template matching search in some embodiments. Figure 14 shows a table of search patterns for adaptive motion vector resolution (AMVR) and merge mode in some examples. FIG. 15 shows examples of pseudocodes for template matching in some embodiments. FIG. 16 shows an example of prediction in an intra-block copy mode according to an embodiment of the present disclosure. FIG. 17 shows a flowchart schematically illustrating a process according to some embodiments of the present disclosure. FIG. 18 shows a flowchart outlining another process according to some embodiments of the present disclosure. FIG. 19 is a schematic illustration of a computer system according to an embodiment. Specific details for implementing the invention

[0034] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices capable of communicating with each other, for example, through a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected through the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) perform unidirectional transmission of data. For example, a terminal device (310) may encode video data (for example, a stream of video pictures captured by the terminal device (310)) to be transmitted to another terminal device (320) through the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) receives coded video data from the network (350), decodes the coded video data to recover video pictures, and can display video pictures according to the recovered video data. Unidirectional data transmission may be common in media serving applications and similar applications.

[0035] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of coded video data that may occur, for example, during video conferencing. For bidirectional data transmission, in the example, each terminal device among the terminal devices (330) and (340) may code video data (e.g., a stream of video pictures captured by the terminal device) to transmit to another terminal device among the terminal devices (330) and (340) via the network (350). Each terminal device among the terminal devices (330) and (340) may also receive the coded video data transmitted by the other terminal device among the terminal devices (330) and (340), decode the coded video data to recover the video pictures, and display the video pictures on an accessible display device according to the recovered video data.

[0036] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) may be illustrated as servers, personal computers, and smartphones, but the principles of the present disclosure are not limited thereto. Embodiments of the present disclosure find applications in laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (350) represents any number of networks that transmit coded video data between terminal devices (310, (320), (330), and (340), including, for example, wired and / or wireless communication networks. The communication network (350) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may not be important to the operation of the present disclosure unless described herein.

[0037] FIG. 4 illustrates the arrangement of a video encoder and a video decoder in a streaming environment as an example of an application to the disclosed subject. The disclosed subject may be equally applicable to, for example, other video-enabled applications including video conferencing and digital TV, and to storing compressed video on digital media including CDs, DVDs, memory sticks and similar devices.

[0038] A streaming system may include a capture subsystem (413) which may include, for example, a digital camera, a video source (401) that generates a stream (402) of uncompressed video pictures. In the example, the stream (402) of video pictures includes samples captured by the digital camera. The stream (402) of video pictures, depicted in bold lines to emphasize the high data volume compared to encoded video data (404) (or coded video bitstreams), may be processed by an electronic device (420) comprising a video encoder (403) coupled to the video source (401). The video encoder (403) may enable or implement the embodiments of the disclosed subject matter as described in more detail below, including hardware, software, or a combination thereof. Encoded video data (404) (or encoded video bitstream (404)), depicted as a thin line to emphasize a lower data volume compared to a stream (402) of video pictures, may be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) of FIG. 4, may access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include a video decoder (410) within, for example, an electronic device (430). The video decoder (410) decodes an incoming copy (407) of the encoded video data and generates an outgoing stream (411) of video pictures that can be rendered on a display (412) (e.g., a display screen) or another rendering device (not shown). In some streaming systems, encoded video data (404), (407) and (409) (e.g., video bitstreams) may be encoded according to specific video coding / compression standards.Examples of these standards include ITU-T Recommendation H.265. In the example, the video coding standard under development is informally known as Multipurpose Video Coding (VVC). The disclosed topics may be used in the context of VVC.

[0039] It should be noted that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0040] FIG. 5 shows a block diagram of a video decoder (510) according to an embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) in the example of FIG. 4.

[0041] A receiver (531) may receive one or more coded video sequences to be decoded by a video decoder (510); in the same or other embodiments, may receive one coded video sequence at a time, wherein the decoding of each coded video sequence is independent of other coded video sequences. A coded video sequence may be received from a channel (501) which may be a hardware / software link to a storage device storing encoded video data. The receiver (531) may receive the encoded video data along with other data, e.g., encoded audio data and / or auxiliary data streams, which may be forwarded to their respective use entities (not shown). The receiver (531) may separate the coded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In other applications, it may be outside the video decoder (510) (not shown). In other applications, for example, the buffer memory (not shown) may be outside the video decoder (510) to prevent network jitter, and for example, another buffer memory (515) may be additionally located inside the video decoder (510) to handle playback timing. When the receiver (531) receives data from a storage / forward device or an isosynchronous network with sufficient bandwidth and controllability, the buffer memory (515) may not be needed or may be small.For use in best-effort packet networks such as the Internet, a buffer memory (515) may be required, may be relatively large and advantageously may have an adaptive size, and may be implemented at least partially in similar elements (not shown) outside of the operating system or video decoder (510).

[0042] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from a coded video sequence. Categories of these symbols include information used to manage the operation of the video decoder (510), and potentially information for controlling a rendering device, such as a rendering device (512) (e.g., a display screen) that can be coupled to the electronic device (530) as illustrated in FIG. 5, rather than being an integrated part of the electronic device (530). The control information for the rendering device(s) may be in the form of Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not illustrated). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (520) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to a group. The subgroups may include Groups of Pictures (GOP), pictures, tiles, slices, macroblocks, Coding Units (CU), blocks, Transform Units (TU), Prediction Units (PU), etc. The parser (520) may also extract coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0043] The parser (520) can generate symbols (521) by performing an entropy decoding / parsing operation on a video sequence received from a buffer memory (515).

[0044] The reconstruction of the symbols (521) may involve multiple different units depending on the type of the coded video picture or its parts (e.g., inter- and intra-picture, inter- and intra-block), and other factors. Which units and how they may be involved may be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. The flow of this subgroup control information between the parser (520) and the multiple units below is not illustrated for clarity.

[0045] Beyond the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with one another and be at least partially integrated with one another. However, for the sake of illustrating the subject matter disclosed, the conceptual subdivision into the functional units below is appropriate.

[0046] The first unit is a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives control information from the parser (520) as symbol(s) (521), including the quantized transform coefficients, the transform to be used, the block size, the quantization factor, the quantization scaling matrix, etc. The scaler / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0047] In some cases, the output samples of the scaler / inverse transform (551) may belong to an intra-coded block; that is, a block that does not use prediction information from previously reconstructed pictures, but may use prediction information from previously reconstructed parts of the current picture. This prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses surrounding already reconstructed information fetched from the current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information generated by the intra-prediction unit (552) on a sample-by-sample basis to the output sample information provided by the scaler / inverse transform unit (551).

[0048] In other cases, the output samples of the scaler / inverse unit (551) may be intercoded and potentially belong to a motion-compensated block. In such cases, the motion-compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (521) belonging to the block, these samples may be added to the output of the scaler / inverse unit (551) by the aggregator (555) to generate output sample information (referred to here as residual samples or residual signals). The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches prediction samples may be controlled by motion vectors available to the motion-compensated prediction unit (553), for example, in the form of symbols (521) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from reference picture memory (557) when accurate motion vectors of sub-samples are in use, motion vector prediction mechanisms, etc.

[0049] The output samples of the aggregator (555) may undergo various loop filtering techniques within the loop filter unit (556). Video compression techniques are included in the coded video sequence (also referred to as the coded video bitstream) and are controlled by parameters that make the loop filter unit (556) available as symbols (521) from the parser (520), but may also include in-loop filter techniques that can respond to meta-information obtained during the decoding of previous parts (in the decoding order) of the coded picture or coded video sequence, as well as respond to previously reconstructed and loop-filtered sample values.

[0050] The output of the loop filter unit (556) may be a sample stream that can be output to the rendering device (512) and also stored in the reference picture memory (557) for use in future inter-picture prediction.

[0051] Certain coded pictures can be used as reference pictures for future predictions once they have been fully reconstructed. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by a parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and the new current picture buffer can be reallocated before starting the reconstruction of the subsequent coded picture.

[0052] A video decoder (510) can perform decoding operations according to a predetermined video compression technique of a standard such as ITU-T Rec. H.265. In that the coded video sequence complies with both the syntax of the video compression technique or standard and the profiles documented in the video compression technique or standard, the coded video sequence may follow the syntax specified by the video compression technique or standard used. Specifically, the profile may select specific tools from all tools available in the video compression technique or standard as the only tools available for use in that profile. Additionally, for compliance, the complexity of the coded video sequence may be within the boundaries defined by the levels of the video compression technique or standard. In some cases, the levels limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the levels may be further restricted through the Hypothetical Reference Decoder (HRD) specification signaled in the coded video sequence and metadata for HRD buffer management.

[0053] In an embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or reconstruct the original video data more accurately. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0054] FIG. 6 shows a block diagram of a video encoder (603) according to an embodiment of the present disclosure. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of the video encoder (403) of the example of FIG. 4.

[0055] The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6) capable of capturing video image(s) to be encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0056] A video source (601) may provide a source video sequence to be coded by a video encoder (603) in the form of a digital video sample stream having any appropriate bit depth (e.g., 8-bit, 10-bit, 12-bit,...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any appropriate sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that impart motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sampling structure, color space, etc. being used. A person skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0057] According to an embodiment, the video encoder (603) can code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate coding rate is one of the functions of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to other functional units. The coupling is not illustrated for clarity. Parameters set by the controller (650) may include rate control-related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique, …), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions belonging to the video encoder (603) optimized for a specific system design.

[0058] In some embodiments, the video encoder (603) is configured to operate in a coding loop. For the sake of oversimplification, in the example, the coding loop may include a (local) decoder (633) embedded in the video encoder (603), and a source coder (630) that is responsible for generating symbols (e.g., a symbol stream based on the input picture to be coded and reference picture(s). (Since any compression between symbols and the coded video bitstream in the video compression techniques considered in the disclosed subject is lossless) the decoder (633) reconstructs the symbols to generate sample data in a manner similar to that which the (remote) decoder also generates. The reconstructed sample stream (sample data) is input into the reference picture memory (634). Because the decoding of the symbol stream leads to bit-exact results independently of the decoder location (local or remote), the content of the reference picture memory (634) is also bit exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" the reference picture samples exactly the same sample values ​​that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift in cases where synchronization cannot be maintained, for example, due to channel error) is also used in some related techniques.

[0059] The operation of the “local” decoder (633) may be the same as the operation of the “remote” decoder, such as the video decoder (510), which has already been described in detail above in relation to FIG. 5. However, with brief reference also to FIG. 5, since symbols are available and the encoding / decoding of symbols for a coded video sequence by the entropy coder (645) and the parser (520) may be lossless, the entropy decoding parts of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0060] An observation that can be made at this point is that any decoder technique, excluding the parsing / entropy decoding present in the decoder, must also exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject focuses on decoder operations. Descriptions of encoder techniques may be abbreviated as they are the opposite of the comprehensively described decoder techniques. Further details are required only in specific areas and are provided below.

[0061] During operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes the input picture by referencing one or more previously coded pictures from a video sequence designated as "reference pictures." In this way, the coding engine (632) codes the differences between the pixel blocks of the reference picture(s) that can be selected as predictive reference(s) for the input picture and the pixel blocks of the input picture.

[0062] The local video decoder (633) can decode the coded video data of pictures that can be designated as reference pictures based on the symbols generated by the source coder (630). The operations of the coding engine (632) may advantageously be lossy processes. When the coded video data can be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a copy of the source video sequence having some errors. The local video decoder (633) can replicate the decoding processes that can be performed by the video decoder on the reference pictures and allow the reconstructed reference pictures to be stored in the reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures having common content as reconstructed reference pictures to be acquired by the far-end video decoder (without transmission errors).

[0063] The predictor (635) can perform predictive searches for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., which can serve as appropriate predictive references for the new pictures. The predictor (635) can operate on a sample block-by-pixel block basis to find appropriate predictive references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory (634).

[0064] The controller (650) can manage the coding operations of the source coder (630), including the settings of parameters and subgroup parameters used to encode video data, for example.

[0065] The outputs of all the function units mentioned above may undergo entropy coding in an entropy coder (645). The entropy coder (645) converts the symbols generated by the various function units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0066] The transmitter (640) may buffer the coded video sequence(s) generated by the entropy coder (645) and prepare for transmission through a communication channel (660), which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter (640) may merge the coded video data from the video coder (603) with other data to be transmitted, e.g., coded audio data and / or auxiliary data streams (sources not shown).

[0067] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a specific coded picture type to each coded picture that can affect the coding techniques that can be applied to the individual pictures. For example, pictures can often be assigned as one of the following picture types:

[0068] An intra-picture (I-picture) may be one that can be coded and decoded without using any other picture within the sequence as a source for prediction. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. A person skilled in the art is aware of these variations of I-pictures, and their individual applications and features.

[0069] The prediction picture (P picture) may be coded and decoded using inter-prediction or intra-prediction, which uses at most one motion vector and reference index to predict sample values ​​of each block.

[0070] A bidirectional prediction picture (Picture B) may be coded and decoded using inter-prediction or intra-prediction, which uses up to two motion vectors and reference indices to predict sample values ​​of each block. Similarly, multi-prediction pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0071] Source pictures are generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and can be coded block by block. Blocks can be predictively coded by referencing other (already coded) blocks as determined by the coding assignment applied to the individual pictures of the blocks. For example, blocks of I pictures can be non-predictively coded or predictively coded by referencing already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of P pictures can be predictively coded via spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks of B pictures can be predictively coded via spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0072] The video encoder (603) can perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In its operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancies within the input video sequence. Thus, the coded video data can follow the syntax specified by the video coding technique or standard used.

[0073] In an embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0074] Video can be captured as multiple source pictures (video pictures) in temporal order. Intra-picture prediction (often abbreviated as intra prediction) uses spatial correlations within a given picture, while inter-picture prediction uses (temporal or other) correlations between pictures. In the example, a specific picture being encoded / decoded, referred to as the current picture, is divided into blocks. When a block within the current picture is similar to a reference block within a previously encoded and still buffered reference picture in the video, the block within the current picture can be encoded by a vector referred to as a motion vector. The motion vector points to the reference block of the reference picture and may have a third dimension identifying the reference picture if multiple reference pictures are in use.

[0075] In some embodiments, a bi-prediction technique may be used in inter-picture prediction. According to the bi-prediction technique, two reference pictures, such as a first reference picture and a second reference picture, are used for the current picture in the video, both of which are ahead in the decoding order (but may be past and future, respectively, in the display order). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture, and a second motion vector pointing to a second reference block in the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.

[0076] In addition, to improve coding efficiency, merge mode techniques can be used in inter-picture prediction.

[0077] According to some embodiments of the present disclosure, predictions such as inter-picture predictions and intra-picture predictions are performed in blocks. For example, according to the HEVC standard, a picture within a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs within the picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), which are one luma CTB and two chroma CTBs. Each CTU may be a quadtree that is recursively divided into one or more coding units (CUs). For example, a 64x64 pixel CTU may be divided into one 64x64 pixel CU, four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In the example, each CU is analyzed to determine the prediction type for the CU, such as an inter-prediction type or an intra-prediction type. The CU is divided into one or more prediction units (PUs) based on temporal and / or spatial predictability. Typically, each PU includes a luminal prediction block (PB) and two chroma PBs. In the embodiment, the prediction operation in coding (encoding / decoding) is performed in the units of the prediction block. Using a luminal prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luminal values) for pixels such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, and similar ones.

[0078] FIG. 7 shows a drawing of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures and to encode the processing block into a coded picture that is part of a coded video sequence. In the example, the video encoder (703) is used in place of the video encoder (403) of FIG. 4.

[0079] In the HEVC example, the video encoder (703) receives a matrix of sample values ​​for a prediction block of 8x8 samples and a processing block of similar values. The video encoder (703) determines whether the processing block is best coded using intra-mode, inter-mode, or a dual prediction mode, for example, using rate distortion optimization. When the processing block is to be coded in intra-mode, the video encoder (703) may use an intra-prediction technique to encode the processing block into a coded picture; when the processing block is to be coded in inter-mode or dual prediction mode, the video encoder (703) may use an inter-prediction or dual-prediction technique, respectively, to encode the processing block into a coded picture. In a particular video coding technique, the merge mode may be an inter-picture prediction submode in which motion vectors are derived from one or more motion vector predictors without the gain of a coded motion vector component outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the target block. In the example, the video encoder (703) includes other components such as a mode determination module (not shown) for determining the mode of the processing blocks.

[0080] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residue calculator (723), a switch (726), a residue encoder (724), a general controller (721), and an entropy encoder (725) combined together as shown in FIG. 7.

[0081] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks within reference pictures (e.g., blocks within previous and later pictures), generate inter-prediction information (e.g., description of redundant information according to inter-encoding techniques, motion vectors, merge mode information), and calculate inter-prediction results (e.g., prediction blocks) based on the inter-prediction information using any suitable technique. In some examples, the reference pictures are decoded reference pictures that are decoded based on encoded video information.

[0082] The intra-encoder (722) is configured to receive samples of the current block (e.g., processing block), compare the block with already coded blocks within the same picture in some cases, generate quantized coefficients after conversion, and in some cases also generate intra-prediction information (e.g., intra-prediction direction information according to one or more intra-encoding techniques). In an example, the intra-encoder (722) also calculates intra-prediction results (e.g., predicted blocks) based on the intra-prediction information and reference blocks within the same picture.

[0083] The general controller (721) is configured to determine general control data and to control other components of the video encoder (703) based on the general control data. In the example, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select an intra mode result to be used by the remaining calculator (723), controls the entropy encoder (725) to select intra prediction information, and includes the intra prediction information in the bitstream; when the mode is inter mode, the general controller (721) controls the switch (726) to select an inter prediction result to be used by the remaining calculator (723), controls the entropy encoder (725) to select inter prediction information, and includes the inter prediction information in the bitstream.

[0084] The remainder calculator (723) is configured to calculate the difference (remainder data) between the received block and the prediction results selected from the intra-encoder (722) or the inter-encoder (730). The remainder encoder (724) is configured to operate based on the remainder data to encode the remainder data and generate transformation coefficients. In an example, the remainder encoder (724) is configured to convert the remainder data from the spatial domain to the frequency domain and generate transformation coefficients. Next, the transformation coefficients may undergo quantization processing to obtain quantized transformation coefficients. In various embodiments, the video encoder (703) also includes a remainder decoder (728). The remainder decoder (728) is configured to perform an inverse transformation and generate decoded remainder data. The decoded remainder data can be appropriately used in the intra-encoder (722) and the inter-encoder (730). For example, the inter encoder (730) can generate decoded blocks based on the decoded remainder data and inter prediction information, and the intra encoder (722) can generate decoded blocks based on the decoded remainder data and intra prediction information. The decoded blocks are appropriately processed to generate decoded pictures, and the decoded pictures can be buffered in a memory circuit (not shown) and used as reference pictures in some examples.

[0085] The entropy encoder (725) is configured to format the bitstream to include the encoded block. The entropy encoder (725) is configured to include various information according to an appropriate standard, such as the HEVC standard. In the example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), remainder information, and other appropriate information within the bitstream. According to the subject matter disclosed, it should be noted that when coding a block in the inter mode or the merged submode of the dual prediction mode, there is no remainder information.

[0086] FIG. 8 shows a drawing of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and to decode the coded pictures to generate reconstructed pictures. In the example, the video decoder (810) is used in place of the video decoder (410) of FIG. 4.

[0087] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a remainder decoder (873), a reconstruction module (874), and an intra decoder (872) combined together as shown in FIG. 8.

[0088] The entropy decoder (871) may be configured to reconstruct specific symbols representing the syntax elements constituting the coded picture from the coded picture. Such symbols may include prediction information (e.g., intra-prediction information or inter-prediction information) capable of identifying specific samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), respectively, the mode in which the block is coded (e.g., intra-mode, inter-mode, dual-prediction mode, the latter two in a merged submode or other submode), residual information in the form of quantized transformation coefficients, and similar information. In the example, when the prediction mode is inter- or dual-prediction mode, the inter-prediction information is provided to the inter-decoder (880); when the prediction type is intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may undergo inverse quantization and is provided to the remainder decoder (873).

[0089] The inter decoder (880) is configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0090] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0091] The remainder decoder (873) is configured to perform inverse quantization to extract inverse quantized transform coefficients and to process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The remainder decoder (873) may also require specific control information (to include quantization parameters (QP)), and that information may be provided by the entropy decoder (871) (the data path is not shown as this may be only low-volume control information).

[0092] The reconstruction module (874) is configured to combine the residuals and prediction results (output by the inter or intra prediction module, depending on the case) output by the remainder decoder (873) in the spatial domain to form a reconstructed block that may be part of a reconstructed picture that may eventually be part of a reconstructed video. It should be noted that other appropriate operations, such as a deblocking operation and similar ones, may be performed to improve visual quality.

[0093] It should be noted that video encoders (403), (603), and (703), and video decoders (410), (510), and (810) can be implemented using any suitable technology. In an embodiment, video encoders (403), (603), and (703), and video decoders (410), (510), and (810) can be implemented using one or more integrated circuits. In another embodiment, video encoders (403), (603), and (603), and video decoders (410), (510), and (810) can be implemented using one or more processors that execute software instructions.

[0094] Block-based compensation can be used for inter-prediction and intra-prediction. For inter-prediction, block-based compensation from different pictures is known as motion compensation. Block-based compensation can also be performed from previously reconstructed regions within the same picture, as in intra-prediction. Block-based compensation from reconstructed regions within the same picture is referred to as intra-picture block compensation, current picture referencing (CPR), or intra-block copy (IBC). A displacement vector representing the offset between the current block and the reference block (also referred to as the prediction block) within the same picture is referred to as the block vector (BV), where the current block can be encoded / decoded based on the reference block. Unlike the motion vector of motion compensation, which can be any value (positive or negative in the x or y direction), the BV has several constraints to ensure that the reference block is available and has already been reconstructed. Additionally, in some examples, for the consideration of parallel processing, some reference regions that are tile boundaries, slice boundaries, or wavefront ladder shape boundaries are excluded.

[0095] In some examples, IBC mode can be used to significantly improve the coding efficiency of screen content materials. Generally, IBC mode can be implemented as a block-level coding mode. On the encoder side, the encoder can perform block matching (BM) to find the optimal block vector for each CU. In some examples, on the encoder side, hash-based motion estimation (also referred to as hash-based search) is performed on the CUs of IBC mode. The encoder can perform rate distortion (RD) checks on blocks whose width or height is not greater than 16 luma samples. For non-merged mode, block vector search is first performed using hash-based search. If hash search does not return a valid candidate, block matching-based local search may be performed.

[0096] In some examples, in a hash-based search, hash key matching (32-bit CRC) between the current block and the reference block is extended to all allowed block sizes in the current picture. The hash key calculation for all positions within the current picture can be based on 4x4 subblocks. For a larger current block, when all hash keys of all 4x4 subblocks match the hash keys at the corresponding reference positions, the hash key can be determined to match the hash key of the reference block. If hash keys of multiple reference blocks are found to match the hash key of the current block, the block vector costs of each of the matching reference blocks can be calculated, and then, one matching reference block with the minimum cost is selected as the result of the hash-based search.

[0097] In some examples, block matching search searches the area local to the current block. For instance, in a block matching search, the search range is set to encompass both previous and current CTUs.

[0098] In some implementations, the resolution of the block vector is limited to integer positions. In some other systems, the block vector may be allowed to point to fractional positions. In some examples, the luminal block vector of an IBC-coded CU is at integer precision. The chroma block vector may also be rounded to integer precision. In some examples, the IBC mode may be combined with Adaptive Motion Vector Resolution (AMVR) and may switch between 1 pixel (pel) and 4-pel motion vector precision. In some implementations, the IBC mode is treated as a third prediction mode rather than an intra or inter prediction mode. In some examples, the IBC mode is applicable to CUs where both width and height are 64 luminal samples or less.

[0099] The coding of block vectors can be explicit or implicit. In explicit mode, the BV difference between the block vector and its predictor is signaled. In implicit mode, the block vector is recovered from the predictor (referred to as the block vector predictor) without using the BV difference, in a manner similar to motion vectors in merge mode. In some examples, explicit mode may be referred to as non-merged BV prediction mode or IBC AMVP mode. In some examples, implicit mode may be referred to as merged BV prediction mode, IBC merge mode, or IBC skip mode.

[0100] There may be variations of IBC modes. In the example, IBC mode is treated as a third mode distinct from intra-prediction mode and inter-prediction mode. Therefore, BV prediction in implicit mode (or IBC merge mode) and explicit mode (IBC AMVP mode) is separated from regular inter-mode. In some examples, a separate merge candidate list may be defined for IBC mode, where the items in the separate merge candidate list are BVs. Similarly, in the example, the BV prediction candidate list for IBC explicit mode (IBC AMVP mode) contains only BVs. The general rule applicable to the two lists (i.e., the separate merge candidate list for IBC merge mode and the BV prediction candidate list for IBC AMVP mode) is that, in terms of the candidate derivation process, the two lists may follow the same logic as the list of advanced motion vector prediction (AMVP) predictors used in regular AMVP mode (used in inter-prediction) or the list of merge candidates used in regular merge mode (used in inter-prediction). For example, five spatial neighbor locations (e.g., A0, A1 and B0, B1, B2 in Fig. 2), for example, HEVC or VVC intermerging modes are accessed for the IBC merging mode to derive a separate list of merge candidates for the IBC merging mode.

[0101] In the implementation example, block-level flags are used to signal whether IBC AMVP mode or IBC Skip / Merge mode is used for the CU. In the example, when the flag (e.g., indicated by merge_flag) is true, IBC Skip / Merge mode is used; when the flag is false, IBC AMVP mode is used.

[0102] In some examples, for IBC skip / merge modes, the merge candidate index may be signaled to indicate which of the block vectors in the merge candidate list from neighboring candidate IBC coding blocks is used as the BV predictor for predicting the current block. In some examples, the merge candidate list may include spatial, history-based motion vector prediction (HMVP), and pairwise candidates.

[0103] In some examples, for IBC AMVP mode, the block vector difference is encoded in the encoded bitstream in the same way as the motion vector difference. Block vector prediction in IBC AMVP mode can use two candidates as predictors: one from the left neighbor and one from the up neighbor (when IBC encoded). If neither neighbor is available, the default block vector can be used as the predictor. In the example, a flag is signaled in the encoded bitstream to indicate the index of the BV predictor.

[0104] As explained above, the BV of the current block being reconstructed within the picture may have specific constraints, and therefore the reference block for the current block is within the IBC reference area.

[0105] An IBC reference region refers to a portion of a picture from which a reference block can be selected. For example, an IBC reference region may be located within specific parts of a reconstructed area within the picture. The size, position, shape, and / or similar characteristics of an IBC reference region may be constrained. Alternatively, a BV may be constrained. In an example, a BV is a two-dimensional vector containing x and y components, and at least one of the x and y components may be constrained. Constraints may be specified in relation to the BV, the IBC reference region, or a combination of the BV and the IBC reference region. In various examples, when specific constraints are specified for the BV, the IBC reference region is constrained accordingly. Likewise, when specific constraints are specified in relation to the IBC reference region, the BV is constrained accordingly.

[0106] FIG. 9 shows an example of intra-block copying according to an embodiment of the present disclosure. The current picture (900) will be reconstructed under decoding. The current picture (900) includes a reconstructed area (910) (gray area) and an area to be decoded (920) (white area). The current block (930) is being reconstructed by the decoder. The current block (930) may be reconstructed from a reference block (940) located within the reconstructed area (910). The position offset between the reference block (940) and the current block (930) is referred to as a block vector (950) (or BV (950)). In the example of FIG. 9, the IBC reference area (960) is within the reconstructed area (910), the reference block (940) is within the IBC reference area (960), and the block vector (950) is constrained to point to the reference block (940) within the IBC reference area (960).

[0107] Various constraints may be applied to the BV and / or IBC reference regions. In the embodiment, the IBC reference region for the current block being reconstructed within the current CTB is constrained to be within the current CTB.

[0108] In the embodiment, the effective memory requirement for storing reference samples to be used in intra-block copying is 1 CTB in size. In the example, the CTB size is 128x128 samples. The current CTB contains the current region being reconstructed. The current region has a size of 64x64 samples. Since the reference memory can also store the reconstructed samples within the current region, when the reference memory size is equal to the CTB size of 128x128 samples, the reference memory can store three additional regions of 64x64 samples. Thus, the IBC reference region may include specific parts of the previously reconstructed CTB, but the total memory requirement for storing reference samples does not change (e.g., 1 CTB size of 128x128 samples or a total of four 64x64 reference samples). In the example, the previously reconstructed CTB is the left neighbor of the current CTB, as shown in FIG. 10.

[0109] FIG. 10 shows an example of intra-block copying according to an embodiment of the present disclosure. The current picture (1001) includes a current CTB (1015) being reconstructed and a previously reconstructed CTB (1010) which is a left neighbor of the current CTB (1015). The CTBs within the current picture (1001) have a CTB size of 128x128 samples and a CTB width of 128 samples. The current CTB (1015) includes four regions (1016)-(1019), and the current region (1016) is being reconstructed. The current region (1016) includes a plurality of coding blocks (1021)-(1029). Likewise, the previously reconstructed CTB (1010) includes four regions (1011)-(1014). Coding blocks (1021)-(1025) are reconfigured, the current block (1026) is being reconfigured, and coding blocks (1026)-(1027) and regions (1017)-(1019) must be reconfigured.

[0110] The current region (1016) has a collocated region (i.e., a region (1011) within the previously reconstructed CTB (1010)). The relative position of the collocated region (1011) to the previously reconstructed CTB (1010) may be the same as the relative position of the current region (1016) to the current CTB (1015). In the example illustrated in FIG. 10, the current region (1016) is the upper-left region of the current CTB (1015), and thus the collocated region (1011) is also the upper-left region of the previously reconstructed CTB (1010). Since the position of the previously reconstructed CTB (1010) is offset by the width of the CTB from the position of the current CTB (1015), the position of the collocated region (1011) is offset by the width of the CTB from the position of the current region (1016).

[0111] In an embodiment, the juxtaposed area of ​​the current area (1016) is within a previously reconfigured CTB, wherein the position of the previously reconfigured CTB is offset by 1 CTB width or a multiple thereof from the position of the current CTB (1015), and thus, the position of the juxtaposed area is also offset by a corresponding 1 CTB width or a multiple thereof from the position of the current area (1016). The position of the juxtaposed area may be shifted to the left or upward from the current area (1016) or similarly.

[0112] As described above, the size of the IBC reference area for the current block (1026) is constrained by the CTB size. In the example of FIG. 10, the IBC reference area may include parts of the already reconstructed current area (1016), such as areas (1012)-(1014) and coding blocks (1021)-(1025) within the previously reconstructed CTB (1010). To ensure that the size of the search range is within the CTB size, the IBC reference area may exclude the juxtaposed area (1011). Referring to FIG. 10, the reference block (1091) is located within the area (1014) of the previously reconstructed CTB (1010). The block vector (1020) represents the offset between the current block (1026) and the individual reference block (1091). The reference block (1091) is within the IBC reference area.

[0113] The example illustrated in FIG. 10 can be appropriately adapted to other scenarios where the current area is located at a different location within the current CTB (1015). In the example, when the current block is within area (1017), the parallel area for the current block is area (1012). Thus, the IBC reference area may include areas (1013)-(1014), area (1016), and a portion of the already reconstructed area (1017). To ensure that the size of the IBC reference area is within the size of the CTB, the search range further excludes area (1011) and the parallel area (1012). In the example, when the current block is within area (1018), the parallel area for the current block is area (1013). Thus, the IBC reference area may include areas (1014), areas (1016)-(1017), and a portion of the already reconstructed area (1018). To ensure that the size of the IBC reference area is within the size of the CTB, the IBC reference area further excludes areas (1011)-(1012) and the juxtaposed area (1013). In the example, when the current block is within area (1019), the juxtaposed area for the current block is area (1014). Thus, the IBC reference area may include areas (1016)-(1018) and a portion of the already reconstructed area (1019). To ensure that the size of the IBC reference area is within the size of the CTB, the IBC reference area further excludes the previously reconstructed CTB (1010).

[0114] In the above description, the reference block may be in a previously reconfigured CTB (1010) or a current CTB (1015).

[0115] In an example, the IBC reference region may be specified as follows. In the example, the current picture is a luminance picture, the current CTB is a luminance CTB containing multiple luminance samples, and the BV(mvL) satisfies the following constraints for bitstream conformance. In the example, the BV(mvL) has fractional resolution (e.g., 1 / 16-pel resolution).

[0116] The constraints include first conditions that the reference block for the current block has already been reconstructed. If the reference block has a rectangular shape, a neighboring block availability checking process (or reference block availability checking process) may be implemented to check whether the top-left sample and bottom-right sample of the reference block are reconstructed. When both the top-left sample and the bottom-right sample of the reference block are reconstructed, the reference block is determined to be reconstructed.

[0117] For example, when the derivation process for reference block availability is called with inputs the position (xCurr, yCurr) of the top-left sample of the current block set to (xCb, yCb) and the position (xCb + (mvL[0]≫4), yCb + (mvL[1]≫4)) of the top-left sample of the reference block, if the top-left sample of the reference block is reconstructed, the output is equal to TRUE, where the block vector mvL is a 2D vector with x component mvL[0] and y component mvL[1]. When BV(mvL) has a fractional resolution such as 1 / 16-pel resolution, the x component mvL[0] and y component mvL[1] are shifted to have an integer resolution, denoted as (mvL[0]≫4 and mvL[1]≫4, respectively.

[0118] Likewise, when the derivation process for block availability is called with input the position (xCurr, yCurr) of the top-left sample of the current block set to (xCb, yCb), and the position (xCb + (mvL[0]≫4) + cbWidth - 1, yCb + (mvL[1]≫4) + cbHeight - 1) of the bottom-right sample of the reference block, if the bottom-right sample of the reference block is reconfigured, the output is the same as TRUE. The parameters cbWidth and cbHeight represent the width and height of the reference block.

[0119] The constraints may also include at least one of the following second conditions: 1) the value of (mvL[0]≫4) + cbWidth is 0 or less, indicating that the reference block is to the left of the current block and does not overlap with the current block; 2) the value of (mvL[1]≫4) + cbHeight is 0 or less, indicating that the reference block is above the current block and does not overlap with the current block.

[0120] The constraints may also include the following third conditions being satisfied by the block vector mvL:

[0121] (1)

[0122] (2)

[0123] (3)

[0124] (4)

[0125] Here, the parameter CtbLog2SizeY represents the CTB width in log2 format. For example, if the CTB width is 128 samples, CtbLog2SizeY is 7. Equations (1)-(2) specify that the CTB containing the reference block is in the same CTB row as the current CTB (for example, when the reference block is in the previously reconstructed CTB (1010), the previously reconstructed CTB (1010) is in the same row as the current CTB (1015). Equations (3)-(4) specify that the CTB containing the reference block is in either the left CTB column of the current CTB or the same CTB column as the current CTB. The third condition described by Equations (1)-(4), as described with reference to FIG. 10, specifies that the CTB containing the reference block is either the current CTB, such as the current CTB (1015), or the left neighbor of the current CTB, such as the previously reconstructed CTB (1010).

[0126] The constraints may further include a fourth condition: when the reference block is in the left neighbor of the current CTB, the juxtaposed region for the reference block is not reconstructed (i.e., no samples within the juxtaposed region are reconstructed). Also, the juxtaposed region for the reference block is in the current CTB. In the example of FIG. 10, the juxtaposed region for the reference block (1091) is the region (1019) offset by the width of the CTB from the region (1014) where the reference block (1091) is located, and the region (1019) is not reconstructed. Thus, the block vector (1020) and the reference block (1091) satisfy the fourth condition described above.

[0127] In the example, the fourth conditions can be specified as follows: when (xCb + (mvL[0]≫4)) ≫ CtbLog2SizeY is equal to (xCb ≫ CtbLog2SizeY) - 1, the position of the current block (xCurr, yCurr) set to (xCb, yCb), and the position (((xCb + (mvL[0] ≫ 4) + CtbSizeY) ≫ (CtbLog2SizeY - 1)) ≪ (CtbLog2SizeY - 1), ((yCb + (mvL[1] ≫ 4)) ≫ (CtbLog2SizeY - 1)) ≪ (CtbLog2SizeY - 1)) are inputs, and the output is equal to FALSE, which means that the juxtaposed region is not reconstructed as shown in FIG. 10. It indicates that it is not.

[0128] Constraints on the IBC reference region and / or block vector may include appropriate combinations of the first, second, third, and fourth conditions described above. In an example, the constraints include the first, second, third, and fourth conditions as illustrated in FIG. 10. In an example, the first, second, third, and / or fourth conditions may be modified, and the constraints include the modified first, second, third, and / or fourth conditions.

[0129] According to the fourth condition, when one of the coding blocks (1022)-(1029) is the current block, the reference block cannot be in area (1011), so the IBC reference area for such one of the coding blocks (1022)-(1029) excludes area (1011). The reason why area (1011) is excluded is specified below: if the reference block is in area (1011), the parallel area for the reference block is area (1016), but the fourth condition is violated because at least the samples of the coding block (1021) have been reconstructed. Meanwhile, for a coding block to be reconstructed first in the current area, such as the coding block (1121) of area (1116) in FIG. 11, the fourth condition does not prevent the reference block from being in area (1111) because the parallel area (1116) for the reference block has not yet been reconstructed.

[0130] FIG. 11 shows an example of intra-block copying according to an embodiment of the present disclosure. The current picture (1101) includes the current CTB (1115) being reconstructed and the previously reconstructed CTB (1110) which is the left neighbor of the current CTB (1115). The CTBs within the current picture (1101) have a CTB size and a CTB width. The current CTB (1115) includes four regions (1116)-(1119), wherein the current region (1116) is being reconstructed. The current region (1116) includes a plurality of coding blocks (1121)-(1129). Likewise, the previously reconstructed CTB (1110) includes four regions (1111)-(1114). In the current region (1116), the current block (1121) being reconstructed is reconstructed first, and the coding blocks (1122)-(1129) will be reconstructed. In the example, the CTB size is 128x128 samples, and regions (1111)-(1114) and (1116)-(1119) are each 64x64 samples. Since the reference memory size is the same as the CTB size and is 128x128 samples, the IBC reference region when bounded by the reference memory size includes three regions and a portion of an additional region.

[0131] As described with reference to FIG. 10, the current region (1116) has a parallel region (i.e., a region (1111) within the previously reconstructed CTB (1110)). According to the fourth conditions described above, the reference block for the current block (1121) may be within region (1111), and thus the IBC reference region may include regions (1111)-(1114). For example, when the reference block is within region (1111), the parallel region of the reference block is region (1116), where no sample within region (1116) was reconstructed prior to the reconstruction of the current block (1121). However, as described with reference to FIG. 10 and the fourth conditions, for example, after the reconstruction of the coding block (1121), region (1111) is no longer available to be included in the IBC reference region for reconstructing the coding block (1122). Therefore, tight synchronization and timing control of the reference memory buffer must be used, and this can be difficult.

[0132] According to some embodiments, when the current block is first reconstructed in the current region of the current CTB, the IBC reference region may exclude the parallel region of the current region within the previously reconstructed CTB, wherein the current CTB and the previously reconstructed CTB are within the same current picture. The block vector may be determined such that the reference block is within the IBC reference region excluding the parallel region within the previously reconstructed CTB. In an embodiment, the IBC reference region includes coding blocks reconstructed after the parallel region and before the current block in the decoding order.

[0133] In the descriptions below, the CTB size may vary, and the maximum CTB size is set to be equal to the reference memory size. In the example, the reference memory size or maximum CTB size is 128x128 samples. The description may be appropriately adapted to other reference memory sizes or maximum CTB sizes.

[0134] In the embodiment, the CTB size is the same as the reference memory size. The previously reconfigured CTB is the left neighbor of the current CTB, the position of the juxtaposed region is offset from the position of the current region by the width of the CTB, and the coding blocks within the IBC reference region are located in at least one of the current CTB and the previously reconfigured CTB.

[0135] FIGS. 12a through 12d illustrate examples of intra-block copying according to embodiments of the present disclosure. Referring to FIGS. 12a through 12d, the current picture (1201) includes the current CTB (1215) currently being reconstructed and the previously reconstructed CTB (1210) which is the left neighbor of the current CTB (1215). The CTBs within the current picture (1201) have a CTB size and a CTB width. The current CTB (1215) includes four regions (1216)–(1219). Likewise, the previously reconstructed CTB (1210) includes four regions (1211)–(1214). In the embodiment, the CTB size is the maximum CTB size and is equal to the reference memory size. In the example, the CTB size and reference memory size are 128x128 samples, and thus the regions (1211)-(1214), and (1216)-(1219) each have a size of 64x64 samples.

[0136] In the examples illustrated in FIGS. 12a through 12d, the current CTB (1215) includes a top-left area, a top-right area, a bottom-left area, and a bottom-right area corresponding to regions (1216)-(1219), respectively. The previously reconstructed CTB (1210) includes a top-left area, a top-right area, a bottom-left area, and a bottom-right area corresponding to regions (1211)-(1214), respectively.

[0137] Referring to FIG. 12a, the current region (1216) is being reconstructed. The current region (1216) may include a plurality of coding blocks (1221)-(1229). The current region (1216) has a parallel region within the previously reconstructed CTB (1210), i.e., region (1211). An IBC reference region for one of the coding blocks (1221)-(1229) to be reconstructed may exclude the parallel region (1211). The IBC reference region may include regions (1212)-(1214) of the previously reconstructed CTB (1210) that were reconstructed after the parallel region (1211) and before the current region (1216) in the decoding order.

[0138] Referring to FIG. 12a, the position of the juxtaposed region (1211) is offset from the position of the current region (1216) by a CTB width equal to 128 samples. For example, the position of the juxtaposed region (1211) is shifted to the left by 128 samples from the position of the current region (1216).

[0139] Referring again to FIG. 12a, when the current area (1216) is the upper-left area of ​​the current CTB (1215), the juxtaposed area (1211) is the upper-left area of ​​the previously reconstructed CTB (1210), and the search area excludes the upper-left area of ​​the previously reconstructed CTB.

[0140] As illustrated in FIG. 12a, the current block belongs to the upper left 64x64 block (e.g., current area (1216)) of the current CTU (e.g., illustrated by CTB (1215)), and then, in addition to the already reconstructed samples within the current CTU (e.g., illustrated by CTB (1215)), samples within the lower right 64x64 blocks (e.g., illustrated by (1214)) of the left CTU (e.g., illustrated by CTB (1210)) can be referenced as reference samples, for example, using the current picture referencing (CPR) mode. Additionally, the current block can use CPR mode to refer to samples in the lower left 64x64 block (e.g., illustrated by (1213)) of the left CTU (e.g., illustrated by CTB (1210)), and samples in the upper right 64x64 block (e.g., illustrated by (1212)) of the left CTU (e.g., illustrated by CTB (1210)).

[0141] Referring to FIG. 12b, the current region (1217) is being reconfigured. The current region (1217) may include a plurality of coding blocks (1241)-(1249). The current region (1217) has a parallel region (i.e., a region (1212) within the previously reconfigured CTB (1210)). An IBC reference region for one of the plurality of coding blocks (1241)-(1249) may exclude the parallel region (1212). The IBC reference region includes regions (1213)-(1214) of the previously reconfigured CTB (1210), and a region (1216) within the current CTB (1215) reconfigured after the parallel region (1212) and before the current region (1217). The IBC reference region further excludes region (1211) due to constraints on the reference memory size (i.e., 1 CTB size). Likewise, the position of the juxtaposed region (1212) is offset from the position of the current region (1217) by a CTB width equal to 128 samples.

[0142] In the example of FIG. 12b, the current area (1217) is the upper right area of ​​the current CTB (1215), the juxtaposed area (1212) is also the upper right area of ​​the previously reconstructed CTB (1210), and the search area excludes the upper right area of ​​the previously reconstructed CTB (1210).

[0143] As illustrated in FIG. 12b, if the current block belongs to the upper right 64x64 block (e.g., illustrated by (1217)) of the current CTU (e.g., illustrated by CTB (1215)), in addition to the already reconstructed samples (e.g., illustrated by (1216)) within the current CTU, if the luma location (0, 64) for the current CTU has not yet been reconstructed, the current block may also refer to samples within the lower left 64x64 block (e.g., illustrated by (1213)) and lower right 64x64 block (e.g., illustrated by (1214)) of the left CTU using CPR mode; otherwise, the current block may also refer to reference samples within the lower right 64x64 block (e.g., illustrated by (1214)) of the left CTU.

[0144] Referring to FIG. 12c, the current region (1218) is being reconfigured. The current region (1218) may include a plurality of coding blocks (1261)-(1269). The current region (1218) has a parallel region (i.e., region (1213)) within the previously reconfigured CTB (1210). An IBC reference region for one of the plurality of coding blocks (1261)-(1269) may exclude the parallel region (1213). The IBC reference region includes the region (1214) of the previously reconfigured CTB (1210), and regions (1216)-(1217) within the current CTB (1215) that were reconfigured after the parallel region (1213) and before the current region (1218). Likewise, the IBC reference region further excludes regions (1211)-(1212) due to constraints on the reference memory size. The position of the juxtaposed area (1213) is offset from the position of the current area (1218) by a CTB width equal to 128 samples. In the example of FIG. 12c, if the current area (1218) is the lower-left area of ​​the current CTB (1215), the juxtaposed area (1213) is also the lower-left area of ​​the previously reconstructed CTB (1210), and the search area excludes the lower-left area of ​​the previously reconstructed CTB (1210).

[0145] As illustrated in FIG. 12c, if the current block belongs to the lower left 64x64 block (e.g., illustrated by (1218)) of the current CTU (e.g., illustrated by CTB (1215)), in addition to the already reconstructed samples (e.g., (1216) and (1217)) within the current CTU, if the luma location (64, 0) for the current CTU has not yet been reconstructed, the current block can also refer to the samples within the lower right 64x64 block (e.g., illustrated by (1214)) of the left CTU using CPR mode.

[0146] Referring to FIG. 12d, the current region (1219) is being reconstructed. The current region (1219) may include a plurality of coding blocks (1281)-(1289). The current region (1219) has a parallel region (i.e., region (1214)) within the previously reconstructed CTB (1210). An IBC reference region for one of the plurality of coding blocks (1281)-(1289) may exclude the parallel region (1214). The IBC reference region includes regions (1216)-(1218) within the current CTB (1215) that were reconstructed after the parallel region (1214) and before the current region (1219) in the decoding order. The IBC reference region excludes regions (1211)-(1213) due to constraints on the reference memory size, and thus the IBC reference region excludes the previously reconstructed CTB (1210). Likewise, the position of the juxtaposed area (1214) is offset from the position of the current area (1219) by a CTB width equal to 128 samples. In the example of FIG. 12d, if the current area (1219) is the lower right area of ​​the current CTB (1215), the juxtaposed area (1214) is also the lower right area of ​​the previously reconstructed CTB (1210), and the search area excludes the lower right area of ​​the previously reconstructed CTB (1210).

[0147] In FIG. 12d, the current block belongs to the bottom right 64x64 block of the current CTU (illustrated by (1219)), and it can refer only to already reconstructed samples within the current CTU (e.g., (1216), (1217) and (1218)) using CPR mode.

[0148] Using the constraints illustrated in FIGS. 12a–12d, the IBC mode can be implemented using local on-chip memory for hardware implementations in some examples.

[0149] According to an aspect of the present disclosure, template matching (TM) search refers to a decoder-side MV derivation technique for refining motion information of the current CU by finding the closest match between a template in the current picture (i.e., the top and / or left neighbor blocks of the current CU) and a block in the reference picture (i.e., the same size as the template, referred to as the reference template).

[0150] FIG. 13 illustrates an example of a template matching search in some embodiments. In the example of FIG. 13, the current picture (1310) includes the current CU being reconstructed, and the current template for the current CU includes an upper block and a left block adjacent to the current CU. The upper block and the left block form the current template for the current CU. In the reference picture (1320) for the current picture (1310), the initial motion vector (MV) points to the reference position, and the search area (1350) is defined as an area within [-8 pel, +8 pel] of the reference position in the X and Y directions.

[0151] In the example, a template matching search is used to search for a better MV that has the closest match between the reference template and the current template within the search area (1350). In some examples, template matching techniques may use a search step size determined based on the AMVR mode, and the template matching search may be cascaded with a bidirectional matching process in merge modes.

[0152] In some examples, in AMVP mode, an MVP candidate is selected from a list of candidates based on a template error calculation that selects a motion vector predictor (MVP) candidate having the minimum difference between the current template and the reference template. Next, a template matching search is performed based on this specific MVP candidate for MV refinement. In some examples, the template matching search refines this MVP candidate using an iterative diamond search pattern, starting from full-pel motion vector difference (MVD) precision (or 4-pel for 4-pel AMVR mode) within the [-8, +8]-pel search range. The AMVP candidate can be further refined using a cross-search pattern of full-pel MVD precision (or 4-pel for 4-pel AMVR mode), and then sequentially using half-pel cross-search patterns and 1 / 4-pel cross-search patterns depending on AMVR. This search process ensures that the MVP candidate maintains the same MV precision as that indicated by AMVR mode after the TM search process.

[0153] FIG. 14 illustrates a table (1400) of search patterns for AMVR and merge modes in some examples.

[0154] In some examples, in merge mode, a similar search method is applied to merge candidates indicated by the merge index. As illustrated in the table in Fig. 14, the TM search may be performed down to the end of 1 / 8-pel MVD precision or skipped over those exceeding half-pel MVD precision, depending on whether an alternative interpolation filter (used when AMVR is in half-pel mode) is used according to the merged motion information. For example, a variable AltIF of 0 indicates that no alternative interpolation filter is used and the TM search can be performed down to the end of 1 / 8-pel MVD precision; a variable AltIF of 1 indicates that an alternative interpolation filter is used and the TM search can be performed up to half-pel precision. Additionally, when the TM mode is enabled, template matching may operate as an independent process between block-based and sub-block-based bidirectional matching methods or as an additional MV refinement process, depending on whether bidirectional matching based on an activation condition check can be enabled.

[0155] According to some aspects of the present disclosure, template matching techniques may be used in IBC mode to perform template matching search based on IBC merge candidates to achieve block vector refinements.

[0156] According to an aspect of the present disclosure, for a merge candidate list in IBC mode, only a specific number of candidates may be on the merge candidate list, and the merge index is signaled to indicate that it is within the merge candidate list. However, by using a template matching method, the decoder may derive the best block vector by searching for a nearby block vector based on the templates. Additionally, while constructing the IBC merge candidate list, candidate BVs outside the IBC reference regions are considered invalid, and invalid candidate BVs are removed from the merge candidate list. However, there are cases where the best BV emerges from such invalid candidate BVs.

[0157] Some aspects of the present disclosure provide techniques that allow invalid BVs in a merge candidate list, and template matching search may be performed based on invalid BVs to achieve better BVs (e.g., fewer template errors) in IBC mode.

[0158] In the example, a flag may be used to indicate whether template matching is applied to IBC merge candidates.

[0159] FIG. 15 illustrates examples of pseudocodes (1500) for template matching in some embodiments.

[0160] In the example of FIG. 15, when the current block (PU) is in IBC mode, a first flag (e.g., indicated by merge_flag) is signaled in the coded bitstream to indicate whether the current block is coded in IBC AMVP mode or IBC merge mode. The first flag can be parsed from the bitstream as shown in (1501) of FIG. 15. When the first flag (merge_flag) is equal to 1, the current block is coded using IBC merge mode. When the current block is in IBC merge mode, a second flag (e.g., indicated by template_matching_ibc_merge_flag) and a merge index (e.g., indicated by merge_idx) are signaled in the coded bitstream. The second flag can be parsed from the coded bitstream as illustrated by (1502) in FIG. 15, and the merge index can be parsed from the coded bitstream as illustrated by (1503) in FIG. 15.

[0161] The merge index represents a candidate BV in the IBC merge candidate list. If the second flag (e.g., template_matching_ibc_merge_flag) is equal to 1, a template matching search can be performed using the candidate BV represented by the merge index in the IBC merge candidate list as the initial BV. In the example, when the second flag (template_matching_ibc_merge_flag) is equal to 0, the template matching search is disabled, and the candidate BV represented by the merge index is used as a block vector pointing to a reference block to reconstruct the current block (current PU).

[0162] In some embodiments, the template matching search is associated with the IBC merge mode. Thus, the template matching search can be applied in the IBC merge mode without signaling a second flag (e.g., template_matching_ibc_merge_flag). The merge index can be signaled to indicate a candidate BV from the IBC merge candidate list to be used as the initial BV for template matching.

[0163] According to another aspect of the present disclosure, during the construction of an IBC merge candidate list, in the process of validating each IBC merge candidate, the template matching search range (Sr) considers the IBC reference range instead of using only the IBC reference range.

[0164] FIG. 16 illustrates an example of an IBC according to an embodiment of the present disclosure. In FIG. 16, the current picture (1601) includes the current CTB (1615) being reconstructed and the previously reconstructed CTB (1610) which is the left neighbor of the current CTB (1615). The current CTB (1615) includes four regions (1616)-(1619). Likewise, the previously reconstructed CTB (1610) includes four regions (1611)-(1614). In the embodiment, the CTB size is the maximum CTB size and is equal to the reference memory size. In the example, since the CTB size and the reference memory size are 128x128 samples, the regions (1611)-(1614) and (1616)-(1619) each have a size of 64x64 samples.

[0165] In the example of FIG. 16, the current region (1616) is being reconstructed. The current region (1616) has a region (1611) that is juxtaposed with the previously reconstructed CTB (1610). The IBC reference region for the current block (1650) to be reconstructed may exclude the juxtaposed region (1611). The IBC reference region may include regions (1612)–(1614) of the previously reconstructed CTB (1610) that are reconstructed after the juxtaposed region (1611) and before the current region (1616) in the decoding order.

[0166] In the example of FIG. 16, the arrow (1651) indicates a candidate BV within the IBC merge candidate list. The candidate BV points to position A. The rectangular box (1652) around position A is the template matching search range that can be indicated by Sr. In the example, position A has two-dimensional coordinates (x0, y0). Then, the rectangular box (1652) can be defined by four corners: (x0-Sr, y0-Sr), (x0+Sr, y0-Sr), (x0-Sr, y0+Sr), and (x0+Sr, y0+Sr). In some related examples, because position A is outside the IBC reference area (the area (1611) is excluded from the IBC reference area), the candidate BV (1651) is considered invalid and is not included in the IBC merge candidate list.

[0167] In some embodiments, invalid merge candidates, such as candidate BV (1651), may be inserted into the IBC merge candidate list. To perform a template matching search based on candidate BV (1651), position A (pointed to by candidate BV (1651)) is moved into the IBC reference range, for example, to position C, which is the nearest position within the valid IBC reference range for position A. Next, the template matching search may begin additional search and refinement within the template matching search range from position C. In the example of FIG. 16, if the top-left corner B of region (1612) has 2D coordinates (x1, y1), position C may have 2D coordinates (x1, y0) in the example.

[0168] In some embodiments, during the validation of IBC merge candidates, a candidate BV is considered valid when the template matching search range of the BV partially or completely parallels the IBC reference region. In the example of FIG. 16, since the template matching search region (1652) for candidate BV (1651) partially overlaps with the IBC reference region (including region (1612), region (1613), and region (1614)), candidate BV (1651) is considered valid and can be included in the IBC merge candidate list. In another example, if x0+Sr > x1, candidate BV (1651) is valid. Likewise, if y0+Sr > y1, candidate BV (1651) is valid.

[0169] Additionally, according to an embodiment of the present disclosure, during a template matching search, when an intermediate block vector is outside the IBC reference area, the intermediate block vector is not considered, and template matching calculations associated with the intermediate block vector (e.g., calculation of the difference between the reference template pointed to by the intermediate block vector and the current template) may be skipped.

[0170] FIG. 17 illustrates a flowchart schematically describing a process (1700) according to an embodiment of the present disclosure. The process (1700) may be used in a video encoder. In various embodiments, the process (1700) is executed by a processing circuit in terminal devices (310), (320), (330), and (340), a processing circuit that performs the functions of a video encoder (403), a processing circuit that performs the functions of a video encoder (603), a processing circuit that performs the functions of a video encoder (703), and a processing circuit similar to the same. In some embodiments, the process (1700) is implemented by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit performs the process (1700). The process starts at (S17301) and proceeds to (S1710).

[0171] In (S1710), when the current block is in IBC mode, an initial block vector for predicting the current block is determined in the current coding tree unit (CTU).

[0172] In (S1720), a template matching search is performed based on the initial block vector to determine a refined block vector pointing to a reference block within the same picture as the current block.

[0173] In (S1730), the reconstructed current block is generated based on the reference block.

[0174] In (S1740), the coded video bitstream carrying the picture is encoded to include information representing the initial block vector.

[0175] In some examples, a merge index representing block vector candidates from a list of merge candidates is encoded in the coded video bitstream, and the initial block vector is determined based on the block vector candidates. In some examples, the block vector candidates are determined by an encoder using at least one of block matching or hash-based motion estimation.

[0176] In the example, a first flag indicating an IBC merge mode is encoded in the video bitstream, and a second flag indicating to perform a template matching search is encoded in the video bitstream.

[0177] In another example, a first flag indicating IBC merge mode and performing template matching search encoding is encoded in the coded video bitstream.

[0178] In some embodiments, the merge candidate list is configured to include at least a first block vector candidate pointing to a first position outside the IBC reference area. In the example, to determine an initial block vector based on the first block vector candidate, the position closest to the first position in the IBC reference area is determined; and the initial block vector is determined to point to the closest position.

[0179] In some embodiments, to construct a merge candidate list, a first block vector candidate is inserted into the merge candidate list in response to a determination that the template matching search area of ​​the first block vector candidate overlaps with the IBC reference area.

[0180] In some examples, the IBC reference area includes a reconfigured portion of the current CTU and an area of ​​the left CTU, which are cached in a memory space sized to store the CTU.

[0181] In some examples, to perform a template matching search, parts of the template matching search area outside the IBC reference area are excluded from the template matching search.

[0182] Next, the process proceeds to (S1799) and terminates.

[0183] The process (1700) can be appropriately adapted. The step(s) of the process (1700) can be modified and / or omitted. Additional steps can be added. Any appropriate order of implementation can be used.

[0184] FIG. 18 illustrates a flowchart schematically describing a process (1800) according to an embodiment of the present disclosure. The process (1800) may be used in a video decoder. In various embodiments, the process (1800) is executed by a processing circuit in terminal devices (310), (320), (330), and (340), a processing circuit that performs the functions of a video decoder (410), a processing circuit that performs the functions of a video decoder (510), and a processing circuit similar to the same. In some embodiments, the process (1800) is implemented by software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit performs the process (1800). The process starts at (S1801) and proceeds to (S1810).

[0185] In (S1810), when the current block is in IBC mode, an initial block vector for predicting the current block is determined in the current coding tree unit (CTU).

[0186] In (S1820), a template matching search is performed based on the initial block vector to determine a refined block vector pointing to a reference block within the same picture as the current block.

[0187] In (S1830), the reconstructed current block is generated based on the reference block.

[0188] In some examples, the initial block vector is determined from the coded video bitstream carrying the picture based on the merge index. The merge index represents a block vector candidate from the merge candidate list.

[0189] In the example, a first flag indicating the IBC merge mode is parsed from the coded video bitstream. Next, a second flag indicating to perform a template matching search is parsed from the coded video bitstream.

[0190] In another example, from a coded video bitstream carrying a picture, a flag is parsed indicating the IBC merge mode and performing a template matching search.

[0191] In some examples, the merge candidate list is configured to include at least a first block vector candidate pointing to a first position outside the IBC reference area. In response to a merge index representing the first block vector candidate, the position closest to the first position in the IBC reference area is determined, and the initial block vector is determined to point to the closest position.

[0192] In some examples, to construct a merge candidate list, the first block vector candidate is inserted into the merge candidate list in response to the determination that the template matching search area of ​​the first block vector candidate overlaps with the IBC reference area.

[0193] In some examples, to perform a template matching search, parts of the template matching search area outside the IBC reference area are excluded from the template matching search.

[0194] In some examples, the IBC reference area includes a reconfigured portion of the current CTU and an area of ​​the left CTU, which are cached in a memory space sized to store the CTU.

[0195] Next, the process proceeds to (S1899) and terminates.

[0196] The process (1800) can be appropriately adapted. The step(s) of the process (1800) can be modified and / or omitted. Additional steps can be added. Any appropriate order of implementation can be used.

[0197] The technologies described above may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media. For example, FIG. 19 shows a computer system (1900) suitable for implementing specific embodiments of the disclosed subject matter.

[0198] Computer software may be coded using any suitable machine code or computer language that may depend on assembly, compilation, linking, or similar mechanisms for generating code containing instructions that can be executed directly or through translation, micro-code execution, and similar means by one or more computer central processing units (CPUs), graphics processing units (GPUs), and similar things.

[0199] The commands can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and similar devices.

[0200] The components shown in FIG. 19 for the computer system (1900) are essentially exemplary and are not intended to suggest any limitations on the scope of use or function of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be interpreted as having any dependencies or requirements related to any of the components shown in the exemplary embodiments of the computer system (1900) or any combination thereof.

[0201] The computer system (1900) may include specific human interface input devices. Such human interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, applause), visual input (e.g., gestures), and olfactory input (not shown). Human interface devices may also be used to capture specific media that are not necessarily directly related to conscious input by humans, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0202] Input human interface devices may include one or more of a keyboard (1901), a mouse (1902), a trackpad (1903), a touch screen (1910), a data glove (not shown), a joystick (1905), a microphone (1906), a scanner (1907), and a camera (1908) (only one of each is shown).

[0203] The computer system (1900) may also include specific human interface output devices. These human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. These human interface output devices may include tactile output devices (e.g., touch screen (1910), tactile feedback via a data glove (not shown) or joystick (1905), but there may also be tactile feedback devices that do not serve as input devices), audio output devices (e.g., speakers (1909), headphones (not shown)), visual output devices (e.g., screens (1910) including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have touch screen input capabilities, each of which may or may not have tactile feedback capabilities, some of which may output two-dimensional visual output, or stereoscopic output, output beyond three dimensions via means such as virtual reality glasses (not shown), holographic displays and smoke tanks (not shown), and printers (not shown).

[0204] The computer system (1900) may also include optical media including a CD / DVD ROM / RW (1920) having a CD / DVD or similar media (1921), a thumb drive (1922), a removable hard drive or solid-state drive (1923), legacy magnetic media such as tape and floppy disk (not shown), specialized ROM / ASIC / PLD based devices such as a security dongle (not shown), and human-accessible storage devices such as similar ones and their associated media.

[0205] A person skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transient signals.

[0206] The computer system (1900) may also include an interface (1954) to one or more communication networks (1955). The networks may be, for example, wireless, wired, or optical. The networks may also be local, wide-area, metropolitan, automotive and industrial, real-time, latency-tolerant, etc. Examples of networks may include local area networks such as Ethernet and wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE and similar ones, TV wired or wireless wide-area digital networks including cable TV, satellite TV and terrestrial broadcast TV, automotive and industrial networks including CANBus, etc. Certain networks generally require external network interface adapters attached to specific general-purpose data ports (e.g., USB ports of the computer system (1900)) or peripheral buses (1949); others are generally integrated into the core of the computer system (1900) by attaching to the system bus as described below (e.g., an Ethernet interface for a PC computer system, or a cellular network interface for a smartphone computer system). Using any of these networks, the computer system (1900) can communicate with other entities. Such communication may be unidirectional, receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., from a CANbus to specific CANbus devices), or bidirectional to other computer systems using, for example, a local area or wide area digital network. Specific protocols and protocol stacks may be used in each of the networks and network interfaces as described above.

[0207] The human interface devices, human-accessible storage devices, and network interfaces mentioned above can be attached to the core (1940) of the computer system (1900).

[0208] The core (1940) may include one or more central processing units (CPUs) (1941), graphics processing units (GPUs) (1942), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (1943), hardware accelerators (1944) for specific tasks, graphics adapters (1950), etc. These devices may be connected via a system bus (1948), along with read-only memory (ROM) (1945), random access memory (1946), internal mass storage such as an internal hard drive that is inaccessible to the user, SSDs, and similar devices (1947). In some computer systems, the system bus (1948) may be accessible in the form of one or more physical plugs that enable expansion by additional CPUs, GPUs, and similar devices. Peripheral devices may be attached directly to the core's system bus (1948) or via a peripheral bus (1949). In the example, the screen (1910) can be connected to a graphics adapter (1950). Architectures for peripheral buses include PCI, USB, and similar ones.

[0209] CPUs (1941), GPUs (1942), FPGAs (1943), and accelerators (1944) can be combined to execute specific instructions that can construct the computer code mentioned above. The computer code may be stored in ROM (1945) or RAM (1946). Temporary data may also be stored in RAM (1946), while permanent data may be stored, for example, in internal mass storage (1947). High-speed storage and retrieval of any of the memory devices may be made possible through the use of cache memory that may be closely associated with one or more CPUs (1941), GPUs (1942), mass storage (1947), ROM (1945), RAM (1946), and similar devices.

[0210] A computer-readable medium may have computer code for performing various computer-implemented operations. The medium and the computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type widely known and available to those skilled in the art of computer software.

[0211] As an example rather than a limitation, a computer system having an architecture (1900) and, in particular, a core (1940) may provide functionality as a result of processor(s) (including CPUs, GPUs, FPGAs, accelerators, and similar ones) executing software implemented on one or more types of tangible computer-readable media. Such computer-readable media may be media associated with the user-accessible mass storage introduced above, as well as specific storage of the core (1940) having non-transient characteristics, such as internal mass storage (1947) or ROM (1945). Software implementing various embodiments of the present disclosure may be stored on these devices and executed by the core (1940). The computer-readable media may include one or more memory devices or chips depending on specific requirements. Software may enable the core (1940) and, in particular, internal processors (including CPUs, GPUs, FPGAs, and similar ones) to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (1946) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide function as a result of logic embedded in or otherwise implemented in a circuit (e.g., accelerator (1944)) capable of operating to execute specific processes or specific parts of specific processes described herein on behalf of or together with the software. References to software may include logic, and where appropriate, vice versa. References to computer-readable media may include a circuit storing software for execution (e.g., an integrated circuit (IC)), a circuit implementing logic for execution, or, where appropriate, both. The present disclosure covers any suitable combination of hardware and software.

[0212] Appendix A: Abbreviations

[0213] JEM (joint exploration model): Joint exploration model

[0214] VVC (versatile video coding): Multipurpose video coding

[0215] BMS(benchmark set): Benchmark set

[0216] MV (Motion Vector): Motion Vector

[0217] HEVC (High Efficiency Video Coding): High-efficiency video coding

[0218] SEI (Supplementary Enhancement Information): Additional Enhancement Information

[0219] VUI (Video Usability Information): Video usability information

[0220] GOP (Group of Picture): Picture Group

[0221] TU (Transform Unit): Transform Unit

[0222] PU (Prediction Unit): Prediction Unit

[0223] CTU (Coding Tree Unit): Coding Tree Unit

[0224] CTB (Coding Tree Block): Coding Tree Block

[0225] PB(Prediction Block): Prediction Block

[0226] HRD (Hypothetical Reference Decoder): Virtual Reference Decoder

[0227] SNR (Signal Noise Ratio): Signal-noise ratio

[0228] CPU (Central Processing Unit): Central Processing Unit

[0229] GPU (Graphics Processing Unit): Graphics Processing Unit

[0230] CRT (Cathode Ray Tube): Cathode Ray Tube

[0231] LCD (Liquid-Crystal Display): Liquid Crystal Display

[0232] OLED (Organic Light-Emitting Diode): Organic Light-Emitting Diode

[0233] CD (Compact Disc): Compact Disc

[0234] DVD (Digital Video Disc): Digital Video Disc

[0235] ROM (Read-Only Memory): Read-only memory

[0236] RAM (Random Access Memory): Random Access Memory

[0237] ASIC (Application-Specific Integrated Circuit): Application-Specific Integrated Circuit

[0238] PLD (Programmable Logic Device): Programmable logic device

[0239] LAN (Local Area Network): Local Area Network

[0240] GSM (Global System for Mobile Communications): Global System for Mobile Communications

[0241] LTE (Long-Term Evolution): Long-Term Evolution

[0242] CANBus (Controller Area Network Bus): Controller Area Network Bus

[0243] USB (Universal Serial Bus): Universal Serial Bus

[0244] PCI (Peripheral Component Interconnect): Peripheral Component Interconnect

[0245] FPGA (Field Programmable Gate Areas): Field Programmable Gate Area

[0246] SSD (solid-state drive): Solid-state drive

[0247] IC (Integrated Circuit): Integrated Circuit

[0248] CU (Coding Unit): Coding Unit

[0249] Although the present disclosure describes some exemplary embodiments, there are modifications, permutations, and various alternative equivalents that fall within the scope of the disclosure. Accordingly, those skilled in the art will understand that numerous systems and methods can be created that implement the principles of the present disclosure and thus fall within the spirit and scope of the present disclosure, even though they are not expressly illustrated or described herein.

Claims

Claim 1 A method for video processing in a decoder, comprising: determining an initial block vector for predicting a current block in a current coding tree unit (CTU) in response to a current block being predicted in an intra block copy (IBC) mode by a processor; constructing an IBC merge candidate list, wherein the step of constructing the IBC merge candidate list comprises inserting the first block vector candidate into the IBC merge candidate list in response to a determination that the template matching search region of the first block vector candidate overlaps at least partially with an IBC reference region, wherein the first block vector candidate points to a first position outside the IBC reference region; performing template matching based on the initial block vector to determine a refined block vector pointing to a reference block in a picture as the current block by the processor; and reconstructing the current block based on the reference block by the processor. Claim 2 A method according to claim 1, further comprising the step of determining the initial block vector based on a merge index included in a coded video bitstream, wherein the merge index represents a block vector candidate within the IBC merge candidate list comprising a plurality of IBC candidates in an IBC merge mode. Claim 3 A method according to claim 2, further comprising: a step of parsing a first flag indicating the IBC merge mode from the coded video bitstream; and a step of parsing a second flag indicating whether the template matching is applied to the block vector candidate among the plurality of IBC candidates in the IBC merge mode indicated by the merge index from the coded video bitstream. Claim 4 A method according to claim 2, further comprising the step of parsing a flag indicating the IBC merge mode from the coded video bitstream, wherein the template matching is applied to block vector candidates of each block within the CTU predicted in the IBC merge mode. Claim 5 The method according to claim 2, wherein the step of determining the initial block vector based on the merge index comprises: determining the position closest to the first position within the IBC reference region in response to the merge index representing the first block vector candidate; and further comprising the step of determining the initial block vector to point to the closest position. Claim 6 A method according to claim 1, wherein the IBC reference area comprises a reconstructed portion of the current CTU and an area of ​​the left CTU, which are cached in a memory space of a size for storing the CTU. Claim 7 A method according to claim 1, wherein the step of performing the template matching further comprises the step of excluding a portion of the template matching search area outside the IBC reference area from the template matching. Claim 8 A device for video decoding, comprising a processing circuit configured to perform the method of any one of claims 1 to 7. Claim 9 A non-transient computer-readable storage medium storing instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 7. Claim 10 delete Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images