Method for performing intra block copy padding, decoding apparatus and storage medium

Abstract

The present disclosure relates generally to video coding, and more specifically to intra block copy (IBC) padding mode. A method for performing IBC padding is disclosed. The method includes receiving a coded video bitstream, extracting a first syntax element from the coded video bitstream, the first syntax element indicating that the IBC padding is available for a current video block, extracting a second syntax element from the coded video bitstream, the second syntax element indicating that the current video block is predicted using the IBC padding, extracting at least one syntax element from the coded video bitstream, partitioning the current video block into at least two sub-partitions based on the at least one syntax element, wherein the at least two sub-partitions include a padding sub-partition and an IBC sub-partition, predicting the padding sub-partition based on a predefined padding rule, and predicting the IBC sub-partition based on a predefined IBC rule.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 252,079, filed October 4, 2021. This application also claims priority to U.S. Non-Provisional Patent Application No. 17 / 943,449, filed September 13, 2022. The entire contents of the earlier applications are incorporated herein by reference. Technical Field

[0003] This disclosure describes a series of advanced video coding techniques. Specifically, the disclosed techniques relate to the implementation and enhancement of Intra Block Copy (IntraBC or IBC) with sample padding in video coding and decoding. Background Technology

[0004] The background description provided herein is for the purpose of presenting the general content of this disclosure. Within the scope described in this background section, neither the work of the currently named inventor nor any aspect of this description that could not otherwise qualify as prior art at the time of filing of this application is expressly or impliedly acknowledged as prior art to this disclosure.

[0005] Video encoding and decoding can be performed using inter-frame picture prediction with motion compensation. Uncompressed digital video can comprise a series of pictures, each with a spatial dimension of, for example, 1920x1080 luminance samples and associated full-color or subsampled chrominance samples. This series of pictures can have a fixed or variable picture rate (optionally referred to as the frame rate), such as 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, video with a pixel resolution of 1920x1080, a frame rate of 60 frames per second, and 4:2:0 chrominance subsampling at 8 bits per pixel per color channel requires a bandwidth close to 1.5 Gbit / s. One hour of such video would require more than 600 gigabytes of storage space.

[0006] One objective of video encoding and decoding is to reduce redundancy in the uncompressed input video signal through compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by two orders of magnitude or more. Lossless compression, lossy compression, and combinations thereof can be employed. Lossless compression refers to a technique that allows an exact copy of the original signal to be reconstructed from the compressed original signal through a decoding process. Lossy compression refers to an encoding / decoding process in which the original video information cannot be fully preserved during encoding and cannot be fully recovered during decoding. When using lossy compression, the reconstructed signal may differ from the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application despite some information loss. In the case of video, lossy compression is widely used in many applications. The tolerable amount of distortion depends on the application. For example, users of some consumer video streaming applications may tolerate higher distortion than users of film or television broadcasting applications. The compression ratio achieved by a particular encoding algorithm can be selected or adjusted to reflect various distortion tolerances: higher tolerable distortion generally allows the encoding algorithm to produce higher losses and higher compression ratios.

[0007] Video encoders and decoders can employ techniques from a wide range of categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0008] Video codec techniques can include a variety of techniques known as intra-frame coding. In intra-frame coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference images. In some video codecs, images are spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the image can be called an intra-frame picture. Intra-frame pictures and their derived pictures (e.g., independent decoder refresh pictures) can be used to reset the decoder state and thus can be used as the first picture in an encoded video stream and video session, or as a still picture. The samples of the intra-predicted block can then be frequency-domain transformed, and the resulting transform coefficients can be quantized before entropy coding. Intra-frame prediction represents a technique that minimizes the sample values ​​in the pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficients, the fewer bits are needed to represent the entropy-coded block for a given quantization step size.

[0009] Traditional intra-frame coding (such as intra-frame coding known from techniques like MPEG-2 generation coding) does not use intra-frame prediction. However, some newer video compression techniques include attempts to encode / decode blocks based on, for example, surrounding sample data and / or metadata, which precede the data blocks being intra-coded / decoded in the decoding order. Such techniques are referred to below as "intra-frame prediction" techniques. It is noteworthy that, at least in some cases, intra-frame prediction uses reference data from the current frame being reconstructed while other frames are reference data.

[0010] There can be many different forms of intra-prediction. When more than one such technique can be used in a given video coding technique, the technique used can be called an intra-prediction mode. A particular codec can have one or more intra-prediction modes. In some cases, a mode can have multiple sub-modes and / or can be associated with multiple parameters, and mode / sub-mode information and the internal coding parameters of the video block can be encoded separately or jointly contained in the mode codeword. The choice of codeword used for a given mode / sub-mode / parameter combination can affect the coding efficiency gain through intra-prediction, and the entropy coding techniques used to convert codewords into bitstreams are similar.

[0011] H.264 introduced an intra-prediction mode, which was improved in H.265 and further refined in newer coding techniques such as Joint Exploration Model (JEM), Universal Video Coding (VVC), and Baseline Set (BMS). Typically, for intra-prediction, prediction blocks are formed using neighboring sample values ​​that are already available samples. For example, available values ​​from a specific set of neighboring samples are copied into the prediction block along a specific direction and / or line. The reference to the direction used can be encoded in the bitstream or can be predicted itself.

[0012] refer to Figure 1A ,exist Figure 1A The lower right corner depicts a subset of nine prediction directions from the 33 possible prediction directions specified by H.265 (corresponding to the 33 angular modes of the 35 intra-frame modes specified by H.265). The point (101) where the arrows converge represents the sample being predicted. The arrows indicate the direction of the sample at point 101 predicted using neighboring samples. For example, arrow (102) indicates that sample (101) is predicted based on one or more neighboring samples to the upper right at a 45-degree angle to the horizontal direction. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more neighboring samples to the lower left of sample (101) at a 22.5-degree angle to the horizontal direction.

[0013] Still referencing Figure 1A ,exist Figure 1AThe upper left corner depicts a 4×4 square block (104) of samples (represented by a bold dashed line). The square block (104) comprises 16 samples, each labeled “S”, with its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (starting from the top) and the first sample in the X dimension (starting from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, sample S44 is located in the lower right corner. An exemplary reference sample following a similar numbering scheme is also shown. The reference sample is labeled R, with its Y position (e.g., row index) and X position (column index) relative to the block (104). In H.264 and H.265, predicted samples adjacent to the block being reconstructed will be used.

[0014] Intra-frame image prediction for block (104) can begin by copying reference sample values ​​from neighboring samples in the prediction direction indicated by a signal. For example, suppose the encoded video stream includes signaling for the prediction direction of block (104) relative to arrow (102), i.e., predicting samples based on one or more prediction samples at a 45-degree angle to the upper right of the horizontal direction. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Sample S44 is then predicted based on reference sample R08.

[0015] In some cases, especially when the predicted direction is not divisible by 45 degrees, the values ​​of multiple reference samples can be combined, for example, by interpolation, to calculate the reference sample.

[0016] With the development of video coding technology, the number of possible prediction directions has continuously increased. For example, in H.264 (2003), nine different directions were available for intra-frame prediction. This increased to 33 in H.265 (2013), while JEM / VVC / BMS, when publicly released, could support up to 65 directions. Experiments have been conducted to help identify the most suitable intra-frame prediction directions, and certain techniques in entropy coding can be used to encode those most suitable directions with a small number of bits, thus incurring a bit penalty for the direction. Furthermore, sometimes the direction itself can be predicted based on adjacent directions used in the intra-frame prediction of adjacent decoded blocks.

[0017] Figure 1B A schematic diagram (180) depicting 65 intra-frame prediction directions according to JEM is shown to illustrate the increase in the number of prediction directions over time in various coding techniques.

[0018] The mapping of intra-prediction direction bits representing direction in an encoded video bitstream to the predicted direction can vary depending on the video coding technique, and the mapping range can be, for example, a simple direct mapping from the predicted direction to the intra-prediction mode, to codewords, to complex adaptive schemes involving the most probable mode (and similar techniques). However, in all cases, some directions used for intra-prediction are statistically less likely to appear in the video content compared to other directions. Since the goal of video compression is to reduce redundancy, in well-designed video coding techniques, those less likely directions may require more bits to represent than the more likely directions.

[0019] Inter-frame or intra-frame image prediction may be based on motion compensation. In motion compensation, blocks of sample data from a previously reconstructed image or a portion thereof (the reference image), after being spatially offset along a direction indicated by a motion vector (hereafter referred to as MV), can be used to predict newly reconstructed images or image portions (e.g., blocks). In some cases, the reference image can be the same as the image currently being reconstructed. MV can have two dimensions, X and Y, or three dimensions, with the third dimension indicating the reference image being used (similar to a temporal dimension).

[0020] In some video compression techniques, the current MV applicable to a certain region can be predicted based on other MVs (e.g., those MVs that are spatially adjacent to the region being reconstructed and whose decoding order precedes that MV). Doing so eliminates redundancy in the related MVs, significantly reducing the total amount of data required to encode the MV, thus increasing the compression ratio. MV prediction can work effectively, for example, because when encoding the input video signal obtained from the camera (called natural video), there is a statistical probability that a larger region than the region applicable to a single MV moves in similar directions in the video sequence. Therefore, in some cases, this larger region can be predicted using similar motion vectors derived from the MVs of neighboring regions. This makes the actual MV of a given region similar to or the same as the MV predicted based on the surrounding MVs. Furthermore, after entropy coding, this inverse MV can be represented with fewer bits than when directly encoding the MV (rather than predicting from neighboring MVs). In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating predictions based on multiple surrounding MVs.

[0021] H.265 / HEVC (ITU-T H.265 Recommendation, “High Efficiency Video Coding”, December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms specified in H.265, this application describes the technique hereinafter referred to as “spatial combining”.

[0022] Specifically, refer to Figure 2 The current block (201) includes samples that have been discovered by the encoder during the motion search process and can be predicted based on previous blocks of the same size that have generated spatial offsets. Alternatively, the MV can be derived from metadata associated with one or more reference images, rather than being directly encoded. For example, using the MV associated with any of the five surrounding samples A0, A1, and B0, B1, B2 (corresponding to 202 to 206 respectively), the MV can be derived from metadata associated with one or more reference images (from the nearest reference image in decoding order). In H.265, MV prediction can use predictions from the same reference images that are also being used in adjacent blocks. Summary of the Invention

[0023] This disclosure describes various embodiments of methods, apparatuses, and computer-readable storage media for performing intra-block copy (IBC) padding in a current video block.

[0024] According to one aspect, embodiments of this disclosure provide a method for performing intra-block copy (IBC) padding in a current video block of a video stream. The method includes: receiving a current video block from an encoded video stream via a device, wherein a corresponding reference block partially matches the current video block. The device includes a memory storing instructions and a processor in communication with the memory. The method further includes: extracting a first syntax element from the encoded video stream via the device, the first syntax element indicating that IBC padding can be used for the current video block; extracting a second syntax element from the encoded video stream via the device, the second syntax element indicating that IBC padding is used to predict the current video block; extracting at least one syntax element from the encoded video stream via the device; segmenting the current video block into at least two sub-partitions based on the at least one syntax element via the device, wherein the at least two sub-partitions include a padding sub-partition and an IBC sub-partition; predicting the padding sub-partition based on a predefined padding rule via the device; and predicting the IBC sub-partition based on a predefined IBC rule via the device.

[0025] According to another aspect, embodiments of this disclosure provide an apparatus for performing intra-block copy (IBC) padding in a current video block. The apparatus includes a memory storing instructions and a processor communicating with the memory. When the processor executes the instructions, the processor is configured to cause the apparatus to perform the aforementioned video decoding and / or encoding methods.

[0026] According to another aspect, embodiments of this disclosure provide a non-transitory computer-readable storage medium for storing instructions, wherein, when executed by a computer, the instructions are configured to cause the computer to perform the aforementioned video decoding and / or encoding methods.

[0027] The above and other aspects, and their implementations, will be described in more detail in the drawings, specification, and claims. Attached Figure Description

[0028] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, wherein:

[0029] Figure 1A A schematic diagram illustrating an exemplary subset of intra-frame prediction modes.

[0030] Figure 1B A schematic diagram illustrating an exemplary intra-frame prediction direction is shown.

[0031] Figure 2 This diagram illustrates the current block based on an example and the surrounding spatial merging candidates used for motion vector prediction.

[0032] Figure 3 A schematic diagram showing a simplified block diagram of a communication system (300) according to an exemplary embodiment.

[0033] Figure 4 A schematic diagram showing a simplified block diagram of a communication system (400) according to an exemplary embodiment is shown.

[0034] Figure 5 A schematic diagram showing a simplified block diagram of a decoder according to an exemplary embodiment is provided.

[0035] Figure 6 A schematic diagram showing a simplified block diagram of an encoder according to an exemplary embodiment is provided.

[0036] Figure 7 A block diagram of a video encoder according to another exemplary embodiment is shown.

[0037] Figure 8 A block diagram of a video decoder according to another exemplary embodiment is shown.

[0038] Figure 9This illustrates a scheme for segmenting coded blocks according to an exemplary embodiment of the present disclosure.

[0039] Figure 10 Another scheme for coded block segmentation according to an exemplary embodiment of this disclosure is shown.

[0040] Figure 11 Another scheme for coded block segmentation according to an exemplary embodiment of this disclosure is shown.

[0041] Figure 12 This illustrates an example of dividing a basic block into coded blocks according to an exemplary segmentation scheme.

[0042] Figure 13 An exemplary ternary tree partitioning scheme is shown.

[0043] Figure 14 An exemplary scheme for quadtree-binary tree coding block segmentation is shown.

[0044] Figure 15 This illustration shows a scheme for dividing a coded block into pairs of transform blocks and the encoding order of the transform blocks according to exemplary embodiments of the present disclosure.

[0045] Figure 16 This illustrates another scheme for dividing a coded block into pairs of transform blocks according to exemplary embodiments of the present disclosure, and the encoding order of the transform blocks.

[0046] Figure 17 The figure illustrates another scheme for dividing a coded block into pairs of transform blocks according to an exemplary embodiment of the present disclosure.

[0047] Figure 18 This illustrates the concept of intra-block copy (IBC), which uses reconstructed coded blocks to predict the current coded block within the same frame.

[0048] Figure 19 An exemplary reconstructed sample is shown that can be used as a reference sample for IBC.

[0049] Figure 20 An exemplary reconstructed sample is shown that can be used as a reference sample for IBC, and the reconstructed sample has some exemplary limitations.

[0050] Figure 21 An example of an on-chip reference sample memory (RSM) update mechanism for IBC is shown.

[0051] Figure 22 Show Figure 21 Spatial view of an exemplary on-chip RSM update mechanism.

[0052] Figure 23This illustrates another example of an on-chip reference sample memory (RSM) update mechanism for IBC.

[0053] Figure 24 A comparison of spatial views of the exemplary RSM update mechanism for horizontally and vertically segmented superblocks used in IBC is shown.

[0054] Figure 25 A flowchart illustrating a method according to an exemplary embodiment of this disclosure is shown.

[0055] Figure 26A Some examples of partial IBC predictions are shown.

[0056] Figure 26B Some examples of partial IBC predictions are shown.

[0057] Figure 27 An example of a filled area in a video block is shown.

[0058] Figure 28 A schematic diagram of a computer system according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0059] The invention will now be described in detail with reference to the accompanying drawings, which form part of the invention and illustrate specific examples of embodiments by way of illustration. However, it should be noted that the invention can be embodied in various different forms, and therefore, the subject matter covered or claimed is intended to be construed as not being limited to any of the embodiments set forth below. It should also be noted that the invention can be embodied as a method, apparatus, component, or system. Therefore, embodiments of the invention can take the form of, for example, hardware, software, firmware, or any combination thereof.

[0060] Throughout the specification and claims, terms may have subtle meanings implied or implied in the context that go beyond their expressly stated meanings. The expressions “in one embodiment” or “in some embodiments” as used herein do not necessarily refer to the same embodiment, and the expressions “in another embodiment” or “in other embodiments” as used herein do not necessarily refer to different embodiments. Similarly, the expressions “in one implementation” or “in some implementations” as used herein do not necessarily refer to the same implementation, and the expressions “in another implementation” or “in other implementations” as used herein do not necessarily refer to different implementations. For example, the claimed subject matter includes combinations of all or part of the exemplary embodiments / implementations.

[0061] Generally, terms can be understood at least in part based on their usage in context. For example, terms such as “and,” “or,” or “and / or” as used herein can include a variety of meanings, which can depend at least in part on the context in which they are used. Typically, the term “or,” when used in an associative list (such as A, B, or C), is intended to mean A, B, and C (in an inclusive sense) and A, B, or C (in an exclusive sense). Furthermore, the terms “one or more” or “at least one” as used herein depend at least in part on the context and can be used to describe any feature, structure, or characteristic in a singular sense, or to describe a combination of features, structures, or characteristics in a plural sense. Similarly, terms such as “a,” “an,” or “described” can also be understood to express either a singular or plural usage, depending at least in part on the context. Additionally, terms such as “based on” or “determined by…” can be understood not necessarily to express a set of exclusive factors, but rather to allow for the existence of other factors that are not necessarily explicitly described, which also depends at least in part on the context.

[0062] Figure 3 This is a simplified block diagram of a communication system (300) according to an embodiment disclosed in this application. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first terminal device (310) and a second terminal device (320) interconnected via a network (350). Figure 3 In the example, the first terminal device (310) and the second terminal device (320) can perform unidirectional data transmission. For example, the first terminal device (310) can encode video data (e.g., a video image stream captured by the terminal device (310)) for transmission over a network (350) to the second terminal device (320). The encoded video data is transmitted in the form of one or more encoded video streams. The second terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to recover the video images, and display the video images based on the recovered video data. Unidirectional data transmission is implemented in applications such as media services.

[0063] In another example, the communication system (300) includes a third terminal device (330) and a fourth terminal device (340) that perform bidirectional transmission of encoded video data, which can be implemented, for example, during a video conferencing application. For bidirectional data transmission, in one example, each of the third terminal device (330) and the fourth terminal device (340) can encode video data (e.g., a stream of video images captured by the terminal device) for transmission over a network (350) to the other terminal device among the third terminal device (330) and the fourth terminal device (340). Each of the third terminal device (330) and the fourth terminal device (340) can also receive encoded video data transmitted by the other terminal device among the third terminal device (330) and the fourth terminal device (340), and can decode the encoded video data to recover video images, and can display these video images on an accessible display device based on the recovered video data.

[0064] exist Figure 3 In the examples, the first terminal device (310), the second terminal device (320), the third terminal device (330), and the fourth terminal device (340) can be servers, personal computers, and smartphones, but the adaptability of the basic principles disclosed in this application may not be limited thereto. The embodiments disclosed in this application can also be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, etc. Network (350) refers to any number or type of network that transmits encoded video data between the first terminal device (310), the second terminal device (320), the third terminal device (330), and the fourth terminal device (340), including, for example, wired (connected) and / or wireless communication networks. The communication network (350) can exchange data in circuit-switched, packet-switched, and / or other types of channels. The network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this application, unless explicitly explained below, the architecture and topology of the network (350) may be irrelevant to the operation of this application.

[0065] As an example of the application of the disclosed subject matter Figure 4 This illustration shows the placement of a video encoder and a video decoder in a streaming environment. The subject matter disclosed in this application is equally applicable to other video applications, including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0066] The video streaming system may include an acquisition subsystem (413) that may include a video source (401) such as a digital camera for creating uncompressed video pictures or image streams (402). In the example, the video picture stream (402) includes samples recorded by the digital camera of the video source (401). The video picture stream (402) is depicted as a thick line to emphasize the high data volume compared to encoded video data (404) (or encoded video bitstream), and the video picture stream (402) may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. Compared to the uncompressed video image stream (402), the encoded video data (404) (or the encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume. This encoded video data can be stored on a streaming server (405) for future use or directly stored on a downstream video device (not shown). One or more streaming client subsystems, such as Figure 4 Client subsystems (406) and (408) can access a streaming server (405) to retrieve copies (407) and (409) of encoded video data (404). Client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and produces an uncompressed output video picture stream (411) that can be displayed on a display (412) (e.g., a screen) or another presentation device (not depicted). The video decoder 410 may be configured to perform some or all of the functions described in this application. In some streaming systems, the encoded video data (404), video data (407), and video data (409) (e.g., video streams) may be encoded according to certain video encoding / compression standards. Examples of such standards include ITU-T H.265. In this example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). This application can be used in the context of the VVC standard and other video coding standards.

[0067] It should be noted that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0068] Figure 5This is a block diagram of a video decoder (510) according to any embodiment disclosed in this application below. The video decoder (510) may be disposed in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used in place of Figure 4 The video decoder (410) in the example.

[0069] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510). In the same embodiment or another embodiment, one encoded video sequence is decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Each video sequence can be associated with multiple video frames or images. The encoded video sequences can be received from a channel (501), which can be a hardware / software link to a storage device that stores the encoded video data or a video stream source that transmits the encoded video data. The receiver (531) can receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that can be forwarded to their respective processing circuits (not shown). The receiver (531) can separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (515) can be placed between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be located externally (not shown) to the video decoder (510) or separate from it. In other applications, a buffer memory (not shown) may be located externally to the video decoder (510) to, for example, prevent network jitter, and another buffer memory (515) may be configured internally to, for example, handle broadcast timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, a buffer memory (515) may not be necessary, or it may be made smaller. For use on packet networks such as the Internet, a buffer memory (515) of sufficient size may be required, and its size may be relatively large. The buffer memory may be implemented with an adaptive size and may be at least partially implemented in an operating system or a similar component (not shown) external to the video decoder (510).

[0070] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols may or may not be part of the electronic device (530), but may be coupled to the electronic device (530), such as... Figure 5 As shown in the figure. The control information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse / entropy decode the received encoded video sequence. The entropy coding of the encoded video sequence may be performed according to video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup from the encoded video sequence for use in the video decoder based on at least one parameter corresponding to the subgroup. The subgroup may include Group of Pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (520) can also extract information from the encoded video sequence, such as transform coefficients (Fourier transform coefficients), quantizer parameter values, motion vectors, etc.

[0071] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0072] Depending on the type of encoded video frames or portions thereof (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different processing or functional units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed from the encoded video sequence by the parser (520). For brevity, the flow of such subgroup control information between the parser (520) and the various processing or functional units described below is not described.

[0073] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these functional units interact closely with each other and can be integrated with one another. However, for the purpose of clearly describing the various functions of the disclosed subject matter, the conceptually subdivided functional units are used in the disclosure below of this application.

[0074] The first unit may include a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) may receive quantization transform coefficients as symbols (521) from the parser (520) and control information, including information indicating which inverse transform type to use, block size, quantization factor / parameter, and quantization scaling matrix. The scaler / inverse transform unit (551) may output a block containing sample values, which may be input into the aggregator (555).

[0075] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may generate blocks of the same size and shape as the blocks being reconstructed, using surrounding block information that has been reconstructed and stored in the current picture buffer (558). For example, the current picture buffer (558) buffers partially reconstructed current images and / or fully reconstructed current images. In some implementations, the aggregator (555) adds the predictive information generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) based on each sample.

[0076] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (553) can access the reference image memory (557) to extract samples for inter-frame prediction of the image. After motion compensation is performed on the extracted samples according to the symbol (521), these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (the output of unit (551) is called the residual sample or residual signal), thereby generating output sample information. The motion compensation prediction unit (553) can obtain the prediction samples from the address in the reference image memory (557) under the control of a motion vector, and the motion vector is available to the motion compensation prediction unit (553) in the form of the symbol (521), which, for example, includes X, Y components (motion) and a reference image component (time). Motion compensation may also include interpolation of sample values ​​extracted from a reference image memory (557) when using subsample precise motion vectors, and motion compensation may also be associated with a motion vector prediction mechanism, etc.

[0077] The output samples of the aggregator (555) can be employed by various loop filtering techniques in the loop filter unit (556). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream), which are used as symbols (521) from the parser (520) in the loop filter unit (556). However, video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values. Various types of loop filters can be included in the loop filter unit 556 in different orders, as will be described in further detail below.

[0078] The output of the loop filter unit (556) can be a sample stream, which can be output to the display device (512) and stored in the reference image memory (557) for subsequent inter-frame image prediction.

[0079] Once fully reconstructed, certain encoded images can be used as reference images for future inter-frame prediction. For example, once the encoded image corresponding to the current image has been fully reconstructed and the encoded image (by, for example, the parser (520)) is identified as the reference image, the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0080] The video decoder (510) can perform decoding operations according to a predetermined video compression technique, such as that used in the ITU-T Rec.H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under that configuration file. To conform to the standard, the complexity of the encoded video sequence may also be required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0081] In some exemplary embodiments, the receiver (531) may receive supplemental (redundant) data along with the encoded video. This supplemental data may be a portion of the encoded video sequence. The supplemental data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The supplemental data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0082] Figure 6 This is a block diagram of a video encoder (603) according to an embodiment disclosed in this application. The video encoder (603) may be disposed in an electronic device (620). The electronic device (620) may also include a transmitter (640) (e.g., a transmission circuit). The video encoder (603) may be used in place of Figure 4 The video encoder (403) in the embodiment.

[0083] The video encoder (603) can obtain data from the video source (601) (not) Figure 6 In one embodiment, a portion of the electronic device (620) receives video samples, the video source being capable of capturing video images to be encoded by a video encoder (603). In another example, the video source (601) may be implemented as part of the electronic device (620).

[0084] A video source (601) can provide a sequence of source video samples to be encoded by a video encoder (603) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, XYZ, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) can be a storage device capable of storing previously prepared video. In a video conferencing system, the video source (601) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures or images, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc., used. The relationship between pixels and samples can be easily understood by those skilled in the art. The following focuses on describing samples.

[0085] According to some exemplary embodiments, a video encoder (603) can encode and compress images of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate constitutes a function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller (650) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be used with other suitable functions related to a video encoder (603) optimized for a particular system design.

[0086] In some exemplary embodiments, the video encoder (603) may be configured to operate within an encoding loop. As a simplified description, in an embodiment, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (633) embedded within the video encoder (603). Even if the embedded decoder 633 does not entropy encode the video stream through the source encoder 630, the decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered in this application, any compression between the symbols in entropy encoding and the encoded video stream may be lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). Since decoding of the symbol stream produces bit-precise results independent of the decoder's location (local or remote), the contents of the reference image memory (634) are also bit-precisely corresponding between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is used to improve coding quality.

[0087] The operation of the “local” decoder (633) can be combined with, for example, the above-described... Figure 5 The video decoder (510) is described in detail as the same as the "remote" decoder. However, a further brief reference is provided. Figure 5 When symbols are available and the entropy encoder (645) and parser (520) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (510), including the buffer (515) and parser (520), may not be fully implemented in the encoder’s local decoder (633).

[0088] It can be observed that any decoder technique, other than parsing / entropy decoding which may only exist in the decoder, may also necessarily exist in the corresponding encoder in a substantially identical functional form. For this reason, the subject matter disclosed in this application sometimes focuses on decoder operations that are associated with the decoding portion of the encoder. Therefore, the description of the encoder technique can be simplified, as the encoder technique is the inverse of the fully described decoder technique. A more detailed description of the encoder is only required in certain areas or aspects, and is provided below.

[0089] During operation, in some exemplary embodiments, the source encoder (630) may perform motion-compensated predictive coding. This motion-compensated predictive coding predictively encodes the input image, referencing one or more previously encoded images from the video sequence designated as "reference images." In this manner, the encoding engine (632) encodes the differences (residuals) in color channels between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image. The term "residual" and its adjective form "residual" are used interchangeably.

[0090] The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture, based on symbols created by the source encoder (630). The operation of the encoding engine (632) can be a lossy process. When the encoded video data can be decoded by the video decoder (633), Figure 6 When the source video sequence (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process, which can be performed by the video decoder on the reference image, and allows the reconstructed reference image to be stored in a reference image cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference image that shares the same content (no transmission errors) as the reconstructed reference image to be obtained by the remote video decoder.

[0091] The predictor (635) can perform a prediction search against the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search in the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (635) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (635), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (634).

[0092] The controller (650) can manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0093] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by various functional units according to techniques such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0094] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0095] The controller (650) manages the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:

[0096] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Variations of I-pictures and their corresponding applications and characteristics are familiar to those skilled in the art.

[0097] A predictive picture (P-picture) can be a picture that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0098] A bidirectional predictive picture (B-picture) can be a picture that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

[0099] Source images are typically spatially subdivided into multiple sample coding blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and coded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined by the coding assignment of the corresponding images applied to the block. For example, a block of an I image can be non-predictively coded, or it can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. A pixel block of a P image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. A block of a B image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction. For other purposes, source images or intermediate images can be subdivided into other types of blocks. The partitioning of coding blocks and other types of blocks may or may not follow the same pattern, as described in detail below.

[0100] The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0101] In some exemplary embodiments, the transmitter (640) may transmit both encoded data and additional data. The source encoder (630) may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, redundant images and slices, other forms of redundant data, SEI messages, VUI parameter set fragments, etc.

[0102] The captured video can be presented as multiple source images (video images) in a time-series format. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. For example, a specific image being encoded / decoded can be segmented into blocks; this specific image being encoded / decoded is called the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when using multiple reference images, the motion vector can have a third dimension that identifies the reference image.

[0103] In some exemplary embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to such bidirectional prediction techniques, two reference images are used, such as a first reference image and a second reference image, both preceding the current image in the video in decoding order (but possibly past and future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block can be jointly predicted using a combination of the first and second reference blocks.

[0104] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0105] According to some exemplary embodiments disclosed in this application, predictions such as inter-frame image prediction and intra-frame image prediction are performed on a block-by-block basis. For example, images in a video image sequence are segmented into coding tree units (CTUs) for compression. The CTUs in the images have the same size, such as 128×128 pixels, 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU may include three parallel coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Furthermore, each CTU may be segmented into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU may be segmented into one 64×64 pixel CU, or four 32×32 pixel CUs. Each or more of these 32×32 blocks may be further segmented into four 16×16 pixel CUs. In some exemplary embodiments, each CU can be analyzed during encoding to determine the prediction type for the CU from various prediction types, such as inter-frame prediction or intra-frame prediction. Depending on temporal and / or spatial predictability, the CU can be segmented into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in encoding (encoding / decoding) are performed on a per-prediction-block basis. The CU can be segmented into PUs (or PBs for different color channels) in different spatial patterns. For example, the luma PB and chroma PB can include matrices of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0106] Figure 7This is a diagram of a video encoder (703) according to another exemplary embodiment disclosed in this application. The video encoder (703) is used to receive a processing block (e.g., a prediction block) of sample values ​​within a current video image in a video image sequence, and to encode the processing block into an encoded image that is part of an encoded video sequence. The exemplary video encoder (703) can be used instead of Figure 4 The video encoder (403) in the example.

[0107] For example, a video encoder (703) receives a matrix of sample values ​​for a processing block, such as an 8×8 sample prediction block. The video encoder (703) then uses, for example, rate-distortion optimization (RDO) to determine whether to encode the processing block using intra-frame mode, inter-frame mode, or bidirectional prediction mode. When it is determined that the processing block is to be encoded in intra-frame mode, the video encoder (703) can use intra-frame prediction techniques to encode the processing block into an encoded picture; and when it is determined that the processing block is to be encoded in inter-frame mode or bidirectional prediction mode, the video encoder (703) can use inter-frame prediction or bidirectional prediction techniques to encode the processing block into an encoded picture, respectively. In some exemplary embodiments, a merging mode can be used as a sub-mode for inter-frame picture prediction, wherein motion vectors are derived from one or more motion vector prediction values ​​without relying on encoded motion vector components outside the prediction values. In some exemplary embodiments, motion vector components applicable to the subject block may exist. Therefore, the video encoder (703) may include Figure 7 Other components not explicitly shown include, for example, the pattern decision module used to determine the prediction pattern of the processing block.

[0108] exist Figure 7 In one embodiment, the video encoder (703) includes, as shown below: Figure 7 An exemplary arrangement shows an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0109] An inter-frame encoder (730) is used to receive samples of the current block (e.g., the processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and subsequent images in display order), generate inter-frame prediction information (e.g., descriptions of redundancy information, motion vectors, and merging mode information based on inter-frame coding techniques), and calculate inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is used using embedded... Figure 6 Decoding unit 633 in example encoder 620 (e.g. Figure 7(As shown in the residual decoder 728 in the image, which will be described in further detail below) Decoded reference image based on encoded video information.

[0110] The intra encoder (722) is used to receive samples of the current block (e.g., the processing block), compare the block with encoded blocks in the same image, generate quantization coefficients after transformation, and in some cases, generate intra prediction information (e.g., based on intra prediction direction information of one or more intra coding techniques). The intra encoder (722) can also calculate intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same image.

[0111] A general controller (721) can be used to determine general control data and, based on this general control data, control other components of the video encoder (703). In an example, the general controller (721) determines the prediction mode of a block and, based on this prediction mode, provides control signals to a switch (726). For example, when the prediction mode is an intra-frame mode, the general controller (721) controls the switch (726) to select the intra-frame mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-frame prediction information and add the intra-frame prediction information to the bitstream; and when the prediction mode for the block is an inter-frame mode, the general controller (721) controls the switch (726) to select the inter-frame prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-frame prediction information and add the inter-frame prediction information to the bitstream.

[0112] A residual calculator (723) can be used to calculate the difference (residual data) between the received block and the prediction result of a block selected from the intra encoder (722) or the inter encoder (730). A residual encoder (724) can be used to encode the residual data to generate transform coefficients. For example, the residual encoder (724) can be used to transform the residual data from the time domain to the frequency domain to generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various examples, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform the inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, and in some embodiments, the decoded image may be buffered in a memory circuit (not shown) and used as a reference image.

[0113] An entropy encoder (725) can be used to format the bitstream to produce encoded blocks and to perform entropy encoding. The entropy encoder (725) is used to generate various information within the bitstream. For example, the entropy encoder (725) can be used to obtain general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information within the bitstream. Residual information may not be available when encoding blocks in inter-frame mode or a merged sub-mode of bidirectional prediction mode.

[0114] Figure 8 This is a diagram of an exemplary video decoder (810) according to another embodiment disclosed in this application. The video decoder (810) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In the example, the video decoder (810) can be used instead of Figure 4 The video decoder (410) in the example.

[0115] exist Figure 8 In the example, the video decoder (810) includes, for example, Figure 8 An exemplary arrangement shows an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together.

[0116] An entropy decoder (871) can be used to reconstruct certain symbols from an encoded image, representing the syntax elements constituting the encoded image. Such symbols may include, for example, the mode used to encode the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, merged sub-mode, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can be identified for prediction by either the intra-frame decoder (872) or the inter-frame decoder (880), residual information in the form of, for example, quantized transform coefficients, and so on. In the example, when the prediction mode is inter-frame or bidirectional prediction mode, inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is intra-frame prediction type, intra-frame prediction information is provided to the intra-frame decoder (872). Residual information may be provided to the residual decoder (873) via inverse quantization.

[0117] The inter-frame decoder (880) can be used to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

[0118] The intra-frame decoder (872) can be used to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

[0119] The residual decoder (873) can be used to perform inverse quantization to extract the dequantized transform coefficients, and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) can also utilize certain control information (to obtain the quantizer parameter (QP)), and this information can be provided by the entropy decoder (871) (the data path is not indicated because this is only low-level control information).

[0120] The reconstruction module (874) can be used to combine the residual output by the residual decoder (873) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which may be part of a reconstructed image, which in turn may be part of a reconstructed video. It should be noted that other suitable operations, such as deblocking, can also be performed to improve visual quality.

[0121] It should be noted that any suitable technology can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In some exemplary embodiments, one or more integrated circuits can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In another embodiment, one or more processors executing software instructions can be used to implement the video encoder (403), video encoder (603), and video encoder (603), as well as the video decoder (410), video decoder (510), and video decoder (810).

[0122] Moving on to block partitioning for encoding and decoding, partitioning typically begins with a basic block and can follow a predefined set of rules, a specific pattern, a partition tree, or any partitioning structure or scheme. Partitioning can be hierarchical and recursive. After partitioning or segmenting the basic block according to any of the exemplary partitioning processes or other processes or combinations thereof described below, a final set of partitions or coding blocks is obtained. Each of these partitions can reside at one of the different partitioning levels in the hierarchical partitioning structure and can have a different shape. Each partition can be referred to as a coding block (CB). The implementation of various exemplary partitions is further described below; each generated CB can have any allowed size and partitioning level. Such partitions can be called coding blocks because they can form units against which some basic encoding / decoding decisions can be made, and encoding / decoding parameters can be optimized, determined, and signaled within the encoded video stream. The highest or deepest level in the final partition represents the depth of the coding block partitioning structure of the tree. Coding blocks can be luma coding blocks or chroma coding blocks. The CB tree structure for each color can be referred to as a coding block tree (CBT).

[0123] The coding blocks for all color channels can be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels can be collectively referred to as a coding tree unit (CTU). The segmentation patterns or structures of various color channels in a CTU may be the same or different.

[0124] In some implementations, the partition tree scheme or structure used for the luma and chroma channels may not need to be the same. In other words, the luma and chroma channels can have separate coding tree structures or patterns. Furthermore, whether the luma and chroma channels use the same or different coding partition tree structures, and the actual coding partition tree structure to be used, can depend on whether the slice being encoded is a P-slice, a B-slice, or an I-slice. For example, for an I-slice, the chroma and luma channels may have separate coding partition tree structures or coding partition tree structure patterns, while for a P-slice or a B-slice, the luma and chroma channels can share the same coding partition tree scheme. When using separate coding partition tree structures or patterns, the luma channel can be partitioned into multiple CBs using one coding partition tree structure, and the chroma channel can be partitioned into multiple chroma CBs using another coding partition tree structure.

[0125] In some exemplary implementations, a predetermined segmentation pattern can be applied to the basic blocks. For example... Figure 9As shown, an exemplary 4-way partitioning tree can begin at a first predefined level (e.g., a 64×64 block level or other size, as the basic block size), and the basic blocks can be hierarchically partitioned down to a predefined lowest level (e.g., a 4×4 level). For example, the basic blocks can be subject to four predefined partitioning options or partitioning patterns represented by 902, 904, 906, and 908, where a partition denoted as R is allowed for recursive partitioning, repeating at lower scales as shown. Figure 9 The same segmentation options are shown, up to the lowest level (e.g., 4x4 level). In some implementations, this can be... Figure 9 Additional constraints are imposed on the partitioning scheme. Figure 9 The implementation may allow rectangular partitioning (e.g., 1:2 / 2:1 rectangular partitioning), but may not allow these rectangular partitions to be recursive; instead, it may allow square partitioning to be recursive. If necessary, according to... Figure 9 The recursive segmentation method generates a final set of encoded blocks. The encoding tree depth can be further defined to indicate the segmentation depth starting from the root node or root block. For example, the encoding tree depth of the root node or root block (e.g., a 64×64 block) can be set to 0, and the segmentation depth can be further defined according to the recursive segmentation method. Figure 9 After further partitioning the root block, the coding tree depth increases by 1. For the above scheme, the maximum or deepest level from the 64×64 basic block to the 4×4 minimum partition will be 4 (starting from level 0). This partitioning scheme can be applied to one or more color channels. It can be followed as... Figure 9 The scheme independently splits each color channel (e.g., for each color channel at each layer level, the splitting mode or splitting option in a predefined pattern can be determined independently). Alternatively, two or more color channels can share... Figure 9 The same hierarchical pattern tree (e.g., for two or more color channels at each hierarchical level, the same splitting pattern or splitting option in a predefined pattern can be selected).

[0126] Figure 10 Another exemplary predefined splitting pattern is shown, which allows recursive splitting to form a split tree. For example... Figure 10 As shown, an exemplary 10-way partitioning structure or partitioning pattern can be predefined. The root block can start at a predefined level (e.g., starting with a basic block at a 128×128 level or a 64×64 level). Figure 10 Exemplary partitioning structures include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. These can be... Figure 10The partition types 1002, 1004, 1006, and 1008, shown in the second row, with three sub-partitions, are referred to as "T-type" partitions. These "T-type" partitions 1002, 1004, 1006, and 1008 can be respectively called left T-type, top T-type, right T-type, and bottom T-type. In some exemplary embodiments, partitions are not allowed... Figure 10 Any of the rectangular partitions can be further subdivided. The encoding tree depth can be further defined to indicate the segmentation depth starting from the root node or root block. For example, the encoding tree depth of the root node or root block (e.g., a 128×128 block) can be set to 0, and in accordance with... Figure 10 After further segmenting the root block, the coding tree depth increases by 1. In some implementations, only all square partitions within 1010 are allowed to be processed according to... Figure 10 The pattern is recursively segmented to the next level of the segmentation tree. In other words, recursively segmenting the square partitions within T-shaped models 1002, 1004, 1006, and 1008 may not be allowed. See [reference needed] if necessary. Figure 10 The recursive segmentation process generates a final set of coded blocks. This scheme can be applied to one or more color channels. In some implementations, more flexibility can be added to the use of partitions below 8×8 levels. For example, in some cases, 2×2 chroma inter-frame prediction can be used.

[0127] In some other exemplary implementations of coded block segmentation, quadtree structures can be used to segment basic blocks or intermediate blocks into quadtree partitions. This quadtree segmentation can be applied hierarchically and recursively to any square partition. Whether further quadtree segmentation is performed on the basic block or the intermediate block / partition, it can be adapted to various local features of the basic block or intermediate block / partition. Quadtree segmentation can be further adjusted at image boundaries. For example, implicit quadtree segmentation can be performed at image boundaries such that the block will maintain its quadtree segmentation until its size fits the image boundaries.

[0128] In some other exemplary embodiments, a hierarchical binary tree partitioning starting from a base block can be used. For this approach, the base block or intermediate block can be partitioned into two partitions. The binary tree partitioning can be horizontal or vertical. For example, a horizontal binary tree partitioning can partition the base block or intermediate block into equal right and left partitions. Similarly, a vertical binary tree partitioning can partition the base block or intermediate block into equal upper and lower partitions. This binary tree partitioning can be hierarchical and recursive. At each point in the base block or intermediate block, it can be determined whether the binary tree partitioning scheme should continue, and if so, whether to use a horizontal or vertical binary tree partitioning. In some embodiments, further partitioning can stop at a predefined minimum partition size (in one or two dimensions). Alternatively, further partitioning can stop once a predefined partitioning level or depth starting from the base block is reached. In some embodiments, the aspect ratio of the partitions can be limited. For example, the aspect ratio of the partitions can be no less than 1:4 (or greater than 4:1). Therefore, a vertical strip partition with a vertical-to-horizontal aspect ratio of 4:1 can be further divided into an upper partition and a lower partition by a vertical binary tree, each of which has a vertical-to-horizontal aspect ratio of 2:1.

[0129] In some other examples, such as Figure 13 As shown, the ternary tree partitioning scheme can be used to partition basic blocks or any intermediate blocks. The ternary tree pattern can be implemented vertically (e.g., ...). Figure 13 As shown in 1302), or horizontally implement the ternary tree pattern (such as...). Figure 13 (As shown in 1304). Although Figure 13 The exemplary partitioning ratio is shown vertically or horizontally as 1:2:1, but other ratios can be predefined. In some implementations, two or more different ratios can be predefined. This ternary tree partitioning scheme can be used to complement quadtree or binary tree partitioning structures because it captures objects located at the block center in a neighboring partition, whereas quadtrees and binary trees always partition along the block center, thus dividing objects into separate partitions. In some implementations, the width and height of the partitions in the exemplary ternary tree are always powers of 2 to avoid additional transformations.

[0130] The above partitioning schemes can be combined in any way at different partitioning levels. As an example, the quadtree partitioning scheme and the binary tree partitioning scheme can be combined to partition the basic block into a quadtree-binary-tree (QTBT) structure. In this scheme, if a set of predefined conditions is specified, the basic block or intermediate block / partition can be partitioned into a quadtree or a binary tree. Figure 14 A specific example is shown in the text. Figure 14 In the example, the basic block is first divided into four partitions using a first quadtree partition, as shown in 1402, 1404, 1406, and 1408. Subsequently, each of the resulting partitions is either partitioned by the quadtree into four further partitions (e.g., 1408), or by the binary tree at the next level into two further partitions (horizontal or vertical partitions, e.g., 1402 or 1406, both symmetrical), or not partitioned at all (e.g., 1404). Recursive binary or quadtree partitioning can be allowed for square partitions, as shown in the overall exemplary partitioning pattern of 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree partitioning and dashed lines represent binary tree partitioning. Flags can be used for each binary tree partition node (non-leaf binary tree partition) to indicate whether the binary tree partition is horizontal or vertical. For example, as shown in 1420, consistent with the partitioning structure of 1410, the flag "0" can indicate a horizontal binary tree partition, and the flag "1" can indicate a vertical binary tree partition. For quadtree partitions, there is no need to indicate the partition type, because quadtree partitions always divide blocks or partitions horizontally and vertically to generate four equal-sized sub-blocks / sub-partitions. In some implementations, the flag "1" can indicate a horizontal binary tree partition, and the flag "0" can indicate a vertical binary tree partition.

[0131] In some exemplary implementations of QTBT, the quadtree partitioning rule set and the binary tree partitioning rule set can be represented by the following predefined parameters and their associated corresponding functions:

[0132] -CTU size: The size of the root node of the quadtree (the size of the basic block).

[0133] -MinQTSize: The minimum allowed size of a quadtree leaf node.

[0134] -MaxBTSize: The maximum allowed size of the root node of the binary tree.

[0135] -MaxBTDepth: Maximum allowed binary tree depth

[0136] -MinBTSize: The minimum allowed size of a binary leaf node.

[0137] In some example implementations of the QTBT segmentation structure, the CTU size can be set to 128×128 luminance samples with two corresponding 64×64 chrominance sample blocks (when considering and using exemplary chrominance subsampling), the MinQTSize can be set to 16×16, the MaxBTSize can be set to 64×64, the MinBTSize (for width and height) can be set to 4×4, and the MaxBTDepth can be set to 4. The CTU can first be quadtree-segmented to generate quadtree leaf nodes. The size of the quadtree leaf nodes can range from the minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., the CTU size). If a node is 128×128, it will not be binarytree-segmented first because its size exceeds the MaxBTSize (i.e., 64×64). Otherwise, nodes not exceeding the MaxBTSize can be binarytree-segmented. Figure 14 In the example, the basic block is 128×128. According to the predefined rule set, only quadtree partitioning is possible on the basic block. The partition depth of the basic block is 0. Each of the four generated partitions is 64×64, not exceeding MaxBTSize, and can be further quadtree or binary tree partitioned at level 1. This process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), further partitioning can be considered. When the width of a binary tree node equals MinBTSize (i.e., 4), further horizontal partitioning can be considered. Similarly, when the height of a binary tree node equals MinBTSize, further vertical partitioning can be considered.

[0138] In some exemplary embodiments, the QTBT scheme described above can be configured to support the flexibility of having the same QTBT structure for luma and chroma, or separate QTBT structures. For example, for P-slices and B-slices, the luma CTB and chroma CTB in a CTU can share the same QTBT structure. However, for I-slices, the luma CTB can be segmented into CBs using a QTBT structure, and the chroma CTB can be segmented into chroma CBs using another QTBT structure. This means that a CU can be used to indicate different color channels in an I-slice; for example, an I-slice can be composed of coded blocks of the luma component or coded blocks of the two chroma components, and a CU in a P-slice or B-slice can be composed of coded blocks of all three color components.

[0139] In some other implementations, the ternary tree scheme described above can be used to supplement the QTBT scheme. Such implementations can be referred to as multi-type-tree (MTT) structures. For example, in addition to binary tree partitioning of nodes, alternatives can be selected... Figure 13This is one of the ternary tree partitioning modes. In some implementations, ternary tree partitioning can only be performed on square nodes. Additional flags can be used to indicate whether the ternary tree partition is horizontal or vertical.

[0140] The primary motivation for driving the design of two-level or multi-level trees is likely to reduce complexity, as seen in implementations of QTBTs and QTBTs supplemented by ternary tree partitioning. Theoretically, the complexity of traversing a tree is T. D , where T represents the number of partition types and D is the depth of the tree. A trade-off can be made by using multiple types (T) while reducing the depth (D).

[0141] In some implementations, the CB can be further segmented. For example, to perform intra-frame or inter-frame prediction during encoding and decoding, the CB can be further segmented into prediction blocks (PBs). In other words, the CB can be further segmented into different sub-partitions, where separate prediction decisions / prediction configurations can be made. In parallel, to depict the levels at which transforms or inverse transforms of the video data are performed, the CB can be further segmented into transform blocks (TBs). The schemes for segmenting the CB into PBs and TBs can be the same or different. For example, each segmentation scheme can be performed using its own process based on, for example, various characteristics of the video data. In some exemplary implementations, the PB segmentation scheme and the TB segmentation scheme can be independent. In some other exemplary implementations, the PB segmentation scheme and the TB segmentation scheme and boundaries can be related. In some implementations, for example, TBs can be segmented after PBs are segmented, and specifically, each PB can be determined after the segmentation of the coded blocks, and then each PB can be further segmented into one or more TBs. For example, in some implementations, a PB can be segmented into one, two, four, or other numbers of TBs.

[0142] In some implementations, different processing can be applied to the luma and chroma channels to segment the basic block into coded blocks and further into prediction and / or transform blocks. For example, in some implementations, segmenting the coded block into prediction and / or transform blocks is permitted for the luma channel, while not for the chroma channel. Therefore, in such implementations, transformation and / or prediction of the luma block can be performed only at the coded block level. For another example, the minimum transform block size for the luma and chroma channels can be different; for example, segmenting the coded block for the luma channel into smaller transform and / or predict blocks than the chroma channel can be permitted. For yet another example, the maximum depth at which the coded block is segmented into transform and / or predict blocks may differ between the luma and chroma channels; for example, segmenting the coded block for the luma channel into deeper transform and / or predict blocks than the chroma channel can be permitted. For a specific example, the luma-coded block can be divided into transform blocks of multiple sizes, which can be represented by recursive segmentation down to a maximum of two levels, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, as well as transform block sizes from 4×4 to 64×64, are allowed. However, for the chroma block, only the largest possible transform block can be specified for the luma block.

[0143] In some exemplary implementations of segmenting a coded block into a PB, the depth, shape, and / or other characteristics of the PB segmentation may depend on whether the PB is intra-frame coded or inter-frame coded.

[0144] The segmentation of a coded block (or prediction block) into transform blocks can be implemented in various exemplary schemes, including but not limited to: recursive or non-recursive quadtree segmentation and predefined pattern segmentation, with the additional consideration of transform blocks at the boundaries of the coded or prediction blocks. Typically, the generated transform blocks may be at different segmentation levels, may vary in size, and may not need to be square (e.g., these transform blocks can be rectangles with allowed sizes and aspect ratios). The following will combine... Figure 15 , 16 and Figure 17 Further examples will be described in more detail.

[0145] However, in some other implementations, the CB obtained through any of the above-described segmentation schemes can be used as a basic coding block or a minimum coding block for prediction and / or transformation. In other words, no further segmentation will be performed for the purposes of inter-frame prediction / intra-frame prediction and / or transformation. For example, the CB obtained from the above-described QTBT scheme can be directly used as the unit for performing prediction. Specifically, this QTBT structure eliminates the concept of multiple partition types, that is, it eliminates the separation of CU, PU, ​​and TU, and supports greater flexibility in the shape of the CU / CB partitions as described above. In this QTBT block structure, the shape of the CU / CB can be square or rectangular. The leaf nodes of this QTBT can be used as units for prediction and transformation processing without any further segmentation. This means that in this exemplary QTBT coding block structure, the CU, PU, ​​and TU have the same block size.

[0146] The various CB segmentation schemes described above can be combined in any way, and the CB can be further segmented into PB and / or TB (including without PB / TB segmentation). The following specific implementations are provided as non-limiting examples.

[0147] The following describes specific exemplary implementations of coded block and transform block segmentation. In such exemplary implementations, recursive quadtree segmentation or the predefined segmentation patterns described above (e.g., Figure 9 and Figure 10 (As shown) The basic block is divided into multiple coding blocks. At each level, the characteristics of the local video data can determine whether further quadtree partitioning of a particular partition should continue. The resulting multiple CBs can be at different quadtree partitioning levels and can have different sizes. A decision can be made at the CB level (or CU level, for all three color channels) whether to use inter-picture (temporal) prediction or intra-picture (spatial) prediction to encode the picture region. Each CB can be further divided into one, two, four, or other numbers of PBs according to a predefined PB partitioning type. Within a PB, the same prediction process can be applied, and relevant information can be sent to the decoder based on the PB. After obtaining the remaining blocks by applying the prediction process based on the PB partitioning type, the CB can be partitioned into multiple TBs according to another quadtree structure similar to the CB coding tree. In this particular implementation, the CB or TB can be, but is not limited to, squares. Furthermore, in this particular example, for inter-frame prediction, the PB can be square or rectangular, while for intra-frame prediction, the PB can only be square. The coding block can be divided into, for example, four square TBs. Each TB can be further recursively partitioned (using quadtree partitioning) into smaller TBs, called a Residual QuadTree (RQT).

[0148] The following describes another exemplary implementation of dividing a basic block into multiple CBs, PBs, and / or TBs. For example, instead of using methods such as Figure 9 or Figure 10 Instead of the segmentation unit types shown, a quadtree structure with nested multi-type trees using binary and ternary tree segmentation structures can be used (e.g., QTBT as described above or QTBT with ternary tree segmentation). Unless the size of multiple CBs is too large for the maximum transform length, the separation of CBs, PBs, and TBs can be abandoned (i.e., segmenting a CB into multiple PBs and / or multiple TBs, and segmenting multiple PBs into multiple TBs), where further segmentation of such CBs may be necessary. This exemplary segmentation scheme can be designed to support greater flexibility in the shape of CB segmentations, allowing prediction and transformation to be performed at the CB level without further segmentation. In this coding tree structure, the shape of the CB can be square or rectangular. Specifically, the coding tree block (CTB) can be segmented first by a quadtree structure. Then, the leaf nodes of the quadtree can be further segmented using nested multi-type tree structures. Figure 11 The example shown is a nested multi-type tree structure using binary and ternary tree partitioning structures. Specifically, Figure 11 The exemplary multi-type tree structure includes four partition types, referred to as vertical binary tree partition (SPLIT_BT_VER) (1102), horizontal binary tree partition (SPLIT_BT_HOR) (1104), vertical ternary tree partition (SPLIT_TT_VER) (1106), and horizontal ternary tree partition (SPLIT_TT_HOR) (1108). Multiple CBs then correspond to the leaves of the multi-type tree. In this exemplary implementation, unless the CB is too large for the maximum transform length, the partition will be used for prediction and transform processes without any further partitioning. This means that, in most cases, the CB, PB, and TB have the same block size in the quadtree of the nested multi-type tree coding block structure. An anomaly will occur when the maximum supported transform length is less than the width or height of the color component of the CB. In some implementations, in addition to binary and ternary tree partitions, Figure 11 Nested patterns can also include quadtree partitioning.

[0149] Figure 12 The diagram illustrates a concrete example of a quadtree (including quadtree splitting, binary tree splitting, and ternary tree splitting options) with a nested multi-type tree-coded block structure where a basic block has block partitions. More detailed... Figure 12 The diagram illustrates the partitioning of the basic block 1200 quadtree into four square partitions: 1202, 1204, 1206, and 1208. For each of the partitions generated by the quadtree partitioning, further uses can be determined. Figure 11Further segmentation is performed on various types of tree structures and quadtrees. In Figure 12 In the example, partition 1204 is not further divided. Partitions 1202 and 1208 are divided using another quadtree partitioning method. In partition 1202, the upper-left partition generated from the second-level quadtree partitioning is divided using a third-level quadtree partitioning method, and the upper-right partition generated from the second-level quadtree partitioning is divided using a third-level quadtree partitioning method. Figure 11 The horizontal binary tree partitioning 1104, and the lower left partition generated by the second-level quadtree partitioning, are used... Figure 11 The non-segmentation in the middle, and the lower right partition generated by the segmentation of the second-level quadtree, are adopted Figure 11 The horizontal ternary tree in the middle is used to partition 1108. Another quadtree is used to partition partition 1208, and the upper left partition of the second-level quadtree partition is partitioned using the third-level quadtree. Figure 11 The vertical ternary tree partitioning 1106, and the upper right partitioning of the second-level quadtree partitioning are performed using... Figure 11 The non-splitting method in the middle, and the lower left partition of the second-level quadtree partitioning are used. Figure 11 The non-splitting in the middle, and the lower right partition of the second-level quadtree partition, are adopted. Figure 11 The horizontal binary tree in the diagram is divided into 1104 segments. These segments are based on... Figure 11 The horizontal binary tree partition 1104 and the horizontal ternary tree partition 1108 are used to further partition the two sub-partitions of the upper left partition 1208 at the third level. This is followed by... Figure 11 The second-level partitioning pattern after the vertical binary tree partition 1102 divides partition 1206 into two partitions, and then according to... Figure 11 The horizontal ternary tree partitioning 1108 and the vertical binary tree partitioning 1102 are used to perform a third-level partitioning on the two generated partitions. According to... Figure 11 The horizontal binary tree is divided into 1104 partitions, and then a fourth-level partition is applied to one of the resulting partitions.

[0150] For the specific example above, the maximum luminance transformation size can be 64×64, and the maximum supported chrominance transformation size can differ from, for example, the luminance at 32×32. Even if not typically... Figure 12 The aforementioned exemplary CB is further divided into smaller PB and / or TB. When the width or height of the luminance coding block or chrominance coding block is greater than the maximum transform width or height, the luminance coding block or chrominance coding block can be automatically divided in the horizontal and / or vertical directions to meet the transform size limit in the direction.

[0151] In the specific example above where a basic block is divided into multiple color blocks (CBs), and as described above, the coding tree scheme can support the ability for luma and chroma to have separate block tree structures. For example, for P-slices and B-slices, the luma CTB and chroma CTB in a CTU can share the same coding tree structure. For example, for I-slices, luma and chroma can have separate coding block tree structures. When applying separate block tree structures, the luma CTB can be divided into multiple luma CBs using one coding tree structure, and the chroma CTB can be divided into multiple chroma CBs using another coding tree structure. This means that a CU in an I-slice can be composed of coding blocks of the luma component or coding blocks of the two chroma components, and a CU in a P-slice or B-slice is always composed of coding blocks of all three color components, unless the video is monochrome.

[0152] When a coded block is further divided into transform blocks, these transform blocks can be ordered in the bitstream in various orders or scanning methods. Exemplary implementations of dividing a coded block or prediction block into transform blocks, and the encoding order of the transform blocks, will be described in further detail below. In some exemplary implementations, as described above, transform partitioning can support transform blocks of multiple shapes, such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, where the transform block size ranges from, for example, 4×4 to 64×64. In some implementations, if the coded block is less than or equal to 64×64, transform block partitioning may be applied only to the luma component, such that for the chroma block, the transform block size is the same as the coded block size. Otherwise, if the width or height of the coded block is greater than 64, the luma coded block and the chroma coded block can be implicitly partitioned into multiples of the minimum (W, 64) × minimum (H, 64) and minimum (W, 32) × minimum (H, 32) transform blocks, respectively.

[0153] In some exemplary implementations of transform block segmentation, for intra-frame coded blocks and inter-frame coded blocks, the coded blocks can be further segmented into transform blocks with a segmentation depth of up to a predetermined number of levels (e.g., 2 levels). The transform block segmentation depth and size can be correlated. For some exemplary implementations, the mapping from the transform size at the current depth to the transform size at the next depth is shown in Table 1 below.

[0154] Table 1: Settings for Changing Partition Size

[0155]

[0156] Based on the exemplary mapping in Table 1, for a 1:1 square block, the next-level transform partition can create four 1:1 square sub-transform blocks. The transform partition can stop at, for example, 4×4. Therefore, a transform size of 4×4 at the current depth corresponds to the same size 4×4 at the next depth. In the examples in Table 1, for a 1:2 / 2:1 non-square block, the next-level transform partition can create two 1:1 square sub-transform blocks, while for a 1:4 / 4:1 non-square block, the next-level transform partition can create two 1:2 / 2:1 sub-transform blocks.

[0157] In some exemplary implementations, additional constraints related to transform block segmentation can be imposed on the luma component of an intra-coded block. For example, for each level of transform segmentation, all sub-transform blocks can be constrained to have equal sizes. For instance, for a 32×16 coded block, level 1 transform segmentation can create two 16×16 sub-transform blocks, and level 2 transform segmentation can create eight 8×8 sub-transform blocks. In other words, the second-level segmentation must be applied to all first-level sub-blocks to maintain equal transform unit sizes. Figure 15 The following table shows an example of transform block segmentation for intra-coded square blocks as shown in Table 1, with arrows indicating the encoding order. Specifically, 1502 shows a square coded block. 1504 shows a first-level segmentation, where the block is divided into four equal-sized transform blocks according to Table 1, with arrows indicating the encoding order. 1506 shows a second-level segmentation, where all the generated first-level equal-sized blocks are divided into 16 equal-sized transform blocks according to Table 1, with arrows indicating the encoding order.

[0158] In some exemplary embodiments, the aforementioned intra-frame coding restrictions may not apply to the luma component of inter-frame coded blocks. For example, after the first level of transform segmentation, any of the sub-transform blocks can be further independently segmented into more than one level. Therefore, the resulting transform blocks may be the same size or different sizes. Figure 16 The image shows an example of dividing an inter-frame coded block into transform blocks according to the coding order. Figure 16 In the example, according to Table 1, the inter-coded block 1602 is divided into two levels of transform blocks. At the first level, the inter-coded block is divided into four transform blocks of equal size. Then, only one (not all) of the four transform blocks is further divided into four sub-transform blocks, resulting in a total of seven transform blocks of two different sizes, as shown in 1604. Figure 16 The arrows in 1604 show an exemplary encoding order for these 7 transform blocks.

[0159] In some exemplary implementations, additional constraints may be imposed on the transform block for the chroma component. For example, the transform block size may be the same as the coding block size for the chroma component, but not smaller than a predefined size, such as 8×8.

[0160] In some other exemplary embodiments, for coded blocks with a width (W) or height (H) greater than 64, the luminance coded block and chrominance coded block may be implicitly divided into multiples of minimum (W, 64) × minimum (H, 64) and minimum (W, 32) × minimum (H, 32) transform units, respectively. Here, in this disclosure, "minimum (a, b)" may return the smaller value between a and b.

[0161] Figure 17 Further, an exemplary alternative to segmenting a coded block or prediction block into transform blocks is illustrated. For example... Figure 17 As shown, a predefined set of segmentation types can be applied to the coded block based on its transform type, instead of using recursive transform segmentation. Figure 17 In the specific example shown, one of six exemplary segmentation types can be applied to segment the coded block into a different number of transform blocks. This transform block segmentation scheme can be applied to either the coded block or the prediction block.

[0162] To be more detailed, Figure 17 The partitioning scheme provides up to six exemplary partition types for any given transform type (transform type refers to, for example, the type of the main transform (e.g., ADST, etc.)). In this scheme, a transform partition type can be assigned to each coding block or prediction block based on, for example, rate-distortion cost. In the example, the transform partition type assigned to a coding block or prediction block can be determined based on the transform type of the coding block or prediction block. A specific transform partition type may correspond to the transform block partition size and partitioning mode, such as... Figure 17 The six transform partition types are shown in the diagram. The correspondence between various transform types and various transform partition types can be predefined. An example is shown below, where uppercase labels indicate the transform partition type that can be assigned to a coded block or a prediction block based on rate-distortion cost:

[0163] Partition_None: Allocates a transform size equal to the block size.

[0164] • Partition_Splitter: Allocate a transform size with a width that is half the width of the block size and a height that is half the height of the block size.

[0165] • Partition_HORZ(PARTITION_HORZ): Allocates a transform size with the same width as the block size and a height that is half the height of the block size.

[0166] ··Partition_VERT(PARTITION_VERT): Allocates a transform size with a width that is half the width of the block size and a height that is the same as the block size.

[0167] ··Partition_HORZ4(PARTITION_HORZ4): Allocates a transform size with the same width as the block size and a height of 1 / 4 of the block size.

[0168] ··Partition_VERT4(PARTITION_VERT4): Allocates a transform size with a width of 1 / 4 of the block size and a height equal to the block size.

[0169] In the example above, such as Figure 17 The transform partition types shown all include a uniform transform size for the transformed blocks used for partitioning. This is merely an example and not a limitation. In some other implementations, mixed transform block sizes may be used for the transformed blocks partitioned within a specific partition type (or pattern).

[0170] Video blocks (PB or CB, also called PB when not further divided into multiple prediction blocks) can be predicted in various ways instead of being directly encoded, thereby leveraging the various correlations and redundancies in the video data to improve compression efficiency. Accordingly, this prediction can be performed in various modes. For example, video blocks can be predicted via intra-frame prediction or inter-frame prediction. Especially in inter-frame prediction mode, video blocks can be predicted via single-reference inter-frame prediction or composite-reference inter-frame prediction using one or more other reference blocks or inter-frame prediction blocks of one or more other picture frames. To achieve inter-frame prediction, a reference block can be specified by its frame identifier (the temporal position of the reference block) and a motion vector (the spatial position of the reference block) indicating the spatial offset between the current block being encoded or decoded and the reference block. The reference frame identifier and motion vector can be signaled in the bitstream. The motion vector, as the spatial block offset, can be directly signaled, or it can be predicted by another reference motion vector or the predictor motion vector itself. For example, the current motion vector can be predicted directly from a reference motion vector (e.g., the motion vector of a candidate neighboring block), or it can be predicted from a combination of the reference motion vector and the motion vector difference (MVD) between the current motion vector and the reference motion vector. The latter can be referred to as Merge mode with Motion Vector Difference (MMVD). In the bitstream, the reference motion vector can be identified as a pointer to, for example, a spatially adjacent block or a temporally adjacent but spatially juxtaposed block.

[0171] In some other exemplary implementations, intra-block copy (IBC) prediction can be used. In IBC, the current block in the current frame is predicted using another block in the current frame (rather than a temporally different picture frame, hence the term "intra-frame") combined with a block vector (BV), which indicates the offset of the position of the intra-frame predictor or reference block relative to the position of the block being predicted. The position of the coded block can be represented, for example, by pixel coordinates relative to the upper left of the current frame (or slice). Thus, IBC mode uses a similar concept of inter-frame prediction within the current frame. For example, a BV can be predicted directly from another reference BV, or a BV can be predicted by combining the BV difference between the current BV and the reference BV, which is consistent with using a reference MV and MV difference to predict MV in inter-frame prediction. IBC is very useful in terms of improving coding efficiency, especially when encoding and decoding video frames with screen content having, for example, a large number of repetitive patterns (e.g., text information), where the same text fragments (letters, symbols, words, phases, etc.) appear in different parts of the same frame and can be used for mutual prediction.

[0172] In some implementations, IBC can be viewed as a separate prediction mode in addition to the normal intra-prediction mode and the normal inter-prediction mode. Therefore, the prediction mode for a specific block can be selected and signaled from three different prediction modes: intra-prediction, inter-precoding, and IBC mode. In these implementations, flexibility can be built into each of these modes to optimize the coding efficiency of each. In some other implementations, IBC can be viewed as a sub-mode or branch of the inter-prediction mode, using similar motion vector determination mechanisms, reference mechanisms, and coding mechanisms. In such implementations (integrated inter-prediction mode and IBC mode), the flexibility of IBC may be somewhat limited in order to coordinate the usual inter-prediction mode and IBC mode. However, such implementations are less complex and can still utilize IBC to improve the coding efficiency of video frames characterized, for example, screen content. In some exemplary implementations, inter-prediction modes can be extended to support IBC using existing pre-specified mechanisms for separate inter-prediction and intra-prediction modes.

[0173] These prediction modes can be selected at various levels, including but not limited to: sequence level, frame level, image level, slice level, CTU level, CT level, CU level, CB level, or PB level. For example, for IBC purposes, a decision on whether to use an IBC mode can be made and signaled at the CTU level. If the CTU receives a signal to use the IBC mode, then all coded blocks in the entire CTU can be predicted by IBC. In some other implementations, IBC predictions can be determined at the SuperBlock (SB) level. Each SB can be divided into multiple CTUs or partitions (e.g., quadtree partitions) in different ways. Further examples are provided below.

[0174] Figure 18 This is an exemplary snapshot of a portion of the current frame containing multiple CTUs, shown from the decoder's perspective. Each square (e.g., 1802) represents a CTU. A CTU can be one of various predefined sizes as described in detail above, such as an SB. Each CTU can include one or more coded blocks (or prediction blocks, for a specific color channel). CTUs covered by horizontal lines represent those that have been reconstructed. CTU 1804 represents the current CTU being reconstructed. In the current CTU 1804, the coded blocks covered by horizontal lines represent those that have been reconstructed in the current CTU, the coded block 1806 covered by diagonal lines is currently being reconstructed, and the uncovered coded blocks in the current CTU 1804 are waiting to be reconstructed. Other uncovered CTUs have not yet been processed.

[0175] The position or offset of the reference block (relative to the current block) used in the IBC for predicting the current coded block can be indicated by the BV, such as Figure 18 The example arrows in the diagram illustrate this. For instance, BV can indicate a reference block in vector form (in...). Figure 18 The positional difference between the top-left block (marked as "Ref") and the current block. Figure 18 The use of a CTU as the basic IBC unit is illustrated. The basic principles apply to implementations that use a SB as the basic IBC unit. As described in more detail below, in such implementations, each superblock can be divided into multiple CTUs, and each CTU can be further divided into multiple coding blocks.

[0176] As will be disclosed in further detail below, a reference CTU / SB can be referred to as a local CTU / SB or a non-local CTU / SB, depending on its position relative to the current CTU / SB in the IBC. A local CTU / SB can refer to a CTU / SB that coincides with the current CTU / SB, or it can refer to a CTU / SB that is near the current CTU / SB and has been reconstructed (e.g., the left neighbor of the current CTU / SB). A non-local CTU / SB can refer to a CTU / SB that is farther away from the current CTU / SB. When performing IBC prediction on the current coded block, either or both of the local and non-local CTU / SBs can be searched to find the reference block. The specific implementation of IBC can depend on whether the reference CTU / SB is a local CTU / SB or a non-local CTU / SB, because the on-chip and off-chip memory management of the reconstructed samples (e.g., off-chip picture buffers, DPBs, and / or on-chip memory) used for local or non-local CTU / SB references can differ. For example, reconstructed local CTU / SB samples may be suitable for storage in the on-chip memory of the encoder or decoder used for IBC. Conversely, reconstructed non-local CTU / SB samples may be stored in off-chip DPB memory or external memory.

[0177] In some implementations, the location of the reconstructed block that might be used as a reference block for the current coded block 1804 may be restricted. Such restrictions can be the result of various factors and may depend on whether IBC is implemented as an integrated part of the normal inter-frame prediction mode, as a special extension of the inter-frame prediction mode, or as a separate and independent IBC mode. In some examples, only the currently reconstructed CTU / SB samples can be searched to identify the IBC reference block. In some other examples, the currently reconstructed CTU / SB sample and another adjacent reconstructed CTU / SB sample (e.g., the left adjacent CTU / SB) can be used to search for and select the reference block, such as... Figure 18 The thick dashed box 1808 is shown. For this type of implementation, only the reconstructed local CTU / SB sample can be used to search and select IBC reference blocks. In some other examples, certain CTU / SBs may not be available for searching and selecting IBC reference blocks for various other reasons. For example, Figure 18 The CTU / SB1810 marked with an intersection may not be used to search for and select the reference block of the current block 1804, as it may be used for special purposes as further described below (e.g., wavefront parallel processing).

[0178] In some implementations, such as Figure 18As shown, the region formed by the thick dashed box 1808 can be called the local search region. Samples within the local search region can be stored in on-chip memory.

[0179] In some implementations, loop filters are disabled when intra-block copying (IntraBC or IBC) is enabled. These loop filters include a deblocking filter, a constrained directional enhancement filter (CDEF), and loop restoration (LR). By doing so, a second picture buffer dedicated to enabling / supporting intra-block copying is avoided.

[0180] In some implementations, the use of parallel decoding may impose a limitation on allowing the reconstructed CTU / SB to be used to provide an IBC reference block or reference sample, as multiple coded blocks are decoded simultaneously in parallel decoding. Figure 19 In one example shown, each square represents a CTU / SB. Parallel decoding is possible, where multiple consecutive rows and multiple CTUs / SBs every other column (every two columns) can be reconstructed in parallel processing, as shown below. Figure 19 The CTU / SB is shown in the diagram, covered by diagonal lines. Other CTU / SBs covered by horizontal lines have been reconstructed, while those not covered are those not yet reconstructed. Through this parallel processing, for the current parallel-processed CTU / SB at the top-left coordinate (x0, y0), the reconstructed sample at (x, y) can only be used to predict the current CTU / SB in the IBC if the ordinate y is less than y0 and the abscissa x is less than x0 + 2(y0 - y). Therefore, the reconstructed CTU / SBs covered by horizontal lines can serve as a reference for the current block in parallel processing. Note that coordinate units (such as (x0, y0) and (x, y)) can include pixels, blocks (e.g., SBs), etc.

[0181] In some implementations, particularly when the off-chip DPB is used to hold the IBC reference sample, the write-back latency of writing the immediately reconstructed sample to the off-chip DPB may further limit the CTU / SB that can be used to provide the IBC reference sample for the current block. Figure 20 An example is shown in the figure, where it is possible to Figure 19 Additional restrictions are imposed on top of the existing limitations. Specifically, to allow for hardware write-back latency, IBC predicts that it may not use access to instantly reconstructed regions for searching and selecting reference blocks. The number of instantly reconstructed regions that can be restricted or prohibited can be 1 to n CTU / SB, where n is a positive number and can be positively correlated with the duration of the write-back latency. Therefore, in Figure 19Above the specific parallel processing constraints, for the current CTU / SB (covered by the slash), if the coordinates of the upper left position of a current CTU / SB are (x0, y0), then IBC can obtain a prediction of the position (x, y), where the ordinate y is less than y0 and the abscissa y is less than x0+2(y0-y)-D, where D represents the number of immediately reconstructed regions (e.g., regions to the left of the current CTU / SB) that are restricted / prohibited from being used as IBC references. Figure 20 This illustrates an additional CTU / SB of this type, which is restricted to an IBC reference sample when D=2 (in blocks or 2×128 pixels, or pixels when each block is a 128×128 SB). The areas covered by the backslashes cannot be used as additional CTU / SBs for IBC reference.

[0182] In some implementations, such as Figure 20 As shown, the area formed by the block covered by the horizontal line can be called the non-local search area, and the samples in this area can be stored in external memory.

[0183] In some implementations, the IBC architecture can form a dedicated coding mode, where the IBC mode is a third prediction mode in addition to intra-frame prediction and inter-frame prediction modes. When the block size is 64×64 or smaller, the bitstream carries IBC syntax elements indicating the IBC mode of the coding unit. Therefore, the maximum CU size of the IBC can be 64×64 to implement a continuous memory update mechanism for the Reference Sample Memory (RSM). However, the reference sample addressing mechanism remains the same as in the HEVC SCC extension by representing the two-dimensional offset and reusing the inter-frame prediction vector coding process.

[0184] In some implementations, when the Chroma Separate Tree (CST) is active, another special case may occur in which the encoder cannot derive the chroma BV from the luma BV, so that the IBC is used only for the luma coding block.

[0185] In some embodiments, which are also described in further detail below, both the local CTU / SB search area and the non-local CTU / SB search area can be used to search for and select IBC reference blocks. Additionally, when using on-chip memory, some limitations on write-back latency using a reconstructed CTU / SB as an IBC reference may be relaxed or eliminated. In some further embodiments, the methods used when local CTU / SB and non-local CTU / SB coexist may differ due to differences in buffer management of reference blocks, such as when using on-chip memory or off-chip memory. These embodiments will be described in more detail in the following disclosure.

[0186] In some implementations, IBC can be implemented as an extension of the inter-frame prediction mode, treating the current image frame as the reference frame in the inter-frame prediction mode, allowing blocks within the current frame to be used as prediction references. Therefore, even if the IBC process only involves the current image frame, this IBC implementation can follow the coding path used for inter-frame prediction. In such implementations, the reference structure of the inter-frame prediction mode can be applied to IBC, where the representation using BV (Block Value) addressing of reference samples can be similar to the motion vector (MV) in inter-frame prediction. Therefore, based on the current frame as the reference frame, and relying on similar or identical syntax structures and decoding processes, IBC can be implemented as a special type of inter-frame prediction mode.

[0187] In such implementations, since IBC can be considered an inter-frame prediction mode, only intra-frame prediction slices must be allowed to be prediction slices that allow the use of IBC. In other words, IBC is not allowed for this type of prediction that is only for intra-frame slices because inter-frame prediction mode does not invoke any inter-frame prediction processing path. When IBC is applicable, the encoder expands the list of reference pictures to an entry pointing to the current picture. Therefore, the current picture can occupy up to one picture-sized buffer in the shared decoded picture buffer (DPB). In inter-frame prediction mode, the signaling for using IBC may be implicit in the selection of the reference frame. For example, when the selected reference picture points to the current picture, and if necessary and available, the coding unit will use an IBC with a coding path similar to inter-frame prediction and a special IBC extension. In some specific implementations, unlike regular inter-frame prediction, the reference samples in IBC processing are not loop-filtered before being used for prediction. Furthermore, the corresponding reference current picture can be a long-term reference frame, as it will be close to the next frame to be encoded or decoded. In some implementations, to minimize memory requirements, the encoder can release the buffer immediately after reconstructing the current image. The encoder can then fill the DPB with the filtered, reconstructed image as a short-term reference for subsequent frames in true inter-frame prediction, even if it is unfiltered when used for IBC.

[0188] In the exemplary implementation described above, even if IBC is merely an extension of the inter-frame prediction mode, it can be processed using several special procedures that may deviate from normal inter-frame prediction. For example, the IBC reference sample can also be unfiltered. In other words, reconstructed samples prior to in-loop filtering processes, including DeBlocking Filtering (DBF), Sample Adaptive Offset (SAO) filtering, Cross-Component Sample Offset (CCSO) filtering, etc., can be used for IBC prediction, whereas the normal inter-frame prediction mode uses filtered samples for prediction. For another example, IBC may not be interpolated without luma samples, and chroma sample interpolation may only be necessary if the chroma BV is non-integer when derived from the luma BV. For yet another example, when the chroma BV is non-integer and the IBC reference block is close to the boundary of the available area used for IBC reference, surrounding reconstructed samples can be chroma interpolated outside the boundary. BVs pointing to a single adjacent boundary line may not avoid this situation.

[0189] In such implementations, IBC's prediction of the current block can reuse the prediction and coding mechanisms of inter-frame prediction processing, including using a reference BV to predict the current BV and, for example, an additional BV difference. However, in some specific implementations, the luma BV can be implemented at integer resolution instead of with fractional precision, such as that used for regular inter-frame prediction (MV).

[0190] In some implementations... Figure 18 All CTUs and SBs covered by the mid-horizontal line can be used to search for and select IBC reference blocks, except for those used to allow Wavefront Parallel Processing (WPP) within the current CTU (by...). Figure 18 The two CTUs to the right and above (e.g., the cross representation in the text) Figure 18 (As shown in 1810). Therefore, with some exceptions for parallel processing purposes, the entire current image is almost entirely a reconstructed region.

[0191] In some other implementations, the area that the IBC reference block can search and select may be limited to the local CTU / SB. Figure 18The thick dashed box 1808 illustrates an example. In this example, the CTU / SB to the left of the current CTU can serve as a reference sample region for IBC at the start of the current CTU reconstruction process. When using such a local reference region, instead of allocating additional external memory space in the DPB, on-chip memory space can be allocated to keep the local CTU / SB for IBC reference. In some implementations, IBC can use fixed on-chip memory, thereby reducing the complexity of implementing IBC in the hardware architecture. Therefore, a dedicated IBC mode independent of normal inter-frame prediction can be implemented using on-chip memory, rather than simply being implemented as an extension of the inter-frame prediction mode.

[0192] For example, the fixed on-chip memory size for each color component to store local IBC reference samples (e.g., left CTU or SB) can be 128×128. In some implementations, the maximum CTU size can also be 128×128. In this case, the Reference Sample Memory (RSM) can hold samples of a single CTU size. In some other alternative implementations, the CTU size may be smaller. For example, the CTU size can be 64×64. Therefore, the RSM can hold multiple CTUs simultaneously (4 CTUs in this example case). In still other implementations, the RSM can hold multiple SBs, each SB may include one or more CTUs, and each CTU may include multiple coding blocks.

[0193] In some implementations of the local on-chip IBC reference, a CTU is stored in the on-chip RSM and a continuous update mechanism can be implemented to replace the reconstructed sample of the left adjacent CTU with the reconstructed sample of the current CTU. Figure 21 A simplified example of this continuous RSM update mechanism at four intermediate time points during the reconstruction process is shown. Figure 21 In the example, the RSM has a fixed size that can hold a CTU. The CTU may include an implicit partition. For example, the CTU may be implicitly divided into four separate regions (e.g., quadtree partitions). Each region may include multiple coded blocks. The size of the CTU may be 128×128, while for the exemplary quadtree partition, the size of each example region or partition may be 64×64. At each intermediate time, the region / partition of the RSM covered by the horizontal line holds the reconstructed reference sample corresponding to the left adjacent CTU, and the region / partition covered by the gray vertical line holds the reconstructed reference sample corresponding to the current CTU. The coded blocks of the RSM covered by the diagonal lines indicate the current coded block in the current region being encoded / decoded / reconstructed.

[0194] At the first intermediate time indicating the start of reconstruction of the current CTU, the RSM may include only the reconstructed reference samples of the left adjacent CTU for each of the four example regions, as shown in 2102. At the other three intermediate times, the reconstruction process gradually replaces the reconstructed reference samples of the left adjacent CTU with the reconstructed samples of the current CTU. When the encoder processes the first coding block of this region / partition, a 64×64 region / partition reset may occur in the RSM. When a region of the RSM is reset, it is considered blank and is considered not to have any reconstructed reference samples of the IBC saved (in other words, this region of the RSM cannot be used as an IBC reference sample). When the corresponding current coding block in this region is processed, the corresponding block in the RSM is archived together with the reconstructed samples of the corresponding block of the current CTU for use as the IBC reference sample of the next current block, as shown in 2102. Figure 21 The figures for intermediate times 2104, 2106, and 2108 are shown in the table. Once all coded blocks have been processed, corresponding to the RSM regions / partitions, the entire region is filled with reconstructed samples of these current coded blocks as IBC reference samples, as shown in the table. Figure 21 The regions completely covered by vertical lines at different intermediate times are shown. Therefore, at intermediate times 2104 and 2106, some regions / partitions in the RSM retain IBC reference samples from adjacent CTUs, some other regions / partitions completely retain reference samples from the current CTU, while some regions / partitions partially retain the current CTU's reference samples and are partially blank (not used for IBC reference due to the reset process described above). When processing the last region (e.g., the lower right region), all other three regions retain the reconstructed samples of the current CTU as IBC reference samples, while the last region / partition partially retains the reconstructed samples of the corresponding coded block in the current CTU and is partially blank, until the last coded block of the CTU is reconstructed. At this point, the entire RSM saves the reconstructed samples of the current CTU, and if also encoded in IBC mode, the RSM is ready for use with the next CTU.

[0195] Figure 22 This illustrates an implementation of the above-described RSM with continuous spatial updates at a specific intermediate time, where the left adjacent CTU and the current CTU having the current coded block (the block covered by the slash) are both shown in the appendix. Figure 22 In the example, the corresponding reconstructed samples of these two CTUs are covered by horizontal and vertical lines, respectively. These two CTUs are in RSM and effectively serve as IBC reference samples for the current coding block. At a specific reconstruction time in this example, in RSM, the process has replaced the samples included in the uncovered area of ​​the left adjacent CTU with the area covered by the vertical line of the current CTU. The remaining effect samples from the adjacent CTUs are shown as covered by horizontal lines.

[0196] In the exemplary implementation described above, when the fixed RSM size is the same as the CTU size, the RSM is implemented to contain one CTU. In other implementations, if the CTU size is small, the RSM can contain multiple CTUs, more than a single CTU. For example, the CTU size can be 32×32, while the fixed RSM size can be 128×128. Therefore, the RSM can hold samples of 16 CTUs. Following the same basic RSM update principle described above, the RSM can hold 16 adjacent CTUs of the current 128×128 block before being reconstructed. Once processing of the first coded block of the current 128×128 block begins, the first 32×32 region of the RSM is initially filled with reconstructed samples of one adjacent CTU, and an RSM holding a single CTU can be updated as described above. The remaining 15 32×32 regions contain 15 adjacent CTUs as reference samples for the IBC. Once the CTU corresponding to the first 32×32 region of the current 128×128 block being decoded has been reconstructed, the reconstructed sample of that CTU is used to update the first 32×32 region of the RSM. Then, the CTU corresponding to the second 32×32 region of the current 128×128 block can be processed and finally updated using the reconstructed sample. This process continues until the 16 32×32 regions of the RSM contain the reconstructed samples of the current 128×128 block (all 15 CTUs). The decoding process then moves to the next 128×128 block.

[0197] In some other implementations, BV coding can use the processing specified for inter-frame prediction, but it uses simpler rules to construct the candidate list. Specifically, the candidate list structure for inter-frame prediction can consist of five spatial candidates, one temporal candidate, and six history-based candidates. For history-based candidates, multiple candidates need to be compared to avoid duplicate entries in the final candidate list. Additionally, list construction can include pairwise averaging of candidate candidates. In contrast, the IBC list construction process can consider only two spatially adjacent BVs and five history-based BV predictors (HBVPs), where only the first HBVP is compared with the spatial candidate when added to the candidate list. While regular inter-frame prediction uses two different candidate lists, one for merge mode and one for regular mode, the candidate list in IBC is suitable for both. However, merge mode can use up to six candidates from the list, while regular mode only uses the first two. Block Vector Difference (BVD) encoding uses the Motion Vector Difference (MVD) process to generate a final BV of any size. This also means that the reconstructed BV may point to a region outside the reference sample area, requiring correction by removing the absolute offset in each direction using the modulo operation of the width and height of the RSM.

[0198] In some other implementations, as Figure 21 and Figure 22 With this extension, the RSM can maintain a set of adjacent CTUs. Each time a current CTU is processed, the RSM portion of the maintained furthest neighboring CTU is updated in the manner described above using the reconstructed current CTU. Similarly, for the next current CTU, the furthest neighboring CTU in the RSM can be updated and replaced. Therefore, a moving window of adjacent CTUs in the IBS is used to update multiple CTUs maintained in a fixed-size RSM.

[0199] Figure 23 Another specific exemplary implementation of local IBC using on-chip RSM is shown. In this example, the maximum block size of the IBC mode may be limited. For example, the maximum IBC block could be 64×64. The on-chip RSM can be configured to a fixed size corresponding to the superblock (SB), such as 128×128. Figure 23 The implementation of RSM uses and Figure 21 and Figure 22 The implementation method is based on a similar fundamental principle. Figure 23In this context, RSM can maintain multiple adjacent and / or current CTUs as IBC reference samples. Figure 23 In the example, the SB can be quadtree-partitioned. Accordingly, the RSM quadtree can be divided into four regions or units, each 64×64. Each of these regions can hold one or more coded blocks. Alternatively, each of these regions can hold one or more CTUs, and each CTU can hold one or more coded blocks. The encoding order of the quadtree regions can be predefined. For example, the encoding order could be top-left, top-right, bottom-left, bottom-right. Figure 23 The quadtree partitioning of the SB is merely an example. In some other alternative implementations, the SB can be partitioned according to any other scheme. The implementation of RSM updates for local IBC described herein applies to those alternative partitioning schemes.

[0200] In such local IBC implementations, there may be limitations on the local reference block that can be used for IBC prediction. For example, it may be required that the reference block and the current block be in the same SB line. Specifically, the local reference block may be located only in the current SB, or it may be located in an SB to the left of the current SB. Figure 23 The dashed arrows in the diagram illustrate an exemplary current block predicted by another permitted coded block in the IBC. When the current SB or the left SB is used for IBC reference, the reference sample update process in the RSM can follow the reset process described above. For example, when any of the 64×64 cell reference sample memories begins updating using reconstructed samples from the current SB, previously stored reference samples (from the left SB) across the entire 64×64 cell are marked as unusable for generating IBC prediction samples, and the cell is gradually updated with reconstructed samples from the current block.

[0201] Figure 23 Five exemplary states of the RSM during local IBC decoding of the current SB in panel 2302 are shown. Similarly, in each exemplary state, the RSM area covered by the horizontal line holds the corresponding reference sample of the corresponding quadtree region of the left adjacent SB, and the area / partition covered by the gray vertical line holds the corresponding reference sample of the current SB. The encoded block of the RSM covered by the diagonal line represents the current encoded block within the current quadtree region being encoded / decoded. At the start of encoding of each current SB, the RSM stores samples from the previously encoded SB ( Figure 23The RSM is in state (0). When the current block is located in one of the four 64×64 quadtree regions in the current SB, the corresponding region in the RSM is reset and used to store the samples of the current 64×64 encoded region. In this way, the samples in each 64×64 quadtree region of the RSM are gradually updated by the samples in the current SB (states (1) to (3)). When the current SB has been encoded, the entire RSM is filled with all the samples of the current SB (state (4)).

[0202] Figure 23 Each of the 64×64 regions in interface 2302 is labeled using a spatially encoded sequence number. Sequences 0-3 represent the four 64×64 quadtree regions of the left adjacent SB, while sequences 4-7 represent the four 64×64 quadtree regions of the current SB interface. Figure 23 In the middle, interface 2304 further shows Figure 23 The corresponding spatial distribution of RSM states (1), (2), and (3) of interface 2302 in the left neighbor of the reference sample and the current SB in a 128×28 RSM. Coverage areas without intersections represent regions in the RSM with reconstructed samples. Coverage areas with intersections represent regions in the RSM where the reconstructed samples of the left SB have been reset (and therefore cannot be used as reference samples for the local IBC).

[0203] The encoding order of a 64×64 region and the corresponding RSM update order can follow a horizontal scan (as described above). Figure 23 (As shown) or vertical scan. The horizontal scan order is: top left, top right, bottom left, bottom right. The vertical scan order is: top left, bottom left, top right, bottom left. Figure 24 Interfaces 2402 and 2404 illustrate the update process for the left adjacent SB and the current SB reference sample during horizontal and vertical scans, respectively, for comparison when reconstructing each of the four 64×64 regions of the current SB. Figure 24 In the diagram, a 64×64 area covered by unintersecting horizontal lines represents a region with samples available for IBC. Areas covered by intersecting horizontal lines represent the region of the left adjacent SB of the corresponding reconstructed sample that has been updated to the current SB. Uncovered areas represent unprocessed regions of the current SB. Blocks covered by diagonal lines represent the currently processed coding block.

[0204] like Figure 24 As shown, the following restrictions may apply to the reference block of the IBC, depending on the position of the current coded block relative to the current SB.

[0205] If the current block is located in the top-left 64×64 region of the current SB, then in addition to the reconstructed samples in the current SB, reference samples in the bottom-right, bottom-left, and top-right 64×64 blocks of the left SB can also be referenced, such as... Figure 24 The 2412 (for horizontal scanning) and 2422 (for vertical scanning) are shown.

[0206] If the current block is located in the upper right 64×64 block of the current SB, then in addition to the reconstructed samples in the current SB, if the brightness sample at position (0, 64) relative to the current SB has not yet been reconstructed, the current block can also refer to the reference samples in the lower left and lower right 64×64 blocks of the left SB. Figure 24 (2414). Otherwise, the current block of the IBC can also reference the sample in the lower right 64×64 block of the left SB (24×64). Figure 24 (of 2426).

[0207] If the current block is located in the lower left 64×64 block of the current SB, then in addition to the samples already reconstructed in the current SB, if the brightness position (64, 0) relative to the current SB has not yet been reconstructed, the current block can also refer to the reference samples in the upper right 64×64 block and the lower right 64×64 block of the left SB. Figure 24 (2424). Otherwise, the current block of the IBC can also refer to the reference sample in the lower right 64×64 block of the left SB (2424). Figure 24 (of 2416).

[0208] If the current block is located in the lower right 64×64 block of the current SB, it can only reference the reconstructed samples in the current SB of the IBC. Figure 24 (2418 and 2428).

[0209] As described above, in some exemplary implementations, one or both of the local CTU / SB and non-local CTU / SB can be used to search for and select the IBC reference block. Furthermore, when the on-chip RSM is used for local reference, some limitations on the availability of write-back latency for the reconstructed CTU / SB as the IBC reference may be relaxed or eliminated. Such implementations can be applied regardless of whether parallel decoding is employed.

[0210] In some implementations using traditional intra-frame prediction, one problem might be that the current coded block needs to find a reference block of the same size within the current image. When a portion of the reference block does not match the text pattern of the current block, the encoder may have to split the current block into smaller blocks, so that each smaller block can find a good reference block. Alternatively, the encoder might continue to use existing reference blocks, thus impacting prediction efficiency.

[0211] The various embodiments and / or implementations described in this disclosure can be performed individually or in any combination in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). One or more processors execute a program stored in a non-transitory computer-readable medium. In this disclosure, the term "block" can be interpreted as a prediction block, coding block, or coding unit (CU).

[0212] Various embodiments in this disclosure include segmenting an intra-frame BC-coded block into more than one sub-partition. Some of these sub-partitions are predicted using a conventional reference block, indicated by an associated block vector; at least one sub-partition of the block is predicted using a padding method, wherein the padding sample values ​​can be derived from context or signaled in a predefined manner. This seed partition predicted using the padding method within the intra-frame BC-coded block can be referred to as a padding region (or padding sub-partition).

[0213] In the various embodiments of this disclosure, the two sub-partitions in an intra-BC coded block are used only as non-limiting examples, and the scope of the various embodiments of this disclosure is not limited to two sub-partitions, and an intra-BC coded block can be divided / include more than two sub-partitions.

[0214] Figure 25 A flowchart 2500 illustrates an exemplary method following the basic principles of the implementation of IBC prediction using sample padding described above. The exemplary decoding method flow begins at step S2501 and may include some or all of the following steps: step S2510, receiving a current video block in an encoded video stream, wherein a corresponding reference block partially matches the current video block; step S2520, extracting a first syntax element from the encoded video stream, the first syntax element indicating that IBC padding can be used for the current video block; step S2530, extracting a second syntax element from the encoded video stream, the second syntax element indicating that IBC padding is used to predict the current video block; step S2540, extracting at least one syntax element from the encoded video stream; step S2550, segmenting the current video block into at least two sub-partitions based on the at least one syntax element, wherein the at least two sub-partitions include a padding sub-partition and an IBC sub-partition; step S2560, predicting the padding sub-partition based on predefined padding rules; and / or, step S2570, predicting the IBC sub-partition based on predefined IBC rules. The exemplary method is stopped at step S2599.

[0215] In various embodiments of this disclosure, grammatical elements may be referred to as tags.

[0216] In some implementations, at least one syntax element includes a first syntax element and a second syntax element; and / or step S2550 may include: dividing the current video block into at least two sub-partitions using horizontal or vertical segmentation based on the first syntax element, and / or, determining whether the filling sub-partition is the first sub-partition or the second sub-partition of the current video block based on the second syntax element.

[0217] In some implementations, at least one syntax element includes a third syntax element; and / or, the third syntax element indicates the fraction of the fill sub-partition relative to the size of the current video block.

[0218] In some implementations, the fraction includes at least one of the following: 1 / 4, 1 / 2, or 3 / 4.

[0219] In some implementations, step S2550 may include: dividing the current video block into at least two sub-partitions based on at least one syntax element, along a direction defined other than the horizontal or vertical direction.

[0220] In some implementations, step S2560 may include: predicting the filling sub-partition based on the reconstructed adjacent sub-partitions.

[0221] For a non-restrictive example, based on the reconstructed neighboring sub-partitions, predicting the filling sub-partitions includes: horizontally extending the rightmost column of the left neighboring sub-partition to the filling sub-partition; or vertically extending the bottom row of the upper neighboring sub-partition to the filling sub-partition.

[0222] For another non-restrictive example, based on the reconstructed neighboring sub-partitions, predicting the filling sub-partition includes: filling the filling sub-partition with the top sample of the rightmost column of the left neighboring sub-partition; or, filling the filling sub-partition with the leftmost sample of the bottom row of the upper neighboring sub-partition.

[0223] In some implementations, step S2560 may include: predicting filling sub-partitions based on reconstructed adjacent video blocks.

[0224] For a non-limiting example, based on reconstructed neighboring video blocks, predicting the filling sub-partition includes: vertically extending samples from the bottom row of the corresponding sub-partition in the upper neighboring video block to the filling sub-partition; or horizontally extending samples from the rightmost column of the corresponding sub-partition in the left neighboring video block to the filling sub-partition.

[0225] For another non-limiting example, based on reconstructed neighboring video blocks, predicting a filled sub-partition includes: filling the filled sub-partition with the leftmost sample of the bottom row of the corresponding sub-partition in the upper neighboring video block; or, filling the filled sub-partition with the topmost sample of the rightmost column of the corresponding sub-partition in the left neighboring video block.

[0226] In some implementations, method 2500 may further include: extracting an index from the encoded video stream, the index indicating a sample value in a sample buffer; and / or, step S2560 may include: filling a fill sub-partition with the sample value indicated by the index.

[0227] In some implementations, the sample buffer includes at least one of the following: a history-based buffer, or a space-based buffer.

[0228] In some implementations, method 2500 may further include: determining an intra-prediction mode with predefined padding rules; and / or, step S2560 may include: padding a padding sub-partition based on at least one reconstructed neighboring sample according to the intra-prediction mode.

[0229] In some implementations, the intra-prediction mode is determined based on at least one of the following methods: signaled by the encoded video bitstream, derived from adjacent encoded information, and / or selected from at least one predefined intra-prediction mode.

[0230] In some implementations, step S2560 may include: in response to the reconstruction of the upper adjacent video block, vertically extending the samples of the bottom row of the corresponding sub-partition in the upper adjacent video block to the filling sub-partition; in response to the reconstruction of the left adjacent video block, horizontally extending the samples of the rightmost column of the corresponding sub-partition in the left adjacent video block to the filling sub-partition; and / or, in response to the reconstruction of the upper and left adjacent video blocks: extracting a third syntax element from the encoded video stream, the third syntax element indicating whether to use the upper or left adjacent video block to predict the filling sub-partition; in response to the third syntax element indicating that the upper adjacent video block should be used to predict the filling sub-partition, vertically extending the samples of the bottom row of the corresponding sub-partition in the upper adjacent video block to the filling sub-partition; and / or, in response to the third syntax element indicating that the left adjacent video block should be used to predict the filling sub-partition, horizontally extending the samples of the rightmost column of the corresponding sub-partition in the left adjacent video block to the filling sub-partition.

[0231] In various embodiments, an intra-frame BC-coded block can be divided into two or more sub-partitions.

[0232] In some implementations, the segmentation of the coded block can be horizontal. At least one of these sub-partitions is predicted by padding sample values. The segmentation can be symmetric or asymmetric.

[0233] In some implementations, the segmentation of the coded block can be performed vertically. At least one of these sub-partitions is predicted by padding sample values. The segmentation can be symmetric or asymmetric.

[0234] In some implementations, the coded block can be segmented along a defined direction other than the horizontal or vertical direction; for example, the defined direction could be a diagonal direction (45 degrees or 135 degrees diagonal). At least one of these sub-partitions is predicted by padding sample values. The segmentation can be symmetric or asymmetric.

[0235] In various embodiments, at least one sub-partition of an intra-frame BC-coded block can be filled using at least one of the following padding methods.

[0236] One approach uses reconstructed neighboring sub-partitions to predict the filling area. For example, refer to... Figure 26A The second partition (or right subpartition) (2616) of the current block (2610) is populated by horizontally expanding the samples from the rightmost column (2613) of the first partition (or left subpartition) (2612) of the current block (2610). Each row of the second partition (2616) can be predicted by the same values ​​as the corresponding samples of the rightmost column (2613). The first partition (2612) can be predicted using IBC based on the reference block (2670).

[0237] In various embodiments / implementations of this disclosure, the first (or second) partition / subpartition may refer to more than just "one" partition / subpartition; however, when two partitions / subpartitions are arranged side-by-side, the first (or second) partition / subpartition may specifically refer to the left (or right) partition / subpartition, or when two partitions / subpartitions are arranged vertically, the first (or second) partition / subpartition may specifically refer to the upper (or lower) partition / subpartition.

[0238] In another approach, reconstructed neighboring samples outside the current coding block are used to predict the filling region. For example, refer to... Figure 26B The second partition (or right sub-partition) (2616) of the current block (2610) is filled by vertically expanding the samples from the bottom row (2620) of the adjacent reconstructed regions. The bottom row (2620) may correspond to the second partition (2616) and may have the same number of sample values ​​as the width (or horizontal direction) of the second partition (2616). Each column of the second partition (2616) can be predicted by the same values ​​as the corresponding samples of the bottom row (2620). The first partition (2612) can be predicted using IBC based on the reference block (2670).

[0239] In the third method, reconstructed samples from adjacent sub-partitions are used to predict the filling region. The selection of sample values ​​from the set of possible candidate objects is signaled or derived by both the encoder and decoder using the same rules. For example, refer to... Figure 26A The second partition (or right subpartition) (2616) of the current block (2610) is filled using the top sample value (2614) of the rightmost column (2613) in the first partition (or left subpartition) (2612) of the current block (2610). The second partition (2616) can be predicted using the same value as the sample (2614). The first partition (2612) can be predicted using IBC based on the reference block (2670).

[0240] In the fourth method, reconstructed neighboring sample values ​​outside the current coding block are used to predict the filling region. The selection of sample values ​​from the set of possible candidates is signaled or derived by both the encoder and decoder using the same rules. For example, refer to... Figure 26B The second partition (or right sub-partition) (2616) of the current block (2610) is filled using the leftmost sample (2621) of the bottom row (2620) of the adjacent reconstructed regions. The bottom row (2620) can correspond to the second partition (2616) and can have the same number of sample values ​​as the width (or horizontal direction) of the second partition (2616). The second partition (2616) can be predicted using the same values ​​as the sample (2621). The first partition (2612) can be predicted using IBC based on the reference block (2670).

[0241] In some implementations, the first partition of the current block can be filled, and the second partition can be predicted using IBC based on a reference block, similar to the various implementations / embodiments described in this disclosure. For example, for vertical block partitioning, vertical filling can be used for the first partition.

[0242] In some implementations, the current block can be divided using horizontal block partitioning, where the first partition can refer to the top partition and the second partition can refer to the bottom partition. Similar to the various implementations / embodiments described in this disclosure, the second partition of the current block can be filled using horizontal block partitioning, and the first partition can be predicted using IBC based on a reference block. Alternatively, the first partition of the current block can be filled, and the second partition can be predicted using IBC based on a reference block, similar to the various implementations / embodiments described in this disclosure; for example, for horizontal block partitioning, horizontal padding can be used for the first partition.

[0243] In the fifth method, existing sample values ​​stored in a history-based buffer are used to predict the filling region. An index pointing to an entry in the buffer indicates which sample in the buffer is used to predict the current filling region. The size of the history-based buffer is fixed. A first-in, first-out (FIFO) approach can be used to maintain this buffer. The rules for placing samples into the history buffer may vary. For example, sample values ​​that frequently appear in the current CTU or in other window sizes of recently encoded samples can be placed into the history buffer. In another example, only sample values ​​appearing from adjacent reconstructed regions, such as the left-adjacent rightmost column, the top-adjacent bottommost row, etc., can be considered.

[0244] In the sixth method, reconstructed neighboring samples are used to predict the filling region. Intra-prediction processing is used to perform the prediction, where neighboring reconstructed samples serve as input reference samples. The intra-prediction mode can be explicitly signaled, implicitly derived using neighboring encoded information, or selected from one or more predefined intra-prediction modes (e.g., DC, Smoothing, SMOOTH-H, SMOOTH-V).

[0245] Various embodiments in this disclosure may include signaling for partitioning or filling sample values.

[0246] In one approach, syntax elements using high-level syntax (e.g., slice header, image header, sequence header, VPS, PPS, SPS, etc.) indicate that the proposed method can be used for intra-frame BC prediction. This syntax element can be referred to as a flag, such as ibc_padding_enable_flag.

[0247] When ibc_padding_enable_flag is true, at the block level, another syntax element can be used to indicate whether to use regular intra-BC prediction or to use padding regions to predict the current block (assuming it is an intra-BC encoded block). This syntax element can be called another flag, such as ibc_padding_use_flag.

[0248] When ibc_padding_use_flag is true, other syntax elements can be used to indicate which part of the block is predicted by the padding operation. (See also...) Figure 27 Examples of four options for horizontal or vertical partitioning are shown, where the padding area is equal to 1 / 4 of the total block size. Additional partitioning of the padding area within the encoded block is also possible.

[0249] In some implementations, a syntax element is used to indicate whether a horizontal or vertical split is used; another syntax element is used to indicate whether the filled area is the first or second sub-partition of the block (from left to right, from top to bottom).

[0250] In some implementations, an index is used to indicate the size of the fill sub-partition. In one example, the size of the fill sub-partition can be selected from 3 / 4, 1 / 2, 1 / 4, ... of the whole block.

[0251] In various embodiments, when determining the filling region, the value used to predict the filling region can be determined in one of the following non-limiting examples.

[0252] For a non-restrictive example, for padding outside the current coded block using extended neighboring samples, the top (or left) neighboring sample can always be chosen when both the top and left neighboring samples are available. Alternatively, a syntax element can be used to indicate which neighboring sample to use. When only one side is available, that side can always be used. When no neighboring samples are available, such as when the current block is the first block in an image, tile, or slice, padding may not be used at all.

[0253] For another non-restricted example, several samples from reconstructed columns or rows in adjacent blocks or adjacent sub-partitions of the current block can be labeled as candidates for predicting the filling region. For example, the top, middle, and bottom samples of the rightmost adjacent column could be a set of candidates for selection. An index pointing to one of these can be signaled for selection at the block level.

[0254] In the embodiments and implementations of this disclosure, any steps and / or operations can be combined or arranged in any number or order as needed. Two or more steps and / or operations can be performed in parallel. The embodiments and implementations of this disclosure can be used individually or in combination in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. The embodiments of this disclosure can be applied to luma blocks or chroma blocks. The term block can be interpreted as a prediction block, coding block, or coding unit, i.e., CU. The term block in this document can also be used to refer to a transform block. In the following terminology, when referring to block size, it can refer to the width or height of the block, the maximum value of the width and height, the minimum value of the width and height, the area size of the block (width * height), or the aspect ratio (width:height, or, height:width).

[0255] The above techniques can be implemented as computer software that uses computer-readable instructions and is physically stored in one or more computer-readable media. For example, Figure 28 A computer system (2800) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0256] Computer software can be coded using any suitable machine code or computer language. Any suitable machine code or computer language can be assembled, compiled, linked, or similarly processed to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through decoding, microcode, etc.

[0257] The instructions can be executed on various types of computers or their components, including personal computers, tablets, servers, smartphones, gaming devices, and Internet of Things devices.

[0258] Figure 28 The components of the computer system (2800) shown are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependencies or requirements relating to any one or a combination of components shown in the exemplary embodiments of the computer system (2800).

[0259] The computer system (2800) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement of a data glove), audio input (e.g., speech, clapping), visual input (e.g., gestures), and olfactory input (not depicted). The human-machine interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographs acquired from a still-image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0260] The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard (2801), mouse (2802), touchpad (2803), touch screen (2810), data glove (not shown), joystick (2805), microphone (2806), scanner (2807), camera (2808).

[0261] The computer system (2800) may also include certain human-machine interface output devices. Such human-machine interface output devices may, for example, stimulate the senses of one or more human users through tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen (2810), data gloves (not shown), or joysticks (2805), but may also be tactile feedback devices that are not input devices), audio output devices (e.g., speakers (2809), headphones (not depicted)), visual output devices (e.g., screens (2810) including CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touch screen input functionality, each screen may or may not have tactile feedback functionality, some of which are capable of outputting two-dimensional visual output or more than three-dimensional output through devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

[0262] The computer system (2800) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2820) having media such as CD / DVD (2821), finger drives (2822), removable hard disk drives or solid-state drives (2823), conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC / PLD such as security dongles (not depicted), etc.

[0263] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

[0264] The computer system (2800) may also include an interface (2854) leading to one or more communication networks (2855). The network may be, for example, a wireless network, a wired network, or an optical network. The network may further be a local area network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a latency-tolerant network, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CANBus, etc. Some networks typically require an external network interface adapter (e.g., a USB port of the computer system (2800)) to connect to some general-purpose data port or peripheral bus (2849); other network interfaces are typically integrated into the core of the computer system (2800) by connecting to a system bus (e.g., an Ethernet interface connected to a PC computer system or a cellular network interface connected to a smartphone computer system). The computer system (2800) can use any of these networks to communicate with other entities. Such communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., CANbus connected to certain CANbus devices), or bidirectional, such as connecting to other computer systems using a local area network (LAN) or wide area network (WAN) digital network. As mentioned above, certain protocols and protocol stacks can be used on each of those networks and network interfaces.

[0265] The aforementioned human-machine interface device, human-machine accessible storage device, and network interface can be attached to the kernel (2840) of the computer system (2800).

[0266] The kernel (2840) may include one or more central processing units (CPU) (2841), graphics processing units (GPUs) (2842), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (2843), hardware accelerators (2844) for certain tasks, graphics adapters (2850), etc. These devices, along with read-only memory (ROM) (2845), random access memory (2846), and internal mass storage (2847) such as internal non-user-accessible hard disk drives, SSDs, etc., may be connected via a system bus (2848). In some computer systems, the system bus (2848) may be accessed in the form of one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly connected to the kernel's system bus (2848) or connected via a peripheral bus (2849). In this example, a display (2810) may be connected to a graphics adapter (2850). Peripheral bus architectures include PCI, USB, etc.

[0267] The CPU (2841), GPU (2842), FPGA (2843), and accelerator (2844) can execute certain instructions, which can be combined to form the aforementioned computer code. This computer code can be stored in ROM (2845) or RAM (2846). Transient data can also be stored in RAM (2846), while permanent data can be stored, for example, in internal mass storage (2847). Fast storage and retrieval to any storage device can be achieved using a cache, which can be closely associated with one or more CPUs (2841), GPUs (2842), mass storage (2847), ROM (2845), RAM (2846), etc.

[0268] Computer-readable media may have computer code thereon that performs various computer-implemented operations. The media and computer code may be media and computer code specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

[0269] As a non-limiting example, a computer system having an architecture (2800), particularly a kernel (2840), can be made functional by one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, and memory of certain non-transitory kernels (2840), such as internal kernel mass storage (2847) or ROM (2845). Software implementing various embodiments of this disclosure can be stored in such devices and executed by the kernel (2840). Depending on specific needs, the computer-readable media may include one or more storage devices or chips. The software can cause the kernel (2840), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (2846) and modifying such data structures according to software-defined processes. Additionally or alternatively, the computer system may provide functionality due to hard-wired or otherwise embodied logic in circuitry (e.g., the accelerator (2844)), which may replace or operate with the software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry embodying logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0270] Although several exemplary embodiments have been described in this disclosure, modifications, substitutions, and various equivalent alternatives that fall within the scope of this disclosure exist. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and thus fall within its spirit and scope.

[0271] Appendix A: Abbreviations

[0272] IBC: Intra-Block Coding

[0273] IntraBC: Intra-block coding

[0274] JEM: Joint Exploration Mode

[0275] VVC: Universal Video Coding

[0276] BMS: Benchmark Set

[0277] MV: Motion Vector

[0278] HEVC: High-Efficiency Video Coding

[0279] SEI: Supplemental Enhancement Information

[0280] VUI: Video Availability Information

[0281] GOPs: Image Groups

[0282] TUs: Transformation Unit

[0283] PUs: Prediction Units

[0284] CTUs: Coding Tree Units

[0285] CTBs: Coded Tree Blocks

[0286] PBs: Predicted Blocks

[0287] HRD: Hypothetical Reference Decoder

[0288] SNR: Signal-to-noise ratio

[0289] CPUs: Central Processing Unit

[0290] GPUs: Graphics Processing Units

[0291] CRT: Cathode Ray Tube

[0292] LCD: Liquid Crystal Display

[0293] OLED: Organic Light Emitting Diode

[0294] CD: CD-ROM

[0295] DVD: Digital Video Disc

[0296] ROM: Read-Only Memory

[0297] RAM: Random Access Memory

[0298] ASIC: Application-Specific Integrated Circuit

[0299] PLD: Programmable Logic Device

[0300] LAN: Local Area Network

[0301] GSM: Global System for Mobile Communications

[0302] LTE: Long Term Evolution

[0303] CANBus: Controller Area Network Bus

[0304] USB: Universal Serial Bus

[0305] PCI: Interconnect Peripheral Devices

[0306] FPGA: Field Programmable Gate Domain

[0307] SSD: Solid State Drive

[0308] IC: Integrated Circuit

[0309] HDR: High Dynamic Range

[0310] SDR: Standard Dynamic Range

[0311] JVET: Joint Video Exploration Team

[0312] MPM: Most Likely Pattern

[0313] WAIP: Wide-angle Intra-frame Prediction

[0314] CU: Encoding Unit

[0315] PU: Prediction Unit

[0316] TU: Transformation Unit

[0317] CTU: Coding Tree Unit

[0318] PDPC: Location-dependent prediction combination

[0319] ISP: Intra-Frame Sub-Partition

[0320] SPS: Sequence Parameter Set

[0321] PPS: Image Parameter Set

[0322] APS: Adaptation Parameter Set; VPS: Video Parameter Set

[0323] DPS: Decoding Parameter Set

[0324] ALF: Adaptive Loop Filter

[0325] SAO: Sample Adaptive Offset; CC-ALF: Cross Component Adaptive Loop Filter; CDEF: Constraint Direction Enhancement Filter

[0326] CCSO: Cross Component Sample Offset

[0327] LSO: Local Sample Offset

[0328] LR: Loop Recovery Filter

[0329] AV1: Open Media Alliance Video 1

[0330] AV2: Open Media Alliance Video 2

[0331] RPS: Reference Image Set

[0332] DPB: Decode Image Buffer

[0333] MMVD: Fusion Mode with Motion Vector Difference; IntraBC or IBC: Intra-Block Coding

[0334] BV: Block Vector

[0335] BVD: Block Vector Difference

[0336] RSM: Reference Sample Memory

Claims

1. A method for performing intra-block copy (IBC) padding in a current video block of a video bitstream, the method comprising: The device receives the current video block from the encoded video stream. The device includes a memory storing instructions and a processor communicating with the memory. A corresponding reference block partially matches the current video block. The reference block is an IBC reference block, and the IBC reference block is searched and selected in both local CTU / SB search and non-local CTU / SB search areas. The IBC reference block is identified by searching currently reconstructed CTU / SB samples. When the device uses on-chip memory, the write-back latency limitation for using reconstructed CTU / SB as the IBC reference is relaxed or eliminated. The memory usage differs when the local CTU / SB and the non-local CTU / SB coexist. The device extracts a first syntax element from the encoded video stream, the first syntax element indicating that the IBC padding can be used for the current video block; The device extracts a second syntax element from the encoded video stream, the second syntax element indicating the use of the IBC to fill and predict the current video block; At least one syntax element is extracted from the encoded video stream using the device; The device divides the current video block into at least two sub-partitions based on the at least one syntax element, wherein the at least two sub-partitions include a padding sub-partition and an IBC sub-partition; The device predicts the filled sub-partitions based on predefined filling rules. The device predicts the IBC sub-partition based on predefined IBC rules. The step of predicting the filled sub-partition based on predefined filling rules includes: when the IBC sub-partition has been reconstructed and the IBC sub-partition and the filled sub-partition are arranged horizontally, the samples of the rightmost column of the IBC sub-partition are horizontally extended to the filled sub-partition, and each row of the filled sub-partition is predicted using the same value as the corresponding sample of the rightmost column of the IBC sub-partition; or, when the IBC sub-partition has been reconstructed and the IBC sub-partition and the filled sub-partition are arranged vertically, the samples of the bottom row of the IBC sub-partition are vertically extended to the filled sub-partition, and each column of the filled sub-partition is predicted using the same value as the corresponding sample of the bottom row of the IBC sub-partition.

2. The method according to claim 1, wherein: The at least one syntax element includes a fourth syntax element and a fifth syntax element; as well as The step of dividing the current video block into at least two sub-partitions based on the at least one syntax element includes: Based on the fourth syntax element, the current video block is divided into at least two sub-partitions using horizontal or vertical splitting, and Based on the fifth syntax element, it is determined whether the filling sub-partition is the first sub-partition of the current video block or the second sub-partition of the current video block.

3. The method according to claim 2, wherein: The at least one syntax element includes a third syntax element; and The third syntax element indicates the fraction of the filled sub-partition relative to the size of the current video block.

4. The method according to claim 3, wherein: The fraction includes at least one of the following: 1 / 4, 1 / 2, or 3 / 4.

5. The method according to claim 1, wherein, The step of dividing the current video block into at least two sub-partitions based on the at least one syntax element includes: Based on the at least one syntax element, the current video block is divided into the at least two sub-partitions along directions defined other than horizontal or vertical.

6. An apparatus for decoding a current video block of a video stream, the apparatus comprising: Memory that stores instructions; as well as A processor communicating with the memory, wherein when the processor executes the instructions, the processor is configured to cause the device to perform the method of any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium storing instructions, wherein, When the instructions are executed by a processor, the instructions are configured to cause the processor to perform the method of any one of claims 1 to 5.

8. A method for processing video bitstreams, characterized in that, The video stream is decoded based on the method for performing intra-block copy (IBC) padding in the current video block of the video stream as described in any one of claims 1 to 5.

Images