Encoding / decoding video picture data

Abstract

The present disclosure relates to intra prediction of a target block of a video picture based on template matching that compares a target template patch to a candidate reference template patch. The intra prediction includes identifying a candidate reference block neighborhood from a target block neighborhood and from a geometric transform type; obtaining the candidate reference template patch by applying a geometric transform identified by the geometric transform type to the candidate reference block neighborhood; selecting a candidate reference block having a minimum cost obtained by comparing the candidate reference template patch to the target template patch; and generating an intra prediction block for the target block by applying a geometric transform identified by the geometric transform type associated with the selected candidate reference block to the selected candidate reference block.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and benefit to European Patent Application No. 22306422.1, filed on 27 September 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application generally relates to video frame encoding and decoding. In particular, but not limited thereto, the technical field of this application relates to the prediction of video frame blocks based on intra-frame template matching. Background Technology

[0004] This section aims to introduce the reader to various aspects of the art that may relate to various aspects of at least one embodiment of the present application described below and / or claimed. This discussion is intended to help provide the reader with background information to facilitate a better understanding of the various aspects of the present application. Therefore, these statements should be understood to be read in this light rather than as an admission of prior art.

[0005] In state-of-the-art video compression systems such as HEVC (ISO / IEC 23008-2 High Efficiency Video Coding, ITU-T Recommendation H.265, https: / / www.itu.int / rec / T-REC-H.265-202108-P / en) or VVC (ISO / IEC 23090-3 Versatile Video Coding, ITU-T Recommendation H.266, https: / / www.itu.int / rec / T-REC-H.266-202008-I / en), low-level and high-level picture partitioning is provided to divide the video frame into picture regions, called codec tree units (CTUs). Their size is typically between 16x16 and 64x64 pixels for HEVC, and between 32x32, 64x64, or 128x128 pixels for VVC.

[0006] The CTU (Central Unit) division of a video frame forms a grid of fixed-size CTUs, known as the CTU grid. Its upper and left boundaries coincide spatially with the top and left boundaries of the video frame. The CTU grid represents the spatial partitioning of the video frame.

[0007] In VVC and HEVC, the CTU size (CTU width and CTU height) of all CTUs in the CTU mesh is equal to the same default CTU size (default CTU width CTU DW and default CTU height CTU DH). For example, the default CTU size (default CTU height, default CTU width) can be equal to 128 (CTU DW = CTU DH = 128). The default CTU size (height, width) is encoded into the bitstream, for example, at the sequence level in the Sequence Parameter Set (SPS).

[0008] The spatial position of a CTU within the CTU mesh is determined by its address ctuAddr, which defines the spatial position of the CTU's top-left corner relative to the origin. For example... Figure 1 As shown, the CTU address can define a spatial location starting from the top left corner of the higher-level spatial structure S containing the CTU.

[0009] Each CTU is associated with a codec tree to determine the tree partitioning of the CTU.

[0010] like Figure 1 As shown, in HEVC, the codec tree is a quadtree partition of the CTU, where each leaf is called a codec unit (CU). The spatial location of a CU in a video frame is defined by the CU index cuIdx, which indicates the spatial location starting from the top left corner of the CTU. CUs are spatially partitioned into one or more prediction units (PUs). The spatial location of a PU in a video frame (VP) is defined by the PU index puIdx, which defines the spatial location starting from the top left corner of the CTU, and the spatial location of the elements of a partitioned PU is defined by the PU partition index puPartIdx, which defines the spatial location starting from the top left corner of the PU. Each PU is assigned some intra-frame or inter-frame prediction data.

[0011] The intra-frame or inter-frame encoding / decoding mode is assigned at the CU level. This means that although the prediction parameters vary from PU to CU, each PU is assigned the same intra-frame / inter-frame encoding / decoding mode.

[0012] According to a quadtree called a transform tree, a CU can also be spatially partitioned into one or more transform units (TUs). A transform unit is a leaf of the transform tree. The spatial location of a TU in a video frame is defined by a TU index tuIdx, which defines the spatial location starting from the top-left corner of the CU. Each TU is assigned a number of transform parameters. The transform type is assigned at the TU level, and individual 2D transforms are performed at the TU level during the encoding / decoding or decoding of a frame block.

[0013] Figure 2The diagram illustrates the existing PU partition types in HEVC. These include square partitions (2N×2N and N×N), which are the only partitions used in both intra-frame and inter-frame prediction CUs; ​​symmetric non-square partitions (2N×N, N×2N, used only in inter-frame prediction CUs); and asymmetric partitions (used only in inter-frame prediction CUs). For example, PU type 2NxnU represents an asymmetric horizontal partition of the PU, where the smaller partition is located at the top of the PU. According to another example, PU type 2NxnL represents an asymmetric horizontal partition of the PU, where the smaller partition is located at the top of the PU.

[0014] like Figure 3 As shown, in VVC, the codec tree starts from the root node (i.e., CTU). Next, a quadtree (or quaternion tree) split divides the root node into four nodes, corresponding to four child blocks of equal size (solid lines). Then, the leaves of the quadtree (or quaternion tree) can be further partitioned using what is called a multi-type tree, which involves... Figure 4 One of the four splitting patterns shown is used for binary or ternary splitting. These splitting types are vertical and horizontal binary splitting patterns (denoted as SBTV and SBTH) and vertical and horizontal ternary splitting patterns SPTTV and STTH.

[0015] In the case of a joint codec tree where luminance and chrominance components share a common codec tree, the leaf of the CTU's codec tree is the CU.

[0016] In contrast to HEVC, in VVC, CU, PU, ​​and TU have the same size in most cases, which means that, except in some specific codec modes, codec units are generally not partitioned into PU or TU.

[0017] Figure 5 and Figure 6 It provides an overview of video encoding / decoding methods used in current video standard compression systems such as VVC.

[0018] Figure 5 A schematic block diagram illustrating the steps of a method 100 for encoding a video frame VP according to the prior art is shown.

[0019] In step 110, the video frame VP is partitioned into a CTU grid, and the partition information data is signaled and transmitted to the bitstream. Each CTU in the CTU grid is associated with a codec tree, and each CU in the codec tree associated with each CTU is a block of samples of the video frame VP. In short, the CU of a CTU is a block.

[0020] Following the scan order, typically the raster scan order of the video frame, consider the CTU grid of the CTU. Also following the scan order, typically the raster scan order of the blocks of the CTU, consider each block of the CTU.

[0021] Then, each block of each CTU is encoded using intra-frame or inter-frame predictive coding / decoding modes.

[0022] Intra-frame prediction (step 120) involves predicting the current block using blocks predicted based on encoded, decoded, and reconstructed samples located around the current block, typically at the top and left of the current block. Intra-frame prediction is performed in the spatial domain.

[0023] In inter-frame prediction mode, motion estimation (step 130) and motion compensation (135) are performed. Motion estimation searches for candidate reference blocks in one or more reference video frames used for predictively encoding the current video frame to serve as good predictors for the current block. For example, a good predictor for the current block is a predictor similar to the current block. The output of motion estimation step 130 is one or more motion vectors and reference frame indices associated with the current block. Next, motion compensation (step 135) obtains the prediction block using the motion vector(s) and reference frame indices(s) determined by motion estimation step 130. Essentially, the block belonging to the selected reference frame and pointed to by the motion vector can be used as the prediction block for the current block. Furthermore, since motion vectors are expressed as fractions of integer pixel positions (this is referred to as sub-pixel precision motion vector representation), motion compensation generally involves spatial interpolation of some reconstructed samples of the reference frames to compute prediction block samples.

[0024] The prediction information data is signaled and transmitted to bitstream B. The prediction information may include prediction mode, prediction information encoding / decoding mode, intra-frame prediction mode, or one or more motion vectors and reference frame indices, as well as any other information used to obtain the same prediction block on the decoding side.

[0025] Taking into account the encoding of the calculated predictive residual block (e.g., by subtracting the candidate predictive block from the current block) and the signaling transmission of the predictive information data required to determine the candidate predictive block on the decoding side, method 100 selects one of the intra-frame mode or inter-frame encoding / decoding mode by optimizing the rate-distortion tradeoff.

[0026] Typically, the best prediction mode is given as the prediction mode of the best encoding / decoding mode p* for the current block, as given by the following formula:

[0027]

[0028] Where P is the set of all candidate encoding / decoding modes for the current block, p represents the candidate encoding / decoding mode in this set, and RD cost (p) is the rate-distortion cost of the candidate encoding / decoding mode p, usually expressed as:

[0029] RD cost(p)=D(p)+λ.R(p)

[0030] D(p) is the distortion between the current block and the reconstructed block obtained after encoding / decoding the current block with candidate encoding / decoding mode p, R(p) is the rate cost associated with encoding / decoding the current block with encoding / decoding mode p, and λ is the Lagrangian parameter representing the rate constraint for encoding / decoding the current block and is usually calculated based on the quantization parameters used to encode the current block.

[0031] The current block is typically encoded from the prediction residual block PR. More precisely, for example, the prediction residual block PR is calculated by subtracting the best prediction block from the current block. The prediction residual block PR is then transformed (step 140) using a transform of type DCT (Discrete Cosine Transform) or DST (Discrete Sine Transform), and the resulting transformed coefficient block is quantized (step 150).

[0032] In a variant, method 100 can also skip transform step 140 according to a so-called transform-skip encoding / decoding mode and apply quantization directly to the predicted residual block PR.

[0033] Encode the quantized transform coefficient block (or quantized prediction residual block) into bitstream B (step 160).

[0034] Next, the quantized transform coefficient block (or quantized residual block) is dequantized (step 170) and inverse transformed (180) (or not) to obtain the decoded prediction residual block. Then, the decoded prediction residual block and the prediction block are combined (usually summed) to provide the reconstructed block.

[0035] Entropy encoding can also be applied to other information data to encode the current block of the video frame VP.

[0036] A loop filter (step 190) can be applied to the reconstructed image (including reconstructed blocks) to reduce compression artifacts. Loop filters can be applied after all image blocks have been reconstructed. Examples include deblocking filters, Sample Adaptive Offset (SAO) filters, or adaptive loop filters.

[0037] The reconstructed block or the filtered reconstructed block forms a reference frame, which can be stored in the decoded frame buffer (DPB) so that it can be used as a reference frame for encoding the next current block of the video frame VP or as a reference frame for encoding the next video frame to be encoded.

[0038] Figure 6 A schematic block diagram illustrating the steps of a method 200 for decoding a video frame VP according to the prior art is shown.

[0039] In step 210, partition information data, prediction information data, and quantized transform coefficient blocks (or quantized residual blocks) are obtained by entropy decoding bit stream B.

[0040] The partition information data defines the CTU grid (layout) on the video frame. The CTU grid divides the video frame (VP) into multiple CTUs. The CTUs of the CTU grid are considered along the scan order, typically the raster scan order of the video frame (VP). Similarly, the blocks of the considered CTUs are considered along the scan order, typically the raster scan order of the blocks of CTUs.

[0041] Other information data can also be decoded from bitstream B to decode the current block of the current CTU in the CTU grid.

[0042] In step 220, each current block of the current CTU is entropy decoded.

[0043] Each decoded current block can be a quantized transform coefficient block or a quantized prediction residual block.

[0044] In step 230, the current block (of the current CTU) is dequantized and possibly inverse transformed (step 240) to obtain the decoded prediction residual block.

[0045] On the other hand, the current block is predicted using prediction information data. The predicted block is obtained through its intra-frame prediction (step 250) or its motion-compensated temporal prediction (step 260). The prediction process performed on the decoding side is the same as the prediction process performed on the encoding side.

[0046] Next, the decoded prediction residual block and prediction block are combined (usually by summing), which provides the reconstructed block.

[0047] In step 270, the loop filter can be applied to the reconstructed frame (including the reconstructed block), and the reconstructed block or the filtered reconstructed block forms a reference frame, which can be stored in the decoded frame buffer (DPB), as discussed above. Figure 5 ).

[0048] Intra-Frame Template Matching Prediction (ITMP) mode is a special intra-frame prediction mode for the target block (the current block of the video frame to be predicted). It copies the best block (the prediction block) from the reconstructed block of the video frame, and its L-shaped template matches the L-shaped template of the target block. Essentially, for a predefined search region (search range), the encoder searches for the L-shaped template of the reconstructed block of the video frame that is most similar to the L-shaped template of the target block, and uses the corresponding block as the best prediction block. The encoder then signals the use of the ITMP mode as prediction information and performs the same prediction operation on the decoding side.

[0049] In ECM (Algorithm description of Enhanced Compression Model 5 (ECM 5), M. Coban et al., JVET-Z 2025, April 2022), ITMP mode is enabled for blocks with a size less than or equal to a maximum size of 64 (width and height). This maximum size is configurable.

[0050] Figure 7 The diagram illustrates a block diagram of a method 300 for intra-frame template matching prediction (ITMP) of target blocks in a video frame according to existing technology.

[0051] Method 300 is implemented in the same way as both Method 100 (encoding) and Method 200 (decoding), thus saving signaling data with controlled additional complexity. Only ITMP usage can be signaled using a single flag carried at the CU level via the dedicated syntax element intra_tmp_flag. Figure 20 On both the encoding and decoding sides, the IMTP mode can be tested as an intra-predictor candidate for predicting target blocks.

[0052] In step 310, the target block neighborhood is analyzed to determine the space of the search region R. The search region R typically includes multiple available CTUs, and a search region boundary is determined for each available CTU.

[0053] Possibly, if such a CTU is available, the search area R may slightly extend beyond the boundaries of that CTU. For implementation purposes, there may be slight overlap between two different search areas.

[0054] Figure 8 The search area is shown as defined by existing technology for predicting target blocks (TB) of video frames.

[0055] Typically, the search area consists of four spatial regions: R1, R2, R3, and R4. Region R1 is the current CTU, which includes the target block TB (the block to be predicted in the video frame), region R2 is the top left CTU, R3 is the top CTU, and R4 is the left CTU.

[0056] Set the width SearchRange_w and height SearchRange_h of region R1-R3 to be proportional to the width BlkW and height BlkH of the target block to have a fixed number of comparisons per pixel. That is:

[0057] SearchRange_w=a*BlkW

[0058] SearchRange_h = a * BlkH (2)

[0059] Here, 'a' is a constant integer value that controls the gain / complexity tradeoff. The coefficient 'a' can depend on the width and height of the target block, the width of the target template (e.g., fixed at 4 samples), and / or the CTU size. In practice, 'a' equals 5.

[0060] In step 320, a target block neighborhood (TBN) is defined based on the availability of reconstructed samples in the current CTU. The TBN includes the target block TB and the surrounding CTUs. The TBN defines the locations of available reconstructed samples for constructing the target template patch (TTP), as discussed later.

[0061] For example, such as Figure 8 As illustrated in the diagram, when the target block TB is vertically or horizontally flipped, the adjacent samples located below or to the right of the target block are unavailable (not yet reconstructed) and cannot be used to generate [the desired data]. Figure 9 The target block neighborhood (TBN) used in the template matching candidate search.

[0062] If reconstructed samples are available in the L-shaped neighborhood of the target block, then the target block neighborhood TBN is of the "L-shaped template" type (e.g., ...). Figure 8 As illustrated, if only the reconstructed samples in the left part of the target block TB are available (i.e., when TB reaches the top boundary of the video frame or the search range R, the upper CTU samples are usually unavailable), then the target block neighborhood TBN is of the "left template" type. If only the reconstructed samples in the upper part of the target block TB are available (i.e., when TB reaches the left boundary of the video frame or the search range R, the left CTU samples are usually unavailable), then the target block neighborhood TBN is of the "upper template" type. Or if no reconstructed samples are available around the target block (typically, when the target block TB reaches the top left corner of the video frame or the search range R), then the target block neighborhood TBN is of the "no template" type.

[0063] If the target block neighborhood TBN is "templateless", then in step 330 the prediction block used to predict the target block TB is set to the DC (direct encoding / decoding) value, for example, 1 << (1 – bit depth). Y ), where bitdepth Y This represents the bit depth of the brightness sample in the video image.

[0064] Otherwise, in step 340, a candidate reference block RB is selected from the search range R based on the template matching candidate search. The predicted block of the target block is the candidate reference block RB.

[0065] Figure 9 The diagram illustrates a block diagram of template matching candidate search (step 340) according to the prior art.

[0066] The Target Template Block (TTP) is associated with the Target Block (TB). The TTP is constructed from available reconstructed samples in the Target Block Neighborhood (TBN). The availability of reconstructed samples in the TBN depends on the position of the Target Block (TB) relative to the video frame or search region boundary.

[0067] For each current position in the search region R, in step 341, candidate reference blocks RB located at the current position are considered, and candidate reference template blocks RTP are associated with the candidate reference blocks RB. The candidate reference template RTP is constructed from reconstructed samples in the neighborhood RBN of the candidate reference block.

[0068] The candidate reference block neighborhood (RBN) is associated with the candidate reference block (RB). The candidate reference block neighborhood (RBN) defines the position of the reconstructed sample in the search region R within the neighborhood of the candidate reference block (RB). The candidate reference block neighborhood (RBN) is defined based on the target block neighborhood (TBN), meaning the position defined by the candidate reference block neighborhood (RBN) is defined based on the position of the target block neighborhood (TBN). The candidate reference block neighborhood (RBN) and the target block neighborhood (TBN) have the same shape and layout.

[0069] In step 342, the cost is correlated with the candidate reference block RB to evaluate the matching between the reconstruction samples of the target template block TTP and the reconstruction samples of the candidate reference template block RTP.

[0070] If the current cost does not improve compared to the optimal cost, then consider another location in the search area (step 341).

[0071] If the current cost improves the optimal cost, in step 343, the optimal cost is set to equal the current cost, and the candidate reference block RB is set as the optimal candidate reference block.

[0072] Step 343 is followed by step 341, and so on, until all positions in the search area have been considered.

[0073] For example, cost is the sum of absolute differences (SAD) between the reconstruction samples of the candidate reference block RTP and the reconstruction samples of the target template block TTP.

[0074] Template matching is an important area in image and video processing. However, due to its inherent complexity, it has been excluded from video standardization activities in the past. With the introduction of template matching into ECM by proposals JVET-U0048 (Evaluation of Template Matching Prediction for VVC, K. Naser et al., JVET-U0048, January 2021; and EE2: Intra Template Matching, K. Naser et al., JVET-V0130, April 2021), template matching has been applied to intra-predictive mode coding and decoding. In fact, ECM has lower requirements in terms of affordability of complexity because compression performance dominated this exploratory phase before the assumed standardization phase itself.

[0075] The target block neighborhood (TBN) (or candidate reference block neighborhood (RBN)) defined in the prior art limits the matching possibilities. It consists of a horizontal (when the upper reconstructed sample is available) band and / or a vertical (when the left reconstructed sample is available) band of 4 samples next to the target block, which matches the candidate reference template block (the band of 4 samples next to the candidate reference block) defined within the search area.

[0076] The technical problem addressed by this disclosure is to improve the compression efficiency and usability of the ITMP mode.

[0077] At least one embodiment of this application was designed with the above circumstances in mind. Summary of the Invention

[0078] The following section provides a brief overview of at least one embodiment to provide a basic understanding of some aspects of this application. This overview is not an exhaustive summary of the embodiments. Its purpose is not to identify key or core elements of the embodiments. The following overview presents only some aspects of at least one embodiment in a simplified form as a prelude to a more detailed description provided elsewhere in this document.

[0079] According to a first aspect of this application, a method for intra-frame prediction of target blocks in a video frame based on template matching is provided. The template matching compares a target template block including reconstructed samples in the neighborhood of the target block with a candidate reference template block including reconstructed samples in the neighborhood of a candidate reference block located in a search region. The method includes:

[0080] - Reconstruct sample availability based on the target block neighborhood and determine candidate reference block neighborhoods based on the geometric transformation type;

[0081] - By applying a geometric transformation identified by the geometric transformation type to the reconstruction samples in the neighborhood of the candidate reference block, reconstruction samples of the candidate reference template block are obtained;

[0082] - Associate the cost with a candidate reference block, which is obtained by comparing the reconstruction samples of the candidate reference template block with the reconstruction samples of the target template block;

[0083] - Select a candidate reference block with the lowest cost, the selected candidate reference block being associated with a geometric transformation type identified by the geometric transformation; and

[0084] - By applying a geometric transformation identified by the geometric transformation type associated with the selected candidate reference block to the reconstructed samples of the selected candidate reference block, an intra-prediction block for the target block is generated.

[0085] This disclosure is advantageous because template matching candidate search incurs very low cost even when the sample pattern represented in the candidate reference template block is symmetrical to the target template block. Sample pattern symmetry is likely to occur in computer graphics or screen content. In other words, this disclosure captures the similarity between visual content in the neighborhood of the target block and visual content in the neighborhood of the candidate reference block, even if one of these visual contents is a geometric transformation of the other. This disclosure can be applied to both IMTP applied to luminance samples of the target block TB and IMTP applied to chrominance samples of the target block.

[0086] In one embodiment, the method further includes adjusting the search area based on the geometric transformation type and the position of the target block in the video frame.

[0087] In one embodiment, for each indication of a geometric transformation having a set of geometric transformations or an indication of a current geometric transformation type having at least two combinations of such geometric transformations,

[0088] - Determine the candidate reference block neighborhood based on the availability of reconstruction samples in the target block neighborhood and the current geometric transformation type;

[0089] - For each current position in the search region, obtain a reconstruction sample of the candidate reference template block of the candidate reference block located at the current position in the search region, and associate the cost with the candidate reference block located at the current position in the search region;

[0090] -Having already considered all locations within the search region, consider another type of geometric transformation.

[0091] -After considering all geometric transformation types, the selected candidate reference block is the candidate reference block with the lowest cost (450).

[0092] In one embodiment, for each current position in the search region, the current geometric transformation type, which indicates a set of geometric transformations or a combination of at least two such geometric transformations, is considered.

[0093] - Determine the candidate reference block neighborhood based on the availability of reconstruction samples in the target block neighborhood and the current geometric transformation type;

[0094] - Obtain a reconstruction sample of the candidate reference template block of the candidate reference block located at the current position in the search area;

[0095] - Associate the cost with a candidate reference block located at the current position;

[0096] -Once all types of geometric transformations have been considered, consider another current location within the search area;

[0097] -When all locations in the search area have been considered, the selected candidate reference block is the candidate reference block with the lowest cost.

[0098] In one embodiment, for each geometric transformation type indicating a set of geometric transformations or indicating a combination of at least two of those geometric transformations, the current position in the search region is considered.

[0099] - Determine the candidate reference block neighborhood based on the availability of reconstruction samples and the type of geometric transformation in the target block neighborhood;

[0100] - For each current position in the search region, obtain a reconstruction sample of the candidate reference template block of the candidate reference block located at the current position in the search region, and associate the cost with the candidate reference block located at the current position in the search region;

[0101] -When all locations in the search area have been considered, the selected candidate reference block is the candidate reference block with the lowest cost.

[0102] In one embodiment, the cost associated with a candidate reference block is evaluated by the sum of the absolute differences between the transformed and reordered samples of the reference template block and the reconstructed samples of the target template block.

[0103] In one embodiment, the geometric transformation type indicates at least one of the following geometric transformations or a combination of at least two of the following geometric transformations:

[0104] - Flip horizontally;

[0105] - Flip vertically;

[0106] - Rotation;

[0107] -Constant.

[0108] In one embodiment, the rotation angle is a multiple of 90°.

[0109] In one embodiment, the method further includes signaling the geometric transformation type associated with the selected candidate reference block.

[0110] According to a second aspect of this application, a method is provided for encoding a block of a video frame using block-based prediction blocks, wherein the prediction blocks are obtained according to any of the methods of the first aspect.

[0111] According to a third aspect of this application, a method for decoding a block of a video frame based on block-based prediction blocks is provided, wherein the prediction blocks are obtained according to any of the methods of the first aspect.

[0112] According to a fourth aspect of this application, a bitstream is provided, which is formatted to include encoded video image data obtained by the method according to the first aspect.

[0113] According to a fifth aspect of this application, an apparatus is provided, comprising tools for performing one of the methods according to the first, second, or third aspects.

[0114] According to a sixth aspect of this application, a computer program product including instructions is provided, which, when executed by one or more processors, causes the one or more processors to perform the method according to the first, second, or third aspect.

[0115] According to a seventh aspect of this application, a non-transient storage medium is provided that carries instructions for executing program code according to the methods of the first, second, or third aspects.

[0116] According to an eighth aspect of this application, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor is configured to perform one of the methods according to the first, second, or third aspect.

[0117] The specific nature of at least one embodiment, as well as other objects, advantages, features, and uses of said at least one embodiment, will become apparent from the following description of the examples taken in conjunction with the accompanying drawings. Attached Figure Description

[0118] Reference will now be made to the accompanying drawings, which illustrate embodiments of this application by way of example, wherein:

[0119] Figure 1 An example of a codec tree unit based on HEVC is shown;

[0120] Figure 2 An example of partitioning codec units into prediction units according to HEVC is shown;

[0121] Figure 3 An example of CTU partitioning based on VVC is shown;

[0122] Figure 4 An example of the splitting patterns supported in VVC's multi-type tree partitioning is shown;

[0123] Figure 5 A schematic block diagram illustrating the steps of a method 100 for encoding a video frame VP according to the prior art is shown;

[0124] Figure 6 A schematic block diagram of the steps of a method 200 for decoding a video frame VP according to the prior art is shown;

[0125] Figure 7 The diagram illustrates a block diagram of a method 300 for intra-frame template matching prediction of target blocks in a video frame based on existing technology.

[0126] Figure 8 The search area used in ITMP according to existing technology is shown;

[0127] Figure 9 The diagram illustrates a block diagram of template matching candidate search (step 340) according to the prior art;

[0128] Figure 10 The schematic diagram illustrates a block diagram of a method 400 for intra-frame prediction of target blocks of a video frame based on template matching according to an embodiment.

[0129] Figure 11 and Figure 12 The illustration shows an example of adjusting the search area at the boundary of a video frame or search area according to an embodiment;

[0130] Figure 13 The illustration shows the geometric transformation according to an embodiment when it is a horizontal flip. Figure 10 Method 400;

[0131] Figure 14 The illustrations show some instances determined according to a neighborhood template based on a flip (geometric transformation type) according to an embodiment;

[0132] Figure 15 The illustrations show some instances of neighborhood templates determined according to a rotation (geometric transformation type) based on an embodiment;

[0133] Figure 16 A block diagram of method 400 according to an embodiment is schematically illustrated;

[0134] Figure 17 A block diagram of method 400 according to an embodiment is schematically illustrated;

[0135] Figure 18 A block diagram of method 400 according to an embodiment is schematically illustrated;

[0136] Figure 19 The diagram illustrates the syntactic element structure for transmitting geometric transformation types of signals according to an embodiment;

[0137] Figure 20 The illustration shows a schematic block diagram of an example of a system in which various aspects and embodiments are implemented.

[0138] Similar or identical elements are referenced using the same figure labels. Detailed Implementation

[0139] At least one embodiment will be described more fully below with reference to the accompanying drawings, which depict examples of at least one embodiment. However, embodiments may be implemented in various alternative forms and should not be construed as limited to the examples set forth herein. Accordingly, it should be understood that the invention is not intended to limit the embodiments to the specific forms disclosed. Rather, this application is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of this application.

[0140] At least one of these aspects generally relates to video image encoding and decoding, another generally relates to the transmission of a bit stream provided or encoded, and yet another relates to the reception / access of a decoded bit stream.

[0141] At least one of the embodiments is described in relation to encoding / decoding a single video frame, but this is extended to encoding / decoding multiple video frames (a sequence of frames) as each video frame is encoded / decoded sequentially as described below.

[0142] Furthermore, at least one embodiment is not limited to the current version of VVC. At least one embodiment can be applied to VVC and recommended existing or future developments and extensions. Unless otherwise stated or technically excluded, the aspects described in this application may be used alone or in combination.

[0143] A pixel corresponds to the smallest display unit on the screen, and it can consist of one or more light sources (one for a monochrome screen and three or more for a color screen).

[0144] A video frame, also known as a frame or video frame, includes at least one component (also known as a video component or channel) defined by a specific video / video format that specifies all information related to pixel values ​​and all information that can be used by a display unit and / or any other device to display and / or decode video frame data associated with the video frame.

[0145] The video frame includes at least one component, which is typically represented as an array of samples.

[0146] Monochrome video footage consists of a single component, while color video footage can consist of three components.

[0147] For example, when the picture / video format is a well-known (Y,Cb,Cr) format, a color video picture may include one luminance (or brightness) component and two chrominance components, or when the picture / video format is a well-known (R,G,B) format, a color video picture may include three color components (one for red, one for green, and one for blue).

[0148] Each component of a video frame can include a number of samples relative to the number of pixels on the screen on which the video frame is to be displayed. In variations, the number of samples included in a component can be a multiple (or fraction) of the number of samples included in another component of the same video frame.

[0149] For example, in a video format that includes one luminance component and two chrominance components (such as the (Y,Cb,Cr) format), depending on the color format under consideration, the chrominance component may contain half the number of samples in width and / or height relative to the luminance component.

[0150] A sample is the smallest unit of visual information that makes up a video frame. Sample values ​​can be, for example, luminance or chrominance values, or color values ​​in (R,G,B) format.

[0151] The reconstructed sample is a sample that has been encoded according to method 100 and decoded according to method 200.

[0152] A pixel value is the value of a pixel on the screen. For monochrome video, a pixel value can be represented by a single sample, while for color video, a pixel value can be represented by multiple samples at co-locations. A co-location sample associated with a pixel refers to the sample corresponding to the pixel's position on the screen.

[0153] Video frames are typically viewed as a set of pixel values, with each pixel represented by at least one sample.

[0154] A block of video frame is a set of samples of one component of the video frame. When the frame / video format is a well-known (Y,Cb,Cr) format, a block of at least one luminance sample or a block of at least one chrominance sample can be considered; or when the frame / video format is a well-known (R,G,B) format, a block of at least one color sample can be considered.

[0155] At least one embodiment is not limited to a specific image / video format.

[0156] Generally, this disclosure relates to a method for intra-frame prediction of target blocks in a video frame based on template matching, wherein the template matching compares a target template block including reconstructed samples in the neighborhood of the target block with a candidate reference template block including reconstructed samples in the neighborhood of a candidate reference block located in a search region. The method includes: determining a candidate reference block neighborhood based on the availability of reconstructed samples in the target block neighborhood and based on a geometric transformation type; obtaining reconstructed samples of the candidate reference template block by applying a geometric transformation identified by a geometric transformation type to the reconstructed samples in the candidate reference block neighborhood; associating a cost with a candidate reference block obtained by comparing the reconstructed samples of the candidate reference template block and the reconstructed samples of the target template block; selecting a candidate reference block with the lowest cost, said selected candidate reference block being associated with a geometric transformation type identified by a geometric transformation; and generating an intra-frame predicted block for the target block by applying a geometric transformation identified by the geometric transformation type associated with the selected candidate reference block to the reconstructed samples of the selected candidate reference block.

[0157] Figure 10 The schematic diagram illustrates a block diagram of a method 400 for intra-frame prediction of a target block (TB) of a video frame based on template matching according to an embodiment.

[0158] Method 400 is based on template matching candidate search, which compares the target template block TTP associated with the target block TB with the candidate reference template block RTP associated with the candidate reference block RB located in the search region R.

[0159] In step 410, the availability of reconstructed samples and the geometric transformation type GTT (see example) are determined based on the target block neighborhood. Figure 14 and Figure 15 Determine the candidate reference block neighborhood (RBN).

[0160] The geometric transformation type GTT identifies the geometric transformation GT.

[0161] In step 430, a candidate reference template block RTP associated with a candidate reference block RB located in the search region is obtained. The candidate reference template block RTP includes a reconstruction sample. This reconstruction sample is obtained by applying a geometric transformation identified by the geometric transformation type GTT to the reconstruction sample in the neighborhood RBN of the candidate reference block.

[0162] For example, if the target block neighborhood TBN indicates that the upper and left adjacent reconstructed samples of the target block TB are available, then the target template block TTP includes the upper and left adjacent reconstructed samples of the target block TB. If the geometric transformation is a horizontal flip, then the transformed reconstructed sample positions correspond to the upper and right reconstructed samples in the candidate reference block neighborhood RTP.

[0163] In another instance, if the target block neighborhood TBN indicates that the upper and left adjacent reconstructed samples of the target block TB are available, then the target template block TTP includes the upper and left adjacent reconstructed samples of the target block TB. If the geometric transformation is a vertical flip, then the transformed reconstructed sample positions correspond to the bottom and left reconstructed samples in the candidate reference block neighborhood RTP.

[0164] The reconstructed samples of the target block do not directly correspond to the reconstructed samples of the candidate reference template block RTP (i.e., they are not co-located relative to the neighborhood of the reference and target blocks) because a geometric transformation has already been applied to the reconstructed sample locations.

[0165] When the geometric transformation GT is a horizontal flip Figure 13 Step 3 provides an example of a reordered, transformed sample.

[0166] In step 440, the cost is associated with the candidate reference block RB by comparing the reconstruction sample of the candidate reference template block RTP with the reconstruction sample of the target template block TTP.

[0167] In one embodiment, the cost is evaluated by the sum of absolute differences (SAD) between the reconstructed samples of the reference template block RTP and the reconstructed samples of the target template block TTP.

[0168] In step 450, a candidate reference block is selected. This corresponds to the candidate reference template block RTP that provides the lowest cost. The selected candidate reference block RB is associated with a geometric transformation type GTT, which identifies the geometric transformation GT applied at the reconstructed sample location.

[0169] In step 460, an intra-prediction block for the target block TB is generated by applying a geometric transformation GT identified by the geometric transformation type GTT associated with the selected candidate reference block RB to the reconstructed samples of the selected candidate reference block RB.

[0170] In one embodiment, method 400 further includes step 420, in which the search area is adjusted according to the geometric transformation type GTT and the position of the target block TB in the video frame.

[0171] Compared to method 300 (such as the ITMP method defined in the prior art), the boundaries of the search region have been adjusted. Therefore, the search region must contain candidate reference and target template block reconstruction samples. In fact, in the case of vertical and / or horizontal flipping, candidate reference template block reconstruction samples at the bottom edge and / or right edge of the candidate reference block RB should be available.

[0172] Therefore, in the case of vertical flipping, the search area must take into account the template strip (e.g., 4) added to the height of the reference block when it is near the bottom edge of the video frame or near the target block.

[0173] When horizontally flipped, the search area must take into account the band size added to the right of the reference block width when it is near the right edge of the video frame or near the target block.

[0174] Corresponding to the flip, the candidate reference template area located to the right and / or bottom of the candidate reference block RB should be included in the search area.

[0175] Conversely, when the flip is horizontal and / or vertical, since the reconstructed samples on the left and / or top are not used for candidate reference blocks (RBs), the reference block position in the search area can be moved to the left and / or top of the video frame or to the boundary of the search area.

[0176] Therefore, the search region boundary is determined based on the target block neighborhood TBN and the geometric transformation type (especially when flipped), so that candidate reference template reconstruction samples can be used for comparison with target template block reconstruction samples.

[0177] Figure 11 and Figure 12 The illustration shows an example of search area adjustment (and results at possible candidate reference block RB locations) at the boundaries of a video frame or search area, according to an embodiment.

[0178] For example, when the target block TB is located at the bottom edge of the video frame ( Figure 11 (See the image above) And when the geometric transformation type GTT is vertical and horizontal flip, the search region boundary is adjusted so that the candidate reference block neighborhood RBN includes the lower boundary of the video frame, and the left edge sample of the candidate reference block RB matches the left edge of the video frame. When the target block TB is located at the left edge of the video frame (… Figure 11 (See the image below) And when the geometric transformation type GTT is vertical flip, adjust the search region boundary so that the candidate reference block neighborhood RBN includes the target block neighborhood TBN rows. When the target block TB is located at the top boundary of the video frame ( Figure 12 (See the image above) and when the Geometric Type Transformation (GTT) is a horizontal flip, the search region boundaries are adjusted so that the candidate reference block RB can move to the top left edge of the video frame. When the target block is located at the right edge of the video frame (…), Figure 12 (See the image below) And if the geometric transformation type (GTT) is horizontal and vertical flip, then adjust the search region boundary so that the candidate reference block neighborhood RBN includes at least one sample (e.g., 4 samples) located along the right boundary of the video frame.

[0179] In one embodiment, the geometric transformation type indicates at least one of the following geometric transformations or a combination of at least two of the following geometric transformations:

[0180] - Flip horizontally;

[0181] - Flip vertically;

[0182] - Rotation;

[0183] -Constant.

[0184] In one embodiment, the rotation angle is a multiple of 90°.

[0185] In some implementations (with high-end processing capabilities), combining rotation and flip transformations is advantageous because it produces geometric transformations that cannot be generated from a simple combination of flip or rotation, thereby increasing the number of candidates, i.e., improving the adaptability of signal transmission.

[0186] In another implementation (with limited memory, such as in an embedded system), combining only flips or only rotations can avoid executing rotation or flip routines (e.g., a 180° rotation can be implemented as a combination of first horizontal flip and then vertical flip), thereby reducing the implementation space required for geometric transformations applied to the reconstructed sample.

[0187] In one embodiment of method 400, in step 470, the geometric transformation type GTT signal associated with the selected candidate reference block RB is transmitted to the bit stream B.

[0188] This embodiment increases the bandwidth used to transmit bit stream B but limits the complexity of the decoder.

[0189] In one embodiment, the geometric transformation type GTT associated with the selected candidate reference block RB is not transmitted to the bit stream B, and method 400 considers each geometric transformation of a set of geometric transformations of a set of candidate reference blocks RB located in the search region at both the encoder and the decoder.

[0190] In one embodiment, the geometric transformation type GTT is inherited from the reconstructed block. As an example, if one of the neighboring blocks of the target block TB is encoded and decoded in ITMP mode, the geometric transformation type GTT2 associated with this neighboring block is used as the predictor of the geometric transformation type GTT.

[0191] Advantageously, if the geometric transformation type GTT2 is not the optimal geometric transformation type (determined on the encoding side), the geometric transformation GTT is explicitly signaled into the bit stream.

[0192] In one variant, if the target block TB encoded and decoded in ITMP mode has more than one neighboring block and their respective geometric transformation types are different, the geometric transformation type that appears most frequently is selected as the geometric transformation type GTT predictor.

[0193] In another variant, the index signal transmits the location of neighboring blocks encoded and decoded using ITMP mode, and selects the associated geometric transformation type.

[0194] These final embodiments limit the bandwidth required to transmit bitstream B, but increase the complexity of the decoder.

[0195] Figure 13 The illustration shows a method 400 according to an embodiment when the geometric transformation is a horizontal flip.

[0196] In step 1, the target block neighborhood is analyzed to determine the space of the search region R, and available reconstruction samples in the target block neighborhood TBN are identified.

[0197] In this instance, only reconstructed samples from the left neighborhood of the target block TB are available. The candidate reference block neighborhood (RBN) is of the "left template" type (a band of 4 samples next to the target block TB).

[0198] In step 2, candidate reference block neighborhoods (RBNs) are determined based on the availability of reconstruction samples from the target block neighborhood (TBN) and the geometric transformation type (GTT) (see [link to relevant documentation]). Figure 14 This is such that, in this instance of considering a horizontally flipped GTT, the candidate reference block neighborhood RBN is located to the right of the candidate reference block RB.

[0199] In step 3, the geometric transformation GT (here, horizontal flip) associated with the geometric transformation type GTT is applied to the reconstructed samples of the candidate reference block neighborhood RBN. This includes a horizontal reordering of the sample locations. The candidate reference template block RTP comprises the reconstructed samples located at the reordered reconstructed sample locations. The output of step 3 produces the samples that constitute the reference template block RTP.

[0200] When horizontal flipping is applied to samples of candidate reference block RB, such as Figure 13 As illustrated, the transformed reconstructed samples are reordered based on their distance from the center of the candidate reference block. Figure 13 As illustrated, the horizontal flip has four reconstructed sample positions with indices 1, 2, 3, and 4, which correspond to the transformed reconstructed sample position indices 4, 3, 2, and 1, respectively. When the geometric transformation is a rotation based on direction (clockwise or counterclockwise), the sample positions in the candidate reference block neighborhood RBN of the candidate reference block RB sample will be reordered according to the rotation direction.

[0201] In step 4, for a given target block TB, the cost is associated with each candidate reference block RB by comparing the reconstruction samples of candidate reference template blocks RTP with the reconstruction samples of the target template block TTP.

[0202] In step 5, a candidate reference block (RB) associated with the geometric transformation type GTT is selected. The selected candidate reference block corresponds to a candidate reference block (RB) associated with minimum cost. The geometric transformation identified by the geometric transformation type GTT associated with the selected candidate reference block (RB) is applied to the reconstructed sample of the selected candidate reference block (RB). Here, the selected candidate reference block (RB) is horizontally flipped, and the intra-prediction block for the target block (TB) is the horizontally flipped selected candidate reference block (RB).

[0203] Figure 9 The common template matching candidate search does not check for matches between the target template block and the candidate reference template block whose position has been geometrically transformed from the position of its reconstructed sample. For example, if the target block neighborhood TBN indicates "above template", only the available reconstructed samples located above the target block TB (i.e., the target template block TTP) are compared with the reconstructed samples located above the candidate reference block RB (i.e., the reference template block RTP). Instead, according to method 400, the reconstructed samples located above the candidate reference RB (reference template block RPT) are geometrically transformed (i.e., reordered) to check whether the transformed versions of these reconstructed samples match the reconstructed samples of the target template block.

[0204] Figure 14 The illustrations show some instances determined according to a neighborhood template based on a flipped (geometric transformation type) according to an embodiment.

[0205] When the upper and left neighborhoods of the target block TB are available and the geometric transformation GT is an identity function, a horizontal flip, a vertical flip, or a vertical and horizontal flip, the candidate reference block neighborhood RBN indicates an "L-shaped template" whose layout relative to the candidate reference block RB depends on the flip mode.

[0206] When only samples from the upper neighborhood of the target block TB are available and the geometric transformation is an identity function or a horizontal flip, the candidate reference block neighborhood RBN indicates the "upper template", the layout of which for the candidate reference block RB depends on the flip mode.

[0207] When only samples from the upper neighborhood of the target block TB are available and the geometric transformation is a vertical flip or a horizontal and vertical flip, the candidate reference block neighborhood RBN indicates the "bottom template", the layout of which for the candidate reference block RB depends on the flip mode.

[0208] When only samples from the left neighborhood of the target block TB are available and the geometric transformation is an identity function or a vertical flip, the candidate reference block neighborhood RBN indicates the "left template," the layout of which for the candidate reference block RB depends on the flip mode.

[0209] When only samples from the left neighbor of the target block TB are available and the geometric transformation is a horizontal flip or a horizontal and vertical flip, the candidate reference block neighborhood RBN indicates the "right template", the layout of which for the candidate reference block RB depends on the flip mode.

[0210] Figure 15 The illustrations show some instances of neighborhood templates determined according to a rotation (geometric transformation type) based on an embodiment.

[0211] When samples from the upper and left neighborhoods of the target block TB are available, the candidate reference block neighborhood RBN indicates an "L-shaped template". The layout of this "L-shaped template" also depends on the degree of rotation.

[0212] When only samples from the upper neighborhood of the target block TB are available, if the rotation degree is 0, the candidate reference block neighborhood RBN is the "upper template"; if the rotation degree is 90, the candidate reference block neighborhood RBN is the "right template"; if the rotation degree is 180, the candidate reference block neighborhood RBN is the "bottom template"; and if the rotation degree is 270, the candidate reference block neighborhood RBN is the "left template".

[0213] When only the left neighbor samples of the target block TB are available, if the rotation degree is 0, the candidate reference block neighborhood RBN is the "left template"; if the rotation degree is 90, the candidate reference block neighborhood RBN is the "top template"; if the rotation degree is 180, the candidate reference block neighborhood RBN is the "right template"; if the rotation degree is 270, the candidate reference block neighborhood RBN is the "bottom template".

[0214] Figure 16 A block diagram of method 400 according to an embodiment is illustrated schematically.

[0215] This embodiment applies only to method 100 (encoding).

[0216] For each geometric transformation type GTT that indicates a set of geometric transformations or a combination of at least two of those geometric transformations, run step 410. Optionally, also run step 420. For each position P in the determined search region, run steps 430 and 440. When all positions in the determined search region have been considered, consider another geometric transformation type (new GTT).

[0217] Having considered all geometric transformation types, the selected candidate reference block is the one with the lowest cost (450).

[0218] Figure 17 A block diagram of method 400 according to an embodiment is illustrated schematically.

[0219] This embodiment applies only to method 100 (encoding).

[0220] For each new location (new P) in the initial search region, consider each newly considered geometric transformation type (new GTT) that indicates a set of geometric transformations or a combination of at least two of those geometric transformations.

[0221] For each newly considered geometric transformation type (new GTT), run steps 410, 430, and 440, and

[0222] Optionally, the initial search area is adjusted according to step 420 (between steps 410 and 430).

[0223] Once all newly considered geometric transformation types (new GTT) have been taken into account, consider another location in the initial search region.

[0224] Having considered all positions in the initial search area, the selected candidate reference block is the candidate reference block with the lowest cost (450).

[0225] Figure 18 A block diagram of method 400 according to an embodiment is illustrated schematically.

[0226] This embodiment applies only to method 200 (decoding).

[0227] The geometric transformation type GTT is obtained by parsing an instance of a bitstream, indicating a set of geometric transformations or a combination of at least two of those geometric transformations.

[0228] Alternatively, also run step 420.

[0229] For each location P in the determined search region, steps 430 and 440 are run. After considering all locations in the determined search region, the candidate reference block is selected as the candidate reference block with the lowest cost (450).

[0230] In one embodiment of step 470, such as Figure 19 As illustrated, when the geometry transformation type indicates a flip mode (i.e., horizontal flip or vertical flip or both horizontal and vertical flip), the geometry transformation type is signaled in bitstream B as the integer value of intra_tmp_flip_type of the codec unit syntax element defined in VVC.

[0231] For example, when `intra_tmp_flip_type` equals 0, the geometric transformation type is disabled (i.e., identity). When `intra_tmp_flip_type` equals 1, the geometric transformation type is a horizontal flip for intra-frame prediction of the target block of the video frame according to method 400. When `intra_tmp_flip_type` equals 2, the geometric transformation type is a vertical flip mode for intra-frame prediction of the block of the video frame according to method 400. When `intra_tmp_flip_type` equals 3, the geometric transformation type is a combination of vertical and horizontal flips for intra-frame prediction of the target block of the video frame according to method 400.

[0232] As another example, intra_tmp_flip_flag is equal to 1 to set intra_tmp_flip_type. If intra_tmp_flip_flag is equal to 0, intra_tmp_flip_type is not signaled and its value is inferred to be 0.

[0233] This disclosure is not limited to a specific structure used to carry syntactic elements of geometric transformation types in a bitstream. Furthermore, Figure 19 The instance does not limit the meaning of the geometric transformation type, but can be extended to indicate other geometric transformations, such as rotation or a combination of rotations or a combination of at least one flip and at least one rotation.

[0234] Figure 20 A schematic block diagram illustrating an example of a system 500 in which various aspects and embodiments are implemented is shown.

[0235] System 500 can be embedded as one or more devices, including the various components described below. In various embodiments, system 500 can be configured to implement one or more aspects described in this application.

[0236] Examples of equipment that may constitute all or part of System 500 include personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, connected vehicles and their associated processing systems, head-mounted display devices (HMDs, see-through glasses), projectors, "cave" (systems including multiple displays), servers, video encoders, video decoders, post-processors that process the output from the video decoder, pre-processors that provide input to the video encoder, web servers, video servers (e.g., broadcast servers, video-on-demand servers, or web servers), still or video cameras, encoding or decoding chips, or any other communication devices. The elements of System 500 may be implemented individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of System 500 may be distributed across multiple ICs and / or discrete components. In various embodiments, System 500 may be communicatively connected to other similar systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports.

[0237] System 500 may include at least one processor 510 configured to execute instructions loaded therein for implementing various aspects, such as those described in this application. Processor 510 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 500 may include at least one memory 520 (e.g., a volatile memory device and / or a non-volatile memory device). System 500 may include a storage device 540, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 540 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0238] System 500 may include an encoder / decoder module 530 configured to, for example, process data to provide encoded / decoded video image data, and the encoder / decoder module 530 may include its own processor and memory. The encoder / decoder module 530 may represent one or more modules that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Furthermore, the encoder / decoder module 530 may be implemented as a separate element of system 500, or may be incorporated into processor 510 as a combination of hardware and software known to those skilled in the art.

[0239] Program code to be loaded into processor 510 or encoder / decoder 530 to execute the various aspects described in this application may be stored in storage device 540 and subsequently loaded into memory 520 for execution by processor 510. According to various embodiments, during the execution of the processes described in this application, one or more of processor 510, memory 520, storage device 540, and encoder / decoder module 530 may store one or more of various items. Such stored items may include, but are not limited to, video frame data, information data for encoding / decoding video frame data, bitstreams, matrices, variables, and intermediate or final results of equations, formulas, operations, and arithmetic logic processing.

[0240] In several embodiments, the memory within processor 510 and / or encoder / decoder module 530 may be used to store instructions and provide working memory for processing that can be performed during encoding or decoding.

[0241] However, in other embodiments, external memory (e.g., the processing device may be processor 510 or encoder / decoder module 530) is used for one or more of these functions. External memory may be memory 520 and / or storage device 540, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external dynamic volatile memory, such as RAM, may be used as working memory for video encoding / decoding operations, for example, for MPEG-2 Part 2 (also known as ITU-T Recommendation H.262 and ISO / IEC 13818-2, also known as MPEG-2 video), AVC, HEVC, EVC, VVC, AVI, etc.

[0242] As indicated in box 590, input to the components of system 500 can be provided through various input devices. Such input devices include, but are not limited to, (i) an RF section capable of receiving RF signals transmitted over the air, for example by a broadcast device, (ii) a composite input terminal, (iii) a USB input terminal, (iv) an HDMI input terminal, and (v) a bus, such as CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data Rate), FlexRay (ISO 17458), or Ethernet (ISO / IEC 802-3) bus, when this disclosure is implemented in the automotive field.

[0243] In various embodiments, the input device of block 590 has associated corresponding input processing elements, as known in the art. For example, the RF section may be associated with elements necessary for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a specific band), (ii) down-converting the selected signal, (iii) further limiting the band to a narrower band to select, for example, the signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF section in various embodiments may include one or more elements performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners performing various functions among these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband.

[0244] In one set-top box embodiment, the RF section and its associated input processing elements can receive RF signals transmitted over a wired (e.g., cable) medium. The RF section can then perform frequency selection by filtering, down-converting, and re-filtering to the desired frequency band.

[0245] Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.

[0246] Adding components may include inserting components between existing components, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section may include an antenna.

[0247] Furthermore, USB and / or HDMI terminals may include corresponding interface processors for connecting system 500 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within processor 510, when necessary. Similarly, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within processor 510, when necessary. The demodulated, error-corrected, and demultiplexed streams may be provided to various processing elements, including, for example, processor 510 and encoder / decoder 530, which operate in conjunction with memory and storage elements to process the data streams for presentation on an output device when necessary.

[0248] Various components of system 500 can be provided within an integrated housing. Within the integrated housing, suitable connection arrangements 590, such as internal buses (including I2C buses), wiring, and printed circuit boards known in the art, can be used to interconnect various components and transfer data between them.

[0249] System 500 may include a communication interface 550 that enables communication with other devices via a communication channel 551. The communication interface 550 may include, but is not limited to, a transceiver configured to send and receive data on the communication channel 551. The communication interface 550 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 551 may be implemented, for example, within a wired and / or wireless medium.

[0250] In various embodiments, data streaming to system 500 can be performed using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signals in these embodiments can be received via a communication channel 551 and a communication interface 550 suitable for Wi-Fi communication. The communication channel 551 in these embodiments can typically be connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top cloud communications.

[0251] Other embodiments may use a set-top box to provide streaming data to system 500, which delivers data via an HDMI connection in input box 590.

[0252] Other embodiments may use the RF connection of input block 590 to provide streaming data to system 500.

[0253] Streamed data can be used as a means of transmitting signaling information by System 500. This signaling information may include bitstream B and / or information such as the number of pixels in a video frame and / or any encoding / decoding settings.

[0254] It should be recognized that signal transmission can be implemented in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc., can be used to transmit signaling information to the corresponding decoder.

[0255] System 500 can provide output signals to various output devices, including a display 561, a speaker 571, and other peripheral devices 581. In various examples of embodiments, other peripheral devices 581 may include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 500.

[0256] In various embodiments, control signals may be communicated between system 500 and display 561, speaker 571 or other peripheral devices 581 using signaling such as AV.Link (audio / video link), CEC (consumer electronics control), or other communication protocols that enable device-to-device control with or without user intervention.

[0257] Output devices can be connected to system 500 via dedicated connections through the corresponding interfaces 560, 570 and 580.

[0258] Alternatively, the output device can be connected to the system 500 via communication interface 550 using communication channel 551. The display 561 and speaker 571 can be integrated with other components of the system 500 into a single unit in an electronic device, such as a television set.

[0259] In various embodiments, the display interface 560 may include a display driver, such as, for example, a timing controller (TCon) chip.

[0260] For example, if the RF portion of input 590 is part of a separate set-top box, then display 561 and speaker 571 may optionally be separate from one or more other components. In various embodiments where display 561 and speaker 571 can be external components, output signals may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0261] exist Figure 1-20 This document describes various methods, each comprising one or more steps or actions to implement the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined.

[0262] Examples of block diagrams and / or operation flowcharts are described. Each block represents a portion of circuitry, a module, or code, which includes one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in other implementations, the functions (one or more) marked in the blocks may occur out of order. For example, depending on the functions involved, two blocks shown sequentially may actually execute substantially concurrently, or sometimes these blocks may be executed in reverse order.

[0263] The embodiments and aspects described herein may be implemented in, for example, methods or processes, apparatus, computer programs, data streams, bit streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), implementations of the discussed features may be implemented in other forms (e.g., apparatus or computer programs).

[0264] The method can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices.

[0265] Furthermore, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) can be stored on a computer-readable storage medium. The computer-readable storage medium can take the form of a computer-readable program product implemented in one or more computer-readable media and having computer-executable computer-readable program code implemented thereon. Considering the inherent ability to store information therein and the inherent ability to retrieve information provided therefrom, the computer-readable storage medium as used herein can be considered a non-transitory storage medium. The computer-readable storage medium can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination of the foregoing. It should be understood that while more specific examples of computer-readable storage media to which this embodiment can be applied are provided below, they are merely illustrative and not exhaustive, as will be readily recognized by those skilled in the art: portable computer floppy disks; hard disks; read-only memory (ROM); erasable programmable read-only memory (EPROM or flash memory); portable optical disc read-only memory (CD-ROM); optical storage devices; magnetic storage devices; or any suitable combination of the foregoing.

[0266] Instructions can form applications that are tangibly implemented on processor-readable media.

[0267] For example, instructions can be found in hardware, firmware, software, or a combination thereof. Instructions can be found, for example, in an operating system, a standalone application, or a combination of both. Therefore, a processor can be characterized as, for example, a device configured to execute a process and a device including a processor-readable medium (such as a storage device) having instructions for executing the process. Additionally, in addition to or instead of instructions, the processor-readable medium can store data values ​​generated by the implementation.

[0268] The device can be implemented, for example, in appropriate hardware, software, and firmware. Examples of such devices include personal computers, laptops, smartphones, tablets, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, head-mounted display devices (HMDs, see-through glasses), projectors, "caves" (systems comprising multiple displays), servers, video encoders, video decoders, post-processors that process the output from the video decoder, pre-processors that provide input to the video encoder, web servers, set-top boxes, and any other devices or communication equipment used to process video images. It should be clear that the equipment can be mobile and even mounted in mobile vehicles.

[0269] The computer software may be implemented by processor 510 or by hardware, or by a combination of hardware and software. As a non-limiting example, embodiments may also be implemented using one or more integrated circuits. Memory 520 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. Processor 510 may be of any type suitable for the technical environment and may encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures, as non-limiting examples.

[0270] As will be apparent to those skilled in the art, implementations can generate various signals formatted to carry, for example, information that can be stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted via various wired or wireless links. The signal may be stored on a processor-readable medium.

[0271] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms “an,” “a,” and “the” may also be intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the terms “include / comprise” and / or “including / comprising” may specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as “in response to,” “connected to,” or “associated with,” another element, it may be directly responsive to, connected to, or associated with another element, or there may be intermediate elements. In contrast, when an element is referred to as “directly responsive to,” “directly connected to,” or “directly associated with,” another element, there are no intermediate elements.

[0272] It should be recognized that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the symbols / terms “ / ,” “and / or,” and “at least one of” can be intended to cover the selection of only the first listed option (A), or only the second listed option (B), or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to cover the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or the selection of all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as are listed.

[0273] Various numerical values ​​may be used in this application. Specific values ​​may be used for illustrative purposes and the aspects described are not limited to these specific values.

[0274] It will be understood that while the terms first, second, etc., may be used herein to describe various elements, these elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the teachings of this application, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. There is no implied order between the first element and the second element.

[0275] The references to “an embodiment” or “an embodiment” or “an implementation” or “implementation” and other variations thereof are frequently used to convey that a particular feature, structure, characteristic, etc. (described in connection with an embodiment / implementation) is included in at least one embodiment / implementation. Therefore, the phrases “in an embodiment” or “in an embodiment” or “in an implementation” or “in an implementation” appearing throughout this application, as well as any other variations, do not necessarily refer to the same embodiment.

[0276] Similarly, references to "according to an embodiment / example / implementation" or "in an embodiment / example / implementation" and their variations are frequently used to convey that a particular feature, structure, or characteristic (described in connection with an embodiment / example / implementation) may be included in at least one embodiment / example / implementation. Therefore, the expressions "according to an embodiment / example / implementation" or "in an embodiment / example / implementation" appearing throughout this application do not necessarily refer to the same embodiment / example / implementation, nor are individual or alternative embodiments / examples / implementations necessarily mutually exclusive with other embodiments / examples / implementations.

[0277] The reference numerals appearing in the claims are for illustrative purposes only and do not limit the scope of the claims. Although not explicitly described, these embodiments / examples and variations may be employed in any combination or sub-combination.

[0278] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0279] While some diagrams include arrows along the communication path to indicate the main direction of communication, it should be understood that communication can occur in the opposite direction to the arrows depicted.

[0280] Various implementations involve decoding. As used herein, "decoding" can encompass all or part of a process performed on, for example, a received video frame (which may include a received bitstream encoded with one or more video frames) to produce a final output suitable for display or further processing in a reconstructed video domain. In various embodiments, such processes include one or more processes typically performed by a decoder. In various embodiments, such processes, for example, also include, or optionally include, processes performed by a decoder of the various embodiments described herein.

[0281] As a further example, in one embodiment "decoding" may refer only to dequantization, in another embodiment "decoding" may refer to entropy decoding, in yet another embodiment "decoding" may refer only to differential decoding, and in still another embodiment "decoding" may refer to a combination of dequantization, entropy decoding, and differential decoding. It will be clear, and believed to be well understood, by those skilled in the art, whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to a broader decoding process, depending on the context of the specific description.

[0282] Various implementations involve encoding. In a manner similar to the above discussion of "decoding," the term "encoding" as used herein can encompass all or part of a process performed on an input video frame to produce an output bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder. In various embodiments, such a process also includes, or optionally includes, a process performed by an encoder of the various embodiments described herein.

[0283] As a further example, in one embodiment "encoding" may refer only to quantization, in another embodiment "encoding" may refer only to entropy encoding, in yet another embodiment "encoding" may refer only to differential encoding, and in still another embodiment "encoding" may refer to a combination of quantization, differential encoding, and entropy encoding. It will be clear, and believed to be well understood, by those skilled in the art, whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally to a broader encoding process, depending on the context of the particular description.

[0284] Furthermore, this application may refer to "obtaining" various types of information. Obtaining information may include one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory, processed information, moved information, copied information, erased information, calculated information, determined information, predicted information, or estimated information.

[0285] Furthermore, this application may refer to "receiving" various messages. Receiving messages may include one or more of the following, such as access information or receiving information from a communication network.

[0286] Moreover, as used herein, the term "signal transmission" specifically refers to instructing the corresponding decoder to do something. For example, in some embodiments, the encoder signals specific information, such as encoding / decoding parameters or encoded video frame data. In this way, in embodiments, the same parameter can be used on both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicit signal transmission) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters as well as other parameters, then signaling can be used without transmission (implicit signal transmission) to simply allow the decoder to know and select specific parameters. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be recognized that signaling can be accomplished in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the term "signal transmission" has been mentioned above, the term "signal transmission" can also be used as a noun herein.

[0287] Several implementations have been described. However, it should be understood that various modifications can be made. For example, elements of different implementations can be combined, supplemented, modified, or removed to produce other implementations. Furthermore, those skilled in the art will understand that other structures and processes can replace the disclosed structures and processes, and the resulting implementations will perform at least substantially the same functions in at least substantially the same manner to achieve at least substantially the same results as the disclosed implementations. Therefore, these and other implementations are contemplated in this application.

Claims

1. A method of intra prediction of a target block (TB) of a video picture based on template matching comparing a target template patch (TTP) comprising reconstructed samples in a target block neighborhood (TBN) with a candidate reference template patch (RTP) comprising reconstructed samples in a candidate reference block neighborhood (RBN) of a candidate reference block (RB) located at a position of a search region, wherein the method comprises: - determining (410) the candidate reference block neighborhood (RBN) from the target block neighborhood (TBN) reconstructed sample availability and from a geometric transform type (GTT); - obtaining (430) the reconstructed samples of the candidate reference template patch (RTP) by applying a geometric transform identified by the geometric transform type (GTT) on the reconstructed samples in the candidate reference block neighborhood (RBN); - associating (440) a cost with the candidate reference block (RB) obtained by comparing the reconstructed samples of the candidate reference template patch (RTP) with the reconstructed samples of the target template patch (TTP); - selecting (450) the candidate reference block having the minimum cost, the selected candidate reference block being associated with a geometric transform type (GTT) identifying a geometric transform (GT); and - generating (460) an intra prediction block for the target block (TB) by applying the geometric transform identified by the geometric transform type (GTT) associated with the selected candidate reference block on the reconstructed samples of the selected candidate reference block.

2. The method of claim 1, wherein the method further comprises: The search region is adjusted (420) according to the geometric transform type (GTT) and the position of the target block (TB) in the video picture.

3. The method of claim 1 or 2, wherein for each current geometric transform type (GTT) indicating a geometric transform of a set of geometric transforms or a combination of at least two of the geometric transforms, - determining (410) a candidate reference block neighborhood (RBN) from the target block neighborhood (TBN) reconstructed sample availability and the current geometric transform type (GTT); - for each current position in the search region, obtaining (430) the reconstructed samples of a candidate reference template patch of a candidate reference block (RB) located at the current position of the search region and associating (440) a cost with the candidate reference block (RB) located at the current position of the search region; - when all the positions in the search region have been considered, considering another geometric transform type, - when all the geometric transform types have been considered, the selected candidate reference block is the candidate reference block having the minimum cost (450).

4. The method of claim 1 or 2, wherein for each current position in the search region, a current geometric transform type indicating a geometric transform of a set of geometric transforms or a combination of at least two of the geometric transforms is considered, - determining (410) a candidate reference block neighborhood (RBN) from the target block neighborhood (TBN) reconstructed sample availability and the current geometric transform type, - obtaining (430) the reconstructed samples of a candidate reference template patch of a candidate reference block (RB) located at the current position of the search region and associating (440) a cost with the candidate reference block (RB) located at the current position of the search region, - when all the positions in the search region have been considered, considering another geometric transform type, - when all the geometric transform types have been considered, the selected candidate reference block is the candidate reference block having the minimum cost (450). - determining (410) a candidate reference block neighborhood (RBN) from the target block neighborhood (TBN) reconstructed sample availability and the current geometric transform type (GTT); - obtaining (430) reconstructed samples of a candidate reference template block of a candidate reference block (RB) located at the current position of the search area; - associating (440) a cost with the candidate reference block (RB) located at the current position of the search area; - considering another current position in the search area when all the geometric transform types have been considered; - the selected candidate reference block being the candidate reference block with the lowest cost (450) when all the positions in the search area have been considered.

5. The method of claim 1 or 2, wherein a current position in the search area is considered for each geometric transform indicating a set of geometric transforms or a current geometric transform type indicating a combination of at least two of the geometric transforms, - determining (410) a candidate reference block neighborhood (RBN) from the target block neighborhood (TBN) reconstructed sample availability and the geometric transform type (GTT); - obtaining (430) reconstructed samples of a candidate reference template block of a candidate reference block (RB) located at the current position of the search area and associating (440) a cost with the candidate reference block (RB) located at the current position of the search area for each current position in the search area; - the selected candidate reference block being the candidate reference block with the lowest cost (450) when all the positions in the search area have been considered.

6. The method of claim 1 or 2, wherein the cost associated with a candidate reference block (RB) is evaluated by the sum of absolute differences (SAD) between the transformed and reordered samples of a reference template block (RTP) and the reconstructed samples of the target template block (TTP).

7. The method of claim 1 or 2, wherein the geometric transform type indicates at least one of the following geometric transforms or a combination of at least two of the following geometric transforms: - horizontal flip; - vertical flip; - rotation; - identity.

8. The method of claim 7, wherein the rotation angle of the rotation is a multiple of 90°.

9. The method of claim 1 or 2, wherein the method further comprises signaling (470) the geometric transform type associated with the selected candidate reference block.

10. A method of encoding a block of a video picture based on a prediction block, wherein the prediction block is obtained according to the method of any of claims 1 to 4 or 6 to 9.

11. A method of decoding a block of a video picture based on a prediction block, wherein the prediction block is obtained according to the method of any of claims 1 to 9.

12. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to carry out the method of any of claims 1 to 11.

13. A non-transitory storage medium carrying instructions for a program code for performing the method according to any one of claims 1 to 11.

14. An electronic device comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor is configured to perform the method according to any one of claims 1 to 11.

Images