Encoding / decoding video picture data

Abstract

The present application relates to encoding a video picture into a bitstream of encoded video picture data / decoding a video picture from a bitstream of encoded video picture data, comprising temporally predicting a video picture block by obtaining a temporally predicted block based on at least one reference block of at least one reference picture pointed to by at least one motion vector associated with the video picture block, wherein, if the temporal prediction of the video picture block is based on a single reference block of a reference picture, the method further comprises padding (302) the reference block based on a motion vector derived from a nearest sub-block of the current block within the video picture.

Description

Encoding / decoding video image data

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to European Patent Application No. 22305478.4, filed on 7 April 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application generally relates to video image encoding and decoding. In particular, but not limited thereto, the technical field of this application relates to the temporal prediction of motion compensation for the overlap of reference blocks and reference frames. Background Technology

[0004] This section is intended to introduce the reader to various aspects of the art that may relate to aspects of at least one exemplary embodiment of the present application described below and / or claimed. This discussion is intended to help provide the reader with background information to facilitate a better understanding of the various aspects of the present application. Therefore, these statements should be understood to be read in this context and not as an admission of prior art.

[0005] In state-of-the-art video compression systems such as HEVC (ISO / IEC 23008-2 High Efficiency Video Coding, ITU-T Recommendation H.265, https: / / www.itu.int / rec / T-REC-H.265-202108-P / en) or VVC (ISO / IEC 23090-3 Versatile Video Coding, ITU-T Recommendation H.266, https: / / www.itu.int / rec / T-REC-H.266-202008-I / en), low-level and high-level picture partitioning is provided to divide the video frame into picture blocks, called codec tree units (CTUs). Their size is typically between 16x16 and 64x64 pixels for HEVC, and between 32x32, 64x64, or 128x128 pixels for VVC.

[0006] The CTU (Central Unit) division of a video frame forms a grid composed of CTUs of fixed size; that is, a CTU grid whose upper and left boundaries coincide spatially with the upper and left boundaries of the video frame. The CTU grid represents the spatial partitioning of the video frame.

[0007] In VVC and HEVC, the CTU size (CTU width and CTU height) of all CTUs in the CTU mesh is equal to the same default CTU size (default CTU width CTU DW and default CTU height CTU DH). For example, the default CTU size (default CTU height, default CTU width) can be equal to 128 (CTU DW = CTU DH = 128). The default CTU size (height, width) is encoded into the bitstream, for example, at the sequence level in the Sequence Parameter Set (SPS).

[0008] The spatial location of a CTU within a CTU mesh is determined by its CTU address ctuAddr, which defines the spatial position of the CTU's top-left corner relative to the origin. As shown in Figure 1, the CTU address can define the spatial position starting from the top-left corner of the higher-level spatial structure S containing the CTU.

[0009] Each CTU is associated with a codec tree to determine the tree partitioning of the CTU.

[0010] As shown in Figure 1, in HEVC, the codec tree is a quadtree partition of the CTU, where each leaf is called a codec unit (CU). The spatial location of a CU in a video frame is defined by the CU index cuIdx, which indicates the spatial location starting from the top left corner of the CTU. CUs are spatially partitioned into one or more prediction units (PUs). The spatial location of a PU in a video frame (VP) is defined by the PU index puIdx, which defines the spatial location starting from the top left corner of the CTU. The spatial location of the elements within a partitioned PU is defined by the PU partition index puPartIdx, which defines the spatial location starting from the top left corner of the PU. Each PU is assigned some intra-frame or inter-frame prediction data.

[0011] The intra-frame or inter-frame encoding / decoding mode is assigned at the CU level. This means that although the prediction parameters vary from PU to CU, each PU is assigned the same intra-frame / inter-frame encoding / decoding mode.

[0012] According to a quadtree called a transform tree, a CU can also be spatially partitioned into one or more transform units (TUs). A transform unit is a leaf of the transform tree. The spatial location of a TU in a video frame is defined by a TU index tuIdx, which defines the spatial location starting from the top-left corner of the CU. Each TU is assigned a number of transform parameters. The transform type is assigned at the TU level, and individual 2D transforms are performed at the TU level during the encoding / decoding or decoding of a frame block.

[0013] Figure 2 illustrates the existing PU partition types in HEVC. These include square partitions (2Nx2N and NxN), which are the only partitions used in both intra-frame and inter-frame prediction CUs; ​​symmetric non-square partitions (2NxN, Nx2N, used only in inter-frame prediction CUs); and asymmetric partitions (used only in inter-frame prediction CUs). For example, PU type 2NxnU represents an asymmetric horizontal partition of the PU, where the smaller partition is located at the top of the PU. According to another example, PU type 2NxnL represents an asymmetric horizontal partition of the PU, where the smaller partition is located at the top of the PU.

[0014] As shown in Figure 3, in VVC, the codec tree starts from the root node (i.e., CTU). Next, a quadtree (or quaternion tree) split divides the root node into four nodes, corresponding to four equal-sized sub-blocks (solid lines). The quadtree (or quaternion tree) leaves can then be further divided using a so-called multi-type tree, which involves binary or ternary splits according to one of the four splitting patterns shown in Figure 4. These splitting types are vertical and horizontal binary splitting patterns (denoted as SBTV and SBTH) and vertical and horizontal ternary splitting patterns SPTTV and STTH.

[0015] In the case of a joint codec tree where luminance and chrominance components share a common codec tree, the leaf of the CTU's codec tree is the CU.

[0016] In contrast to HEVC, in VVC, CU, PU, ​​and TU have the same size in most cases, which means that, except in some specific codec modes, codec units are generally not partitioned into PU or TU.

[0017] Figures 5 and 6 provide an overview of video encoding / decoding methods used in current video standard compression systems such as HEVC or VVC.

[0018] Figure 5 shows a schematic block diagram of the steps of a method 100 for encoding a video frame VP according to the prior art.

[0019] In step 110, the video frame VP is partitioned into sample blocks, and the partition information data is sent to the bitstream via a signal. Each block includes a sample of one component of the video frame VP. Therefore, these blocks include samples that define each component of the video frame VP.

[0020] For example, in HEVC, the frame is divided into codec tree units (CTUs). Each CTU can be further subdivided using a quadtree partition, where each leaf of the quadtree represents a codec unit (CU). The partition information data can then include data describing the CTU and the quadtree subdivision of each CTU.

[0021] Therefore, each sample block (or block for short) can be a CU (if the CU includes a single PU) or a PU of the CU.

[0022] Using intra-frame or inter-frame prediction modes, each block is encoded along the coding cycle (also known as "in the cycle").

[0023] Intra-frame prediction (step 120) uses intra-frame prediction data. Intra-frame prediction involves predicting the current block using blocks of intra-frame prediction based on samples that have been encoded, decoded, and reconstructed, located around the current block, typically at the top and left of the current block. Intra-frame prediction is performed in the spatial domain.

[0024] In inter-frame prediction mode, motion estimation (step 130) and motion compensation (135) are performed. Motion estimation searches for a reference block as a good predictor for the current block in one or more reference frames used for predictively encoding the current video frame. In unidirectional motion estimation / compensation, the candidate reference block belongs to a single reference frame in a list of reference frames represented as L0 or L1, while in bidirectional motion estimation / compensation, the candidate reference block is obtained from the reference blocks of reference frame list L0 and the reference blocks of reference frame list L1.

[0025] For example, a good predictor for the current block is a candidate reference block that is similar to the current block. It can also correspond to a reference block that provides a good trade-off between similarity to the current block and the rate cost of indicating the motion information required for time prediction of the current block.

[0026] The output of motion estimation step 130 is inter-frame prediction data, which includes motion information associated with the current block and other information for obtaining the same prediction block on the encoder / decoder side. Typically, the motion information includes one motion vector and a reference frame index for unidirectional estimation / compensation, and two motion vectors and two reference frame indices for bidirectional estimation / compensation. Next, motion compensation (step 135) obtains the prediction block using the motion vector(s) and reference frame(s) determined by motion estimation step 130. Essentially, a reference block belonging to a selected reference frame and pointed to by a motion vector can be used as the prediction block for the current block. Furthermore, since motion vectors are expressed as fractions of integer pixel positions (this is referred to as sub-pixel accuracy motion vector representation), motion compensation generally involves spatial interpolation of some reconstructed samples from the reference frame to compute the prediction block.

[0027] Prediction information data is signaled to the bitstream. Prediction information may include prediction mode (intra-frame, inter-frame, or skip), intra-frame / inter-frame prediction data, and any other information used to obtain the same prediction block on the decoding side.

[0028] Taking into account the encoding of the calculated prediction residual block (e.g., by subtracting the candidate prediction block from the current block) and the signaling of the prediction information data required to determine the candidate prediction block on the decoding side, method 100 selects a prediction mode (intra-frame or inter-frame prediction mode) by optimizing the rate-distortion tradeoff.

[0029] Typically, the best prediction mode is given as the prediction mode of the best encoding / decoding mode p* for the current block, as given by the following formula:

[0030] p * =Argmin p∈P {RD cost (p)} (1)

[0031] Where P is the set of all candidate codec modes for the current block, p represents the candidate codec mode in this set, and RD cost (p) is the rate-distortion cost of the candidate encoding / decoding mode p, usually expressed as:

[0032] RD cost(p) =D(p)+λ.R(p)

[0033] D(p) is the distortion between the current block and the reconstructed block obtained after encoding / decoding the current block with candidate encoding / decoding mode p, R(p) is the rate cost associated with encoding / decoding the current block with encoding / decoding mode p, and λ is the Lagrangian parameter representing the rate constraint for encoding / decoding the current block and is usually calculated based on the quantization parameters used to encode the current block.

[0034] The current block is typically encoded from the prediction residual block PR. More precisely, for example, the prediction residual block PR is calculated by subtracting the best prediction block from the current block. The prediction residual block PR is then transformed using a transform of type DCT (Discrete Cosine Transform) or DST (Discrete Sine Transform) or any other suitable transform (step 140), and the resulting transformed coefficient block is quantized (step 150).

[0035] In a variant, method 100 can also skip transform step 140 according to a so-called transform-skip encoding / decoding mode and directly apply quantization to the predicted residual block PR (step 150).

[0036] Encode the quantized transform coefficient block (or quantized prediction residual block) into the bit stream (step 160).

[0037] Next, as part of the encoding loop, the quantized transform coefficient block (or quantized residual block) is dequantized (step 170) and inverse transformed (180) (or not) to obtain the decoded prediction residual block. Then, the decoded prediction residual block and the prediction block are combined (usually summed) to provide the reconstructed block.

[0038] In step 160, other information data can also be entropy encoded to encode the current block of the video frame VP.

[0039] A loop filter (step 190) can be applied to the reconstructed frame (including reconstructed blocks) to reduce compression artifacts. Loop filters can be applied after all frame blocks have been reconstructed. Examples include deblocking filters, Sample Adaptive Offset (SAO) filters, or adaptive loop filters.

[0040] The reconstructed block or the filtered reconstructed block forms a reference frame, which can be stored in the decoded frame buffer (DPB) so that it can be used as a reference frame for encoding the next current block of the video frame VP or as a reference frame for encoding the next video frame to be encoded.

[0041] Figure 6 shows a schematic block diagram of the steps of a method 200 for decoding a video frame VP according to the prior art.

[0042] In step 210, partition information data, prediction information data, and quantized transform coefficient blocks (or quantized residual blocks) are obtained by entropy decoding of the bitstream of encoded video frame data. For example, this bitstream has been generated according to method 100.

[0043] It can also perform entropy decoding on other information data to decode the current block of the video frame VP from the bitstream.

[0044] In step 220, the reconstructed frame is divided into current blocks based on the partitioning information. Each current block is entropy-decoded from the bitstream along a decoding loop (also known as "in loop"). Each decoded current block is either a quantized transform coefficient block or a quantized prediction residual block.

[0045] In step 230, the current block is dequantized and possibly inverse transformed (step 240) to obtain the decoded prediction residual block.

[0046] On the other hand, prediction information data is used to predict the current block. The predicted block is obtained through its intra-frame prediction (step 250) or its motion-compensated temporal prediction (step 260). The prediction process performed on the decoding side is exactly the same as the prediction process on the encoding side.

[0047] Next, the decoded prediction residual block and prediction block are combined (usually by summation), which provides the reconstructed block.

[0048] In step 270, the loop filter can be applied to the reconstructed frame (including the reconstructed block), and the reconstructed block or the filtered reconstructed block forms a reference frame, which can be stored in the decoded frame buffer (DPB), as discussed above (Figure 5).

[0049] In VVC, motion information is stored in 4×4 blocks per video frame. This means that once the reference frame is stored in the decoded frame buffer (DPB, Figure 5 or Figure 6), the motion vectors and reference frame indices used for temporal prediction of video frame blocks are stored on a 4×4 block basis. These can be used for temporal prediction of motion information for encoding / decoding subsequent inter-frame predicted video frames.

[0050] In VVC, the timing prediction of the current block can be based on a reference block of a reference frame, which may overlap with the boundary of the reference frame. For example, Figure 7 shows an example of bidirectional motion compensation prediction for the current block based on a first reference block of reference frame list L0 and a second reference block of reference frame list L1. The first (and second) reference blocks are pointed to by a first (and second) motion vector. In the example of Figure 7, the first reference block is partially outside the reference frame of reference frame list L0, while the second reference block is inside the reference frame of reference frame list L1. More generally, a reference block of one of the two reference frames, or both reference blocks of the two reference frames, may be partially or completely outside the boundary of the reference frame.

[0051] In VVC, to handle such situations, the reference frame is expanded using a simple padding method that involves vertically padding the video frame boundary samples bs, as shown in Figure 8. This involves simply repeating the boundary sample values ​​bs along a sample row or column perpendicular to the reference frame boundary, extending beyond this boundary. The reference frame expands by M samples along each of its boundaries.

[0052] In the prior art, the padding method based on inter-frame or intra-frame prediction improves the basic padding of Figure 8.

[0053] Section 2.1.7.5 of JVET contribution JVET-J0014 describes an intra-prediction-based padding method (“Description of SDR, HDR and 360° video coding technology proposal by Fraunhofer HHI”, Joint Video Exploration Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 10th meeting: San Diego, USA, April 10-20, 2018, document JVET-J0014). Essentially, Multi-Directional Boundary Padding (MDBP) uses angular intra-prediction to extend the reference frame bounding box whenever the referenced sample block is partially or entirely outside the reference frame block. The optimal intra-mode is estimated between the encoder and decoder using a template block located within the reconstructed reference frame (as shown in Figure 9). The template block is then filled by deriving sample values ​​from the outermost pixel line using all available intra-codec modes. The optimal intra-frame encoding / decoding mode results in the minimum SAD relative to the original reconstructed sample values.

[0054] Section 3.1 of JVET contribution JVET-J0021 describes the first inter-frame prediction-based padding method (“Description of SDR, HDR and 360° video coding technology proposal by Qualcomm and Technicolor – low and high complexity versions”, Joint Video Exploration Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 10th meeting: San Diego, USA, April 10-20, 2018, document https: / / jvet-experts.org / doc_end_user / documents / 10_San%20Diego / wg11 / JVET-J0021-v5.zip). Essentially, as shown in Figure 10, when the decoder performs motion compensation, if the motion vector MV1 associated with block CB of the current video frame points to block RB that is at least partially located outside the boundary of reference frame 1, then region Z of reference block RB is unavailable. As shown in Figure 11, the surrounding fill area is divided into a sub-region SZ of size 4×M or M×4, and a second sub-region PZ of size 4×(PZ), where P is the size of the complete fill block. For each sub-region SZ along the boundary of the reference frame, the motion vector MV2 is derived from the nearest 4×4 sub-block NB in ​​reference frame 1 and the 4×4 sub-block in reference frame 2. If the nearest 4×4 sub-block NB is intra-coded (intra-predicted), then zero motion vector MV2 is used. If the nearest 4×4 sub-block NB is encoded (predicted) using bidirectional inter-frame prediction, then only motion vectors pointing to samples far from the boundary of its reference frame are used in the motion compensation for fill. After the motion vector derivation, then taking into account the average sample value offset between the nearest 4×4 sub-block and its corresponding block in reference frame 2, motion compensation is performed using the selected motion vector MV2 and reference frame 2 to obtain samples in the fill sub-region SZ. Note that the size M of the expanded region based on inter-frame prediction is derived from the value of motion vector MV2 and is related to the distance between the frame boundary and the position pointed to by motion vector MV2, see Figure 11. Furthermore, the size M can be smaller than the full fill size P of the reference frame. In this case, the inter-frame prediction-based fill is further completed by the basic vertical fill of the VVC. This means that the block filled by inter-frame prediction is expanded by the basic vertical fill mentioned above regarding the reference VVC.

[0055] The second inter-frame prediction-based padding method is described in section 2.1.7.5 of JVET contribution JVET-J0025 (“Description of SDR, HDR and 360° video coding technology proposal by Huawei, GoPro, HiSilicon, and Samsung – general application scenario”, Joint Video Exploration Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 10th meeting: San Diego, USA, April 10-20, 2018, document https: / / jvet-experts.org / doc_end_user / documents / 10_San%20Diego / wg11 / JVET-J0025-v4.zip). Essentially, if the boundary samples of the video frame are obtained through motion compensation, then the padding samples are also obtained using said motion compensation, as shown in Figure 12. For neighboring samples using a reference block, motion compensation is performed on the block referenced by the padding samples. If the block is smaller than the filled block or the boundary block was not obtained through motion compensation, then the left-side filling sample is obtained using the nearest sample. The top-left, top-right, bottom-left, and bottom-right corners of the filled block are determined using the nearest sample at each position. This method is performed after in-loop filtering.

[0056] Section 2.1.7.5 of JVET contribution JVET-Y0125 describes a third inter-frame prediction method for handling out-of-bounds reference blocks (“AHG12: Enhanced bi-directional motion compensation”, Yi-Wen Chen, Che-WeiKuo, Ning Yan, Wei Chen, Xiaoyu Xiu, Xianglin Wang, ITU-T SG 16WP 3 and ISO / IEC JTC1 / SC 29 Joint Video Experts Group (JVET) 25th Meeting, via teleconference, January 12-21, 2022, document JVET-Y0125). Essentially, to avoid the predicted (motion-compensated) block b at least partially exceeding the reference frame boundary (which is less efficient), when combining more than one motion-compensated prediction block, the OOB (out-of-bounds) prediction samples of prediction block b are discarded and only non-OOB (in-bounds) prediction samples are used to generate the final prediction block. More precisely, let (Pos_x) i,j ,Pos_y i,j Let be the Cartesian coordinates of the sample position (i,j) within the current block. Let Let Pos be the Cartesian coordinate (Lx) of the motion vector (MV) associated with the sample position (i,j) within the current block, indicating whether the reference frame belongs to reference frame list L0 (x=0) or reference frame list L1 (x=1). LeftBdry Pos RightBdry Pos TopBdry Pos BottomBdry These are the Cartesian coordinates of the four reference screen boundaries.

[0057] Predicting block P is true when at least one of the following conditions is met. Lx The predicted sample at sample location (i,j) Considered OOB

[0058]

[0059] or

[0060]

[0061] When none of the above conditions are met, the predicted sample It is considered non-OOB. Here, half_sample represents half the distance between two adjacent luminance samples, based on the codec's MV internal representation accuracy (1 / 16 pixel for VVC). Otherwise, the predicted sample... This will be considered OOB. If the predicted block P Lx If at least one of its samples is out of bounds (OOB), then the prediction block will be considered OOB; and if all of its samples are non-OOB, then the prediction block will be considered non-OOB. Here, half_sample represents half the distance between two samples, based on the codec's MV internal representation accuracy (1 / 16 pixel for VVC).

[0062] Next, if the predicted sample It is OOB and predicts samples If it is not OOB, then the final predicted sample Equal to the predicted sample

[0063] Otherwise, if the predicted sample Is it non-OOB and predict the sample? If it's OOB, then the final predicted sample Equal to the predicted sample

[0064] Otherwise, the final predicted sample From two prediction samples and The weighted average is given. This weighted average is usually the mean, so weights of 1 / 2 and 1 / 2 are used.

[0065] The third inter-frame prediction-based padding method has a limitation: it cannot handle unidirectional prediction blocks where the MV points to an OOB reference block. Furthermore, it cannot improve the bidirectional motion compensation method for VVC when all prediction blocks associated with reference frames belonging to both reference frame lists L0 and L1 are OOB.

[0066] The out-of-boundary prediction block management method proposed in JVET-Y0125 still has room for improvement.

[0067] The problem addressed by this invention is to further improve the compression rate of existing video codec standards (such as HEVC or VVC).

[0068] Specifically, the problem addressed is improving the efficiency of time prediction for motion compensation when a motion vector points to a reference block outside the boundary of the reference frame.

[0069] At least one exemplary embodiment of this application is designed with the above circumstances in mind. Summary of the Invention

[0070] The following sections provide a brief overview of at least one exemplary embodiment to provide a basic understanding of some aspects of this application. This overview is not an exhaustive summary of the exemplary embodiments. Its purpose is not to identify key or core elements of the exemplary embodiments. The following overview only gives some aspects of at least one exemplary embodiment in a simplified form, serving as a prelude to a more detailed description provided elsewhere in this document.

[0071] According to a first aspect of this application, a method is provided for encoding video frames into a bitstream of encoded video frame data. The method includes timing a video frame block by obtaining a timing-predicted block based on at least one reference block of at least one reference frame pointed to by at least one motion vector associated with the video frame block. If the timing prediction of the video frame block is based on a single reference block of the reference frame, the method further includes filling the reference block based on motion vectors derived from the nearest sub-block of the current block within the video frame.

[0072] According to a second aspect of this application, a method is provided for decoding a video frame from a bitstream of encoded video frame data. The method includes timing a video frame block by obtaining a timing-predicted block based on at least one reference block of at least one reference frame pointed to by at least one motion vector associated with the video frame block. If the timing prediction of the video frame block is based on a single reference block of the reference frame, the method further includes filling the reference block based on motion vectors derived from the nearest sub-block of the current block within the video frame.

[0073] In one exemplary embodiment, if the temporal prediction of a video frame block is based on a single reference block of a reference frame, and if the reference block is at least partially located outside the boundary of the reference frame, then the method further includes filling the reference block based on motion vectors derived from the nearest sub-block of the current block within the video frame.

[0074] In an exemplary embodiment, if the temporal prediction of a video frame block is based on a first reference block of a first reference frame and a second reference block of a second reference frame, the method further includes obtaining a prediction of the final motion compensation of the current video frame block by averaging the first and second prediction blocks based on samples, depending on whether a sample at a given position of the first prediction block is outside the boundary of the first reference frame and whether a sample at the same position of the second prediction block is outside the boundary of another reference frame.

[0075] In one exemplary embodiment, the first prediction block and the second prediction block are obtained by prediction using unidirectional motion compensation.

[0076] In one exemplary embodiment, a reference block of a reference frame is filled based on motion vectors derived from the nearest sub-block of the current block within the video frame, and / or a reference block of another reference frame is filled based on motion vectors derived from the nearest sub-block of the current block within the video frame, and the prediction of unidirectional motion compensation is based on the reference block or the filled reference block.

[0077] In one exemplary embodiment, the first or second prediction block is considered to be outside the boundary of the first or second reference frame only if the position of the first or second prediction block in the reference frame exceeds the time prediction block of the extended reference frame.

[0078] In an exemplary embodiment, when the first prediction block or the second prediction block is derived from a sub-block of inter-frame coding, the first prediction block or the second prediction block is considered to be outside the boundary of the first reference frame or the second reference frame, while when the first prediction block or the second prediction block is derived from a sub-block of intra-frame coding, the first prediction block or the second prediction block is considered to be inside the boundary of the first reference frame or the second reference frame.

[0079] In one exemplary embodiment, the nearest sub-block is either intra-frame encoded or inter-frame encoded.

[0080] In one exemplary embodiment, syntax elements are signaled to the bitstream to indicate whether the padding of a reference block for a reference frame is based on motion vectors derived from the nearest sub-block of the current block within the video frame or on a padding method based on intra-frame prediction.

[0081] In one exemplary embodiment, syntax elements are signaled at the sequence screen or slice level.

[0082] According to a third aspect of this application, a bitstream of encoded video frame data generated by one of the methods according to the first aspect of this application is provided.

[0083] According to a fourth aspect of this application, an apparatus is provided, comprising components for performing one of the methods according to the first and / or second aspects of this application.

[0084] According to a fifth aspect of this application, a computer program product including instructions is provided, which, when executed by one or more processors, cause the instructions to cause the one or more processors to perform the method according to the first and / or second aspects of this application.

[0085] According to a sixth aspect of this application, a non-transitory storage medium is provided, which carries program code instructions for performing the methods according to the first and / or second aspects of this application.

[0086] The specific properties of at least one of the exemplary embodiments, as well as other objects, advantages, features, and uses of the at least one exemplary embodiment, will become apparent from the following description of the examples taken in conjunction with the accompanying drawings. Attached Figure Description

[0087] Reference will now be made to the accompanying drawings, which illustrate exemplary embodiments of this application, wherein:

[0088] Figure 1 shows an example of a codec tree unit based on HEVC;

[0089] Figure 2 shows an example of partitioning codec units into prediction units according to HEVC;

[0090] Figure 3 shows an example of CTU partitioning based on VVC;

[0091] Figure 4 shows an example of the splitting patterns supported in the multi-type tree partitioning according to VVC;

[0092] Figure 5 shows a schematic block diagram of the steps of a method 100 for encoding a video frame VP according to the prior art;

[0093] Figure 6 shows a schematic block diagram of the steps of a method 200 for decoding video frames VP according to the prior art;

[0094] Figure 7 shows an example of a prediction based on bidirectional motion compensation according to the prior art;

[0095] Figure 8 illustrates a filling method according to the prior art;

[0096] Figure 9 illustrates a pre-existing intra-frame prediction-based padding method;

[0097] Figures 10 and 11 illustrate an example of a first inter-frame prediction-based padding method according to the prior art;

[0098] Figure 12 shows an example of a second inter-frame prediction-based padding method according to the prior art;

[0099] Figure 13 schematically illustrates a block diagram of a method 300 for predicting motion compensation according to at least one exemplary embodiment;

[0100] Figure 14 schematically illustrates a block diagram of a method 400 for filling a reference block of a reference screen according to an exemplary embodiment; and

[0101] Figure 15 illustrates a schematic block diagram of an example of a system in which various aspects and exemplary embodiments are implemented.

[0102] Similar or identical elements are referenced using the same figure mark. Detailed Implementation

[0103] At least one of the exemplary embodiments will be described more fully below with reference to the accompanying drawings, which depict examples of at least one of the exemplary embodiments. However, the exemplary embodiments may be implemented in various alternative forms and should not be construed as limited to the examples set forth herein. Accordingly, it should be understood that the present invention is not intended to limit the exemplary embodiments to the specific forms disclosed. Rather, this application is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of this application.

[0104] At least one of these aspects generally relates to video image encoding and decoding, another generally relates to the transmission of a bit stream provided or encoded, and yet another relates to the reception / access of a decoded bit stream.

[0105] At least one of the exemplary embodiments is described in relation to encoding / decoding a single video frame, but is extended to encoding / decoding multiple video frames (a sequence of frames), since each video frame is encoded / decoded sequentially as described below.

[0106] Moreover, for example, at least one exemplary embodiment is not limited to MPEG standards, such as AVC (ISO / IEC 14496-10 Advanced Video Coding for generic audio-visual services, ITU-T Recommendation H.264, https: / / www.itu.int / rec / T-REC-H.264-202108-P / en), EVC (ISO / IEC 23094-1 Essential video coding), HEVC (ISO / IEC 23008-2 High Efficiency Video Coding, ITU-T Recommendation H.265, https: / / www.itu.int / rec / T-REC-H.265-202108-P / en), and VVC (ISO / IEC 23090-3 Versatile Video Coding, ITU-T Recommendation H.264). H.266 (https: / / www.itu.int / rec / T-REC-H.266-202008-I / en), but may be applied to other standards and recommendations, such as AV1 (AOMedia Video 1, http: / / aomedia.org / av1 / specification / ). At least one exemplary embodiment may be adapted to existing or future developments and extensions of any such standard and recommendation. Unless otherwise stated or technically excluded, the aspects described in this application may be used individually or in combination.

[0107] A pixel corresponds to the smallest display unit on the screen, and it can consist of one or more light sources (one for a monochrome screen and three or more for a color screen).

[0108] A video frame, also known as a frame or video frame, includes at least one component (also known as a frame component or channel) defined by a specific video / video format that specifies all information related to pixel values ​​and all information that can be used by a display unit and / or any other device to display and / or decode video frame data associated with the video frame.

[0109] The video frame includes at least one component, typically represented as an array of samples.

[0110] Monochrome video footage consists of a single component, while color video footage can consist of three components.

[0111] For example, when the picture / video format is the well-known (Y,Cb,Cr) format, a color video picture can include one luminance (or brightness) component and two chrominance components, while when the picture / video format is the well-known (R,G,B) format, a color video picture can include three color components (one for red, one for green, and one for blue).

[0112] Each component of a video frame can include a number of samples relative to the number of pixels on the screen on which the video frame is to be displayed. In variations, the number of samples included in a component can be a multiple (or fraction) of the number of samples included in another component of the same video frame.

[0113] For example, in a video format that includes one luminance component and two chrominance components (such as the (Y,Cb,Cr) format), depending on the color format under consideration, the chrominance component may contain half the number of samples in width and / or height relative to the luminance component.

[0114] A sample is the smallest unit of visual information that makes up a video frame. Sample values ​​can be, for example, luminance or chrominance values, or color values ​​in (R,G,B) format.

[0115] A pixel value is the value of a pixel on the screen. For monochrome video, a pixel value can be represented by a single sample, while for color video, a pixel value can be represented by multiple co-localized samples. A co-localized sample associated with a pixel refers to the sample corresponding to the pixel's position on the screen.

[0116] Video frames are typically viewed as a set of pixel values, with each pixel represented by at least one sample.

[0117] A video frame block is a set of samples of one component of a video frame. When the frame / video format is a well-known (Y,Cb,Cr) format, a block of at least one luminance sample or a block of at least one chrominance sample can be considered; or when the frame / video format is a well-known (R,G,B) format, a block of at least one color sample can be considered.

[0118] At least one exemplary embodiment is not limited to a specific image / video format.

[0119] Generally, this application relates to encoding video frames into a bitstream of encoded video frame data / decoding video frames from the bitstream of the encoded video frame data, including timing prediction of video frame blocks by obtaining a block of timing prediction based on at least one reference block of at least one reference frame pointed to by at least one motion vector associated with the video frame block, wherein if the timing prediction of the video frame block is based on a single reference block of the reference frame, then the method further includes filling (302) the reference block based on motion vectors derived from the nearest sub-block of the current block within the video frame.

[0120] This invention improves upon the third existing inter-frame prediction method discussed above because it provides improved filled texture blocks for reference block portions located outside the reference frame boundary in the case of unidirectional temporal prediction of the current video frame block.

[0121] Padding can be applied to encoding method 100 or decoding method 200 to expand the reference frame stored in the DPB.

[0122] Figure 13 schematically illustrates a block diagram of a method 300 for predicting motion compensation according to at least one exemplary embodiment.

[0123] For the current block of the video frame for which inter-frame prediction is to be performed (the PU of the current CU), in step 301, method 300 checks whether prediction with bidirectional motion compensation is used.

[0124] If a unidirectional motion compensation prediction is used for the current block, i.e., if the motion compensation prediction for the current block is based on a single reference block of a reference frame, then in step 302, the reference block is filled based on the motion vector derived from the nearest sub-block of the current block within the video frame, as discussed above with respect to Figures 10 and 11. Well-known unidirectional motion compensation predictions are performed, for example, based on a reference frame that can be filled, as defined in Section 8.5.6.3 of the VVC specification.

[0125] For both reference blocks that are non-OOB reference blocks and for OOB reference blocks, step 302 is performed. For non-OOB reference blocks, the motion compensation prediction for the current block results in the same outcome as in VVC because all samples of the reference block are within a single reference frame boundary.

[0126] Step 302 is advantageous for OOB reference blocks (i.e., for prediction blocks with at least one OOB prediction sample) because the samples for those reference blocks used for motion-compensated predictions are generated by an inter-frame prediction-based padding method, as discussed with respect to Figures 10 and 11. Specifically, the samples for these OOB reference blocks located outside the reference frame boundaries are generated by motion-compensated predictions using stored motion vectors associated with 4×4 sub-blocks within a single reference frame that are closest to the samples of the reference blocks considered above. Therefore, these motion-compensated prediction reference samples have better quality than those obtained via the VVC vertical padding method.

[0127] In a variant of step 302, in step 3021, method 300 checks whether a reference block (pointed to by a motion vector associated with the current block) is out of bounds (OOB), i.e., whether at least one sample of the reference block is outside the boundary of a single reference frame.

[0128] If a single reference block is non-OOB (i.e., all samples of a single reference block are within the reference frame boundary), then in step 3022, a well-known unidirectional motion compensation prediction (e.g., defined in VVC) is performed based on the reference frame.

[0129] If a single reference block is OOB, then in step 3023, the single reference block is filled based on the motion vector derived from the nearest sub-block of the current block within the video frame, as discussed above with respect to Figures 10 and 11, and a well-known unidirectional motion compensation prediction (e.g., as defined in VVC) is performed based on the filled reference frame.

[0130] According to a variant of step 302, the OOB reference block is filled only based on the motion vector derived from the nearest sub-block of the current block within the video frame.

[0131] This variant avoids padding non-OOB reference blocks, thus limiting computational complexity compared to the case where all reference blocks are padded (regardless of whether they are non-OOB or OOB).

[0132] If a prediction using bidirectional motion compensation is applied to the current block (step 301), then steps 303-305 are executed.

[0133] In step 303, based on the first reference block of the reference frame in the reference frame list (e.g., L0), the first prediction block P of the current block is obtained using well-known unidirectional motion compensation prediction (e.g., as defined in VVC). L0If the first reference block is non-OOB, then the well-known unidirectional motion compensation prediction is based on the first reference block. If the first reference block is OOB, then the first reference block is filled based on the motion vector derived from the nearest 4×4 sub-block of the current block within the video frame, as discussed above with respect to Figures 10 and 11, and the well-known unidirectional motion compensation prediction is based on the first filled reference block.

[0134] In step 304, based on a second reference block of another reference frame list (e.g., L1), a second prediction block P of the current block is obtained using well-known unidirectional motion compensation prediction (e.g., as defined in VVC). L1 If the second reference block is non-OOB, then the well-known unidirectional motion compensation prediction is based on the second reference block. If the second reference block is OOB, then the second reference block is filled based on the motion vector derived from the nearest 4×4 sub-block of the current block within the video frame, as discussed above with respect to Figures 10 and 11, and the well-known unidirectional motion compensation prediction is based on the second filled reference block.

[0135] Alternatively, at steps 303 and 304, the inter-frame prediction-based padding method for the first and second reference blocks can only be applied if the first and second reference blocks associated with reference frame lists L0 and L1, respectively, are considered out-of-frame (OOB). Conversely, if at least one of the first and second reference blocks is considered non-OOB, then the inter-frame prediction-based padding method will not be applied to the first and second reference blocks.

[0136] Compared to the other methods mentioned above, this alternative method has the advantage of reduced complexity because it avoids performing inter-frame prediction-based padding in many cases, i.e., when at least one reference block used for bidirectional prediction is non-OOB.

[0137] In step 305, according to the first prediction block P L0 The sample at a given position (i,j) Is it OOB and according to the second prediction block P L1 Samples at the same position (i,j) Whether it is out of bounds (OOB) is determined by averaging the first and second prediction blocks based on the samples to obtain the final motion-compensated prediction block.

[0138] In an exemplary embodiment of step 305, sub-steps 3051-3056 may be performed for each sample position (i,j) of the current block.

[0139] In sub-step 3051, the first sample position (i,j) of the current block is considered.

[0140] In sub-step 3052, method 300 checks the predicted sample at the current position (i,j) of the first predicted block. Check if it's an Out-of-Band (OOB) error, and examine the predicted sample at the current position (i,j) of the second prediction block. Is it non-OOB? If the predicted sample It is OOB and the predicted sample If it's not OOB, then sub-step 3052 is followed by sub-step 3053. If the predicted sample... Is it non-OOB and the predicted sample If it is OOB, then substep 3052 is followed by substep 3054.

[0141] In sub-step 3053, the predicted sample at the current position (i,j) of the final predicted block P is set to be equal to the predicted sample at the current position (i,j) of the second predicted block.

[0142] In sub-step 3054, method 300 checks the predicted sample at the current position (i,j) of the first predicted block. Check if it is non-OOB, and examine the predicted sample at the current position (i,j) of the second prediction block. Is it out-of-bounds (OOB)? If the predicted sample... Is it non-OOB and the predicted sample If it's OOB, then sub-step 3054 is followed by sub-step 3055. If the predicted sample... It is OOB and the predicted sample If it is not OOB, then substep 3054 is followed by substep 3056.

[0143] In sub-step 3055, the predicted sample at the current position (i,j) of the final predicted block P is set to be equal to the predicted sample at the current position (i,j) of the first predicted block.

[0144] In sub-step 3056, the predicted sample at the current position (i,j) of the final predicted block P is set to the weighted average of the predicted samples at the current position (i,j) of the first and second predicted blocks.

[0145] After substeps 3052, 3053, and 3056, method 300 checks whether all positions of the current block have been considered. If not, then the next position of the current block is considered, and the method iterates from substep 3052. If all positions have been considered, then step 305 terminates.

[0146] According to the exemplary embodiment of step 305, if the location of the single-predicted sample is non-OOB and the other is OOB, then the predicted sample of the final predicted block can be equal to the single-predicted sample of one of the first or second predicted blocks. If both the first and second single-predicted samples are non-OOB, or if both are OOB, then it is equal to the weighted average of the two single-predicted samples. In the latter case, where both single-predicted sample locations are OOB, a third disadvantage of the prior art inter-frame prediction method is that it uses single-predicted samples emitted from the vertical padding method of the VVC specification, which may be of low quality.

[0147] Method 300 improves upon the third prior art inter-frame prediction method discussed above for prediction samples with unidirectional and bidirectional motion compensation. For prediction samples with unidirectional motion compensation, the third prior art inter-frame prediction method does not offer any solution. For prediction samples with bidirectional motion compensation, Method 300 benefits from using an improved filled block in the reference frame block outside the boundary, as discussed above with respect to Figures 10 and 11.

[0148] Therefore, method 300 improves compression efficiency compared to conventional motion-compensated predictions that fill in reference frames, as discussed in relation to any prior art filling methods based on intra-frame or inter-frame prediction (such as the first and second types of inter-frame prediction-based filling discussed above). It also improves compression efficiency compared to a third prior art bidirectional inter-frame prediction method.

[0149] In one exemplary embodiment, the predicted sample is predicted only when its position in the reference frame exceeds the temporally predicted block of the extended reference frame. or Only then can it be considered OOB. For example, when the position of the predicted sample in reference frame 1 of Figure 11 is outside the sub-block SZ, the predicted sample is OOB.

[0150] The exemplary embodiment is advantageous because bidirectional motion-compensated predictions can benefit from well-quality filled reference frame blocks, as they are filled through motion compensation rather than through basic vertical filling (as shown in area PZ of Figure 11). Therefore, the exemplary embodiment provides a synergy between a first prior art inter-frame prediction-based filling method and a third prior art inter-frame prediction method, since both unidirectional and bidirectional motion-compensated predictions generally benefit from better quality prediction samples than those in the prior art. The first inter-frame prediction-based filling method provides a good quality extension of the reference frame, the third prior art inter-frame prediction method provides good quality prediction samples for some bidirectional motion-compensated predictions, and combining these two methods according to the exemplary embodiment also results in more frequent use of the bidirectional motion-compensated predictions of the third prior art inter-frame prediction method, thus improving compression efficiency compared to methods that treat predicted samples as OOB when their location is outside the reference frame boundary.

[0151] In one exemplary embodiment, when predicting samples or When derived through the inter-frame prediction process, a predicted sample whose location is outside the reference frame boundary is considered non-OOB (as shown in Figure 11 for region SZ), while it is considered OOB when it is not derived from the inter-frame prediction process. Therefore, if the nearest 4×4 block to the considered sample has no motion vector and is thus intra-frame encoded / decoded, the location of the considered reference sample is considered OOB.

[0152] Therefore, during the reference frame extension process, the M×4 and 4×M sub-regions (SZ in Figure 11) can be marked for inter-frame encoding and decoding. This marking information is then used to determine a given prediction sample. or Is it OOB or not OOB?

[0153] This previous exemplary embodiment provides a more refined and accurate way to ensure that predictions for bidirectional motion compensation benefit from a high-quality extended reference frame portion, thereby further improving the compression efficiency of the current block.

[0154] In an exemplary embodiment shown in FIG14, a reference block for predicting the current block of a video frame can be filled based on information associated with a 4×4 sub-block within the video frame, which is encoded or decoded intra-frame or inter-frame.

[0155] In one exemplary embodiment, the size M of the extended region based on inter-frame prediction (see Figure 11) can be adjusted according to some characteristics of the current video frame.

[0156] In practice, if the current video frame contains reference frames that have already undergone inter-frame prediction, these reference frames are likely to have good-quality inter-frame prediction-based extended blocks around their boundaries because these extended blocks have already been inter-frame predicted. Therefore, the probability of having good inter-frame prediction-based extended blocks increases with the number of inter-frame encoded / decoded frames preceding the reference frames under consideration. This is typically related to the temporal layer of the current video frame. In fact, the temporal layer of a frame from frame to frame is related to the number of encoded / decoded reference frames that the current frame relies on for inter-frame prediction.

[0157] Therefore, the size of the extended region based on inter-frame prediction can be decoded according to the following decision: if the time layer associated with the considered reference frame is at least 2 and at least two reference frames are inter-frame encoded / decoded frames, then the size M of the extended region based on inter-frame prediction for the considered reference frame increases to M+4. Otherwise, the size M of the extended region based on inter-frame prediction for the considered reference frame remains equal to M.

[0158] In one exemplary embodiment, a distance threshold can be used to determine whether a predicted sample of the first or second prediction block is out-of-bounds (OOB). The distance threshold can be modified based on certain characteristics of a reference frame in the current video frame.

[0159] For example, if the reference image belongs to the time layer and is at least equal to a given value (e.g., 2), then the distance threshold can be assigned a higher distance than the value half_sample used in the method of Figure 13 (steps 3052 and 3054). For example, the distance threshold can be set to twice the distance between two adjacent brightness samples, denoted as double_sample.

[0160] The distance threshold can be selected following a decision-making process: if the time layer associated with the reference frame is at least 2, and at least two of the reference frames under consideration are inter-frame encoded / decoded frames, then the distance threshold is set to double_sample. Otherwise, the distance threshold is set to half_sample.

[0161] In the variant, the distance threshold is signaled in some header of the bitstream, such as in the slice header, picture header, picture parameter set (PPS), or sequence parameter set (SPS).

[0162] In one exemplary embodiment, the motion vector MV2 (Figure 11) used for inter-frame prediction of sub-region SZ can be refined after being obtained from the nearest 4×4 sub-block within the reference frame. This refinement can conform to the known template matching (TM) motion refinement used in ECM (“Algorithm description of Enhanced Compression Model 3 (ECM 3)”, Muhammed Coban, Fabrice Le Lanrnec, Mohammed Golam Sarwer, Jacob The 23rd meeting of the Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, held via teleconference, July 7-16, 2021 (document JVET-X2025). This can include searching for reference blocks around a block pointing to a considered motion vector MV2 by minimizing template matching costs (typically distortion, plus optional motion vector cost measurements). Distortion for TM cost calculations can be calculated between the template region adjacent to the current sub-region SZ and the corresponding template block in the reference frame of the current reference frame.

[0163] In an exemplary embodiment of the first inter-frame prediction-based padding method of the improved Figures 10 and 11, the following inter-frame prediction correction can be applied.

[0164] In the prior art solution shown in Figure 11, a correction is applied to the inter-frame predicted sub-region SZ. This correction is an additive offset, calculated as the average difference between the 4×4 sub-block NB and the 4×4 sub-block NBR pointed to by the motion vector MV2. This average difference is then applied to the predicted sub-region SZ.

[0165] In this exemplary embodiment, such correction can be replaced by inter-frame prediction enhancement based on a linear model, similar to known ECM local illumination compensation.

[0166] This takes the form of a linear model that determines the linked sub-blocks NB and NBR:

[0167] (a * ,b * ) = Argmin a,b {‖NB-(a.NBR+b)‖}

[0168] Then, using the linear model parameters (a,b):SZ final = a.SZ+b Enhance the prediction of sub-region SZ.

[0169] The advantage of this exemplary embodiment is that it improves the quality of the sub-region SZ through enhanced temporal prediction.

[0170] In one exemplary embodiment, the decoded video frame can be extended according to the invention immediately after decoding (on the encoder and decoder sides).

[0171] This allows for the preparation of decoded video frames before they are used as reference frames for the next video frame to be encoded or decoded.

[0172] In a variation of the previous exemplary embodiment, the decoded video frame can be extended according to the invention immediately after decoding the decoded video frame and before encoding / decoding the next video frame.

[0173] In a variant of the previous exemplary embodiment, the decoded video frame may be extended only if it is used as a reference frame for encoding / decoding future video frames.

[0174] Then, during the construction or updating of the reference frame list in encoding method 100 and decoding method 200, a decision is made to expand the decoded video frames. On the decoder side, after the header portion of the video frames has been decoded, the decoder is able to know which video frames are used as reference frames for the current video frame. Therefore, for each reference frame used, if it has not yet been expanded, it is expanded and marked as padded.

[0175] In a variation of the previous exemplary embodiment, a decision to extend the video frame can be made immediately after the video frame has been encoded (encoder side) or decoded (decoder side).

[0176] For example, if the current video frame has all its past reference frames in display order, and if the time layer of the current video frame is different from the maximum time layer in the considered video sequence, or if the current video frame is an intra-frame encoded or IRAP (Intra-Random Access Point) frame, then the current video frame is extended. Otherwise, the current frame is not extended. Figure 14 schematically illustrates a block diagram of a method 400 for filling reference blocks of reference frames according to an exemplary embodiment.

[0177] Basically, each reference block located at the boundary of the reference screen is considered iteratively, and for each reference block, a 4×M or M×4 block b is derived around the reference screen to obtain an expanded reference screen, as shown in Figure 7 (filled block).

[0178] In step 401, the first 4×M or M×4 block b is considered as the current reference block.

[0179] In step 402, the 4×4 sub-block nb that is closest to the current block b in the reference frame is determined, and its encoding / decoding (prediction) mode is obtained.

[0180] In step 403, the encoding / decoding mode of block nb is checked.

[0181] If the encoding / decoding mode is inter-frame encoding / decoding mode, then in step 404, the current reference block is filled based on the motion vector derived from block nb, as discussed above with respect to Figures 10 and 11.

[0182] If the encoding / decoding mode is intra-frame encoding / decoding mode, then in step 405, the current reference block is filled based on the intra-frame prediction-based filling method.

[0183] In a variant of step 405, the intra-prediction-based padding method can be the first intra-prediction-based padding method discussed above, wherein the optimal intra-mode is replaced by the obtained intra-code / decode mode.

[0184] In another variant of step 405, the intra-prediction-based padding method can be the first intra-prediction-based padding method as described above, wherein the optimal intra-prediction mode is replaced by a prediction mode similar to ECM's DIMD (decoder intra-prediction mode derivation) ("Algorithm description of Enhanced Compression Model 3 (ECM3)", M. Coban, F. LeLithm c, M. Sarwer, J. The document JVET-X2025, Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29, 23rd meeting, held via teleconference, July 7-16, 2021 (https: / / jvet-experts.org / doc_end_user / current_document.php?id=11231). Essentially, DIMD involves calculating a histogram of the spatial gradients of the signal reconstructed in 2D within block nb, and deriving the intra-frame prediction mode based on this gradient-based analysis. The basic principle is to extend the directions contained in the reconstructed intra-frame signal to the OOB samples of the current reference block.

[0185] In step 406, method 400 checks whether all reference blocks along the reference frame boundary have been considered. If so, the method terminates. Otherwise, new reference blocks are considered and the method iterates from step 402.

[0186] Method 400 further improves the quality of the extended reference image, thereby improving compression efficiency.

[0187] It should be noted that the main difference between the intra-prediction-based padding described in section 2.1.7.5 of JVET contribution JVET-J0014 and the inter-prediction-based padding described here is that a switching process between intra-prediction and inter-prediction occurs during the extension step of the reference frame.

[0188] In a variant of method 400, in step 403, a signaling syntax element in the bitstream indicates whether the reference block for padding (step 302) of the reference frame is based on a motion vector derived from the nearest sub-block of the current block within the video frame or on a padding method based on intra-frame prediction.

[0189] In variants, syntax elements can be signaled at the sequence level, such as in the SPS (Sequence Parameter Set).

[0190] In the variant, syntax elements can be signaled at the picture level, such as in a Picture Parameter Set (PPS).

[0191] In the variant, syntax elements can be signaled at the slice level.

[0192] In a variant of method 400, a reference block is filled using an intra-frame prediction-based filling method only for specific types of video frames (such as for specific screens and / or graphical video content).

[0193] Figure 15 shows a schematic block diagram illustrating an example of a system 600 that implements various aspects and exemplary embodiments.

[0194] System 600 can be embedded as one or more devices, including the various components described below. In various exemplary embodiments, system 600 can be configured to implement one or more aspects described in this application.

[0195] Examples of equipment that may constitute all or part of System 600 include personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, connected vehicles and their associated processing systems, head-mounted display devices (HMDs, see-through glasses), projectors, "cave" (systems including multiple displays), servers, video encoders, video decoders, post-processors that process the output from the video decoder, pre-processors that provide input to the video encoder, web servers, video servers (e.g., broadcast servers, video-on-demand servers, or web servers), still or video cameras, encoding or decoding chips, or any other communication devices. The elements of System 600 may be implemented individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one exemplary embodiment, the processing and encoder / decoder elements of System 600 may be distributed across multiple ICs and / or discrete components. In various exemplary embodiments, System 600 may be communicatively connected to other similar systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports.

[0196] System 600 may include at least one processor 610 configured to execute instructions loaded therein for implementing various aspects, such as those described in this application. Processor 610 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 600 may include at least one memory 620 (e.g., a volatile memory device and / or a non-volatile memory device). System 600 may include a storage device 640, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 640 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0197] System 600 may include an encoder / decoder module 630 configured to, for example, process data to provide encoded / decoded video image data, and the encoder / decoder module 630 may include its own processor and memory. The encoder / decoder module 630 may represent one or more modules that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Furthermore, the encoder / decoder module 630 may be implemented as a separate element of system 600, or may be incorporated into processor 610 as a combination of hardware and software known to those skilled in the art.

[0198] Program code to be loaded into processor 610 or encoder / decoder 630 to execute the various aspects described in this application may be stored in storage device 640 and subsequently loaded into memory 620 for execution by processor 610. According to various exemplary embodiments, during the execution of the processes described in this application, one or more of processor 610, memory 620, storage device 640, and encoder / decoder module 630 may store one or more of various items. Such stored items may include, but are not limited to, video frame data, information data for encoding / decoding video frame data, bitstreams, matrices, variables, and intermediate or final results of equations, formulas, operations, and arithmetic logic processing.

[0199] In several exemplary embodiments, the memory within the processor 610 and / or encoder / decoder module 630 may be used to store instructions and provide working memory for processes that can be performed during encoding or decoding.

[0200] However, in other exemplary embodiments, external memory (e.g., the processing device may be processor 610 or encoder / decoder module 630) is used for one or more of these functions. External memory may be memory 620 and / or storage device 640, such as dynamic volatile memory and / or non-volatile flash memory. In several exemplary embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one exemplary embodiment, fast external dynamic volatile memory, such as RAM, may be used as working memory for video encoding / decoding operations, for example, for MPEG-2 Part 2 (also known as ITU-T Recommendation H.262 and ISO / IEC 13818-2, also known as MPEG-2 video), AVC, HEVC, EVC, VVC, AVI, etc.

[0201] As indicated in box 690, input to the components of system 600 can be provided through various input devices. Such input devices include, but are not limited to, (i) an RF section capable of receiving, for example, RF signals transmitted over the air by a broadcasting device, (ii) a composite input terminal, (iii) a USB input terminal, (iv) an HDMI input terminal, and (v) a bus, such as CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data Rate), FlexRay (ISO 17458), or Ethernet (ISO / IEC 802-3) bus, when the invention is implemented in the automotive field.

[0202] In various exemplary embodiments, the input device of block 690 has associated corresponding input processing elements, as known in the art. For example, the RF section may be associated with elements necessary for: (i) selecting a desired frequency (also known as selecting a signal, or limiting the signal band to a specific band), (ii) down-converting the selected signal, (iii) further limiting the band to a narrower band to select, for example, the signal band that may be referred to as a channel in some exemplary embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF section of various exemplary embodiments may include one or more elements performing these functions, such as a frequency selector, signal selector, band limiter, channel selector, filter, downconverter, demodulator, error corrector, and demultiplexer. The RF section may include tuners performing various functions among these functions, including, for example, down-converting a received signal to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or baseband.

[0203] In one set-top box embodiment, the RF section and its associated input processing elements can receive RF signals transmitted over a wired (e.g., cable) medium. The RF section can then perform frequency selection by filtering, down-converting, and re-filtering to the desired frequency band.

[0204] Various exemplary embodiments may rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.

[0205] Adding components may include inserting components between existing components, such as, for example, inserting amplifiers and analog-to-digital converters. In various exemplary embodiments, the RF portion may include an antenna.

[0206] Furthermore, the USB and / or HDMI terminals may include corresponding interface processors for connecting system 600 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, within a separate input processing IC or within processor 610, as needed. Similarly, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within processor 610, as needed. The demodulated, error-corrected, and demultiplexed streams may be provided to various processing elements, including, for example, processor 610 and encoder / decoder 630, which operate in conjunction with memory and storage elements to process the data streams for presentation on an output device as needed.

[0207] Various components of system 600 can be provided within an integrated housing. Within the integrated housing, suitable connection arrangements 690, such as internal buses (including I2C buses), wiring, and printed circuit boards known in the art, can be used to interconnect various components and transfer data between them.

[0208] System 600 may include a communication interface 650 that enables communication with other devices via a communication channel 651. The communication interface 650 may include, but is not limited to, a transceiver configured to send and receive data on the communication channel 651. The communication interface 650 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 651 may be implemented, for example, within a wired and / or wireless medium.

[0209] In various exemplary embodiments, a Wi-Fi network such as IEEE 802.11 can be used to stream data to system 600. The Wi-Fi signals of these exemplary embodiments can be received via a communication channel 651 and a communication interface 650 suitable for Wi-Fi communication. The communication channel 651 of these exemplary embodiments can typically be connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top cloud communications.

[0210] Other exemplary embodiments may use a set-top box to provide streaming data to system 600, the set-top box delivering data via an HDMI connection in input block 690.

[0211] Other exemplary embodiments may use the RF connection of input block 690 to provide streaming data to system 600.

[0212] Streamed data can be used as a means of signaling notification information by System 600. Signaling notification information may include bitstream B and / or information such as the number of video frame pixels and / or any encoding / decoding settings parameters.

[0213] It should be recognized that signaling notification can be implemented in a variety of ways. For example, in various exemplary embodiments, one or more syntax elements, flags, etc., can be used to send signaling notification information to the corresponding decoder.

[0214] System 600 can provide output signals to various output devices, including a display 661, a speaker 671, and other peripheral devices 681. In various examples of exemplary embodiments, other peripheral devices 681 may include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 600.

[0215] In various exemplary embodiments, control signals may be communicated between system 600 and display 661, speaker 671 or other peripheral devices 681 using signaling such as AV.Link (audio / video link), CEC (consumer electronics control), or other communication protocols that enable device-to-device control with or without user intervention.

[0216] Output devices can be connected to system 600 via dedicated connections through the corresponding interfaces 660, 670 and 680.

[0217] Alternatively, the output device can be connected to the system 600 via communication interface 650 using communication channel 651. The display 661 and speaker 671 can be integrated with other components of the system 600 into a single unit in an electronic device, such as a television set.

[0218] In various exemplary embodiments, the display interface 660 may include a display driver, such as, for example, a timing controller (TCon) chip.

[0219] For example, if the RF portion of input 690 is part of a separate set-top box, then display 661 and speaker 671 may optionally be separate from one or more other components. In various exemplary embodiments where display 661 and speaker 671 can be external components, output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0220] In Figures 1-15, this paper describes various methods, and each method includes one or more steps or actions to implement the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined.

[0221] Examples of block diagrams and / or operation flowcharts are described. Each block represents a portion of circuitry, a module, or code, which includes one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in other implementations, the functions (one or more) marked in the blocks may occur out of order. For example, depending on the functions involved, two blocks shown sequentially may actually execute substantially concurrently, or sometimes these blocks may be executed in reverse order.

[0222] The embodiments and aspects described herein may be implemented in, for example, methods or processes, apparatus, computer programs, data streams, bit streams, or signals. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), implementations of the discussed features may be implemented in other forms (e.g., apparatus or computer programs).

[0223] The method can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices.

[0224] Furthermore, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) can be stored on a computer-readable storage medium. The computer-readable storage medium can take the form of a computer-readable program product implemented in one or more computer-readable media and having computer-executable computer-readable program code implemented thereon. Considering the inherent ability to store information therein and the inherent ability to retrieve information provided therefrom, the computer-readable storage medium as used herein can be considered a non-transitory storage medium. The computer-readable storage medium can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination of the foregoing. It should be understood that while more specific examples of computer-readable storage media to which this exemplary embodiment can be applied are provided below, they are merely illustrative and not exhaustive, as will be readily recognized by those skilled in the art: portable computer floppy disks; hard disks; read-only memory (ROM); erasable programmable read-only memory (EPROM or flash memory); portable optical disc read-only memory (CD-ROM); optical storage devices; magnetic storage devices; or any suitable combination of the foregoing.

[0225] Instructions can form applications that are tangibly implemented on processor-readable media.

[0226] For example, instructions can be found in hardware, firmware, software, or a combination thereof. Instructions can be found, for example, in an operating system, a standalone application, or a combination of both. Therefore, a processor can be characterized as, for example, a device configured to execute a process and a device including a processor-readable medium (such as a storage device) having instructions for executing the process. Additionally, in addition to or instead of instructions, the processor-readable medium can store data values ​​generated by the implementation.

[0227] The device can be implemented, for example, in appropriate hardware, software, and firmware. Examples of such devices include personal computers, laptops, smartphones, tablets, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, head-mounted display devices (HMDs, see-through glasses), projectors, "caves" (systems comprising multiple displays), servers, video encoders, video decoders, post-processors that process the output from the video decoder, pre-processors that provide input to the video encoder, web servers, set-top boxes, and any other devices or communication equipment used to process video images. It should be clear that the equipment can be mobile and even mounted in mobile vehicles.

[0228] The computer software may be implemented by the processor 610, by hardware, or by a combination of hardware and software. As a non-limiting example, exemplary embodiments may also be implemented by one or more integrated circuits. The memory 620 may be of any type suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 610 may be of any type suitable for the technical environment and may encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures, as non-limiting examples.

[0229] As will be apparent to those skilled in the art, implementations can generate various signals formatted to carry information, such as information that can be stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal may be formatted to carry a bit stream of the described exemplary embodiments. Such a signal may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of the spectrum) or baseband signals. Formatting may include, for example, encoding the data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal can be transmitted via various wired or wireless links. The signal may be stored on a processor-readable medium.

[0230] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, the singular forms “an,” “a,” and “the” may also be intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the terms “include / comprise” and / or “including / comprising” may specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as “in response to,” “connected to,” or “associated with,” another element, it may be directly responsive to, connected to, or associated with another element, or there may be intermediate elements. In contrast, when an element is referred to as “directly responsive to,” “directly connected to,” or “directly associated with,” another element, there are no intermediate elements.

[0231] It should be recognized that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the symbols / terms “ / ,” “and / or,” and “at least one of” can be intended to cover the selection of only the first listed option (A), or only the second listed option (B), or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such wording is intended to cover the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or the selection of all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as are listed.

[0232] Various numerical values ​​may be used in this application. Specific values ​​may be used for illustrative purposes and the aspects described are not limited to these specific values.

[0233] It will be understood that while the terms first, second, etc., may be used herein to describe various elements, these elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the teachings of this application, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. There is no implied order between the first element and the second element.

[0234] References to “an exemplary embodiment” or “an exemplary embodiment” or “an implementation” or “implementation” and other variations thereof are frequently used to convey that a particular feature, structure, characteristic, etc. (described in conjunction with an exemplary embodiment / implementation) is included in at least one exemplary embodiment / implementation. Therefore, the phrases “in an exemplary embodiment” or “in an exemplary embodiment” or “in one implementation” or “in one implementation” appearing throughout this application, as well as any other variations, do not necessarily refer to the same exemplary embodiment.

[0235] Similarly, the references to "according to an exemplary embodiment / example / implementation" or "in an exemplary embodiment / example / implementation" and their variations are frequently used to convey that a particular feature, structure, or characteristic (described in conjunction with an exemplary embodiment / example / implementation) may be included in at least one exemplary embodiment / example / implementation. Therefore, the expressions "according to an exemplary embodiment / example / implementation" or "in an exemplary embodiment / example / implementation" appearing throughout this application do not necessarily refer to the same exemplary embodiment / example / implementation, nor are individual or alternative exemplary embodiments / examples / implementations necessarily mutually exclusive with other exemplary embodiments / examples / implementations.

[0236] The reference numerals appearing in the claims are for illustrative purposes only and do not limit the scope of the claims. Although not explicitly described, these exemplary embodiments / examples and variations may be employed in any combination or subcombination.

[0237] When a diagram is presented as a flowchart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flowchart of the corresponding method / process.

[0238] While some diagrams include arrows along the communication path to indicate the main direction of communication, it should be understood that communication can occur in the opposite direction to the arrows depicted.

[0239] Various implementations involve decoding. As used herein, "decoding" can encompass all or part of a process performed, for example, on a received video frame (which may include a received bitstream encoded with one or more video frames) to produce a final output suitable for display or further processing in a reconstructed video domain. In various exemplary embodiments, such a process includes one or more processes typically performed by a decoder. In various exemplary embodiments, such a process, for example, also includes, or optionally includes, a process performed by a decoder of the various embodiments described herein.

[0240] As a further example, in one exemplary embodiment, "decoding" may refer only to dequantization; in another exemplary embodiment, "decoding" may refer to entropy decoding; in yet another exemplary embodiment, "decoding" may refer only to differential decoding; and in yet another exemplary embodiment, "decoding" may refer to a combination of dequantization, entropy decoding, and differential decoding. It will be clear, and believed to be well understood by those skilled in the art, whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally to a broader decoding process, depending on the context of the specific description.

[0241] Various implementations involve encoding. In a manner similar to the above discussion of "decoding," the term "encoding" as used herein can encompass all or part of a process performed on an input video frame to generate an output bitstream. In various exemplary embodiments, such a process includes one or more processes typically performed by an encoder. In various exemplary embodiments, such a process also includes, or optionally includes, a process performed by an encoder of the various embodiments described herein.

[0242] As a further example, in one exemplary embodiment, "encoding" may refer only to quantization; in another exemplary embodiment, "encoding" may refer only to entropy encoding; in yet another exemplary embodiment, "encoding" may refer only to differential encoding; and in still another exemplary embodiment, "encoding" may refer to a combination of quantization, differential encoding, and entropy encoding. It will be clear, and believed to be well understood, by those skilled in the art, whether the phrase "encoding process" is intended to specifically refer to a subset of operations or generally to a broader encoding process, depending on the context of the particular description.

[0243] Furthermore, this application may refer to "obtaining" various types of information. Obtaining information may include one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory, processed information, moved information, copied information, erased information, calculated information, determined information, predicted information, or estimated information.

[0244] Furthermore, this application may refer to "receiving" various messages. Receiving messages may include one or more of the following, such as access information or receiving information from a communication network.

[0245] Moreover, as used herein, the word "signal" specifically refers to instructing the corresponding decoder to do something. For example, in some exemplary embodiments, the encoder signals specific information, such as encoding / decoding parameters or encoded video frame data. In this way, in exemplary embodiments, the same parameter can be used on both the encoder and decoder sides. Thus, for example, the encoder can transmit (explicitly signal) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without transmission (implicitly signal) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various exemplary embodiments by avoiding the transmission of any actual functionality. It should be recognized that signaling can be accomplished in various ways. For example, in various exemplary embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. Although the verb form of the word "signal" has been referred to above, the word "signal" can also be used as a noun herein.

[0246] Several implementations have been described. However, it should be understood that various modifications can be made. For example, elements of different implementations can be combined, supplemented, modified, or removed to produce other implementations. Furthermore, those skilled in the art will understand that other structures and processes can replace the disclosed structures and processes, and the resulting implementations will perform at least substantially the same functions in at least substantially the same manner to achieve at least substantially the same results as the disclosed implementations. Therefore, these and other implementations are contemplated in this application.

Claims

1. A method for encoding video frames into a bitstream of encoded video frame data, the method comprising using a first prediction block based on a first reference frame (…). ) and the second prediction block of the second reference image ( ) Obtain time prediction blocks to perform time prediction on video frame blocks, where, Only when the first prediction block ( ) or the second prediction block ( The first prediction block or the second prediction block is considered to be outside the boundary of the first reference frame or the second reference frame only when its position in the reference frame exceeds the time prediction area of ​​the extended reference frame. The time prediction area is obtained by filling all blocks located at the boundary of the first or second reference frame.

2. A method for decoding video frames from a bitstream of encoded video frame data, the method comprising decoding a first prediction block based on a first reference frame (…). ) and the second prediction block of the second reference image ( ) Obtain time prediction blocks to perform time prediction on video frame blocks, where, Only when the first prediction block ( ) or the second prediction block ( The first prediction block or the second prediction block is considered to be outside the boundary of the first reference frame or the second reference frame only when its position in the reference frame exceeds the time prediction area of ​​the extended reference frame. The time prediction area is obtained by filling all blocks located at the boundary of the first or second reference frame.

3. The method as described in claim 1 or 2, wherein, If the time prediction of the video frame block is based on a single reference block of a reference frame, and if the reference block is at least partially located outside the boundary of the reference frame, then the method further includes filling the reference block based on motion vectors derived from the nearest sub-block of the current video frame block within the video frame.

4. The method of claim 3, further comprising: According to the first prediction block ( The sample at a given position (i,j) of ) Whether it is outside the boundary of the first reference frame and according to the second prediction block ( Samples at the same position (i,j) of ) Whether it is outside the boundary of another reference frame, by analyzing the first prediction block based on samples ( ) and the second prediction block ( The average is used to obtain the prediction of the final motion compensation for the current video frame.

5. The method of claim 4, wherein the first prediction block and the second prediction block are obtained by prediction using unidirectional motion compensation.

6. The method of claim 5, wherein the reference block of the reference frame is filled based on a motion vector derived from the nearest sub-block of the current video frame block within the video frame, or the reference block of the other reference frame is filled based on a motion vector derived from the nearest sub-block of the current video frame block within the video frame, and the prediction of the unidirectional motion compensation is based on the reference block or the filled reference block.

7. The method of claim 6, wherein when the first prediction block ( ) or the second prediction block ( When derived from a sub-block of inter-frame encoding / decoding, the first prediction block or the second prediction block is considered to be outside the boundary of the first reference frame or the second reference frame, and when the first prediction block ( ) or the second prediction block ( When derived from a sub-block of intra-frame encoding and decoding, the first prediction block or the second prediction block is considered to be within the boundary of the first reference frame or the second reference frame.

8. The method of claim 6, wherein the nearest sub-block has undergone intra-frame encoding / decoding or inter-frame encoding / decoding.

9. The method of claim 1 or 2, wherein the syntax element is signaled to the bitstream to indicate whether the padding of the reference block of the reference frame is based on the motion vector derived from the nearest sub-block of the current video frame block within the video frame or based on an intra-frame prediction padding method.

10. The method of claim 9, wherein the syntax element is signaled at the sequence picture or slice level.

11. A video image processing apparatus, comprising components for performing one of the methods claimed in any one of claims 1 to 10.

12. A method for storing a bit stream, characterized in that, The method of encoding video frames into a bitstream of encoded video frame data according to any one of claims 1 to 10 generates a bitstream; and stores the bitstream.

13. A method for transmitting a bit stream, characterized in that, The method of encoding video frames into a bitstream of encoded video frame data as described in any one of claims 1 to 10 generates a bitstream; And the transmission of the bit stream.

14. A computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method claimed in any one of claims 1 to 10.

15. A non-transitory storage medium carrying program code instructions that, when executed by one or more processors, cause the one or more processors to perform the method claimed in any one of claims 1 to 10.

Images