Encoding and Decoding Data

Abstract

An improvement in predictor processing is disclosed. A list of predictors is obtained, the list of predictors having at least two predictors. A first cost associated with a first predictor in the list and a second cost associated with a second predictor in the list are used to determine whether to modify the list of predictors based on a criterion. The list of predictors is modified based on the determination. The criterion is based on a threshold.

Description

[Technical field]

[0001] The present invention relates to predictor encoding and decoding. [Background technology]

[0002] The Joint Video Experts Team (JVET), a joint team formed by MPEG and ITU-T Study Group 16 VCEG, has introduced a new video coding standard called Versatile Video Coding (VVC). The goal of VVC is to achieve significant compression improvements over the existing HEVC standard (i.e., typically twice as fast). Primary target applications and services include, but are not limited to, 360-degree video and high dynamic range (HDR) video. In particular, the test material has demonstrated its effectiveness in ultra-high definition (UHD) video. Thus, we can expect compression efficiency improvements well beyond the 50% targeted in the final standard.

[0003] After completing the standardization of VVCv1, JVET established the Exploration Software (ECM) and started the exploration phase. ECM aims to improve coding efficiency by gathering additional tools and improvements of existing tools on top of the VVC standard.

[0004] Compared to HEVC, VVC has a modified set of "merge modes" for motion vector prediction that achieves higher coding efficiency at the cost of more complexity. Motion vector prediction is enabled by deriving a list of "motion vector prediction candidates", and the index of the selected candidate is signaled in the bitstream. A merge candidate list is generated per coding unit (CU). However, CUs may be split into smaller blocks for decoder-side motion vector refinement (DMVR) or other methods.

[0005] The composition and order of this list can have a significant impact on coding efficiency, since accurate motion vector prediction reduces the size of the residual or distortion of the block prediction, and having such candidates at the top of the list reduces the number of bits required to signal the selected candidate. The present invention aims to improve on at least one of these aspects.

[0006] The modifications incorporated in VVCv1 and ECM result in a maximum of 10 motion vector predictor candidates. This allows for candidate diversity, but may increase bitrate if candidates lower in the list are selected. The present invention broadly relates to improved derivation and ordering of one or more "pair-wise" motion vector predictor candidates in a list of motion vector predictor candidates. A "pair-wise" motion vector predictor candidate is a candidate that is combined or averaged from two or more other candidates in the candidate list. Summary of the Invention

[0007] The present invention relates to an improvement to the processing of a list of predictors (or predictor candidates), which results in an improved ordering of predictors that provides coding efficiency at the cost of some increased complexity. For example, the method seeks to prevent redundant candidates with similar costs from being adjacent in the candidate list by comparing candidate costs with each other. In an embodiment, a balance is struck between the amount of candidate reordering and the potential gain in cost diversity of the resulting list to which the processing according to the invention is applied. The present invention is particularly, but not exclusively, applicable to predictors (or predictor candidates) for motion prediction of image parts in image or video coding. However, the present invention is broadly applicable to any coding where one data unit or data item may be predicted from another data and a list of predictor candidates is generated for that purpose. For example, it is envisaged that the present invention is applicable to at least point cloud data, mesh data, audio data, as well as still and video image data.

[0008] According to one aspect of the present invention, a method for processing predictors is provided, the method comprising: obtaining a list of predictors, the list of predictors having at least two predictors; determining whether to modify the list of predictors based on a criterion using a first cost associated with a first predictor in the list and a second cost associated with a second predictor in the list; and modifying (or not modifying) the list of predictors based on the determination, the criterion being based on a threshold. The first and second predictors may be adjacent predictors in the list (in other words, one following the other). The predictors may be candidates, such as motion information predictor candidates in inter modes such as merge mode, or block predictor candidates in intra modes.

[0009] The determination includes adding the threshold to the first cost or the second cost, and the criterion relates to comparing a result to one of the first cost and the second cost that did not have the threshold added.

[0010] The criterion is whether or not a result of adding the first cost to the threshold is smaller than the second cost.

[0011] The criterion is whether or not a difference between the first cost and the second cost exceeds the threshold.

[0012] Optionally, said difference is calculated as an absolute value and said criterion is whether it is less than said threshold value.

[0013] Optionally, modifying said list comprises removing the first candidate or the second candidate from said list of predictors.

[0014] Optionally, modifying the list includes changing a position of one or more predictors in the list.

[0015] Optionally, changing the position of one or more candidates comprises moving the first predictor or the second predictor to a different position in said list.

[0016] Changing the order may involve moving the first predictor or the second predictor to a position considered to be the end of the list.

[0017] Optionally, the first predictor and the second predictor follow the zeroth predictor at a position considered to be the start of the list. In some examples, the zeroth predictor may be excluded from modifying the list of predictors (e.g., the zeroth predictor may be fixed at a top position in the list, regardless of the reordering of other predictors in the list). In some examples, the method further includes comparing a zeroth cost associated with the zeroth predictor to a zeroth threshold, and if the zeroth cost exceeds the zeroth threshold, the zeroth predictor is included in modifying the list of predictors. Comparing the zeroth cost associated with the zeroth predictor to a zeroth threshold is performed in a first mode set and not in a second mode set. In other examples, the inclusion of the zeroth predictor in modifying the predictor list may be mode dependent. In examples where comparing a zeroth cost associated with the zeroth predictor to a zeroth threshold is performed, the threshold is less than the threshold used for the first predictor and the second predictor. In other examples, the thresholds may be the same.

[0018] In another example, the first predictor is at a position considered to be the start of the list and the first cost is set to zero. In such an example, the first cost may be set to zero. That is, regardless of the true calculated cost associated with the first predictor, the method may include overwriting this value as a zero cost. Alternatively, no cost may be calculated at all and the first cost may be set as zero.

[0019] Optionally, the first cost and the second cost comprise a template matching cost based on a plurality of neighboring samples of the image portion to be encoded or decoded and a plurality of neighboring samples of the image portion associated with a predictor.

[0020] Optionally, the first cost and the second cost comprise a difference between sample values ​​of two block predictors associated with a predictor.

[0021] Optionally, the first cost and the second cost comprise a difference between a sample value associated with a predictor and a sample value associated with at least one other predictor in the list.

[0022] Optionally, the first cost and the second cost are based on a subset of a number of neighboring samples or a number of predictor samples.

[0023] Optionally, the first cost and the second cost are based on a number of samples corresponding to different images at different resolutions.

[0024] Optionally, the values ​​of the plurality of samples used to calculate said first cost and said second cost are pre-processed.

[0025] Optionally, the first cost and the second cost are based on a coding rate.

[0026] Optionally, the first cost and the second cost are based on a distortion cost, which is any of a sum of absolute differences (SAD), a sum of absolute transformed differences (SATD), a sum of squared errors (SSE), or a structural similarity index measure (SSIM).

[0027] The cost may be proportional to the number of samples in the unit being processed (image portion / prediction unit / unit of predicted data). The cost may be proportional to the number of samples in a template (e.g., a template of neighboring samples). The cost is based on the samples and block size used for distortion.

[0028] Optionally, the method further comprises deriving said threshold value. Alternatively or additionally, the method further comprises receiving information indicative of said threshold value.

[0029] The threshold may be based on a quantization parameter (QP) value. In the method according to any of claims 22 to 24, the threshold is based on a picture order count (POC) distance or a temporal distance between the current frame and one or more other frames. Alternatively or additionally, the threshold is based on a temporal ID of a frame. Alternatively or additionally, the threshold is based on a temporal direction of a reference frame. Alternatively or additionally, the threshold is based on whether bi-prediction is used or not. The threshold may depend on one or more QP values ​​of available reference frames. The threshold may depend on a type of distortion used as at least one of the first cost and the second cost. The threshold may depend on a type of sample used to calculate distortion as a measure of at least one of the first cost and the second cost. The threshold may depend on whether intra prediction or inter prediction is used for encoding or decoding the image portion predicted using the list of predictors. The threshold may be based on whether to encode or decode using one or more of a random access configuration, an RA, a low latency configuration, and an intra-only configuration.

[0030] The threshold value is based on a mathematical formula used for encoding and is obtained by the decoder for use in its calculations.

[0031] Additionally or alternatively, the threshold may be based on a shifted quantization parameter (QP) value. The shifted QP value may be used to determine a Lagrangian parameter (λ) as the threshold. The shifted quantization parameter is used to obtain a Lagrangian parameter (e.g., λ) from a table, where multiple entries in the table associate multiple Lagrangian parameter (e.g., λ) values ​​with multiple quantization parameter values.

[0032] Additionally or alternatively, the threshold may be determined as a minimum cost difference between consecutive predictors in the list that are not the first predictor and the second predictor.

[0033] Additionally or alternatively, the threshold may be determined using cost values ​​of all candidates or the first N candidates in the list, where N is the maximum number of candidates allowed in the list.

[0034] Using the cost value may include calculating an average of the difference in costs of successive candidates from among the candidates used to determine the threshold.

[0035] The threshold is based on a Lagrangian parameter (λ), and the criterion is whether the absolute difference between the first distortion cost and the second distortion cost is smaller than the Lagrangian parameter, e.g., the threshold is the Lagrangian parameter multiplied by the rate difference between the first predictor and the second predictor.

[0036] Optionally, at least one of said first distortion cost and said second distortion cost is multiplied by said threshold value.

[0037] Optionally, the criterion is applied after a first ordering process. The criterion is applied after a first ordering process. The first ordering process may be based on an ordering of costs. The costs may be template matching distortion costs.

[0038] Optionally, a threshold value is added to the result of the difference between said first distortion cost and said second distortion cost resulting from the multiplication.

[0039] Optionally, said decision is made according to said criterion during a process of ordering a number of candidates according to distortion cost.

[0040] Optionally, the decision to modify the list or not is based on a minimum distortion difference and a threshold. The maximum number of permutations may be limited to a predefined number. For example, the maximum number of permutations may be the maximum allowed number of candidates in the list.

[0041] Optionally, the decision is based on a minimum distortion difference and is limited to a maximum number of permutations.

[0042] Optionally, modifying the list includes only deleting candidates. Optionally, the number of candidates after deletion must be equal to or greater than the maximum number of candidates allowed in the list. In an embodiment, when the list is modified by moving a candidate, the maximum number of candidates allowed in the list is subtracted.

[0043] Optionally, if the criterion is met, the second predictor is moved to a first position in the list whose cost difference with respect to the subsequent predictor is greater than the threshold. If the maximum number of possible permutations is greater than the maximum number of candidates allowed in the final list, the modifying includes restricting the position to which the first predictor or the second predictor is moved to the end of the current list. Optionally, the predictor processing is applied to at least one of a normal merge mode, a template matching merge mode, a bilateral matching (BM) merge mode, and an intra block copy (IBC) mode.

[0044] In an embodiment, the second predictor immediately follows the first predictor (in the list), and if it is determined that the criterion is not met, the second predictor is inserted after the subsequent predictor in the list. Optionally, if it is determined that the criterion is not met, the criterion is applied to the second cost and to the individual costs of the previous candidates in the list, and if it is determined that the criterion is not met for the second cost and each of the costs of the previous candidates, the second predictor is not inserted after the subsequent predictor. The maximum number of possible permutations may be set to the allowed number of candidates in the final list of processed candidates. Furthermore, the predictor processing is applied to at least one of merging with MVD, affine merging with MVD, and MVD code prediction of motion vector residuals.

[0045] The list of predictors may be a sub-list of predictors (which may be combined with other sub-lists to form a final list, an intermediate list, or other sub-lists), an intermediate list, or a final list of predictors.

[0046] In one embodiment, when template matching is used and at least one of two templates is available, the cost of all predictors (candidates) in the list may be calculated based on using the weighted cost of the available templates as the cost of the unavailable template. For example, the weighting may be such that the cost is proportional to the height and width of the unavailable template. The template may be the left template or the top template.

[0047] In one embodiment, template matching is used and if at least one of the two templates is available, the Lagrangian or lambda is modified proportionally to the missing template.

[0048] In one embodiment, the motion vector for derivation,MV threshold is the same for all merge predictor (candidate) types.,Such an embodiment may be advantageously applied to,merging modes where different motion vector thresholds are applied,based on the type of predictor (candidate), e.g.,,normal, template matching (TM), or block matching (BM),merging modes.

[0049] In one embodiment, when the method of the present invention described in any of the above aspects and embodiments is enabled (or enabled), the ARMC process is applied to all predictors (candidates) derived and ordered in the ARMC.

[0050] In an embodiment where the proposed method is enabled (or effective), pairwise predictors (candidates) are added during or after the ordering process.

[0051] In a further aspect of the invention, a method for processing a list of predictors is provided, comprising determining a minimum cost difference between two consecutive predictors (or predictor candidates) in the list, and if this minimum difference is less than a threshold (λ), relocating one of the candidates to the last position in the list. Optionally, the last position is before a candidate in the list that is considered redundant. The cost difference may be an absolute cost difference.

[0052] According to another further aspect of the invention, there is provided a method of processing a predictor list comprising: determining a minimum cost difference between two successive predictors (or candidate predictors), and if the minimum difference is less than a threshold (lambda), comparing the distortion of one of the (redundant) predictors with the distortion of a (further) subsequent predictor in the list, identifying the position of the first subsequent predictor having a distortion difference higher than lambda, and placing the predictor in this identified position. The cost may be a distortion. The cost difference may be an absolute cost difference.

[0053] According to yet another aspect of the invention, there is provided a method of processing a list of predictors that includes determining a first subsequent non-redundant predictor (or predictor candidate) in the list for a candidate and inserting the non-redundant predictor after the candidate. A redundant candidate is, for example, a candidate whose cost difference is determined to be less than a threshold. The cost may be a distortion. The cost difference may be an absolute cost difference.

[0054] According to another further aspect of the invention, there is provided a method of processing a predictor list including: - for a predictor (or predictor candidate), determine the first, next, non-redundant predictor in the list and insert this non-redundant candidate after the candidate unless this non-redundant candidate is determined to be redundant compared to all candidates before the candidate. A redundant candidate is, for example, a candidate whose cost difference is determined to be less than a threshold. The cost may be a distortion. The cost difference may be an absolute cost difference.

[0055] According to another further aspect of the present invention, a method for processing predictors is provided, comprising obtaining a list of predictors, determining whether an absolute distortion cost difference between a first distortion cost of a first predictor and a second distortion cost of a second predictor is less than a lambda value (e.g., a Lagrangian), where the second predictor immediately follows the first predictor in the list, and moving the position of the first predictor in the list if the absolute distortion cost difference is less than the lambda value. The second predictor may be moved to a position immediately after the last candidate in the list that is currently considered not redundant and before any redundant candidate. A redundant candidate is, for example, a candidate that has previously been determined to have an absolute distortion difference with a preceding candidate that is less than λ. The λ value may be derived based on a QP value or a shifted QP value. For example, the lambda value may be obtained from a table having entries indexed or associated with the shifted QP value.

[0056] According to another further aspect of the present invention, a method for processing a list of predictors is provided, the method including: obtaining a list of predictors, the list of predictors having at least two predictors; determining whether to modify the list of predictors based on a comparison of a first cost associated with a first predictor in the list of predictors to a threshold; and modifying the list of predictors based on the determination. In some examples, determining whether to modify the list of predictors is based on comparing a respective cost associated with each predictor in the list to a threshold. That is, each cost associated with each predictor in the list may be compared to a threshold. In some examples, modifying the list includes removing the candidate or each candidate from the list of predictors whose cost is less than the threshold. Additionally or alternatively, modifying the list includes moving the candidate or each candidate in the list whose cost is less than the threshold to a position in the list next to a candidate whose cost is greater than the threshold. In yet another example, modifying the list may include moving the candidate or each candidate in the list whose cost is less than the threshold to a position considered to be the end of the list. In all examples, modifying the list of predictors may be performed in a first set of modes and not in a second set of modes, i.e., the reordering of predictors in the list may be mode dependent.

[0057] These further aspects may be usefully combined with any of the foregoing descriptions or embodiments already mentioned.

[0058] The following optional features may be applied in combination with any of the above aspects and optional features: Optionally, the list of predictors is for predicting motion in an image portion. The predictors in the list are either intra block predictor candidates or inter motion information predictor candidates. Alternatively, the predictors in the list may be for predicting a position in a point cloud, encoding a mesh, predicting audio data, predicting still image data, or predicting video data. The method may further comprise associating at least one syntax element with one or more predictors in the list of predictors.

[0059] In one aspect according to the present invention, there is provided a method for encoding image data into a bitstream, comprising processing a list of predictors according to any aspect or embodiment described above.

[0060] In one aspect according to the present invention, there is provided a method for decoding image data from a bitstream, comprising processing a list of predictors according to any aspect or embodiment described above.

[0061] According to an aspect of the present invention there is provided an apparatus for encoding image data into a bitstream, said apparatus configured to process a list of predictors according to any aspect or embodiment described above.

[0062] In an aspect according to the present invention there is provided an apparatus for decoding image data from a bitstream, said apparatus configured to process a list of predictors according to any aspect or embodiment described above.

[0063] According to another aspect of the invention there is provided a (computer) program which, when executed, causes a programmable device to carry out a method according to any aspect or embodiment described above. The program may be stored on a computer readable storage medium.

[0064] The program may be provided by itself or on, by or in a carrier medium. The carrier medium may be non-transitory, such as a storage medium, in particular a computer-readable storage medium. The carrier medium may also be transitory, such as a signal or other transmission medium. The signal may be transmitted via any suitable network, including the Internet. Further features of the invention are characterized by the independent and dependent claims.

[0065] Features in one aspect of the invention may be applied to other aspects of the invention in any appropriate combination, in particular method aspects may be applied to apparatus aspects and vice versa.

[0066] Furthermore, functionality implemented in hardware may be implemented in software and vice versa, and references herein to software and hardware features should be construed accordingly.

[0067] Apparatus features as described herein may also be provided as method features, and vice versa. As used herein, means-plus-function features may alternatively be expressed in terms of corresponding structure, such as a suitably programmed processor and associated memory.

[0068] It is also to be understood that particular combinations of the various features described and defined in any aspect of the present invention can be implemented and / or provided and / or used independently. [Brief description of the drawings]

[0069] Reference will now be made, by way of example only, to the accompanying drawings, in which:

[0070] [Figure 1] FIG. 1 is a diagram for explaining the coding structure used in HEVC. [Diagram 2]1 is a block diagram that illustrates generally a data communications system in which one or more embodiments of the present invention may be implemented; [Diagram 3] 1 is a block diagram illustrating components of a processing device in which one or more embodiments of the present invention may be implemented. [Figure 4] 4 is a flow chart illustrating steps of an encoding method according to an embodiment of the present invention. [Diagram 5] 4 is a flow chart illustrating steps of a decoding method according to an embodiment of the present invention. [Figure 6] FIG. 2 illustrates a labeling scheme used to describe multiple blocks relative to a current block. [Figure 7] FIG. 2 illustrates a labeling scheme used to describe multiple blocks relative to a current block. [Figure 8] 13(a) and (b) are diagrams illustrating affine (sub-block) mode. [Figure 9] 1A, 1B, 1C, and 1D are diagrams showing geometric modes. [Figure 10-1] FIG. 1 illustrates the first step of VVC merge candidate list derivation. [Figure 10-2] FIG. 1 illustrates the first step of VVC merge candidate list derivation. [Figure 11] FIG. 13 illustrates a further step in deriving a merge candidate list for VVC. [Figure 12] FIG. 13 is a diagram illustrating the derivation of pair candidates. [Figure 13] FIG. 1 illustrates a template matching method based on neighboring samples. [Figure 14-1] FIG. 11 is a modification of the first step of deriving the merge candidate list shown in FIG. [Figure 14-2] FIG. 11 is a modification of the first step of deriving the merge candidate list shown in FIG. [Figure 15] FIG. 12 is a modification of the further steps of the merge candidate list derivation shown in FIG. [Figure 16]FIG. 13 is a diagram in which the derivation of pair candidates shown in FIG. 12 is modified. [Figure 17] FIG. 13 illustrates cost determination for list candidates. [Figure 18] FIG. 13 is a diagram illustrating a process of rearranging the merge mode candidate list. [Figure 19] 13A and 13B are diagrams illustrating derivation of pair candidates in a sorting process of a merge mode candidate list. [Figure 20] FIG. 1 illustrates a process for ordering predictors based on distortion values. [Figure 21] FIG. 1 illustrates a process for ordering predictors based on distortion values ​​according to an embodiment of the present invention. [Figure 22] FIG. 13 illustrates a process for reordering predictors based on distortion values ​​according to an embodiment of the present invention. [Figure 23] FIG. 13 illustrates a process for reordering predictors based on distortion values ​​according to an embodiment of the present invention. [Figure 24] FIG. 13 illustrates a process for reordering predictors based on distortion values ​​according to an embodiment of the present invention. [Diagram 25] FIG. 13 illustrates a process for reordering predictors based on distortion values ​​according to an embodiment of the present invention. [Figure 26] FIG. 13 illustrates a process for reordering predictors based on distortion values ​​according to an embodiment of the present invention. [Figure 27] FIG. 13 illustrates a process for reordering predictors based on distortion values ​​according to an embodiment of the present invention. [Figure 28] FIG. 13 illustrates a process for reordering predictors based on distortion values ​​according to an embodiment of the present invention. [Figure 29] FIG. 13 is a diagram illustrating a process of rearranging predictors based on distortion values ​​according to an embodiment of the present invention. [Diagram 30] FIG. 13 is a diagram illustrating an ARMC sorting process of a sub-block merge list. [Diagram 31] FIG. 13 illustrates a process for reordering predictors based on distortion values ​​according to an embodiment of the present invention. [Diagram 32] FIG. 13 illustrates a process for reordering predictors based on distortion values ​​according to an embodiment of the present invention. [Diagram 33]FIG. 1 illustrates a system including an encoder or decoder and a communication network according to an embodiment of the present invention. [Diagram 34] 1 is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention. [Diagram 35] FIG. 1 illustrates a network camera system. [Diagram 36] FIG. 1 illustrates a smartphone. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0071] Figure 1 relates to the coding structure used in the High Efficiency Video Coding (HEVC) video and Versatile Video Coding (VVC) standards. A video sequence 1 is composed of a succession of digital images i, each such digital image being represented by one or more matrices, whose coefficients represent pixels.

[0072] An image 2 of a sequence may be divided into multiple slices 3. A slice may constitute the entire image. These slices are divided into non-overlapping coding tree units (CTUs). A coding tree unit (CTU) is the basic processing unit of the High Efficiency Video Coding (HEVC) video standard and is a structure that conceptually corresponds to the macroblock unit used in some earlier video standards. A CTU is sometimes called a maximal coding unit (LCU). A CTU has luma and chroma component parts, each of which is called a coding tree block (CTB). These different color components are not shown in Figure 1.

[0073] A CTU is typically 64 pixels by 64 pixels in HEVC, but 128 pixels by 128 pixels in VVC. Each CTU can be iteratively divided into smaller variable-sized coding units (CUs) 5 using a quad-tree decomposition.

[0074] A coding unit is a basic coding element and consists of two types of subunits called prediction units (PUs) and transform units (TUs). The maximum size of a PU or TU is equal to the CU size. A prediction unit corresponds to a partition of a CU for predicting pixel values. Various different divisions of a CU into PUs are possible, such as a division into four square PUs and two different divisions into two rectangular PUs, as shown in 606. A transform unit is an element unit that undergoes spatial transformation using DCT. A CU can be divided into TUs based on a quadtree representation 607.

[0075] Each slice is embedded in one network abstraction layer (NAL) unit. Furthermore, the coding parameters of a video sequence are stored in a dedicated NAL unit called a parameter set. HEVC and H.264 / AVC employ two types of parameter set NAL units. First, the sequence parameter set (SPS) NAL unit collects all parameters that do not change throughout the video sequence. It typically deals with the coding profile, the size of the video frames, and other parameters. Second, the picture parameter set (PPS) NAL unit contains parameters that may change from one picture (or frame) of the sequence to another. HEVC also includes the video parameter set (VPS) NAL unit, which contains parameters that describe the overall structure of the bitstream. VPS is a type of parameter set defined in HEVC that applies to all layers of the bitstream. A layer may contain multiple temporal sublayers, while all version 1 bitstreams are limited to one layer. HEVC has specific layer extensions for scalability and multiview, which allow for a backward-compatible version 1 base layer and multiple layers.

[0076] VVC introduces another way of dividing an image: sub-pictures, which are independently coded groups of one or more slices.

[0077] 2 illustrates a data communication system in which one or more embodiments of the present invention may be implemented. The data communication system includes a transmitting device (in this case a server 201) operable to transmit data packets of a data stream to a receiving device (in this case a client terminal 202) via a data communication network 200. The data communication network 200 may be a wide area network (WAN) or a local area network (LAN). Such a network may be, for example, a wireless network (Wifi / 802.11a or b or g), an Ethernet network, an Internet network, or a mixed network including several different networks. In a particular embodiment of the present invention, the data communication system may be a digital television broadcasting system in which the server 201 transmits the same data content to several clients.

[0078] The data streams 204 provided by the server 201 may consist of multimedia data representing video and audio data. The audio and video data streams may, in some embodiments of the invention, be captured by the server 201 using a microphone and a camera, respectively. In some embodiments, the data streams are stored in the server 201, received by the server 201 from other data providers, or generated at the server 201. The server 201 comprises, inter alia, an encoder for encoding the video and audio streams to provide a compressed bitstream for transmission, which is a more compact representation of the data presented as input to the encoder.

[0079] In order to obtain a better ratio of the quality of the transmitted data to the amount of transmitted data, the compression of the video data may for example be according to the HEVC format or the H.264 / AVC format or the VVC format.

[0080] Client 202 receives the transmitted bitstream, decodes the reconstructed bitstream, and plays the video images on a display device and the audio data on a loudspeaker.

[0081] Although the example of FIG. 2 considers a streaming scenario, it will be appreciated that in some embodiments of the invention, data communication between the encoder and the decoder may be performed using a media storage device, such as an optical disc.

[0082] In one or more embodiments of the invention, a video image is transmitted along with data representative of correction offsets that are applied to reconstructed pixels of the image to provide filtered pixels in a final image.

[0083] 3 shows a schematic diagram of a processing device 300 configured to implement at least one embodiment of the present invention. The processing device 300 may be a device such as a microcomputer, a workstation or a lightweight handheld device. The device 300 comprises a communication bus 313 connected to: - a central processing unit 311 (denoted as CPU), such as a microprocessor; - a read-only memory 306 (denoted ROM) for storing a computer program for implementing the invention; a random access memory 312 (denoted RAM) for storing the executable code of the method according to an embodiment of the invention and registers suitable for recording variables and parameters necessary for implementing the method for encoding a sequence of digital images according to an embodiment of the invention and / or the method for decoding a bitstream according to an embodiment of the invention; and a communications interface 302 connected to a communications network 303 over which the digital data to be processed is sent and received;

[0084] Optionally, device 300 may also include the following components: - data storage means 304, such as a hard disk, for storing computer programs for implementing the methods of one or more embodiments of the invention, and data used or generated during the implementation of one or more embodiments of the invention; a disk drive 305 for a disk 306. The disk drive is adapted to read data from or write data to the disk 306; A screen 309 for displaying data and / or acting as a graphical interface with the user, via a keyboard 310 or other pointing means.

[0085] The device 300 can be connected to a variety of peripherals, such as a digital camera 320 and a microphone 308 , each connected to an I / O card (not shown), which provide multimedia data to the device 300 .

[0086] The communication bus provides communication and interoperability between the various elements included in or connected to the device 300. The representation of the bus is not limiting, and in particular the central processing unit is operable to communicate instructions to any element of the device 300 directly or by other elements of the device 300.

[0087] The disk 306 may be replaced by any information medium, such as, for example, a compact disk (CD-ROM), rewritable or not, a ZIP disk or a memory card, and generally speaking, by any information storage means readable by a microcomputer or microprocessor, whether integrated in the device or not, possibly removable, and suitable for storing one or more programs, the execution of which can implement the method of encoding a sequence of digital images and / or the method of decoding a bitstream according to the present invention.

[0088] The executable code may, as mentioned above, either be stored in the read-only memory 306, on the hard disk 304 or on a removable digital medium such as for example the disk 306. According to a variant, the executable code of the program may be received by the communications network 303 via the interface 302, to be stored in one of the storage means of the device 300, for example the hard disk 304, before being executed.

[0089] The central processing unit 311 is adapted to control and direct the execution of instructions or parts of the software code of a program or program according to the invention, instructions stored in one of the storage means mentioned above. On power-up, the program or programs stored in a non-volatile memory (for example the hard disk 304 or the read-only memory 306) are transferred to a random access memory 312, which contains the executable code of the program or programs, as well as registers for storing variables and parameters necessary for the implementation of the invention.

[0090] In this embodiment, the device is a programmable device that uses software to implement the invention, but the invention could alternatively be implemented in hardware (e.g., in the form of an application specific integrated circuit (ASIC)).

[0091] 4 is a block diagram of an encoder according to at least one embodiment of the present invention, the encoder being represented by a number of connected modules, each adapted to implement, for example in the form of programming instructions executed by the CPU 311 of the device 300, at least one corresponding step of a method for implementing at least one embodiment of encoding images of an image sequence according to one or more embodiments of the present invention.

[0092] The encoder 400 receives as input an original sequence of digital images i0 to in401. Each digital image is represented by a collection of samples, also called pixels (hereafter also referred to as pixels).

[0093] A bitstream 410 is output by the encoder 400 after the encoding process is performed. The bitstream 410 includes multiple coding units or slices, each of which includes a slice header for transmitting coded values ​​of coding parameters used to code the slice, and a slice body that includes coded video data.

[0094] An input digital image i0 to 401 is divided by module 402 into blocks of pixels. The blocks correspond to image portions and may be of variable size (for example 4x4, 8x8, 16x16, 32x32, 64x64, 128x128 pixels, and several rectangular block sizes can also be considered). A coding mode is selected for each input block. Two families of coding modes are provided: coding modes based on spatial predictive coding (intra prediction) and coding modes based on temporal predictive coding (inter coding, merge, SKIP). Possible coding modes are tested.

[0095] Module 403 performs an intra prediction process in which a given block to be coded is predicted by a prediction calculated from pixels in the neighborhood of the block to be coded. The indication of the selected intra prediction and the difference between a given block and its prediction are coded to provide a residual when intra coding is selected.

[0096] Temporal prediction is performed by a motion estimation module 404 and a motion compensation module 405. First, a reference image is selected from a set of reference images 416, and the motion estimation module 404 selects a part of the reference image (also called a reference region or image portion), which is the region closest to a given block to be coded (closest in terms of pixel value similarity). The motion compensation module 405 then uses the selected region to predict the block to be coded. The difference between the selected reference region and the given block (also called a residual block) is calculated by the motion compensation module 405. The selected reference region is indicated by a motion vector.

[0097] Therefore, in both cases (spatial and temporal prediction), the residual is calculated by subtracting the predicted value from the original block.

[0098] In the intra prediction implemented by module 403, the prediction direction is coded. In the inter prediction implemented by modules 404, 405, 416, 418, 417, at least one motion vector or data for identifying such a motion vector is coded for the temporal prediction.

[0099] If inter prediction is selected, the motion vector and information related to the residual block are coded. To further reduce the bit rate, assuming the motion is homogeneous, the motion vector is coded by the difference to the motion vector prediction. The motion vector prediction and coding module 417 obtains the motion vector prediction from a set of motion information prediction candidates from the motion vector field 418.

[0100] The encoder 400 further comprises a selection module 406 for selecting a coding mode applying a coding cost criterion, such as a rate-distortion criterion. To further reduce redundancy, a transform (such as a DCT) is applied to the residual block by a transform module 407, and the resulting transformed data is quantized by a quantization module 408 and entropy coded by an entropy coding module 409. Finally, the coded residual block of the current block being coded is inserted into a bitstream 410.

[0101] The encoder 400 also performs decoding of the encoded image to generate reference images (e.g., those of the reference image / picture 416) for motion estimation of subsequent images. This allows the encoder and decoder receiving the bitstream to have the same reference frame (reconstructed image or image portion is used). The inverse quantization ("dequantization") module 411 performs inverse quantization ("dequantization") of the quantized data, followed by inverse transformation by the inverse transformation module 412. The intra prediction module 413 uses the prediction information to decide which prediction to use for a given block, and the motion compensation module 414 actually adds the residual obtained by the module 412 to the reference area obtained from the set of reference images 416.

[0102] Post-filtering is then applied by module 415 to filter the reconstructed frame of pixels (image or image portion). In an embodiment of the invention, an SAO loop filter is used in which a compensation offset is added to the pixel values ​​of the reconstructed pixels of the reconstructed image. It is understood that post-filtering does not always have to be performed. Also, other types of post-filtering can be performed in addition to or instead of SAO loop filtering.

[0103] 5 is a block diagram of a decoder 60 that may be used to receive data from an encoder according to an embodiment of the present invention. The decoder is represented by a number of connected modules, each adapted to implement a corresponding step of a method implemented by the decoder 60, for example in the form of programming instructions executed by the CPU 311 of the device 300.

[0104] The decoder 60 receives a bitstream 61 containing coding units (e.g. data corresponding to blocks or coding units), each including a header containing information about coding parameters and a body containing the coded video data. As explained with respect to Fig. 4, the coded video data is entropy coded, and an index of a motion vector prediction is coded with a given number of bits for a given block. The received coded video data is entropy decoded by module 62. The residual data is inverse quantized by module 63 before an inverse transform is applied by module 64 to obtain pixel values.

[0105] Mode data indicating the encoding mode is also entropy decoded, and intra-type or inter-type decoding is performed on the block (unit / set / group) of encoded image data based on the mode.

[0106] For intra modes, intra prediction is determined by intra prediction module 65 based on the intra prediction mode specified in the bitstream.

[0107] If the mode is inter, motion prediction information is extracted from the bitstream to find (identify) the reference region to be used by the encoder. The motion prediction information includes a reference frame index and a motion vector residual. The motion vector prediction is added to the motion vector residual by the motion vector decoding module 70 to obtain a motion vector. The various motion prediction tools used in VVC are described in more detail below with reference to Figures 6 to 10.

[0108] A motion vector decoding module 70 applies motion vector decoding to each current block that has been coded with motion prediction. Once the index of the motion vector prediction for the current block is obtained, the actual value of the motion vector associated with the current block can be decoded and used to apply motion compensation by module 66. The reference image portion indicated by the decoded motion vector is extracted from reference image 68 for applying motion compensation 66. Motion vector field data 71 is updated with the decoded motion vector for use in predicting subsequent decoded motion vectors.

[0109] Finally, a decoded block is obtained. If necessary, post-filtering is applied by a post-filtering module 67. A decoded video signal 69 is finally obtained and provided by the decoder 60.

[0110] Multiple motion estimation (inter) modes HEVC uses three different inter modes: Inter mode (Advanced Motion Vector Prediction: AMVP), "classical" merge mode (also called "non-affine merge mode" or "normal" merge mode), and "classical" merge Skip mode (also called "non-affine merge skip" mode or "normal" merge skip mode). The main difference between these modes is the data signal in the bitstream. Regarding motion vector coding, the current HEVC standard includes a contention-based scheme for motion vector prediction. This means that for each inter or merge mode (i.e., "classical / normal" merge mode or "classical / normal" merge skip mode), multiple candidates compete with a rate-distortion criterion on the encoder side to find the best motion vector prediction or the best motion information. Then, an index corresponding to the best candidate for the best prediction or motion information is inserted into the bitstream along with a "residual" that represents the difference between the predicted value and the actual value. The decoder can derive the same set of predictions or candidates and uses the best one according to the decoded index. Using the residual, the decoder can recreate the original value.

[0111] In the Screen Content Extension of HEVC, a new coding tool called Intra Block Copy (IBC) is signaled as one of these three inter modes, and the difference between IBC and its equivalent inter modes is that it checks if the reference frame is the current one. This can be implemented, for example, by checking the reference index of list L0 and inferring that if this is the last frame in that list, this is an intra block copy. Another way is to compare the picture order counts of the current and reference frames.

[0112] The design of prediction and candidate derivation is important to achieve the best coding efficiency without disproportionately impacting the complexity. In HEVC, two motion vector derivations are used: one for inter modes (Advanced Motion Vector Prediction (AMVP)) and one for merge modes (merge derivation processes - classical merge mode and classical merge skip mode). In the following, the different motion prediction modes used in VVC are described.

[0113] FIG. 6 illustrates the labeling scheme used herein to describe blocks located relative to the current block (ie, the block currently being coded / decoded) between frames (FIG. 6).

[0114] VVC Merge Mode VVC adds some inter modes compared to HEVC, in particular adding new merge modes to the usual merge modes of HEVC.

[0115] Affine mode (sub-block mode) In HEVC, only the translational motion model is applied for motion compensated prediction (MCP), but in the real world there are many kinds of motion, including zoom, rotation, perspective, and other irregular motions.

[0116] In JEM, a simplified affine transform motion compensation prediction is applied and the general principles of the affine mode are explained below based on an excerpt from document JVET-G1001 presented at the JVET Conference held in Turin on 13-21 July 2017. This document is incorporated herein in its entirety by reference insofar as other algorithms used in JEM are described.

[0117] As shown in FIG. 8(a), the affine motion field of a block is described by two control point motion vectors.

[0118] Affine mode is a motion compensation mode like inter modes (AMVP, "classical" merge, or "classical" merge skip). Its principle is to generate one motion information per pixel according to two or three neighboring motion information. In JEM, affine mode derives one motion information for each 4x4 block as depicted in Fig. 8(a) (each square is a 4x4 block, and the whole block in Fig. 8(a) is a 16x16 block divided into 16 blocks of 4x4 size squares - each 4x4 square block has a motion vector associated with it). Affine mode is available in AMVP and merge modes (i.e., classical merge mode, also called "non-affine merge mode", and classical merge skip mode, also called "non-affine merge skip mode") by enabling affine mode with a flag.

[0119] In the VVC specification, the affine mode is also called the sub-block mode.

[0120] The subblock merging mode of VVC includes a subblock-based temporal merging candidate that inherits the motion vector field of the block of the previous frame pointed to by the spatial motion vector candidate. This subblock candidate follows the inherited affine motion candidate if the neighboring block is coded in the inter-affine mode of subblock merging, and some constructed affine candidates are derived before some zero Mv candidates.

[0121] CIIP In addition to the normal and sub-block merge modes, the VVC standard also includes a Combined Inter-Merge / Intra Prediction (CIIP), also known as the Multi-Hypothesis Intra-Inter (MHII) merge mode.

[0122] Combined Inter Merge / Intra Prediction (CIIP) merge can be considered as a combination of the normal merge mode and the intra mode, and is described below with reference to FIG. 10 (partially illustrated in FIG. 10-1 and FIG. 10-2). The block prediction of the current block (1001) in this mode is the average between the merge prediction block and the intra prediction block, as depicted in FIG. 10. The merge prediction block is obtained by exactly the same process as in the merge mode, and is therefore bi-predictive of a temporal block (1002) or two temporal blocks. Therefore, the merge index is signaled for this mode in the same way as in the normal merge mode. The intra prediction block is obtained based on the neighboring samples (1003) of the current block (1001). However, the amount of intra modes available for the current block is limited compared to intra blocks. Furthermore, there is no chroma intra prediction block signaled for CIIP blocks. Chroma prediction is equal to luma prediction. As a result, 1, 2, or 3 bits are used for the intra prediction signal of a CIIP block.

[0123] The CIIP block prediction is obtained by a weighted average of the merge block prediction and the intra block prediction, where the weighting of the weighted average depends on the selected block size and / or intra prediction block.

[0124] Then, the obtained CIIP prediction is added to the residual of the current block to obtain the reconstructed block. It should be noted that CIIP mode is only valid for non-skip blocks. In fact, using CIIP skip usually leads to a worse compression performance and an increased encoder complexity. This is because CIIP modes often have the opposite block residual to other skip modes. As a result, signaling skip mode increases the bitrate. -CIIP is avoided if the current CU is skip. In fact, in VVC, the only way to signal a block residual equal to 0 for merge mode is to use skip mode, because the CU_CBF flag is inferred to be true for merge mode. And when this CBF flag is true, the block residual cannot be 0.

[0125] Thus, in this specification, CIIP should be interpreted as a mode that combines features of inter prediction and intra prediction, and not necessarily as a label given to one particular mode.

[0126] CIIP uses the same motion vector candidate list as the normal merge mode.

[0127] MMVD The MMVD merge mode is the derivation of a certain regular merge mode candidate. It can be seen as an independent merge candidate list. The MMVD merge candidate selected for the current CU is obtained by adding an offset value to one motion vector component (mvx or mvy) of the first regular merge candidate. The offset value is added to the motion vector of the first list L0 or to the motion vector of the second list L1 depending on the configuration of these reference frames (both backward, both forward, or forward and backward). The first merge candidate is indicated by an index. The offset value is indicated by a distance index between eight possible distances (1 / 4-pel, 1 / 2-pel, 1-pel, 2-pel, 4-pel, 8-pel, 16-pel, 32-pel) and a direction index giving the x- or y-axis and the sign of the offset.

[0128] In VVC, only the first two candidates in the regular merge list are used to derive the MMVD and are signaled with one flag.

[0129] Geometric division mode Geometric (GEO) merge mode is a specific bi-prediction mode. Figure 9 illustrates the generation of this specific block prediction. The block prediction includes one triangle from the first block prediction (901 or 911) and a second triangle from the second block prediction (902 or 912). However, several other possible divisions of the block are possible, as depicted in Figures 9(c) and 9(d). Geometric merge should be interpreted herein as a mode that combines the features of two inter non-square predictions, and not necessarily as a label given to one specific mode.

[0130] In the example of Fig. 9(a), each partition (901 or 902) has a motion vector candidate that is a unidirectional candidate. And for each partition, an index is signaled to get the corresponding motion vector candidate in the list of unidirectional candidates at the decoder. And the first and second cannot be the same candidate. This list of candidates comes from the list of regular merge candidates, and for each candidate, one of the two components (L0 or L1) is removed.

[0131] IBC VVC also allows the enabling of intra-block copy (IBC) merge mode, which has an independent merge candidate derivation process.

[0132] Other movement information improvements DMVR Decoder-side motion vector derivation (DMVR) in VVC improves the accuracy of MV in merge mode. In this method, bilateral matching (BM) based decoder-side motion vector refinement is applied. In this bi-prediction operation, a refined MV is searched around the initial MV in reference picture list L0 and reference picture list L1. The BM method calculates the distortion between two candidate blocks in reference picture list L0 and list L1.

[0133] BDOF VVC also integrates a bidirectional optical flow (BDOF) tool. BDOF, previously called BIO, is used to refine the bidirectional prediction signal of a CU at the 4x4 subblock level. BDOF is applied when a CU satisfies some conditions, in particular when the distances (i.e., POC (Picture Order Count) difference) from two reference pictures to the current picture are the same. As the name suggests, the BDOF mode is based on the concept of optical flow, which assumes that object motion is smooth. For each 4x4 subblock, a motion refinement (v_x,v_y) is calculated by minimizing the difference between the L0 predicted sample and the L1 predicted sample. This motion refinement is then used to adjust the two predicted sample values ​​of the 4x4 subblock.

[0134] PROF Similarly, prediction refinement by optical flow (PROF) is used for the affine modes.

[0135] AMVR and hpelIfIdx VVC also includes Adaptive Motion Vector Resolution (AMVR). AMVR allows the motion vector difference of a CU to be coded with different precision. For example, in AMVP mode, 1 / 4-luma samples, 1 / 2-luma samples, integer-luma samples, or 4-luma samples are considered. The following table from the VVC specification shows the AMVR shifts based on different syntax elements: TIFF2025510535000002.tif45153

[0136] AMVR may affect coding modes other than those using motion vector differential coding as different merging modes. Indeed, for some merging candidates, a parameter hpelIfIdx is propagated, which represents the index of the luma interpolation filter with half-pel precision. For example, in AMVP mode, hpelIfIdx is derived as follows: hpelIfIdx=AmvrShift==3?1:0

[0137] Bi-prediction with CU-level weights (BCW) In VVC, the Bi-prediction with CU-level weighting (BCW) mode is extended to allow not only simple averaging (as performed in HEVC) but also weighted averaging of the two prediction signals P0 and P1 according to the following formula: P bi-pred =((8-w)*P0+w*P1+4)≫3

[0138] Weighted average biprediction allows 5 weights, w∈{-2, 3, 4, 5, 10}.

[0139] For non-merged CUs, the weight index bcwIndex is signaled after the motion vector difference.

[0140] For a merged CU, the weight index is inferred from neighboring blocks based on the merge candidate indexes.

[0141] BCW is only used for CUs with 256 or more luma samples. In addition, for low-latency pictures, all five weights are used. For non-low-latency pictures, only three weights (w∈{3,4,5}) are used.

[0142] Normal merge list derivation In VVC, a normal merge list is derived as shown in Figures 10 and 11. First, if spatial candidates B1 (1002), A1 (1006), B0 (1010), and A0 (1014) (shown in Figure 7) exist, they are added. Then, partial redundancy is performed by adding A1 (1008) between the motion information of A1 and B1 (1007), adding B0 (1012) between the motion information of B0 and B1 (1011), and adding A0 (1016) between the motion information of A0 and A1 (1015).

[0143] When a merge candidate is added, the variable cnt is incremented (1015, 1009, 1013, 1017, 1023, 1027, 1115, 1108).

[0144] If the number of candidates in the list (cnt) is strictly less than four (1018), candidate B2 (1019) is added (1022) if it does not have the same motion information as A1 and B1 (1021).

[0145] Next, temporal candidates are added: if the bottom right candidate (1024) exists (1025), it is added (1026), otherwise the middle temporal candidate (1028) exists (1026), it is added (1029).

[0146] Next, a history-based (HMVP) is added (1101), but only if it has the same motion information as A1, B1 (1103). Furthermore, the number of history-based candidates must not exceed the maximum number of candidates in the merge candidate list minus one (1102). Therefore, there is at least one position missing in the merge candidate list after a history-based candidate.

[0147] Next, if the number of candidates in the list is at least two, we build pair candidates (1106) and add them to the merge candidate list (1107).

[0148] Next, if there are any empty positions in the merge candidate list (1109), a zero candidate is added (1110).

[0149] For spatial and history-based candidates, the parameters BCWidx and useAltHpelIf are set equal to the relevant parameters of the candidate. For spatial and zero candidates, they are set equal to the default value of 0. These default values ​​essentially disable the method.

[0150] For paired candidates, BCldx is set to 0, and hpelIfIdxp is set to the hpelIfIdxp of the first candidate if it is equal to the hpelIfIdxp of the second candidate, otherwise it is set to 0.

[0151] Pair candidates The pair candidates are constructed (1106) according to the algorithm of FIG. 12. As depicted, when there are two candidates in the list (1201), hpelIfIdxp is derived as described above (1204, 1202, 1203). Then, the inter direction (interDir) is set equal to 0 (1205). For each list, L0 and L1, if at least one reference frame is valid (different from -1) (1207), the parameter is set. If both are valid (1208), the mv information of this candidate is derived (1209) and set equal to the reference frame of the first candidate, the motion information is the average between the two motion vectors of this list, and the variable interDir is incremented. If only one of the candidates has motion information for this list (1210), the motion information of the pair candidate is set equal to this candidate (1212, 1211) and the inter direction variable interDir is incremented.

[0152] ECM After the standardization of VVCv1 was completed, JVET launched the Exploration Software (ECM) and started the exploration phase. ECM aims to improve the efficiency of encoding by adding additional tools and improving existing tools on top of the VVC standard. Various additional tools compared to VVC are described in JVET-X2025.

[0153] ECM Merge Mode Among all the tools added, some merging modes have been added. Affine MMVD signal offset is for merging affine candidates as MVVD encoding in normal merging mode. Similarly, GEO_MMVD has also been added. CIIP_PDPC is an extension of CIIP. Also, two template matching merging modes have been added: normal template matching and GEO template matching.

[0154] Normal template matching is performed based on template matching estimation as shown in Figure 13. At the decoder side, for the candidate corresponding to the associated merge index and both available lists (L0, L1), motion estimation based on neighboring samples of the current block (1301) and motion estimation based on neighboring samples of corresponding multiple block positions is performed, the cost is calculated, and the motion information that minimizes the cost is selected. The motion estimation is limited by a search range, and some restrictions on this search range are also used to reduce the complexity.

[0155] In ECM, the normal template matching candidate list is based on the normal merge list, but some additional steps and parameters are added, which may result in different merge candidate lists for the same block. Moreover, the normal merge candidate list for template matching is only available with 4 candidates, while the normal merge candidate list for ECM with common test conditions defined in JVET is 10 candidates.

[0156] Derivation of normal merge lists in ECM In ECM, the normal merge list derivation has been updated. Figure 14 (divided into Figure 14-1 and Figure 14-2) and Figure 15 show the updates based on Figures 10 and 11, respectively. However, for clarity, the modules of the history-based candidates (1101) are collapsed into (1501).

[0157] In this Fig. 15, a new type of merge candidate is added: non-adjacent candidates (1540). These candidates come from blocks that are spatially located in the current frame, but not from adjacent blocks. They are selected by distance and direction. On a history basis, the list of adjacent candidates can be added until it reaches the maximum number of candidates minus one.

[0158] Zero Candidate If the list has not yet reached the maximum number of candidates (Maxcand), zero candidates are added to the list. The zero candidates are added according to the possible reference frames or pairs of reference frames. The following pseudocode gives the derivation of such candidates: int iNumRefIdx=slice.isInterB()?min(MaxRefL0,MaxRefL1):MaxRefL0; int r=0 int refcnt=0; while (nbCand <Maxcand) { if (slice.isInterB()) { addZero(L0(Mv(0,0),Refix(r)),L1(Mv(0,0),Refix(r))); } else { addZero(Mv(0,0),Refix(r)); } nbCand++; if (refcnt==iNumRefIdx-1) { r=0; } else { ++r; ++refcnt; } }

[0159] The pseudocode can be summarized as follows: For each reference frame index (unidirectional) or pair of reference indices (bi-predictive), a zero candidate is added. When all are added, only zero candidates with reference frame index 0 are added until the number of candidates reaches a maximum value. In this way, the merge list can contain multiple zero candidates. In fact, it has been surprisingly found that such phenomena occur frequently in real video sequences, especially at the beginning of slices, frames, and sequences.

[0160] In a recent modification of the derivation of merge candidates, the number of candidates in the list can be larger than the maximum number of candidates in the final list, Maxcand, but this initial list of candidates, MaxCandInitialList, is used in the derivation, resulting in zero candidates being added up to the number of candidates, MaxCandInitialList, but not up to Maxcand.

[0161] BM merge mode BM merge is a merge mode dedicated to the adaptive decoder-side motion vector refinement method that extends the multi-pass DMVR of ECM. As described in JVET-X2025, this mode corresponds to two merge modes, which refine the MVs in only one direction: one merge mode for L0 and one merge mode for L1. In other words, BM merge is enabled only if the DMVR condition is valid. For these two merge modes, only one list of merge candidates is derived, and all candidates respect the DMVR condition.

[0162] Merge candidates for BM merge mode are derived from spatially adjacent coded blocks, TMVP, non-adjacent blocks, HMVP, and pairwise candidates, similar to the normal merge mode. The difference is that only those that satisfy the DMVR condition are added to the candidates. The merge index is coded in the same way as the normal merge mode.

[0163] AMVP merge mode AVMP merge mode, also known as bidirectional prediction, is defined in JVET-X2025 as follows: It consists of AMVP prediction in one direction and merge prediction in the other direction. This mode is valid for a coding block if the selected merge prediction and AMVP prediction satisfy the DMVR condition, there is at least one reference picture from the past and one reference picture from the future relative to the current picture, and the distances from the two reference pictures to the current picture are the same, then bilateral matching MV refinement is applied to the merge MV candidate and the AMVP MVP as starting points. Otherwise, if the template matching function is enabled, template matching MV refinement is applied to the merge prediction or AMVP prediction with the higher template matching cost.

[0164] The AMVP part of the mode is signaled as normal unidirectional AMVP, i.e. the reference index and MVD are signaled, the derived MVP index if template matching is used, or the MVP index if template matching is disabled.

[0165] For an AMVP direction LX (X is 0 or 1), the merge part of the other direction (1-LX) is implicitly derived by minimizing the bilateral matching cost between the AMVP prediction and the merge prediction, i.e., for a pair of AMVP and merge motion vectors. For all merge candidates in the merge candidate list with other direction (1-LX) motion vectors, the bilateral matching cost is calculated using the merge candidate MV and the AMVP MV. The merge candidate with the smallest cost is selected. Starting from the selected merge candidate MV and the AMVP MV, bilateral matching refinement is applied to the coding block.

[0166] The third pass of the multi-pass DMVR, which is the BDOF refinement of the 8x8 sub-PUs of the multi-pass DMVR, is valid for the AMVP-merge mode coding block.

[0167] The mode is indicated by a flag, and if valid, the AMVP direction LX is further indicated by a flag.

[0168] MVD Code Prediction The code prediction method is described in JVET-X0132. Motion vector differential code prediction can also be applied in normal inter modes if the motion vector differential contains non-zero components. In the current ECM version, it is applied to AMVP, affine MVD, and SMVD modes. The possible MVD code combinations are sorted according to the template matching cost, and the index corresponding to the true MVD code is derived and coded in the context model. On the decoding side, the MVD code is derived as follows: 1 / Parse the magnitude of the MVD components. 2 / Parse the context-coded MVD code prediction index. 3 / Create MV candidates by the combination of possible codes and MVD absolute values, and add them to the MV predictor. 4 / Derive the MVD code prediction cost for each MV based on the template matching cost and sorting. 5 / Select the true MVD code using the MVD code prediction index. 6 / Add the true MVD to the MV predictor for the final MV.

[0169] TIM D Regarding intra prediction, template-based intra mode derivation (TIMD) fusion, JVET-X2025 describes it as follows: For each intra prediction mode of MPM, the SATD between the template prediction sample and the reconstructed sample is calculated. First, the two intra prediction modes with the smallest SATD are selected as the TIMD mode. These two TIMD modes are fused with weights after applying PDPC processing, and such weighted intra prediction is used to code the current CU. Position-dependent intra prediction fusion (PDPC) is included in the derivation of TIMD modes.

[0170] Duplicate Check In Figures 14 and 15, a duplicate check for each candidate is added (1440, 1441, 1442, 1443, 1444, 1445, 1530). However, duplicates are also present for non-adjacent candidates (1540) and history-based candidates (1501). This involves comparing the motion information of the current candidate with index cnt with the motion information of other previous candidates. If this motion information is equal, it is considered a duplicate and the variable cnt is not incremented. Of course, by motion information we mean the mutual orientation, the reference frame index and the motion vector for each list (L0,L1). Note that zero candidates corresponding to different reference frames are not considered as duplicates.

[0171] MVTH In ECM, a motion vector threshold is introduced for this overlap check. This parameter modifies the equality check by considering two motion vectors equal if the absolute value of their difference is less than or equal to the motion vector threshold MvTh for each component. In normal merge mode, MvTh is set to 1, which corresponds to the traditional overlap check without a motion vector threshold.

[0172] MvTh is equal to a value that depends on the number of luma samples of the current CU, nbSamples, in template matching normal merge mode defined as follows: if (nbSamples<64) MvTh=1< <MV_FRACTIONAL=16 else if (nbSamples<256) MvTh=2< <MV_FRACTIONAL=32 else MvTh=4< <MV_FRACTIONAL=64

[0173] Here MV_FRACTIONAL corresponds to the inter resolution of the codec, i.e. currently the 16th resolution is used in the ECM, so MV_FRACTIONAL is equal to 4. << is the left shift operator. Also, nbSamples=HeightxWidth (height and width of the current block).

[0174] For example, there is another threshold called MvThBDMVRMvdThreshold, which is used for GEO merge derivation as well as for overlap checks of non-adjacent candidates, which will be described later. if (nbSamples<64) MvThBDMVRMvdThreshold=(1<<MV_FRACTIONAL)> >2=4 else if (nbSamples<256) MvThBDMVRMvdThreshold=(1<<MV_FRACTIONAL)> >1=8 else MvThBDMVRMvdThreshold=(1<<MV_FRACTIONAL)> >0=16

[0175] ARMC In ECM, Adaptive Reordering of Merge Candidates Using Template Matching (ARMC) was added to reduce the number of bits in the merge index. Candidates are reordered based on the cost of each candidate according to the template matching cost calculated as in Figure 13. In this method, only one cost is calculated per candidate. This method is applied only to the first five candidates in the regular merge candidate list after this list is derived. It should be understood that the number five was chosen to balance the complexity of the reordering process with the potential benefits, and a larger number (e.g., all candidates) could be reordered.

[0176] FIG. 18 is an example of this method applied to a regular merge candidate list containing 10 candidates, such as CTC.

[0177] This method is also applied to the sub-block merging mode except for the spatial candidates, and to the regular TM mode for all four candidates.

[0178] In one proposal, this method was also extended to sort and select candidates to be included in the final merge mode candidate list. For example, in JVET-X0087, all possible non-adjacent candidates (1540) and history-based candidates (1501) are considered together with the temporally non-adjacent candidates to create a candidate list. This candidate list is created without considering the maximum number of candidates. This candidate list is then sorted. Only the correct number of candidates from this list are added to the final merge candidate list. The correct number of candidates corresponds to the first N candidates in the list. In this example, the correct number is the maximum number of candidates minus the number of spatial and temporal candidates already in the final list. In other words, the non-adjacent and history-based candidates are processed separately from the adjacent spatial and temporal candidates. The processed list is used to supplement the adjacent spatial and temporal merge candidates already present in the merge candidate list to generate the final merge candidate list.

[0179] In JVET-X0091, ARMC is used to select one temporal candidate from three temporal candidates: bi-dir, L0, and L1. The selected candidate is added to the merge candidate list.

[0180] In JVET-X0133, merge candidates are selected from multiple temporal candidates sorted using ARMC. Similarly, all neighboring candidates are subjected to ARMC and up to nine candidates are added to the merge candidate list.

[0181] All these proposed methods use classical ARMC sorting to sort the final list of merge candidates. JVET-X0087 reuses the costs calculated during sorting of non-adjacent and history-based candidates to avoid additional computational costs. JVET-X0133 applies a systematic sorting to all candidates on the final merge candidate list.

[0182] New ARMC Since the initial implementation of ARMC, the method has been added to several other modes: ARMC applies to IBC, MMVD, affine MMVD, CIIP, CIIP with template matching, and BM merge modes, in addition to normal, template matching, and subblock merge modes. Furthermore, the ARMC principle of sorting candidate lists based on template matching cost is also applied to deriving AMVP merge candidates, as well as the intra-method TIMD for selecting the most probable prediction.

[0183] We also added principles for the sign residual prediction methods of the affine MVD method, the AMVP method, and the SMVD method.

[0184] We also conduct additional testing on the use of this reordering for GEO using GEO merge mode and template matching, and for reference frame index prediction.

[0185] In addition to this mostly systematic use of ARMC, for some candidates some additional modifications were made to the derivation process.

[0186] For example, the derivation of ECM4.0 includes cascaded ARMC processes for the derivation candidates of normal mode, TM mode, and BM merge mode, as shown in Figure 19 for normal mode and TM merge mode.

[0187] First, 10 temporal locations are checked compared to the previous merge candidate derivation and, after overlap check, are added to a temporal candidate list. This temporal list can contain up to 9 candidates. Furthermore, there is a specific threshold for this temporal list since the MV threshold is always 1 and is independent of the merge mode compared to the motion threshold used for merge candidate derivation. Based on the list of up to 9 non-overlapping initial locations, the ARMC process is applied and only the first temporal candidate is added to the conventional merge candidate list if it is non-overlapping compared to the previous candidate.

[0188] In a similar manner, non-adjacent spatial candidates are derived from among the 59 positions. The initial list of non-overlapping candidates amounts to 18 candidates. However, the motion threshold is different from that used for the temporal derivation and the remaining list, and does not depend on the normal or template merge modes. It is set equal to mv in BDMVR. The maximum 18 non-adjacent candidates are sorted and only the first 9 non-adjacent candidates are kept and added to the merge candidate list. After that, other candidates are added unless the list already contains the maximum number of candidates. Furthermore, in TM merge mode, the maximum number of candidates in the list, MaxCandInitialList, is greater than the maximum number of candidates that the final list can contain, Maxcand.

[0189] Then, the ARMC process is applied to all candidates in the intermediate list, including MaxCandInitialList, as shown in Figure 19. The maximum number of candidates, Maxcand, is set in the final list of candidates.

[0190] Similar modifications apply to the BM merge mode as well as the template matching merge mode.

[0191] ARMC template cost algorithm Figure 17 shows the template cost calculation of the ARMC method. The number of candidates considered in this process, NumMergeCandInList, is greater than or equal to the maximum number Maxcand that the list can contain (1712).

[0192] For each candidate in the list (1701), if the cost was not calculated during the first ARMC process for temporal and non-adjacent candidates (1702), the cost is set equal to 0 (1703). In the implementation, the not calculated cost mergeList[i].cost associated with candidate "i" is set equal to the maximum value MAXVAL. If the head template of the current block is available (1704), the distortion compared to the current block template is calculated (1705) and added to the current cost (1706). Next, or if not, if the left template of the current block is available (1707), the distortion compared to the current block template is calculated (1708) and added to the current cost (1709). Then, the cost of the current merge candidate, mergeList[i].cost, is set equal to the calculated cost (1710) and the list is updated (1711). In this example, we consider the current candidate i to be set to its position in terms of cost compared to the costs of the other candidates. Once all candidates have been advanced, the number of candidates in the list, NumMergeCandInList, is set equal to the maximum number of possible candidates in the list, Maxcand.

[0193] Figure 20 shows an explanation of updating the candidate list (1710) of Figure 17. First, the variable Shift is set to 0. Then, as long as Shift is less than the current candidate i of Figure 17 and the associated cost of the current candidate is less than the cost of the previous candidate number i-1-shift (2002), the variable shift is incremented (2003). When this loop ends and the variable Shift is different from 0 (2007), the candidate number i is inserted in the position of i-shift (2010).

[0194] Multiple Hypothesis Prediction (MHP) ECM also adds multiple hypothesis prediction (MHP), which allows up to four motion compensated prediction signals per block (versus two in VVC). These individual prediction signals are superimposed to form an overall prediction signal. The motion parameters for each additional prediction hypothesis can be signaled explicitly by specifying a reference index, a motion vector prediction index, and a motion vector difference, or implicitly by specifying a merge index. A separate multiple hypothesis merge flag distinguishes between these two signaling modes.

[0195] For spatial, non-adjacent, and history-based merging candidates, the multiple hypothesis parameter value "addHypNeighbours" is inherited from the candidate.

[0196] In the case of temporal, zero and paired candidates, the multiple hypothesis parameter value "addHypNeighbours" is not retained (it is explicit).

[0197] LIC In ECM, Local Illumination Compensation (LIC) has been added, which is based on a linear model for illumination changes, which is calculated thanks to the neighboring samples of the current block and the previous block.

[0198] In ECM, LIC is only valid for unidirectional prediction. LIC is signaled by a flag. In merge mode, the LIC flag is not sent, but instead the LIC flag is inherited from the merge candidate in the following way:

[0199] For spatial candidates, non-adjacent merge candidates, and history-based merge candidates, the value of the LIC flag is inherited.

[0200] For temporal candidates and zero candidates, the LIC flag is set to 0.

[0201] For the pair candidates, the value of the LIC flag is set as shown in FIG. 16. This figure is based on FIG. 12, with modules 1620 and 1621 added and modules 1609, 1612, 1611 updated. The variable average is set to false (1620), and if the average of the pair was calculated for the current list, the LIC flag of the pair LICFlag[cnt] is set to false and the variable averageUsed is set to true (1609). If the candidate only has list motion information (1612-1611), and the average was not used, the LIC flag is updated and set equal to the OR operation of the current value and the value of the LIC flag of the candidate.

[0202] Also, if the pair candidate is Bidir (eg, 3), the LIC flag becomes false.

[0203] However, in the algorithm of FIG. 16, LICflag can be different from true only if two candidates have motion information of one list, and each candidate has its own list. For example, candidate 0 has motion information of only L0, and candidate 1 has motion information of only L1. In this case, the LIC flag can be non-zero, but this does not happen because LIC is only unidirectional. Therefore, the LIC flag of the pair is always false. As a result, the pair candidate cannot use LIC when it is potentially needed. This reduces the efficiency of the candidate and avoids the propagation of LIC to the next coding block, resulting in a reduction in coding efficiency.

[0204] Furthermore, the duplicate check of the ECM software brings about some inefficiencies. As depicted in FIGS. 14 and 15, each candidate is added to the list, and the duplicate checks (1440, 1441, 1442, 1443, 1444, 1445, 1530) only affect the increment of the variable cnt (1405, 1409, 1413, 1417, 1423, 1427, 1508). Also, as explained in FIG. 16, the variable BCWidx is not initialized for the paired candidates. As a result, if the candidate last added to the list is a duplicate candidate, the value of BCWidx for the paired candidates becomes the value of the previous duplicate candidate. In VVC, when a candidate is determined to be a duplicate, the candidate is not added, so such a situation does not occur.

[0205] Embodiment In one embodiment, when the list of predictors is sorted according to the cost values calculated for each predictor, this cost value includes a threshold value and a cost calculated based on the decoded samples in order to obtain the cost to be compared. Therefore, the cost is calculated based on samples that are not the original block or the final decoded samples of the current block.

[0206] This embodiment modifies module 2002 in FIG. 20 as follows. "while ((shift < i) AND (mergeList[i].cost + TH < mergeList[i - 1 - shift].cost))", where the threshold value TH is added.

[0207] FIG. 21 is a diagram showing an alternative embodiment, and this diagram is based on FIG. 20. In this diagram, the condition of the loop (2102) is the same as that in FIG. 20. When the shift value is determined (2102)(2103), if the cost obtained by adding the threshold value TH to the mergeList[i].cost of the current candidate is greater than or equal to the cost of the candidate at the position of i - 1 - shift (2108), the candidate is not inserted at the position of i - shift (2109)(2110). That is, the current candidate maintains its position even if its cost, which is mergeList[i].cost, is excellent.

[0208] Alternatively, but mathematically equivalently, in one embodiment, when the list of predictors is ordered according to the cost value calculated for each predictor, the difference or absolute difference between the two costs is compared to a threshold, where the costs include costs based on samples that are not in the original block.

[0209] This embodiment is illustrated by Figures 23 to 28. For example, in Figure 23, the absolute value between the costs is calculated (2305) to determine whether the candidate needs to be set to another location.

[0210] An advantage of these embodiments is that the candidates can be reordered to improve the diversity of the candidate list order, thereby increasing the likelihood that a candidate earlier in the list will be selected, which leads to improved coding efficiency.

[0211] In one embodiment, the list of predictors is ordered according to a cost value calculated for each predictor, and if some candidates in the initial list of candidates are removed, the cost value includes a cost calculated based on a threshold and the decoded samples to obtain a cost to be compared, i.e., the cost is calculated based on samples that are not the original block or the final decoded samples of the current block.

[0212] This embodiment can be obtained by modifying Fig. 21 so that if a candidate does not reach the condition defined by the threshold TH (2108), it is set to the end of the list or completely removed from the list. To set a candidate to the end of the list, the shift value is set equal to -Maxcand+1+i (2109) and the candidate mergeList[i] is inserted at the i-shift position (2110).

[0213] Alternatively, mathematically equivalently, in one embodiment, at least one candidate is removed from the list according to the cost value calculated for each predictor, and the difference value or absolute difference value between the two costs is compared to a threshold, where the cost includes a cost based on samples that are not the original block.

[0214] An advantage of these embodiments is that candidates that do not offer enough diversity in terms of cost are less likely to be selected and can therefore be removed or moved to the end of the list, thus optimizing the list order resulting in improved coding efficiency.

[0215] The list of predictors can be intra or inter.

[0216] Some criteria that may be applied In one embodiment, at least one syntax element in the bitstream is associated with a list of predictors, and this ordering affects the value of the syntax element, so that if an ordering is applied and this ordering differs from a cost-insensitive ordering, the value of the syntax element associated with the predictor list may differ.

[0217] The advantage of this is that the predictors are ordered according to the correlation between them and the data representing the current block, thus improving coding efficiency.

[0218] In one embodiment, each predictor in the list represents a value of a syntax element. For example, the method is applied to the sign MVD prediction method for motion vector residual coding in affine, normal and SMVD inter modes. In these modes, the sorted list is used by one or more syntax elements.

[0219] In one embodiment, some predictors in the list represent indices. This embodiment corresponds to merge index coding for MVD coding of merge MMVD and merge affine MMVD methods, or classical merge index coding for merge modes including IBC and GEO modes. Similarly, indices of inter predictors such as inter modes (AMVP), affine, IBC, SMVD can also be considered. This embodiment is also applicable to TIMD intra modes where the list of predictors is sorted. In particular, the number of sorted candidates is more than the maximum possible index value that the syntax element can signal. This means that MaxCandInitialList is better than MaxCand.

[0220] Possible cost calculation The accuracy of the process of determining the relative "cost" of the candidates at the decoder determines how effective the reordering process is. The following aspects provide refinements to the cost determination process that result in more accurate lists, lower cost calculation complexity, or both.

[0221] It is important to note that the algorithm that performs this sorting can be modified to trade off accuracy versus complexity (speed) of the sorting process.

[0222] The following format is intended to pinpoint relative costs while minimizing the overall computational complexity.

[0223] In one example, the sorting is based on cost, which includes a measure between samples: the sample associated with each candidate is compared to another sample, and a relative amount of cost is calculated.

[0224] For example, the cost of a candidate can be calculated based on the neighboring samples of its predicted block and the neighboring samples of the current block, which are easily available at the decoder side.

[0225] The cost can be calculated between the samples of the two block predictions corresponding to the candidates in the list. For example, if the candidate is a bi-prediction candidate, the cost can be the distortion between the two block predictors. Bi-prediction candidates are described in more detail below.

[0226] Costs can also be calculated relative to other predictor candidates. For example, one other predictor candidate can be the most likely candidate or prediction. The cost of a candidate is calculated using the samples of that candidate and the samples of the most likely candidate.

[0227] The cost can be calculated on a subset of the neighborhood samples or on a subset of the prediction samples, e.g., if there are multiple adjacent samples that can be used to determine the cost, these samples are sampled to reduce computational complexity.

[0228] The cost can be calculated based on samples corresponding to images at another resolution: high similarity with the higher resolution image is a good sign of low cost (i.e. good prediction).

[0229] The sample values ​​used in the cost calculation can be pre-processed. Given that only the relative cost values ​​are needed (i.e., only the order matters), pre-processing the values ​​means the calculation will be simpler and is unlikely to have a significant impact on the effectiveness of the sorting process. Pre-processing will improve the sorting process by calculating the costs.

[0230] In the above example, the cost could be a measure of distortion such as the sum of absolute differences (SAD), the sum of absolute transformed differences (SATD), the sum of squared errors (SSE), or the structural similarity index measure (SSIM).

[0231] Alternatively, the cost could be a measure of distortion, with a weight applied to this distortion. It could also be a rate or an estimated rate. Or it could be a threshold.

[0232] It is also possible to have weighted costs, with different weights depending on the type of prediction or candidate, or on the initial position in the list of candidates.

[0233] In one embodiment, the cost is proportional to the number of samples.

[0234] For example, the number of samples is the number of samples in the current block, and the distortion (cost based on sample values) is divided by this number.

[0235] In an alternative embodiment, the number of samples is the number of samples used to calculate the distortion, or the number of distortion comparisons, and the distortion (cost based on sample values) is divided by this number.

[0236] In one embodiment, the cost based on sample values ​​is divided by the number of distortion comparisons and multiplied by the number of samples in the block. For example, if template matching is performed on the top and left rows of a block (row and row have only one sample), the cost associated with candidate I, mergeList[i].cost, is set equal to mergeList[i].cost = ((height x width) x mergeList[i].cost) / (height + width).

[0237] The advantage of this embodiment is that the cost is proportional to the block size, so that the parameters of the criterion are adaptable to all block sizes. More precisely, it provides a more efficient use of the lambda value defined in the following embodiments.

[0238] Further criteria In one embodiment, the threshold is the Lagrangian parameter lambda (λ) as used in the rate-distortion criterion J=D+λR, which is used at the encoder side to determine the best coding possibility for a block, or more generally to select the best coding possibility from a set of coding possibilities.

[0239] This embodiment significantly improves the coding efficiency. Let two candidates or predictors cand1 and cand2 have distortions D1 and D2 respectively. And although D1 > D2, if their RD costs J1 and J2 are such that J1 < J2, these candidates can be considered to be too close or redundant in terms of RD because the rate for signal cand2 is higher than its distortion merit. This can be written as follows: J1 < J2 ⇔ D1 + λR1 < D2 + λR2 ⇔ D1 - D2 < λ(R2 - R1) ⇔ (D1 - D2) / (R2 - R1) < λ

[0240] Since many indexes in the video codec use unary maximum codes, it can be considered that R2 - R1 = 1. Therefore, if |D1 - D2| < λ or |D1 < D2 + λ|, cand1 and cand2 can be considered redundant.

[0241] Therefore, in this embodiment, when the absolute value of the distortion between the current candidate and another candidate is less than λ, the candidate is regarded as redundant.

[0242] This embodiment is shown in FIGS. 21 to 28. For example, in FIG. 23, the minimum absolute distortion difference (2305) is compared with the lambda value (2308).

[0243] This formula is particularly efficient and is also effective when the rate difference between a candidate and its previous candidate is not 1.

[0244] In the previous embodiments, in order to determine whether two candidates / predictors are redundant, the rate difference between the two candidates / predictors is not considered. Therefore, in order to improve the coding efficiency, this rate difference is calculated to determine whether a certain candidate is redundant compared to another candidate. This embodiment is useful when the syntax elements related to the list of candidates / predictors are coded with something other than unary codes. For example, the merge motion differences of normal MMVD or affine MMVD are coded with Golomb codes. In that case, the number of bins or bit difference between one index value and the next index value can be 0, 1, 2, or more.

[0245] In an additional embodiment, the actual rate is considered. That is, the actual rate when using CABAC is considered. In that case, this embodiment can also be applied when the index is coded with a unary maximum code.

[0246] This embodiment can be implemented by changing the λ value by λ(R2 - R1) from FIGS. 23 to 28, or by changing the cost of the candidate by multiplying the rate by λ. Similarly, for FIGS. 20 to 22, the criterion can be D2 + λxR2 < D1 + λxR1 or D2 + λ(R2 - R1) < D1, where D2 is mergeList[i].cost. (In FIG. 20, the criterion in the loop (2002) is to find a candidate better than the current candidate. However, the criterion for determining whether a candidate is redundant is D1 < D2 + λ(R2 - R1). For example, this embodiment changes the module 2002 in FIG. 20 "while ((shift < i) AND (mergeList[i].cost + TH*mergeList[i].rate < mergeList[i - 1 - shift].cost + TH*mergeList[i - 1 - shift].rate))" including changing it as follows.

[0247] In one embodiment, as described in Figures 23 to 28, when candidates are reordered after a first ordering, the criteria (2305) used to determine that a candidate is redundant includes multiplication of one distortion threshold, such as D2-Th.D1<0 or alternatively Th.D2-D1<0 or Th2.D2-Th1.D1<0.

[0248] The advantage of this embodiment is the improvement of coding efficiency. In particular, this criterion can adapt the redundancy of two candidates based on the proportionality of the distortion. Thanks to this criterion, two candidates with low distortion are not considered as redundant. On the contrary, two candidates with high distortion are considered as redundant more frequently than previous criteria.

[0249] In one embodiment, the criteria includes at least one threshold that balances the two distortions, and the thresholds are added, for example the criteria defined above becomes: |D2-Th.D1|-λ<0, or |Th2.D2-Th1.D1|-λ<0.

[0250] An advantage of this embodiment is that it improves the coding efficiency in some cases, where the penalty for distortion for some candidates may give an improvement in coding efficiency, for example depending on the distance of the current block and the type of the candidate.

[0251] Lambda configuration In the encoder implementation, the lambda value depends on several parameters that affect the tradeoff between distortion and rate, and on how the distortion is calculated and the rate is estimated.

[0252] In one embodiment, λ depends on one or more parameters and the value of λ is calculated at the decoder side, the advantage of which is that no additional rate is required to transmit the λ value.

[0253] In one embodiment, the lambda value (or threshold) is derived or obtained at the encoder side and transmitted at the decoder side (in the bitstream).

[0254] The advantage of this embodiment is that the encoder implementation can set the corresponding lambda value depending on the implementation or sequence type.

[0255] In one embodiment, the lambda used or transmitted at the decoder side depends on the QP value used for the current slice or picture. In a further embodiment, if the QP is set at the block level, the λ value is adapted at the block level based on the block QP value. The QP value is the most important parameter that affects the quality of the reconstructed block with the block residual and, as a result, the rate-distortion compromise.

[0256] In one embodiment, the lambda used or transmitted at the decoder side depends on the picture order count distance or the temporal distance between the current frame and its reference frame. In some modes, such as the normal merge mode, the POC distances of one candidate may differ from each other. Therefore, it is preferable to consider the minimum POC distance or the minimum temporal distance between the current frame and its reference frame when multiple POC distances are possible. The temporal distance between the current frame and its reference frame or the nearest reference frame affects the quality of the block predictor. As a result, it affects the reconstructed block quality with or without block residuals.

[0257] In one embodiment, the lambda used or transmitted at the decoder side depends on a temporal ID. In many video coding standards, frames have a temporal ID that indicates the depth for temporal scalability. Therefore, since this temporal ID may affect the quality compromise required for the frame, the lambda may also depend on this parameter.

[0258] In one embodiment, the lambda used at the decoder side or transmitted depends on the temporal direction. Temporal direction means that the reference frames come from the past, the future, or both directions. It is known that if block predictors come from both directions, the quality between predictors is better on average, thus affecting the rate-distortion compromise. Since there are also modes where the temporal directions of one candidate can be different from each other, such as the normal merge mode, it is preferable that the lambda depends on the number of temporal directions.

[0259] In one embodiment, the lambda used at the decoder side or transmitted lambda depends on uni-prediction or bi-prediction. It is known that if a block predictor comes from two block predictors, the quality of this predictor will be better on average, even if the two reference blocks are both from the same time direction. This will therefore affect the rate-distortion compromise. In some modes, such as the normal merge mode, one candidate may be bi-predicted and the other uni-predicted, so it is desirable for the lambda to depend on whether bi-prediction is used or not.

[0260] In one embodiment, the lambda used or transmitted at the decoder side depends on the QP value of the reference frame of the current slice. In one embodiment, to avoid complex lambda settings for some modes, it depends on the QP value of the closest reference frame in each or both directions, or on the minimum QP value for each reference frame direction. In combination with the previous embodiment, it can depend on the difference in QP values ​​of the QPs selected for the reference frames. The QP of the reference affects the quality of the reference and, consequently, the QP of the current frame, thus affecting the rate-distortion compromise.

[0261] In one embodiment, the lambda used or transmitted at the decoder side depends on the distortion measure used. It is known that λ depends on the distortion (SAD, SSE, etc.) used at the encoder side. Therefore, at the decoder side, λ also depends on the type of distortion.

[0262] In one embodiment, the lambda used or transmitted at the decoder side depends on the type of samples used to calculate the distortion. Indeed, as mentioned before, criteria can be applied for different reordering or removal algorithms based on different types of samples. This can also be taken into account for setting the lambda value. For example, template matching uses fewer samples than the current block, so the distortion is calculated with fewer samples, and this can be taken into account.

[0263] As mentioned above, an alternative to this embodiment is to adjust the distortion depending on the number of samples.

[0264] In one embodiment, the lambda used or transmitted at the decoder side depends on the type of prediction: At the encoder side, the λ value can depend on the type of prediction, intra / inter, and the λ used at the decoder side also depends on this characteristic.

[0265] In one embodiment, the lambda used at the decoder side or transmitted depends on the type of GOP configuration. For example, the lambda may be based on another QP than the QP of the current frame or may be shifted depending on the type of configuration, such as random access configuration RA, low latency configuration, or intra-only configuration.

[0266] In an embodiment, the lambda used or transmitted at the decoder side is based on the lambda formula used at the encoder to select among the best candidates / predictors etc. of a list and is used at the decoder side. The formula to be used can be signaled to the decoder in the bitstream, for example by an index indicating which formula to use from a predefined list of formulas available to calculate λ.

[0267] It should be noted that in other embodiments, the threshold or lambda value may depend on any combination of the above factors and may be optimally implemented to provide the best balance between coding efficiency and complexity.

[0268] One possible implementation for setting the correct lambda is to derive lambda using a traditional formula that generally depends on the QP value, and shift this QP value according to the parameters defined earlier. In the specific example that gives a significant gain, the lambda formula used to select among the best merging candidates is used. If the current frame is in a random access configuration, the lambda formula is set to QP-4, where QP is the QP of the current slice / frame. Also, if the frame is in a low-latency configuration, the QP value used to obtain the lambda value at the decoder is the current QP.

[0269] Of course, lambda can be stored in a predefined table instead of a formula, in which case each value in the table is associated with a QP value.

[0270] Threshold (TH) is not lambda As explained in the previous embodiment, the λ value is theoretically a good value to use as a criterion, but it depends on several parameters and related to the implementation of the encoder, so in other embodiments, other values ​​can be used as thresholds.

[0271] In one embodiment, the threshold is a minimum value as depicted in FIG.

[0272] In one embodiment, the threshold is calculated based on the cost values ​​of all the candidates (NumMergeCandInList) or according to the first Maxcand candidates.

[0273] In one embodiment, the threshold is the average of the difference between the distortions of each candidate compared to its predecessor. With this criterion, all algorithms related to Figures 21 to 28 are valid, and the λ or TH values ​​are replaced by this average value.

[0274] An advantage of these embodiments is that the threshold does not depend on multiple parameters and does not need to be transmitted.

[0275] Possible algorithms The algorithms for sorting the list based on defined criteria and for removing candidates based on criteria can be specialized and are specific to each mode since the costs are calculated at the decoder side from samples that are not of the current block.

[0276] In one embodiment, the criteria of the present invention are applied during the ordering process, which consists of modifying the criteria of the loops, as described in Figure 20. As further examples, Figures 21 and 22 are also relevant to this embodiment.

[0277] The advantage of this embodiment is that it has a small impact on the complexity since only one loop is required to apply the proposed criterion.

[0278] Figure 23 shows one algorithm that is applied after an initial ordering based on cost, i.e., at the beginning of the algorithm, all candidates in the list are ordered according to their cost / distortion, e.g., after the ARMC method. The list of candidates contains NumMergeCandInList candidates, which can be greater than or equal to the maximum number of candidates used in the final list Maxcand.

[0279] In this figure, the number of reorderings, MaxNumReorder, is limited (2302) to the number of candidates in the list NumMergeCandInList minus one (2301). The algorithm is applied (2302) while the number of possible reorderings is greater than one and the minimum difference between a candidate and its next candidate is greater than or equal to a lambda (threshold) value, for example, defined in the embodiment. The minimum cost difference between candidate i and its successor candidate i+1 (2304) (2305) (2306) is determined and the variable CandToMove is set equal to this successor candidate i+1. This candidate is considered redundant compared to its predecessor. If this minimum cost is less than lambda (threshold) (2308), this candidate is moved from its position to a position of MaxNumReorder-1 (2310) (2309). That is, this candidate is set to the furthest position, but before the previous candidate considered redundant by the algorithm. MaxNumReorder is then subtracted.

[0280] The algorithm can be summarized as follows: -Determine the minimum cost difference between two consecutive candidates. - If this minimum difference is less than lambda (or some other threshold), then place this candidate at the furthest position in the list (specifically, ahead of any previous candidates deemed redundant by the algorithm).

[0281] Figure 29 shows an example of the sorting obtained by this algorithm. In this example, λ (i.e. the threshold) is set equal to 16. At the beginning of the first step "Step 1", the list of 6 merge candidates is sorted according to their cost (mergeList[i].cost) from the lowest 2264 to the highest 2750. Then, the absolute value of the cost difference between each candidate and its predecessor is calculated (2305) and the minimum is identified (2306). In this example of Figure 29, in step 1, the minimum absolute cost difference is 5 (min=5) and the associated candidate CandToMove is candidate index i=4. This cost of 5 is smaller than the lambda value (2308) which is equal to 16. As a result, candidate index 4 is moved to the end of the list in "Step 2" and MaxNumReorder is decremented. Therefore, this candidate is not considered in the second step of the loop. In step 2, the list considered contains 5 candidates and the minimum absolute cost difference is 8, which corresponds to the merge candidate index i=1. This candidate is inserted fourth in the merge candidate list for step 3 and is not considered in step 3. During "step 3", the minimum absolute cost difference is 18, which is greater than the lambda value of 16 (2308), so the algorithm stops as defined at (2302). As the end of step 3, the merge list candidates are not ordered according to cost value (minimum to maximum), but according to an algorithm that provides improved diversity by moving candidates that may be redundant (due to their cost similarity to other candidates) to the end of the list. More specifically, redundant candidates are moved to a position following candidates that are not currently considered redundant.

[0282] One advantage of this embodiment compared to the embodiment of FIG. 21 is that the redundancy is ordered based on the most redundant candidate. Another advantage is that the reordering first removes redundant candidates by placing them last according to their redundancy, and then the less redundant candidates are included in the final set. In addition, the number of removals / reorders is also limited. Therefore, an improvement in coding efficiency is obtained compared to the embodiment like FIG. 21. However, in this case, a second reordering process is required, which increases the complexity. In another embodiment, MaxNumReorder is not set equal to the number of candidates, but to Maxcand or a smaller value instead. Indeed, the proposed reordering sets candidates inefficiently when the list is significantly large (NumMergeCandInList) compared to the number of candidates with indexes (Maxcand). So it is desirable to limit the possible reordering to a small number of values. For example, in the case of a merge MMVD with ARMC, the list of MVD positions contains 96 candidates and the maximum index for signaling the MVD is only 12. So it is desirable to set MaxNumReorder to 12 in this case.

[0283] FIG. 25 shows, in an embodiment, an alternative algorithm to that of FIG. 23. In this algorithm, in contrast to the previous embodiment, there is no comparison of λ / threshold (2502) with the minimum cost difference. The most redundant candidate CandToMove is always inserted at the end of the list. To obtain coding efficiency, the maximum number of possible reorderings MaxNumReorder is limited (2502) by the variable LimitReorder, in comparison with the embodiment of FIG. 23. Therefore, only LimitReorder reorderings are applicable in that case. It is noted that this embodiment is similar to FIG. 23 with λ / threshold set equal to a maximum value that is always larger than the possible absolute difference between the two distortions.

[0284] The advantage of this embodiment is that it is adaptable to all encoder selection algorithms, since there is no need to derive or transmit the lambda value at the decoder side, but it reduces the coding efficiency, although in some modes this restriction is effective.

[0285] In an alternative embodiment to that of Figure 23, the algorithm only removes candidates. This can be achieved by modifying the criterion of the loop (2302). In that case, the loop stops when MaxNumReorder is greater than or equal to Maxcand instead of 1. Furthermore, for better coding efficiency, the minimum distortion difference (2304), (2305), (2306), (2307) is determined only in the first part of the list, e.g., the candidates that are (currently) in the final list (Maxcand). This can be achieved by modifying the criterion of 2304 and restricting i to Maxcand.

[0286] Simply removing candidates is less efficient on average than sorting and removing as described in FIG. 23, except when it affects the maximum number of candidates in the final list, Maxcand.

[0287] In one embodiment, candidates that are deemed redundant are removed from the list, resulting in a reduction in the maximum value of the final list.

[0288] Figure 24 illustrates this embodiment, where, like Figure 23, each time a candidate is set to the end of the list (2410), the maximum number of candidates in the final list, Maxcand, is decreased (2412). This is particularly efficient when the associated syntax element is not coded with a unary max code or when the Maxcand value is small. However, this creates a parsing problem, since a full reconstruction is required for the decoder to parse the associated syntax element.

[0289] Figure 26 shows an alternative algorithm to that of Figure 23. In comparison to Figure 23, the position at which the redundant candidate is inserted is not always MaxNumReorder-1 (2309), but the first position (2613) (2614) (2615) at which a candidate becomes non-redundant. This is achieved by identifying the first candidate (2614) among the candidates after the current redundant candidate (2613) (CandToMove) whose distortion difference between the redundant candidate CandToMove and this candidate is greater than Lambda (a threshold). The insertion position isertPos of the redundant candidate is set equal to the position of this second candidate i (2615).

[0290] The algorithm can be summarized as follows: -Determine the minimum cost difference between two consecutive candidates. -If this minimum difference is less than lambda, then the consecutive candidates are considered redundant. -Compare the distortion of the redundant candidate with the distortion of the successor candidate in the list. The first candidate location having a distortion difference greater than −λ is identified. - Setting a redundant candidate at this identified location.

[0291] In an additional embodiment, if the value of the maximum number of possible reorderings, MaxNumReorder, is greater than the maximum number of candidates in the final list, Maxcand (2612), then isertPos remains equal to MaxNumReorder-1.

[0292] Compared to the embodiment of Fig. 23, this embodiment improves the coding efficiency. In fact, this restriction on the repositioning of candidates considered as redundant is often correct, since the distortion cost is not the actual distortion compared to the original block, and the initial reordering is often correct.

[0293] This embodiment is particularly efficient when the normal merge mode, the template matching merge mode, the BM merge mode, and the IBC merge mode are enabled.

[0294] It should be noted that the algorithm in this figure is adaptable. For example, one additional step could be to check if the number of merge candidates in the list NumMergeCandInList is greater than 2 and apply the algorithm, in order to avoid applying the algorithm when it is certain that the order of the candidates will not be changed. In fact, the algorithm can only change the order of the candidates if the list contains more than one candidate. Furthermore, the number of candidates considered in the algorithm could be the number of candidates reordered by ARMC, since the other candidates do not have any relevant distortion. In another possible implementation, step 2602 could be changed to a "for" loop (instead of a "while" loop) from the maximum number of reordering candidates NumMergeCandInList to 2, and step 2611 could be removed. The algorithm stops when the minimum value min is below. All the described algorithms could be based on a table that associates the initial candidate position with its reordering position and another table that contains the associated cost of each candidate. In that case, to avoid complex reordering, the reordering to the final list of reordered candidates could only be done at the end of the ARMC and diversity reordering algorithms.

[0295] Figure 27 shows another possible implementation of the reordering algorithm after the initial reordering based on distortion in an embodiment. In contrast to the algorithm in Figure 23, it is not based on minimum distortion difference, but is specialized to identify non-redundant candidates.

[0296] For each candidate cand (2702) in the list (2701), the algorithm looks for the first following candidate cand2 (2704) that is not redundant in the sense of the criterion defined in the present invention (2705). If there is no non-redundant candidate, the next candidate is considered and the criterion is applied to decide if the candidate is redundant or non-redundant. If not, the first non-redundant candidate (2706) is inserted after the candidate (2707) (isertPos=cand+1). Of course, this insertion implies that this non-redundant candidate is deleted from its original position.

[0297] The algorithm can be summarized as follows: -For a given candidate, determine the first candidate in the list, the next candidate, and the non-redundant candidate. - Insert this non-redundant candidate after the candidate.

[0298] This embodiment is particularly efficient for lists where the candidates are almost all redundant. So, for example, this embodiment is efficient for MMVD merge, affine or regular, where the MVD residual positions are close. Similarly, this algorithm is particularly efficient for MVD symbol prediction for the same reasons.

[0299] Figure 28 shows an additional step from Figure 20. In this figure, when a candidate is considered to be non-redundant (2805), it is also tested whether this candidate is non-redundant compared to all previous candidates (2808) of the current candidate (2809).

[0300] The algorithm can be summarized as follows: -For a given candidate, determine the first candidate in the list, the next candidate, and the non-redundant candidate. - Insert this non-redundant candidate after the candidate unless this non-redundant candidate is redundant compared to all candidates preceding it.

[0301] As with the previous embodiment, this embodiment is particularly efficient for lists where the candidates are almost all redundant. So, for example, this embodiment is efficient for MMVD merge, affine or regular, where the MVD residual position is close. Similarly, this algorithm is particularly efficient for MVD code prediction for the same reason. Thanks to this additional restriction of reordering, a better coding efficiency is obtained compared to the previous one. Even though the algorithm of Fig. 26 gives better results than this embodiment for normal, TM, BM merge modes, this algorithm also gives significant gains for these modes.

[0302] In an alternative embodiment, SizeCandList is not set equal to the number of candidates NumMergeCandInList (2701) (2801), but is set to Maxcand or alternatively a smaller value. Indeed, if the list is significantly large compared to the number of candidates with indices, the proposed reordering will result in inefficient candidate ordering. It is therefore desirable to limit the possible reordering to a small number of values. For example, in a merge MMVD with ARMC, the list of MVD positions for one candidate is 96, and there are only 12 indices that signal the MVD. In this case, it is desirable to set MaxNumReorder to 12. Note that this restriction of Maxcand (2701) (2801) is arbitrary if the difference between NumMergeCandInList and Maxcand is not large.

[0303] Other embodiments The following features can be usefully combined with the above-described embodiments to form other embodiments of the present invention.

[0304] In an embodiment, if the list contains only two candidates, the proposed method is not applied, which is equivalent to setting Th to 0. This results in a gain in coding efficiency, for example when using the embodiment of Fig. 20. Indeed, if there are only two candidates in the candidate list, it is preferable to set the candidate with the best distortion as the first candidate.

[0305] In other embodiments, for example as in FIG. 23, if there are only two candidates, a permutation is applied after the first permutation, but the criteria remain unchanged.

[0306] In one embodiment, if only one template is available for the current block, the cost of all candidates in the list is set proportional to the cost of the available template.

[0307] For example, if the template on the left is unavailable, the cost of the current block is Cost = CostUP + (CostUP x height) / width.

[0308] where height and width correspond to the height and width of the block. Here, the top template is considered to be the line at the top of the block, and the left template is the line at the left of the block. However, longer or shorter templates can be considered by adapting the formula. Multiplications and divisions can be replaced by shift operations to reduce complexity.

[0309] The advantage of this embodiment is an improvement in coding efficiency: in fact, the lambda value is fixed for some rate-distortion compromise, and then λ is adapted to this rate-distortion compromise: for a distortion corresponding to an equivalent number of samples for all blocks of the same size, a coding improvement is observed.

[0310] Alternatively, in this case, instead of modifying each distortion cost, we can modify lambda. In the same example, if the left template is not available, then λ or the threshold is λ = λ / (1 + (height) / width). This calculation can be modified to only use shifts instead of divisions to reduce complexity.

[0311] The advantage of this embodiment compared to the previous embodiment is that the complexity is reduced since the new lambda is calculated once instead of modifying each distortion cost.

[0312] In current ECM, the motion vector thresholds in TM and BM merging modes are usually different for different types of candidates, which creates a kind of diversity in the candidate subsets.

[0313] In one embodiment where the proposed invention is applied, the MV threshold for derivation is the same for all merge candidate types, e.g., the motion vector threshold for temporal and non-adjacent candidates is the same as the motion vector threshold for all other types of candidates.

[0314] This improves coding efficiency since diversity is created by our proposed method, which is more efficient than using different motion vector thresholds.

[0315] Similarly, as shown in FIG. 19, merge-type based ARMC provides some benefit because of the diversity control based on merge type.

[0316] In one embodiment, when the method of the present invention is enabled, the ARMC process is applied to all candidates derived and ordered in the ARMC. For example, in comparison with FIG. 19, up to 9 temporal candidates and up to 22 (18 in some embodiments) non-adjacent candidates are added to the list of other merge candidates (B1, A1, ...). In order to use at least as many ARMC comparisons as there are ARMC processes per merge type, the initial list of candidates is not limited to the defined MaxCandForFirstList, but to MaxCandForFirstList + the number of temporal candidates and the number of non-adjacent candidates. However, this number can be limited to the maximum number that the buffer of merge candidates can contain. For example, it can be fixed to 30.

[0317] One advantage of this embodiment is that it improves coding efficiency since the diversity is preferably controlled by the method proposed in the present invention. Moreover, this embodiment reduces complexity since ARMC is not applied to duplicate candidates in the temporary list or the non-adjacent list.

[0318] In one embodiment, the number of temporal candidates and / or the number of non-adjacent candidates is reduced, the advantage of which is that the cost calculation for all these candidates becomes less complex.

[0319] We found an improvement in coding efficiency when adding pairwise candidates after the ARMC process, but the gain was significantly reduced when competing with the merge-type-based ARMC process, as the number of possible candidates increases significantly.

[0320] In one embodiment, if the proposed method is effective, pairwise candidates are added during or after the ordering process. Optionally, the pairwise candidates in that case are built based on the first candidate given by the ARMC ordering and one or more other candidates in the list. Optionally, these added candidates are sorted according to the criteria proposed in the present invention by comparing their cost with the costs of the other candidates in the list, or a second sorting process is applied after this first sorting process.

[0321] The advantage is improved coding efficiency. In fact, paired candidates may be candidates that are too close to existing candidates in the list. As a result, in a cost-based ordering using ARMC, these too redundant candidates are placed at the top of the list without creating enough diversity. When an embodiment of the present invention is applied to the list, the list becomes more diverse and paired candidates are ordered differently or removed if they are too redundant. Thus, improved coding efficiency can be obtained.

[0322] In an additional embodiment, if pairwise are added after other conventional candidate sorting processes, they are in separate lists. In this case, a table stores the order of each candidate, and another table stores the associated cost. These tables are set up in the ARMC process and the diversity sorting process. Then, the final list is set up based on the candidates in both lists according to the order determined. The advantage is that the maximum size of the buffer can be reduced, thus reducing the complexity.

[0323] In one embodiment, the number of pairwise candidates is limited to a fixed value, for example, 4.

[0324] An advantage of this embodiment is that it limits the maximum number of candidates to which the ARMC process can be applied.

[0325] In one embodiment, the maximum number of pairwise candidates that can be added is defined as the number of target candidates in the final list, or to a fixed value. This maximum number of pairwise candidates is considered to limit the maximum number of candidates in the initial list (before ARMC processing). So, in this case, NumMergeCandInList is set to NumMergeCandInList minus the maximum number of pair candidates that can be added. An offset (which can also be positive or negative) is also set. It should be noted that since the pairwise is constructed after the initial sorting, the actual number of pairwise candidates that can be added cannot be known.

[0326] The advantage of this embodiment is reduced and limited complexity, since the maximum number of candidates with ARMC cost calculations is limited.

[0327] In one embodiment, some modes do not add pairwise candidates, e.g., pairwise candidates are not added to the TM merge list.

[0328] Another benefit is reduced complexity.

[0329] Predictive Coding Applications The above embodiment has been described as an implementation for predicting an image portion (prediction unit) in the context of video or image encoding and decoding. However, it is assumed that the above method is applicable to any compression technique in which a predictor or predictor candidate may be used. For example, in point cloud encoding, the predictor is a point in a cloud or an element of a mesh. Alternatively, the predictor may be for an audio data portion in which the audio data to be encoded or decoded is predicted from another audio data.

[0330] All described embodiments may be combined unless expressly stated otherwise, and in fact many combinations may be synergistic, resulting in efficiency gains greater than the sum of their parts.

[0331] Reordering the first candidates In a particular predictor list, ARMC is not applied to one or more first candidates. As mentioned above, for example, ARMC is not applied to the subblock temporal predictor (STMVP) of the subblock merge list. The diversity criterion and related algorithms can be adapted to these particular cases.

[0332] Furthermore, ARMC is applied to several modes, which are competed together. ARMC sets the most probable predictor to the top of each list. This most probable predictor can be considered as the so-called "zeroth" predictor. This is because it is often desirable to exclude a predictor from reordering, and the zeroth predictor is at the top of the list both initially (as Cand0) and after any reordering process (as NewCand0, see Figure 30). However, since similar candidates are set to the top of each list, there is not enough diversity between prediction modes, which is undesirable from the viewpoint of coding efficiency.

[0333] In one embodiment, the sorting based on the diversity criterion is applied to the candidates with cost. In the example of the STMVP candidate where sorting is not performed and the cost is not calculated, it remains at the top of the list. Similarly, the candidate NewCand1 (illustrated in FIG. 31) obtained after ARMC remains at the second position of the list based on the diversity criterion.

[0334] One alternative embodiment is to calculate the cost of STMVP candidates and apply the diversity algorithm to all sub-block merging candidates. In that case, the calculation of STMVP candidates can be conditioned on the activation of the diversity algorithm or the threshold / lambda value of the diversity criterion. If this is equal to zero, the cost of STMVP is not calculated.

[0335] In one embodiment, the cost of the top candidate in the list that is not evaluated is set equal to 0, and the diversity algorithm uses this candidate as the 0th candidate in the list. In the example of the STMVP candidate, the distortion cost of STMVP is not calculated, but the associated value is set equal to 0, and the diversity algorithm is applied to all subblock candidates in the list, including this candidate. Of course, the diversity algorithm does not change the position of the STMVP candidate, but the position of the "first" candidate (taking the position following the 0th candidate) NewCand1 in Figure 31 may change. In this embodiment, if the first candidate is considered to be sorted, it means that its cost is close to 0.

[0336] The advantage of this embodiment is that it improves the coding efficiency compared to the previous embodiment. In this embodiment, the first candidate is considered too redundant compared to its own cost and diversity criterion thresholds. Compared to the previous embodiment, this candidate can be moved to another position.

[0337] One alternative to this embodiment is to calculate the cost of the STMVP candidate and set it to the top of the list, whatever the calculated cost is, so that the ARMC algorithm does not consider this cost for sorting purposes. The diversity criterion can then be applied to all candidates in the list.

[0338] Compared with the previous embodiment, the diversity criterion can correctly evaluate the diversity between the 0th candidate (i.e., the STMVP candidate) and its successor NewCand1. Note that in the diversity algorithm, the candidate in the second position of the list is not necessarily the same through each loop.

[0339] Thus, the advantage of this embodiment is that the criterion is calculated according to the correct data, thus improving the coding efficiency.

[0340] Considering the 0th candidate for sorting In one embodiment, the diversity algorithm can sort all candidates in the list, or all candidates with a cost. For the top candidate in the list, since there is no previous candidate, its cost is compared to a threshold to be considered by the diversity algorithm. If the threshold is a λ value and D0 is the distortion of NewCand0, then the diversity criterion is: D0<λ.

[0341] Figure 31 shows this embodiment based on Figure 26. In this figure, only step 3103 has been modified compared to step 2603: the minimum cost is initialized with the cost of the first candidate in the list and the variable CandToMove, which represents the index of the moveable candidate, is initialized with the value 0.

[0342] An advantage of this embodiment is that it improves coding efficiency, since candidates at the top of the list are also considered for reordering.

[0343] In a further embodiment, the possibility of reordering the 0th candidate is limited to some modes. For example, the possibility of reordering the 0th candidate is limited to BM merge mode or TM merge mode. In fact, TM and BM are more efficient when they are not enabled to each other. This means that these two modes generate similar block prediction values, as well as normal merge mode. In order to obtain a kind of diversity between the candidate sets of these modes, there is a difference in terms of the derivation of the candidates. However, the proposed method gives better results because it generates diversity between the sets. Compared to FIG. 31, step 3103 is applied only for the related modes, otherwise the minimum value is set equal to MAX.

[0344] The advantage of this embodiment is that the coding efficiency is improved due to the diversity among the candidate set.

[0345] In one embodiment, the threshold for the 0th candidate in the list is different from the threshold used in the diversity algorithm for the other candidates.

[0346] In that case, the criterion is D0<λ1 for the 0th candidate in the list, and |D-Di(i+1)|<λ2 for the other candidates. In the example of Fig. 31, step 3108 is adapted to modify the λ value if CandToMove is equal to 0 or in other cases, similar to step 3114;

[0347] The advantage is that the likelihood of permuting the 0th candidate is different from the permutations of the other predictors, allowing better parameterization.

[0348] In one embodiment, the threshold for the 0th candidate is inferior to the threshold for the diversity criterion.

[0349] In fact, moving or deleting the 0th candidate is more restricted than the other candidates.

[0350] All of the embodiments described above in relation to threshold or lambda values ​​(eg, how to set or calculate the threshold or lambda values) are applicable to the above embodiments.

[0351] In one embodiment, candidates can be moved or removed based on their cost value according to the criterion Di<λ.

[0352] In one embodiment, all candidates with distortion less than λ are removed from the predictor set. For example, if the maximum number of predictors in the current list (MaxNumReorder) is greater than the maximum number of predictors the final list can contain (Maxcand), then all predictors with cost less than λ are removed from the list until MaxNumReorder is set equal to Maxcand.

[0353] In one embodiment, all candidates whose distortion is less than λ are moved to another position where the cost distortion of the previous candidate is greater than the λ value.

[0354] Figure 32 illustrates this embodiment. For each candidate (3201), it is tested whether its cost is less than lambda (3202). If so, this candidate (3203) is moved to the first position whose candidate cost is greater than or equal to lambda 3205, and the current candidate (3203) is inserted in the immediately following position (3207). Of course, this algorithm only applies if there is at least one candidate whose cost value is greater than or equal to λ.

[0355] In an alternative embodiment, candidates with cost less than λ are moved to the end of the list.

[0356] The advantage is that all candidates are considered according to the criterion, and those candidates with a cost worse than λ are penalized more compared to the previous embodiment, which is more efficient when the number of predictors in the predictor list is large.

[0357] In a further embodiment, the proposed move or delete applies only to some modes. For example, the possibility to reorder the initial candidates is limited to the BM merge mode or the TM merge mode. In fact, as explained above, TM and BM are more efficient when the other is not valid.

[0358] The advantage of this embodiment is that the coding efficiency is improved due to the diversity among the candidate set.

[0359] Implementation of the Invention Fig. 33 shows a system 191, 195 including at least one of the encoder 150 or the decoder 100 and a communication network 199 according to an embodiment of the present invention. According to an embodiment, the system 195 is for processing and providing content (e.g. video and audio content for displaying / outputting or streaming the video and audio content) to a user having access to the decoder 100, for example via a user interface of a user terminal constituting the decoder 100 or capable of communicating with the decoder 100. Such a user terminal may be a computer, a mobile phone, a tablet or any other type of device capable of providing / displaying (provided / streamed) content to a user. The system 195 obtains / receives the bitstream 101 (in the form of a continuous stream or signal - for example while a previous video / audio is being displayed / output) via the communication network 199. According to an embodiment, the system 191 is for processing content and storing the processed content, for example the processed video and audio content for later displaying / outputting / streaming. The system 191 obtains / receives content including an original image sequence 151, which is received and processed (including filtered by a deblocking filter according to the invention) by an encoder 150, which generates a bitstream 101 to be communicated to the decoder 100 via a communication network 191. The bitstream 101 is then communicated to the decoder 100 in several ways, for example it may be pre-generated by the encoder 150 and stored as data in a storage device (e.g. on a server or cloud storage) in the communication network 199 until a user requests the content (i.e. bitstream data) from the storage device, at which point the data is communicated / streamed from the storage device to the decoder 100.The system 191 may also comprise a content providing device for providing / streaming to the user (e.g., by communicating data for a user interface to be displayed on the user terminal) content information (e.g., title of the content and other meta / storage location data for identifying, selecting and requesting the content) of the content stored in the storage device and for receiving and processing user requests for content so that the requested content is delivered / streamed from the storage device to the user terminal. Alternatively, the encoder 150 generates the bitstream 101 and communicates / streams it directly to the decoder 100 when the user requests the content. The decoder 100 then receives the bitstream 101 (or signal) and performs filtering with a deblocking filter according to the present invention to obtain / generate a video signal 109 and / or an audio signal, which is used by the user terminal to provide the requested content to the user.

[0360] Any step of the method / process according to the present invention or function described herein may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the step / function is stored or transmitted as one or more instructions, codes, programs, or computer readable media and executed by one or more hardware-based processing units, such as a PC ("personal computer"), a DSP ("digital signal processor"), a programmable computing machine, which may be a circuit, processor and memory, a general-purpose microprocessor or central processing unit, a microcontroller, an ASIC ("application specific integrated circuit"), a field programmable logic array (FPGA), or other equivalent integrated or discrete logic circuit. Thus, the term "processor" as used herein may refer to any of the aforementioned structures, or other structures suitable for implementing the techniques described herein.

[0361] The embodiments of the present invention may also be implemented by a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described herein to illustrate functional aspects of a device / apparatus configured to perform the embodiments, but need not necessarily be implemented by different hardware units. Rather, the various modules / units may be combined into a codec hardware unit or provided by a collection of interoperating hardware units including one or more processors in combination with appropriate software / firmware.

[0362] The embodiments of the present invention may be realized by a computer of a system or apparatus that reads and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium to perform the modules / units / functions of one or more of the above-mentioned embodiments, and / or includes one or more processing units or circuits for performing one or more functions of the above-mentioned embodiments, and may also be realized by a method implemented by the computer of the system or apparatus, e.g., by reading and executing computer executable instructions from a storage medium to perform one or more functions of the above-mentioned embodiments, and / or by controlling one or more processing units or circuits for performing one or more functions of the above-mentioned embodiments. The computer may include a separate computer or a network of separate processing units for reading and executing the computer executable instructions. The computer executable instructions may be provided to the computer from a computer readable medium, e.g., a communication medium via a network or a tangible storage medium. The communication medium may be a signal / bit stream / carrier wave. The tangible storage medium is a "non-transitory computer-readable storage medium" and may include, for example, one or more of a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), a Blu-ray disk (BD)™), a flash memory device, a memory card, etc. At least some of the steps / functions may also be implemented in hardware by machines or dedicated components such as an FPGA ("field programmable gate array") or an ASIC ("application specific integrated circuit").

[0363] Fig. 34 is a schematic block diagram of a computing device 3600 for implementing one or more embodiments of the present invention. The computing device 3600 can be a device such as a microcomputer, a workstation, or a lightweight portable device. The computing device 3600 comprises a communication bus connected to: - a central processing unit (CPU) 3601, such as a microprocessor; - a random access memory (RAM) 3602 for storing executable codes of the method according to the embodiments of the present invention and registers for recording variables and parameters required for implementing the method of encoding or decoding at least a part of an image according to the embodiments of the present invention, the memory capacity of which can be expanded by an optional RAM, for example connected to an expansion port; - a read only memory (ROM) 3603 for storing computer programs for implementing the embodiments of the present invention; - a network interface (NET) 3604, typically connected to a communication network through which digital data to be processed is transmitted and received. The network interface (NET) 3604 can be a single network interface or can be composed of a collection of different network interfaces (for example a wired interface and a wireless interface, or different types of wired or wireless interfaces). Data packets are written to the network interface for transmission or read from the network interface for reception under the control of a software application executed by the CPU 3601. The executable code can be stored either in the ROM 3603, in the HD 3606 or in a removable digital medium, such as a disk. According to a variant, the executable code of the program can be received by the communication network, via the NET 3604, to be stored in one of the storage means of the communication device 3600, such as the HD 3606, before being executed. The CPU 3601 is adapted to control and direct the execution of the program or parts of the program's instructions or software code according to an embodiment of the invention, these instructions being stored in one of the aforementioned storage means.After power-up, CPU 3601 can execute instructions from main RAM memory 3602 associated with software applications after those instructions have been loaded, for example, from program ROM 3603 or HD 3606. Such software applications, when executed by CPU 3601, cause the steps of the method according to the present invention to be performed.

[0364] It is also understood that according to another embodiment of the invention, the decoder according to the aforementioned embodiment is provided in a user terminal such as a computer, a mobile phone (cellular telephone), a table or any other type of device (e.g. a display device) capable of providing / displaying content to a user. According to yet another embodiment, the encoder according to the aforementioned embodiment is provided in an image capture device, also comprising a camera, a video camera or a network camera (e.g. a closed circuit television or a video surveillance camera) that captures and provides content for the encoder to encode. Two such examples are given below with reference to Figures 37 and 38.

[0365] FIG. 35 is a diagram showing a network camera system 3700 including a network camera 3702 and a client device 202 .

[0366] The network camera 3702 includes an imaging unit 3706 , an encoding unit 3708 , a communication unit 3710 , and a control unit 3712 .

[0367] The network camera 3702 and the client device 202 are connected via the network 200 so as to be able to communicate with each other.

[0368] The imaging unit 3706 includes a lens and an image sensor (e.g., a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS)) to capture an image of an object and generate image data based on the image. The image may be a still image or a moving image.

[0369] The encoding unit 3708 encodes the image data using the encoding method described above.

[0370] The communication unit 3710 of the network camera 3702 transmits the encoded image data encoded by the encoding unit 3708 to the client device 202 .

[0371] Furthermore, the communication unit 3710 receives commands from the client device 202. The commands include commands for setting parameters for encoding in the encoding unit 3708.

[0372] The control unit 3712 controls other units within the network camera 3702 according to commands received by the communication unit 3712 .

[0373] The client device 202 includes a communication unit 3714 , a decryption unit 3716 , and a control unit 3718 .

[0374] The communication unit 3714 of the client device 202 transmits a command to the network camera 3702 .

[0375] Furthermore, the communication unit 3714 of the client device 202 receives the encoded image data from the network camera 3712 .

[0376] The decoder 3716 decodes the encoded image data using any of the decoding methods described above, or a combination of the decoding methods described above.

[0377] The control unit 3718 of the client device 202 controls other units within the client device 202 in response to user operations or commands received by the communication unit 3714 .

[0378] The control unit 3718 of the client device 202 controls the display device 2120 to display the image decoded by the decoding unit 3716 .

[0379] In addition, the control unit 3718 of the client device 202 controls the display device 2120 to display a GUI (graphical user interface) for specifying parameter values ​​of the network camera 3702, and also displays the encoding parameters of the encoding unit 3708.

[0380] Furthermore, the control unit 3718 of the client device 202 controls other units within the client device 202 in response to a user's operation input to the GUI displayed by the display device 2120 .

[0381] The control unit 3718 of the client device 202 controls the communication unit 3714 of the client device 202 to send a command specifying a parameter value for the network camera 3702 to the network camera 3702 in response to a user's operational input to the GUI displayed by the display device 2120.

[0382] FIG. 36 is a diagram showing a smartphone 3800.

[0383] The smartphone 3800 includes a communication unit 3802, a decoding unit 3804, a control unit 3806, and a display unit 3808.

[0384] The communication unit 3802 receives the encoded image data via the network 200 .

[0385] The decoding unit 3804 decodes the encoded image data received by the communication unit 3802 .

[0386] The decoding / encoding unit 3804 decodes / encodes the encoded image data using the decoding method described above.

[0387] The control unit 3806 controls other units within the smartphone 3800 in response to a user's operation or a command received by the communication unit 3806 .

[0388] For example, the control unit 3806 controls the display unit 3808 to display the image decoded by the decoding unit 3804. The smartphone 3800 may also include a sensor 3812 and an image recording device 3810. In this manner, the smartphone 3800 can record an image and encode the image (using the methods described above).

[0389] The smartphone 3800 can then decode the encoded image (as described above) and display it via the display device 3808 or transmit the encoded image to other devices via the communication device 3802 and the network 200.

[0390] Alternatives and fixes Although the present invention has been described with reference to the embodiments, it should be understood that the present invention is not limited to the disclosed embodiments. Those skilled in the art will understand that various changes and modifications can be made without departing from the scope of the present invention as defined in the appended claims. All features disclosed in this specification (including the appended claims, abstract and drawings) and / or all steps of any method or process so disclosed can be combined in any combination, except combinations in which at least some of such features and / or steps are mutually exclusive. Each feature disclosed in this specification (including the appended claims, abstract and drawings) can be replaced with an alternative feature serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is an example of a generic series of equivalent or similar features.

[0391] It will also be understood that the result of the above-mentioned comparison, determination, evaluation, selection, execution, performing, or consideration, e.g., a selection made during an encoding or filtering process, may be indicated or determinable / inferable in data in the bitstream, e.g., a flag or data indicating the result, and the indicated or determined / inferred result may be used in processing, e.g., during a decoding process, instead of actually performing the comparison, determination, evaluation, selection, execution, performing, or consideration.

[0392] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

[0393] Any reference numerals appearing in the claims are for illustrative purposes only and shall not be construed as limiting the scope of the claims.

Claims

1. 1. A method for processing predictors, comprising the steps of: obtaining a list of predictors, the list of predictors having at least two predictors; determining whether to modify the list of predictors based on a criterion using a first cost associated with a first predictor in the list and a second cost associated with a second predictor in the list; modifying the list of predictors based on the determination; and Including, The criteria are based on a threshold value. method.

2. The determination includes adding the threshold to the first cost or the second cost, and the criterion relates to comparing a result to one of the first cost and the second cost that did not add the threshold. The method of claim 1.

3. The criterion is whether or not a result of adding the first cost to the threshold is smaller than the second cost. The method of claim 2.

4. The criterion is whether the difference between the first cost and the second cost exceeds the threshold. The method of claim 1.

5. The difference is calculated as an absolute value and the criterion is whether it is less than the threshold value. The method according to claim 4.

6. Modifying the list includes removing the first candidate or the second candidate from the list of predictors.

6. The method according to any one of claims 1 to 5.

7. Modifying the list includes changing the position of one or more predictors in the list.

7. The method according to any one of claims 1 to 6.

8. Changing the position of one or more candidates includes moving the first predictor or the second predictor to a different position in the list. The method according to claim 7.

9. Changing the order includes moving the first predictor or the second predictor to a position considered to be the end of the list. The method according to claim 7.

10. The first and second predictors follow the zeroth predictor at what is considered the start of the list.

10. The method according to any one of claims 1 to 9.

11. The zero predictor is excluded from modifying the list of predictors. The method of claim 10.

12. and comparing a zeroth cost associated with the zeroth predictor to a zeroth threshold, and if the zeroth cost exceeds the zeroth threshold, the zeroth predictor is included in modifying the list of predictors. The method of claim 10.

13. Comparing a zeroth cost associated with the zeroth predictor to a zeroth threshold is performed in a first mode set and is not performed in a second mode set. The method of claim 12.

14. The zero threshold is less than the thresholds used for the first predictor and the second predictor.

14. The method according to claim 12 or 13.

15. The first predictor is at what is considered the start of the list, and the first cost is set to zero.

10. The method according to any one of claims 1 to 9.

16. The first cost and the second cost include a template matching cost based on a plurality of neighboring samples of the image portion to be encoded or decoded and a plurality of neighboring samples of the image portion associated with the predictor.

15. The method according to any one of claims 1 to 14.

17. The first cost and the second cost include a difference between sample values ​​of two block predictors associated with the predictor.

15. The method according to any one of claims 1 to 14.

18. The first cost and the second cost include a difference between a sample value associated with a predictor and a sample value associated with at least one other predictor in the list.

15. The method according to any one of claims 1 to 14.

19. The first cost and the second cost are based on a subset of a plurality of neighboring samples or a plurality of predictor samples.

15. The method according to any one of claims 1 to 14.

20. The first cost and the second cost are based on a plurality of samples corresponding to different images at different resolutions.

15. The method according to any one of claims 1 to 14.

21. The values ​​of the samples used to calculate the first cost and the second cost are preprocessed.

21. The method of any one of claims 1 to 20.

22. The first cost and the second cost are based on a coding rate.

22. The method according to any one of claims 16 to 21.

23. The first cost and the second cost are based on a distortion cost.

22. The method according to any one of claims 16 to 21.

24. The distortion cost is either the sum of absolute differences (SAD), the sum of absolute transformed differences (SATD), the sum of squared errors (SSE), or the structural similarity index measure (SSIM).

24. The method of claim 23.

25. The cost is proportional to the number of samples in the unit being processed.

25. The method of any one of claims 1 to 24.

26. The cost is proportional to the number of samples in the template.

25. The method of claim 23 or 24.

27. The cost is based on the sample and block size used for the distortion.

25. The method of claim 23 or 24.

28. and deriving the threshold value.

28. The method of any one of claims 1 to 27.

29. and receiving information indicative of the threshold value.

28. The method of any one of claims 1 to 27.

30. The threshold is based on a quantization parameter (QP) value.

30. The method of claim 28 or 29.

31. The threshold is based on a picture order count (POC) distance or a temporal distance between the current frame and one or more other frames.

30. The method of claim 28 or 29.

32. The threshold is based on the temporal ID of the frame.

30. The method of claim 28 or 29.

33. The threshold is based on the time direction of the reference frame.

30. The method of claim 28 or 29.

34. The threshold is based on whether bi-prediction is used.

30. The method of claim 28 or 29.

35. The threshold depends on the QP value or values ​​of the available reference frames.

30. The method of claim 28 or 29.

36. The threshold depends on the type of distortion used as at least one of the first cost and the second cost.

30. The method of claim 28 or 29.

37. The threshold is based on a type of sample used to calculate distortion as a measure of at least one of the first cost and the second cost.

30. The method of claim 28 or 29.

38. The threshold is based on whether intra-prediction or inter-prediction is used to encode or decode the image portion predicted using the list of predictors.

30. The method of claim 28 or 29.

39. The threshold is based on whether to encode or decode using one or more of a random access configuration, a RA, a low latency configuration, and an intra-only configuration.

30. The method of claim 28 or 29.

40. The threshold is based on a mathematical formula used for encoding and is obtained by the decoder for use in the calculation of the threshold.

30. The method of claim 28 or 29.

41. The threshold is based on a shifted quantization parameter (QP) value.

30. The method of claim 28 or 29.

42. The shifted QP value is used to determine the Lagrangian parameter (λ) as the threshold.

41. The method of claim 40.

43. The shifted quantization parameter is used to obtain a Lagrangian parameter from a table, where multiple entries in the table associate multiple Lagrangian parameter values ​​with multiple quantization parameter values.

53. A method according to any one of claims 41 to 52.

44. The threshold is determined as the minimum cost difference between consecutive predictors in the list that are not the first predictor and the second predictor.

29. The method of any one of claims 1 to 28.

45. The threshold is determined using the cost values ​​of all candidates in the list or the first N candidates, where N is the maximum number of candidates allowed in the list.

29. The method of any one of claims 1 to 28.

46. Using the cost value includes calculating an average of differences in costs of successive candidates from among the candidates used to determine the threshold.

46. ​​The method of claim 45.

47. The threshold is based on a Lagrangian parameter (λ), and the criterion is whether an absolute difference between a first distortion cost and a second distortion cost is smaller than the Lagrangian parameter.

44. The method of any one of claims 23 to 43.

48. The threshold is the Lagrangian parameter multiplied by the rate difference between the first predictor and the second predictor.

48. The method of claim 47.

49. At least one of the first distortion cost and the second distortion cost is multiplied by the threshold value.

44. The method of any one of claims 23 to 43.

50. The criteria are applied after the first ordering process.

38. The method of claim 37.

51. A further threshold is added to the result of the difference between the first distortion cost and the second distortion cost resulting from the multiplication.

51. The method of claim 37 or 50.

52. The decision is made according to the criterion during a process of ordering a number of candidates according to their distortion cost.

52. The method of any one of claims 1 to 51.

53. The decision to modify the list is based on a minimum distortion difference and a threshold.

53. The method of claim 52.

54. The maximum number of permutations is limited to a given number.

54. The method of claim 53.

55. The maximum number of permutations is the maximum number of candidates allowed in the list.

55. The method of claim 54.

56. The decision is based on the minimum distortion difference and is limited to a maximum number of permutations.

53. The method of any one of claims 1 to 52.

57. Modifying the list only involves removing candidates.

57. The method of any one of claims 52 to 56.

58. The number of candidates after the deletion is equal to or greater than the maximum number of candidates allowed in the list.

58. The method of claim 57.

59. When the list is modified by moving a candidate, the maximum number of candidates allowed in the list is subtracted.

59. The method of any one of claims 1 to 58.

60. If the criterion is met, the second predictor is moved to the first position in the list whose cost difference with respect to the subsequent predictor is greater than the threshold.

56. A method according to any one of claims 53 to 55.

61. If the maximum number of possible permutations is greater than the maximum number of candidates allowed in the final list, the modifying includes limiting the location to which the first predictor or the second predictor is moved to the end of the current list.

61. The method of claim 60.

62. The predictor processing is applied to at least one of a normal merge mode, a template matching merge mode, a bilateral matching (BM) merge mode, and an intra block copy (IBC) merge mode.

62. The method of claim 60 or 61.

63. The second predictor immediately follows the first predictor, and if it is determined that the criterion is not met, the second predictor is inserted after the subsequent predictor in the list.

53. The method of any one of claims 1 to 52.

64. applying the criterion to the second cost and to each of the costs of the previous candidates in the list if it is determined that the criterion is not satisfied, and not inserting the second predictor after the subsequent predictor if it is determined that the criterion is not satisfied for the second cost and each of the costs of the previous candidates.

64. The method of claim 63.

65. The maximum number of permutations possible is set to the allowable number of candidates in the final list of candidates processed.

65. The method of any one of claims 52 to 64.

66. The predictor processing is applied to at least one of merging with MVD, affine merging with MVD, MVD sign prediction of motion vector residuals.

66. The method of claim 64 or 65.

67. The list of predictors is for predicting motion in an image portion.

67. The method of any one of claims 1 to 66.

68. The predictors in the list are either intra block predictor candidates or inter motion information predictor candidates.

68. The method of claim 67.

69. The predictors in the list are one of: a prediction of a position in a point cloud; an encoding of a mesh; a prediction of audio data; a prediction of still image data; and a prediction of video data.

67. The method of any one of claims 1 to 66.

70. and associating at least one syntax element with one or more predictors in the list of predictors. The method of claim 1.

71. 1. A method for processing predictors, comprising the steps of: obtaining a list of predictors, the list of predictors having at least two predictors; determining whether to modify the list of predictors based on a comparison of a first cost associated with a first predictor in the list of predictors to a threshold; modifying the list of predictors based on the determination; and A method comprising:

72. Determining whether to modify the list of predictors is based on comparing a respective cost associated with each predictor in the list to a threshold.

72. The method of claim 71.

73. Revising the list includes removing the or each candidate from the list of predictors whose cost is below the threshold.

73. The method of claim 71 or 72.

74. Modifying the list includes moving the or each candidate in the list whose cost is less than the threshold to a position in the list next to a candidate whose cost is greater than the threshold.

73. The method of claim 71 or 72.

75. Modifying the list includes moving the or each candidate in the list whose cost is less than the threshold to a position considered to be the tail of the list.

73. The method of claim 71 or 72.

76. Modifying the list of predictors is performed in a first mode set and not in a second mode set.

76. The method of any one of claims 71 to 75.

77. 77. A method of encoding image data into a bitstream, comprising processing a list of predictors according to a method according to any one of claims 1 to 76.

78. 77. A method of decoding image data from a bitstream, comprising processing a list of predictors according to a method according to any one of claims 1 to 76.

79. 78. An apparatus for encoding image data into a bitstream, the apparatus being configured to perform the method of claim 77.

80. 80. An apparatus for decoding image data from a bitstream, the apparatus being configured to perform the method of claim 78.

81. A program which when executed causes a programmable device to carry out a method according to any one of claims 1 to 76.

82. 82. A computer readable storage medium having stored thereon the program according to claim 81.

Images